Gestures

From Openmoko

(Difference between revisions)
Jump to: navigation, search
(added the demo movie)
(Compiling for SHR)
 
(32 intermediate revisions by 13 users not shown)
Line 1: Line 1:
Read more on how the project is going [http://www.borza.ro/index.php/category/accelerometer-based-gestures/ here]. <object width="425" height="350"> <param name="movie" value="http://www.youtube.com/v/MX50lXuJ28Q"> </param> <embed src="http://www.youtube.com/v/MX50lXuJ28Q" type="application/x-shockwave-flash" width="425" height="350"> </embed> </object>
+
{{Application|Gestures}}
The gesture recognition library that is used on the Neo FreeRunner is the same one that is used in my video.
+
[[Image:Accelges-Recognize-1.png|thumb]]
  
== Applications (needs some cleaning) ==
+
=Alpha 2 Release (14 Aug 2008) for Neo FreeRunner=
* Mute audio or suspend when screen is facing down;
+
* Go to main menu when shaken;
+
* Volume up/down during call when tilting left/right (still unclear);
+
* Turning the phone face to the user (not the same as taking it to the 
+
ear) to turn on the backlight
+
* Automatic portrait/landscape switching for the UI
+
* Turning the phone screen down to mute sound (and probably turn off the 
+
backlight) or hold call
+
* Swinging in an O-shape in the air to redial
+
* Moving the phone in a firm gesture from one ear to the other to switch 
+
between active and held calls
+
* Scrolling with firm tilts (suggested several times, should see if it's 
+
usable)
+
* Dropping (suggested several times, though it's unclear how to react to 
+
it)
+
* Shaking to get audio feedback (could e.g. imitate balls rolling inside 
+
to the number of unread messages, or liquid splashing to incdicate the 
+
battery level)
+
* Starting driving in a car (if that's detectable -- probably has other 
+
patterns than walking etc) to switch to some “car mode”
+
* Stopping e.g. at a traffic light to choose a better time to notify about 
+
new messages than while driving
+
* Taking off in a plane (should be detectable, but hard to train) to shut 
+
down all RF systems
+
* Similarly, landing to re-enable RF systems
+
  
Vigorous shaking (side to side) while receiving a call could reject it.
+
==Video Demo==
 +
[http://www.youtube.com/watch?v=K2S2rQUETwc Openmoko Neo FreeRunner does Accelerometer-based Gestures, and Screen Orientation]
  
A sideways swing (90degres) out of the wrist could mean general
+
''Remark: Please watch the video, so that you'd know what to expect out of the Alpha 2 Release.
Cancel/Esc/Back
+
A long swing could close a app (more a arm swing than wrist. Same
+
G-forces but longer time).
+
These swing moves could be used on two axis and each in two axis for
+
different usage.
+
A firm wrist tilt backside down could mean global OK.
+
  
Maybe some basic moves like thsese should have absolute global meaning.
+
==Sources==
 +
In October 2008, the code moved from the  [http://code.google.com/p/accelges/source/browse/ Accelges Google Summer of Code] repository to a GIT server located at http://repo.accelsense.org.
 +
=== Compiling for SHR ===
 +
To be completed
 +
oe_runconf  --with-libtool-sysroot=.../sysroots/armv4t-oe-linux-gnueabi/
 +
oe_runmake DESTDIR="../accelges-read-only_destdir" install
  
(like left, right, enter, esc)
+
==Alpha 2 Release: The Quick Way==
 +
SSH Neo FreeRunner:
 +
wget http://accelges.googlecode.com/files/accelges_0.1.0-svnr204-r2_armv4t.ipk
 +
opkg install accelges_0.1.0-svnr204-r2_armv4t.ipk
 +
On The Neo FreeRunner
 +
/etc/init.d/gesl start
 +
Over SSH you must use
 +
DISPLAY=:0 /etc/init.d/gesl start
 +
(''otherwise screen orientation won't work''):
  
Mute phone my hitting it on something hard three times with one side.
+
You're ready: make a gesture, make Neo change its screen orientation (''if you did watch the video, you'll know what to do'').
  
Face down lying still - lock screen
+
===Training: The Quick Way===
Face up lying still - never lock screen
+
'''OPTIONAL''': Stop the gesture recognizer, and listener:
 +
/etc/init.d/gesd-neo2 stop
 +
/etc/init.d/gesl stop
 +
Run
 +
Home > Gestures
 +
Select a gesture, press Train, and follow instructions.
  
* holding the moko out & angling the front of it up repeatedly turns up
+
'''REQUIRED''': Start/restart the gesture recognizer, and listener:
volume
+
/etc/init.d/gesd-neo2 start (from SSH, or from Neo)
 +
/etc/init.d/gesl start ('''only from Neo''')
 +
 
 +
==Alpha 2 Release Details==
 +
 
 +
===Prerequisites===
 +
* You can use any distribution you like; I would suggest using the [[Om 2008.8]] for now. You'll need these dependencies:
 +
dbus, dbus-glib, libxrandr, libnotify, notification-daemon, libcurl, and gtk+
 +
* On the [[Om 2008.8]], you'll be missing:
 +
libnotify, notification-daemon, libsexy2, libwnck-1-18, libstartup-notification-1-0, and libglade-2
 +
opkg install libnotify notification-daemon libsexy2 libwnck-1-18 libstartup-notification-1-0 libglade-2.0-0
 +
''Remark: On the 2008.8, your missing dependencies will be downloaded automatically by the installer.''
 +
 
 +
===Preparing Om 2008.8 for Gestures===
 +
* Turn off Suspend:
 +
  Home > Settings > Suspend > off
 +
* Make sure Neo has Web Access (use SSH):
 +
echo "nameserver xxx.xxx.xxx.xxx" > /etc/resolv.conf
 +
* Install Terminal:
 +
Home > Installer > Development > openmoko-terminal2
 +
* Create a shell script that will start the gesture listener daemon (use SSH):
 +
echo "exec /etc/init.d/gesl start" > /usr/bin/gesl-begin
 +
chmod +x /usr/bin/gesl-begin
 +
* '''Restart Neo!'''
 +
 
 +
===Installing Gestures===
 +
* Download the [http://accelges.googlecode.com/files/accelges_0.1.0-svnr204-r2_armv4t.ipk Alpha 2 Release Package], or (use SSH):
 +
echo "nameserver xxx.xxx.xxx.xxx" > /etc/resolv.conf
 +
wget http://accelges.googlecode.com/files/accelges_0.1.0-svnr204-r2_armv4t.ipk
 +
* Install:
 +
opkg install accelges_0.1.0-svnr204-r2_armv4t.ipk
 +
 
 +
===Training Gestures===
 +
* Launch the Gesture Manager:
 +
Home > Gestures
 +
* Select a gesture and click the first icon on the toolbar; Follow the instructions. Do the same for all the gestures!
 +
** Step 1: Launch Gesture Manager <br />[[Image:Accelges-Train-1.png|200px]]
 +
** Step 2: This is the Gesture Manager <br />[[Image:Accelges-Train-2.png|200px]]
 +
** Step 3: Select a gesture, and click Train (first icon from the toolbar) <br />[[Image:Accelges-Train-3.png|200px]]
 +
** Step 4: Make the gesture (this step creates the gesture) <br />[[Image:Accelges-Train-4.png|200px]]
 +
** Step 5: Make the gesture again (this step trains the gesture) <br />[[Image:Accelges-Train-5.png|200px]]
 +
** Step 6: Select another gesture, and do the same <br />[[Image:Accelges-Train-6.png|200px]]
 +
 
 +
* Restart the Gesture Recognizer (use SSH):
 +
/etc/init.d/gesd-neo2 restart
 +
 
 +
''Remark: You have to restart the Gesture Recognizer, so that it will re-read the newly trained gestures. Every time you train one or more gestures, restart the recognizer!''
 +
 
 +
===Running Gestures===
 +
* Use the script you have created before; '''be sure to run the script from the Neo, no SSH or VNC''':
 +
gesl-begin
 +
 
 +
* There are 12 gestures available (once, you make one, a notification will pop up):
 +
{| border=1 |
 +
| align="center"|Shake-shake gesture: <br />[[Image:Accelges-Recognize-1.png|100px]]
 +
| align="center"|Forward-backward gesture: <br />[[Image:Accelges-Recognize-2.png|100px]]
 +
| align="center"|Horizontal-circle gesture: <br />[[Image:Accelges-Recognize-3.png|100px]]
 +
| align="center"|Z gesture: <br />[[Image:Accelges-Recognize-4.png|100px]]
 +
|-
 +
| align="center"|Right gesture: <br />[[Image:Accelges-Recognize-5.png|100px]]
 +
| align="center"|Left gesture: <br />[[Image:Accelges-Recognize-6.png|100px]]
 +
| align="center"|Up gesture: <br />[[Image:Accelges-Recognize-7.png|100px]]
 +
| align="center"|Down gesture: <br />[[Image:Accelges-Recognize-8.png|100px]]
 +
|-
 +
| align="center"|Right-left gesture: <br />[[Image:Accelges-Recognize-9.png|100px]]
 +
| align="center"|Left-right gesture: <br />[[Image:Accelges-Recognize-10.png|100px]]
 +
| align="center"|Up-down gesture: <br />[[Image:Accelges-Recognize-11.png|100px]]
 +
| align="center"|Down-up gesture: <br />[[Image:Accelges-Recognize-12.png|100px]]
 +
|}
 +
''Remark: If you don't start the Gesture Listener from the Neo, screen orientation won't work''
 +
 
 +
===How To Improve Recognition Accuracy===
 +
* Check to see which gesture gets recognized most often. Try training those gestures again.
 +
* Determine which gestures don't get recognized at all. Try training those gestures again.
 +
 
 +
==Alpha 2 Development Details==
 +
===DBUS===
 +
Gesture Recognizer (gesd) sends signals on:
 +
org.openmoko.accelges
 +
/org/openmoko/accelges/Recognizer
 +
org.openmoko.accelges.Recognizer.Recognized
 +
 
 +
Try (on [[FSO]] distribution)
 +
mdbus -s -l
 +
and make a gesture for more information on DBUS.
 +
 
 +
===Configuration Files===
 +
The configuration file for Neo is:
 +
/etc/accelges/neo2/neo2.cfg
 +
<pre>
 +
# Paul-Valentin Borza <paul@borza.ro>
 +
 
 +
# gestures configuration file for Neo/Wii
 +
 
 +
# classes and probabilities for classifier
 +
#    sclass\tFILE static acceleration class
 +
#    dclass\tFILE dynamic acceleration class
 +
#    sclassp\tVALUE probability for static acceleration class
 +
#    dclassp\tVALUE probability for dynamic acceleration class
 +
sclass s.class
 +
dclass d.class
 +
sclassp 0.4
 +
dclassp 0.6
 +
 
 +
# classes for recognizer (add more than one)
 +
#    class\tid\tFILE static acceleration class
 +
class screen_zzp screen_zzp.class
 +
class screen_zzn screen_zzn.class
 +
class screen_zpz screen_zpz.class
 +
class screen_znz screen_znz.class
 +
class screen_pzz screen_pzz.class
 +
class screen_nzz screen_nzz.class
 +
class screen_npp screen_npp.class
 +
class screen_nnp screen_nnp.class
 +
class screen_pnp screen_pnp.class
 +
class screen_ppp screen_ppp.class
 +
 
 +
# models for recognizer (add more than one)
 +
#    model\tid\tFILE dynamic acceleration model
 +
model left left.model
 +
model left, and return left-and-right.model
 +
model right right.model
 +
model right, and return right-and-left.model
 +
model up up.model
 +
model up, and return up-and-down.model
 +
model down down.model
 +
model down, and return down-and-up.model
 +
model shake, shake shake-shake.model
 +
model horizontal circle horizontal-circle.model
 +
model forward, backward forward-backward.model
 +
model z z.model
 +
</pre>
 +
 
 +
Again, if you modify the configuration file, you'll have to restart the gesture recognizer (gesd).
 +
 
 +
===Daemons===
 +
Gesture Recognizer that uses the top accelerometer (/dev/input/event2):
 +
/etc/init.d/gesd-neo2 start|stop|restart
 +
Gesture Listener that switches screen orientation, and notifies the user of recognized gestures on the screen:
 +
/etc/init.d/gesl start|stop|restart
 +
''Remark: Start the recognizer first, and after that the listener (obvious, but important)''
 +
 
 +
=Components/Applications=
 +
Try playing with:
 +
gesd (gesture recognizer)
 +
gesl (gesture listener)
 +
gesm (gesture manager)
 +
 
 +
=Known Issues=
 +
* One, or both accelerometers might not work. Try:
 +
hexdump /dev/input/event2 (for top accelerometer)
 +
hexdump /dev/input/event3 (for bottom accelerometer)
 +
''Remark: The Alpha 2 Release uses the TOP accelerometer''
 +
* After a while, the accelerometers might stop working.
 +
Restart your Neo!
 +
* After a while, the accelerometer send bogus values (values that are lower than what they should be). If this happens, the classifier won't be able to classify dynamic/static acceleration, and nothing will work - nor the gesture training, nor the gesture recognition.
 +
Restart your Neo!
 +
 
 +
=Use Cases=
 +
* Mute audio or suspend when screen is facing down; - NO SUPPORT FOR MUTE/UNMUTE AUDIO IN NEO FOR NOW
 +
* Go to main menu when shaken; - NO SUPPORT YET
 +
* Volume up/down during call when tilting left/right (still unclear); - NO SUPPORT YET
 +
* Turning the phone face to the user (not the same as taking it to the ear) to turn on the backlight - BACKLIGHT HAS A BUG, CAN'T BE USED YET
 +
* Automatic portrait/landscape switching for the UI - <font color="#00FF00">Done</font>
 +
* Turning the phone screen down to mute sound (and probably turn off the backlight) or hold call
 +
* Swinging in an O-shape in the air to redial
 +
* Moving the phone in a firm gesture from one ear to the other to switch between active and held calls
 +
* Scrolling with firm tilts (suggested several times, should see if it's usable)
 +
* Dropping (suggested several times, though it's unclear how to react to it)
 +
* Shaking to get audio feedback (could e.g. imitate balls rolling inside to the number of unread messages, or liquid splashing to incdicate the battery level)
 +
* Starting driving in a car (if that's detectable -- probably has other  patterns than walking etc) to switch to some “car mode”
 +
* Stopping e.g. at a traffic light to choose a better time to notify about new messages than while driving
 +
* Taking off in a plane (should be detectable, but hard to train) to shut down all RF systems
 +
** This is a bad idea - you need to do that way before it lifts off the ground
 +
* Similarly, landing to re-enable RF systems
 +
** Similarly, you shouldn't turn them back on until way after the wheels touch the tarmac
 +
* Vigorous shaking (side to side) while receiving a call could reject it.
 +
* A sideways swing (90degres) out of the wrist could mean general Cancel/Esc/Back
 +
* A long swing could close a app (more a arm swing than wrist. Same G-forces but longer time).
 +
* These swing moves could be used on two axis and each in two axis for different usage.
 +
* A firm wrist tilt backside down could mean global OK.
 +
* Maybe some basic moves like these should have absolute global meaning. (like left, right, enter, esc)
 +
* Mute phone my hitting it on something hard three times with one side.
 +
* Face down lying still - lock screen
 +
* Face up lying still - never lock screen
 +
* holding the moko out & angling the front of it up repeatedly turns up volume
 
* angling front down repeatedly turns down volume
 
* angling front down repeatedly turns down volume
* a set of 5 or 10 standard, easily distinguishable gestures that the user
+
* a set of 5 or 10 standard, easily distinguishable gestures that the user can map to favorite programs
can map to favorite programs
+
 
 +
 
 +
<onlyinclude>
 +
{{ApplicationBox|
 +
  Name=[[Gestures]]|
 +
  Description=Gestures is a daemon that interpret measurements from the accelerometers and the movements you make with your FreeRunner. It also switches between portrait and landscape if it notices that your FreeRunner has been rotated.|
 +
  Screenshot=Accelges-Recognize-1.png|
 +
  Homepage=http://accelges.googlecode.com|
 +
  TestedOn=Om 2008.8|
 +
  PackageName=accelges
 +
}}
 +
</onlyinclude>
 +
 
 +
[[Category:System Applications]]
 +
[[Category:Accelerometer applications]]

Latest revision as of 14:27, 4 November 2010

Gestures is one of the applications that runs on the Openmoko Phones. For a list of all applications, visit Applications

Accelges-Recognize-1.png

Contents

[edit] Alpha 2 Release (14 Aug 2008) for Neo FreeRunner

[edit] Video Demo

Openmoko Neo FreeRunner does Accelerometer-based Gestures, and Screen Orientation

Remark: Please watch the video, so that you'd know what to expect out of the Alpha 2 Release.

[edit] Sources

In October 2008, the code moved from the Accelges Google Summer of Code repository to a GIT server located at http://repo.accelsense.org.

[edit] Compiling for SHR

To be completed

oe_runconf  --with-libtool-sysroot=.../sysroots/armv4t-oe-linux-gnueabi/
oe_runmake DESTDIR="../accelges-read-only_destdir" install

[edit] Alpha 2 Release: The Quick Way

SSH Neo FreeRunner:

wget http://accelges.googlecode.com/files/accelges_0.1.0-svnr204-r2_armv4t.ipk
opkg install accelges_0.1.0-svnr204-r2_armv4t.ipk

On The Neo FreeRunner

/etc/init.d/gesl start

Over SSH you must use

DISPLAY=:0 /etc/init.d/gesl start

(otherwise screen orientation won't work):

You're ready: make a gesture, make Neo change its screen orientation (if you did watch the video, you'll know what to do).

[edit] Training: The Quick Way

OPTIONAL: Stop the gesture recognizer, and listener:

/etc/init.d/gesd-neo2 stop
/etc/init.d/gesl stop

Run

Home > Gestures

Select a gesture, press Train, and follow instructions.

REQUIRED: Start/restart the gesture recognizer, and listener:

/etc/init.d/gesd-neo2 start (from SSH, or from Neo)
/etc/init.d/gesl start (only from Neo)

[edit] Alpha 2 Release Details

[edit] Prerequisites

  • You can use any distribution you like; I would suggest using the Om 2008.8 for now. You'll need these dependencies:
dbus, dbus-glib, libxrandr, libnotify, notification-daemon, libcurl, and gtk+

libnotify, notification-daemon, libsexy2, libwnck-1-18, libstartup-notification-1-0, and libglade-2

opkg install libnotify notification-daemon libsexy2 libwnck-1-18 libstartup-notification-1-0 libglade-2.0-0

Remark: On the 2008.8, your missing dependencies will be downloaded automatically by the installer.

[edit] Preparing Om 2008.8 for Gestures

  • Turn off Suspend:
 Home > Settings > Suspend > off
  • Make sure Neo has Web Access (use SSH):
echo "nameserver xxx.xxx.xxx.xxx" > /etc/resolv.conf
  • Install Terminal:
Home > Installer > Development > openmoko-terminal2
  • Create a shell script that will start the gesture listener daemon (use SSH):
echo "exec /etc/init.d/gesl start" > /usr/bin/gesl-begin
chmod +x /usr/bin/gesl-begin
  • Restart Neo!

[edit] Installing Gestures

echo "nameserver xxx.xxx.xxx.xxx" > /etc/resolv.conf
wget http://accelges.googlecode.com/files/accelges_0.1.0-svnr204-r2_armv4t.ipk
  • Install:
opkg install accelges_0.1.0-svnr204-r2_armv4t.ipk

[edit] Training Gestures

  • Launch the Gesture Manager:
Home > Gestures
  • Select a gesture and click the first icon on the toolbar; Follow the instructions. Do the same for all the gestures!
    • Step 1: Launch Gesture Manager
      Accelges-Train-1.png
    • Step 2: This is the Gesture Manager
      Accelges-Train-2.png
    • Step 3: Select a gesture, and click Train (first icon from the toolbar)
      Accelges-Train-3.png
    • Step 4: Make the gesture (this step creates the gesture)
      Accelges-Train-4.png
    • Step 5: Make the gesture again (this step trains the gesture)
      Accelges-Train-5.png
    • Step 6: Select another gesture, and do the same
      Accelges-Train-6.png
  • Restart the Gesture Recognizer (use SSH):
/etc/init.d/gesd-neo2 restart

Remark: You have to restart the Gesture Recognizer, so that it will re-read the newly trained gestures. Every time you train one or more gestures, restart the recognizer!

[edit] Running Gestures

  • Use the script you have created before; be sure to run the script from the Neo, no SSH or VNC:
gesl-begin
  • There are 12 gestures available (once, you make one, a notification will pop up):
Shake-shake gesture:
Accelges-Recognize-1.png
Forward-backward gesture:
Accelges-Recognize-2.png
Horizontal-circle gesture:
Accelges-Recognize-3.png
Z gesture:
Accelges-Recognize-4.png
Right gesture:
Accelges-Recognize-5.png
Left gesture:
Accelges-Recognize-6.png
Up gesture:
Accelges-Recognize-7.png
Down gesture:
Accelges-Recognize-8.png
Right-left gesture:
Accelges-Recognize-9.png
Left-right gesture:
Accelges-Recognize-10.png
Up-down gesture:
Accelges-Recognize-11.png
Down-up gesture:
Accelges-Recognize-12.png

Remark: If you don't start the Gesture Listener from the Neo, screen orientation won't work

[edit] How To Improve Recognition Accuracy

  • Check to see which gesture gets recognized most often. Try training those gestures again.
  • Determine which gestures don't get recognized at all. Try training those gestures again.

[edit] Alpha 2 Development Details

[edit] DBUS

Gesture Recognizer (gesd) sends signals on:

org.openmoko.accelges
/org/openmoko/accelges/Recognizer
org.openmoko.accelges.Recognizer.Recognized

Try (on FSO distribution)

mdbus -s -l

and make a gesture for more information on DBUS.

[edit] Configuration Files

The configuration file for Neo is:

/etc/accelges/neo2/neo2.cfg
# Paul-Valentin Borza <paul@borza.ro>

# gestures configuration file for Neo/Wii

# classes and probabilities for classifier
#    sclass\tFILE	static acceleration class
#    dclass\tFILE	dynamic acceleration class
#    sclassp\tVALUE	probability for static acceleration class
#    dclassp\tVALUE	probability for dynamic acceleration class
sclass	s.class
dclass	d.class
sclassp	0.4
dclassp	0.6

# classes for recognizer (add more than one)
#    class\tid\tFILE	static acceleration class
class	screen_zzp	screen_zzp.class
class	screen_zzn	screen_zzn.class
class	screen_zpz	screen_zpz.class
class	screen_znz	screen_znz.class
class	screen_pzz	screen_pzz.class
class	screen_nzz	screen_nzz.class
class	screen_npp	screen_npp.class
class	screen_nnp	screen_nnp.class
class	screen_pnp	screen_pnp.class
class	screen_ppp	screen_ppp.class

# models for recognizer (add more than one)
#    model\tid\tFILE	dynamic acceleration model
model	left	left.model
model	left, and return	left-and-right.model
model	right	right.model
model	right, and return	right-and-left.model
model	up	up.model
model	up, and return	up-and-down.model
model	down	down.model
model	down, and return	down-and-up.model
model	shake, shake	shake-shake.model
model	horizontal circle	horizontal-circle.model
model	forward, backward	forward-backward.model
model	z	z.model

Again, if you modify the configuration file, you'll have to restart the gesture recognizer (gesd).

[edit] Daemons

Gesture Recognizer that uses the top accelerometer (/dev/input/event2):

/etc/init.d/gesd-neo2 start|stop|restart

Gesture Listener that switches screen orientation, and notifies the user of recognized gestures on the screen:

/etc/init.d/gesl start|stop|restart

Remark: Start the recognizer first, and after that the listener (obvious, but important)

[edit] Components/Applications

Try playing with:

gesd (gesture recognizer)
gesl (gesture listener)
gesm (gesture manager)

[edit] Known Issues

  • One, or both accelerometers might not work. Try:
hexdump /dev/input/event2 (for top accelerometer)
hexdump /dev/input/event3 (for bottom accelerometer)

Remark: The Alpha 2 Release uses the TOP accelerometer

  • After a while, the accelerometers might stop working.
Restart your Neo!
  • After a while, the accelerometer send bogus values (values that are lower than what they should be). If this happens, the classifier won't be able to classify dynamic/static acceleration, and nothing will work - nor the gesture training, nor the gesture recognition.
Restart your Neo!

[edit] Use Cases

  • Mute audio or suspend when screen is facing down; - NO SUPPORT FOR MUTE/UNMUTE AUDIO IN NEO FOR NOW
  • Go to main menu when shaken; - NO SUPPORT YET
  • Volume up/down during call when tilting left/right (still unclear); - NO SUPPORT YET
  • Turning the phone face to the user (not the same as taking it to the ear) to turn on the backlight - BACKLIGHT HAS A BUG, CAN'T BE USED YET
  • Automatic portrait/landscape switching for the UI - Done
  • Turning the phone screen down to mute sound (and probably turn off the backlight) or hold call
  • Swinging in an O-shape in the air to redial
  • Moving the phone in a firm gesture from one ear to the other to switch between active and held calls
  • Scrolling with firm tilts (suggested several times, should see if it's usable)
  • Dropping (suggested several times, though it's unclear how to react to it)
  • Shaking to get audio feedback (could e.g. imitate balls rolling inside to the number of unread messages, or liquid splashing to incdicate the battery level)
  • Starting driving in a car (if that's detectable -- probably has other patterns than walking etc) to switch to some “car mode”
  • Stopping e.g. at a traffic light to choose a better time to notify about new messages than while driving
  • Taking off in a plane (should be detectable, but hard to train) to shut down all RF systems
    • This is a bad idea - you need to do that way before it lifts off the ground
  • Similarly, landing to re-enable RF systems
    • Similarly, you shouldn't turn them back on until way after the wheels touch the tarmac
  • Vigorous shaking (side to side) while receiving a call could reject it.
  • A sideways swing (90degres) out of the wrist could mean general Cancel/Esc/Back
  • A long swing could close a app (more a arm swing than wrist. Same G-forces but longer time).
  • These swing moves could be used on two axis and each in two axis for different usage.
  • A firm wrist tilt backside down could mean global OK.
  • Maybe some basic moves like these should have absolute global meaning. (like left, right, enter, esc)
  • Mute phone my hitting it on something hard three times with one side.
  • Face down lying still - lock screen
  • Face up lying still - never lock screen
  • holding the moko out & angling the front of it up repeatedly turns up volume
  • angling front down repeatedly turns down volume
  • a set of 5 or 10 standard, easily distinguishable gestures that the user can map to favorite programs


Accelges-Recognize-1.png Gestures

Gestures is a daemon that interpret measurements from the accelerometers and the movements you make with your FreeRunner. It also switches between portrait and landscape if it notices that your FreeRunner has been rotated.


Homepage: http://accelges.googlecode.com
Package: accelges
Tested on: Om 2008.8

Personal tools

Read more on how the project is going here. <object width="425" height="350"> <param name="movie" value="http://www.youtube.com/v/MX50lXuJ28Q"> </param> <embed src="http://www.youtube.com/v/MX50lXuJ28Q" type="application/x-shockwave-flash" width="425" height="350"> </embed> </object> The gesture recognition library that is used on the Neo FreeRunner is the same one that is used in my video.

Applications (needs some cleaning)

  • Mute audio or suspend when screen is facing down;
  • Go to main menu when shaken;
  • Volume up/down during call when tilting left/right (still unclear);
  • Turning the phone face to the user (not the same as taking it to the

ear) to turn on the backlight

  • Automatic portrait/landscape switching for the UI
  • Turning the phone screen down to mute sound (and probably turn off the

backlight) or hold call

  • Swinging in an O-shape in the air to redial
  • Moving the phone in a firm gesture from one ear to the other to switch

between active and held calls

  • Scrolling with firm tilts (suggested several times, should see if it's

usable)

  • Dropping (suggested several times, though it's unclear how to react to

it)

  • Shaking to get audio feedback (could e.g. imitate balls rolling inside

to the number of unread messages, or liquid splashing to incdicate the battery level)

  • Starting driving in a car (if that's detectable -- probably has other

patterns than walking etc) to switch to some “car mode”

  • Stopping e.g. at a traffic light to choose a better time to notify about

new messages than while driving

  • Taking off in a plane (should be detectable, but hard to train) to shut

down all RF systems

  • Similarly, landing to re-enable RF systems

Vigorous shaking (side to side) while receiving a call could reject it.

A sideways swing (90degres) out of the wrist could mean general Cancel/Esc/Back A long swing could close a app (more a arm swing than wrist. Same G-forces but longer time). These swing moves could be used on two axis and each in two axis for different usage. A firm wrist tilt backside down could mean global OK.

Maybe some basic moves like thsese should have absolute global meaning.

(like left, right, enter, esc)

Mute phone my hitting it on something hard three times with one side.

Face down lying still - lock screen Face up lying still - never lock screen

  • holding the moko out & angling the front of it up repeatedly turns up

volume

  • angling front down repeatedly turns down volume
  • a set of 5 or 10 standard, easily distinguishable gestures that the user

can map to favorite programs