Openmoko - User contributions [en]

Wishlist/LiveUSB distro

2007-12-15T17:33:24Z

Fthiery: /* About USB Booting */

{{wishlist}}

==Introduction==

NOTE: this page may be renamed: portable destkop

An openmoko device could act as '''the perfect geeky swiss knive''': go anywhere with your Linux desktop and tools.

YES, there are tons of liveCDs out there, but CDs/DVDs:
* get easily damaged
* don't fit in a pocket (except for businesscard-sized ones ; too exotic...)
* don't update or save settings/personal data without a supplementary USB device

YES, there are LiveUSB distros but:
* a Neo/OpenMoko device IS sort of an USB stick; one object is better than two
* you may carry my cellphone everywhere, what about usb sticks or cds ?

The biggest argument against these ones is that you'd have to carry an USB cable with you... But it's a standard one, which is good news :) An integrated zippable plug could be an interesting add-on...

Similar functionality can be found in [http://en.wikipedia.org/wiki/Wizpy the Wizpy portable media player], but relies on different methods for working under Windows (cd-rom emulation).

When the openmoko device is in mass storage mode, a host computer should be able to boot on it, presenting a grub menu offering to boot into several images / partitions (payloads) on the transflash: memtest, UBCD (the ultimate boot cd), a lightweight security oriented livecd distro, you name it...

It's sometimes called Live USB: from [http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB definition], ''A live USB is a USB flash drive containing a full operating system which can be booted from. Live USBs are closely related to Live CDs, and are sometimes used interchangeably. Like Live CDs, Live USBs can be used for system administration, data recovery, or the testing of operating system distributions without committing to a permanent installation on the local hard drive. Many of the smaller Linux distributions can also be used from a USB flash drive.''

Usually, all you need for this with an USB storage device (flash USB stick or external drive) is a partition, flagged "bootable" (see fdisk), containing a boot loader (say grub). But the problem is that specifications vary between motherboards, so there are variants and mandatory requirements to define (here would be a good idea), so that one can optimize/maximize the compatibility.

==The perfect geek knife==

We may want to start our own distro with the following constraints:
* Multiboot: small utilities/compilations + bull blown light OS
* Aiming at maximum hardware compatibility, possibly using liveCD technology (knoppix-based distros)
* Aiming at minimum resource+system footprint, and boot speed
* Being able to install software temporary (for the session) or permanently, upgradable
* Being able to use the phone features at the same time (access pim, data & calls)

Optimally, being able to charge the device while plugged in would be great too.

==About USB Booting==

From [http://www.damnsmalllinux.org/wiki/index.php/USB_Booting DSL Wiki]:

Older computer BIOS usually do not support direct booting from a USB device. Around 2001, PC motherboard manufacturers started to add USB boot support.

There are two common BIOS methods for direct USB booting:
* One method is called the "USBHDD" method and it is used to support the booting of standard USB mass storage devices that are configured like a normal PC hard drive.
* The other method is called the "USBZIP" method and it supports booting from a USB storage device that behaves like the original IOMEGA ZIP drive with USB support.

Most computers (just about all Dells, for example) made today have a BIOS that supports the USBHDD method so I expect that this will eventually become the standard way to boot a USB device. However, many motherboards will support BOTH methods, and many older motherboards have USBZIP support.

Some newer BIOSs which support USB 2.0 will not boot from an older pen drive. Using a USB 2.0 compliant one usually solves this problem. Also, some older BIOSs which only support USB 1.1 will not boot from newer drives which support USB 2.0!

Some USB keys don't boot. If this is the case, it may be possible to fix them by installing a new master boot record. (Most keys boot OK by default; some cannot be fixed even by doing this. However, it helps in some cases). Run the command:

lilo -M /dev/$usbdevice

(where usbdevice is the key, ex: /dev/sdb, checking '''dmesg''' shall give the appropriate one when plugging the neo in mass storage mode)

==USB Booting in Neo1973's kernel==

Although there is very little information about it, the g_storage kernel module is responsible for the Neo's mass storage mode. The boot-or-not criteria might rely on this very module.

==TODO==

Testing/further research has to be conducted to determine:
* if a bootable-flagged partition / grub doesn't mess openmoko up
* if the phone can still act as a phone, or if the booted os can use the gprs functions
* if the phone can recharge on mass storage mode (from the usb cable)
* if a dedicated partition for each payload is needed (see memtest example...)
* if multi-boot is possible: grub?

==Linux distro Howto's: liveUSB creation==

In short:
#Mount USB drive, with e.g. mount /dev/sda1 /flash - can be either FAT16 or FAT32!
#Mount ISO image, with e.g. mount /tmp/dsl-3.2.iso /tmp/iso -o loop
#Copy all contents from ISO to USB drive: cp -vr /tmp/iso/* /flash/
#Rename and move syslinux files to root directory: mv /flash/boot/isolinux/* /flash/
#Rename isolinux.cfg: mv /flash/isolinux.cfg /flash/syslinux.cfg
#Unmount USB drive: umount /flash
#Install syslinux: syslinux /dev/sda1 and eventually set the MBR boot flag for this partition (with fdisk).

*Fedora Core's [http://revisor.fedoraunity.org Revisor] enables easy GUI-driven ISO customization, allowing to choose the output medium (CD/DVD/USB)
*[http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive Installing DSL to USB device - using grub]
*[http://wiki.debian.org/BootUsb Debian BootUSB - using Lilo]
*[http://www.debuntu.org/how-to-install-ubuntu-linux-on-usb-bar Howto install ubuntu (or any other distro) on usb stick]
*[http://www.gentoo.org/doc/en/liveusb.xml Gentoo on usb stick]
*[http://pendrivelinux.com/ Windows & Linux tutorials & resources]:
* Flash installation via Windows: XUbuntu, DSL, Knoppix, Slax, MiniMe
* Flash installation via Linux: Ubuntu Edgy, Knoppix, PCLinuxOS

'''Transflash partitioning schema example'''

* 2 Gb: /dev/sd? (where N is the transflash's number)
* 700 Mb: /dev/sd?1 : containing bootable iso, FAT16
* 1300 Mb: /dev/sd?2 : openmoko & bootable os home partition, EXT3?
* + eventually a swap

The limitation of this method is that the booted OS is static (not-self modifiable). There's the option to install linux using the partition, but it's not really mobility-oriented (liveCDs are optimized for maximum autoconfiguration).

==Alternative methods==

===Booting ISO files ?===

* Grub? From [http://www.freesoftwaremagazine.com/articles/grub_intro#comment-12900 How to install grub on an usb pendrive]:
If you want to boot from a iso file on a harddisk, do something in menu.lst like
title Boot from iso on a harddisk
map (hdX,Y)/your.iso (hdZ)
map --rehook
chainloader (hdZ)+1
rootnoverify (hdZ)
boot

Grub & memdisk can boot floppy images, but (AFAIK) not isos. Example from the stock /boot/grub/menu.lst ubuntu:
title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin
quiet
boot

''Using the memdisk kernel from the syslinux package, you can load disk images and execute them in a non-emulated environment.''

* [http://www.knoppix.net/wiki/Cheat_Codes knoppix '''bootfrom''' cheatcode]

bootfrom=/dev/hda1/KNX.iso

Bootfrom needs access to a running Knoppix-System '''with the same Kernel as the Bootkernel''', before it is able to mount the partition / ISO-Image.

* Maybe there's a way to achieve a boot cd iso selecter see ([http://www.linuxquestions.org/questions/showthread.php?t=528840 this post])
* virtualization-oriented distro... ex [http://unit.aist.go.jp/itri/knoppix/xen/index-en.html Xenoppix] Cons: would work best only on modern VT-capable computers. What about a very minimal distribution (say DSL without X) kqemu-enblaed, booting into an ncurses iso-loader? The delicate part would be on the networking side.

===Zero-Install / Networked systems===

*[http://klik.atekon.de/wiki/index.php/Main_Page Klik] with more than 1000 packages ported. See [http://klik.atekon.de/wiki/index.php/Comparison comparison chart].
*[http://0install.net/index.html The Zero Install System], with automatic dependancies, upgrading. See [http://0install.net/matrix.html] comparison chart.
*Why not rolling a [http://wiki.rpath.com/wiki/Conary:Concepts Conary/RPath-based] distro, based on [http://www.rpath.com/rbuilder/project/rpath/ rPath linux] reference distribution ? Or choose within the available ones...
* The FUSE [http://httpfs.sourceforge.net/net_boot.htm httpfs] may allow pure http net booting, downloading files from a distant livecd only when needed (this image may be very minimal though).
* [http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&start=75&tstart=0 some people] are thinking of using Amazon S3 as virtual disk

===Using Qemu to boot Linux from a flash drive within Windows===

The following tutorial explains how to use Qemu to boot Linux from a portable USB flash device while still working within Windows. This Enables the user to have both systems running at the same time eliminating the need to restart the PC and set your BIOS options to boot Linux.

[http://www.pendrivelinux.com/2007/03/09/use-qemu-to-boot-linux-from-windows/ PendriveLinux article]

===Putting a vnc server on openmoko + IP over USB or BlueTooth/Wifi===

That's another alternative :) Will bring a VGA-sized capture screen. However, given the current animated behaviour in openmoko, performance will be poor.

===The BlackDog project===

"When plugged into the host computer’s USB port, BlackDog starts up, '''automatically launching the X Window system for Windows Xming and a software NAT router via an autorun application that the BlackDog presents to the host through a virtual USB CD-ROM. Once those applications are running, the virtual USB CD-ROM drive disconnects, and a virtual USB-to-Ethernet adapter is connected to provide the communications link.'''

The host machine’s monitor, keyboard, mouse, and Internet connection are used by the BlackDog for the duration of the session."

The [http://en.wikipedia.org/wiki/BlackDog BlackDog server] project has lots of interesting concepts (and software). Could [http://www.jasonkolb.com/weblog/2006/11/idea_7_personal.html personal servers] change the face of computing ? Together with biometry, ''they open up the possibility of carrying your personal server, or an extension of it, with you on your physical person. The reason this is so exciting is because it enables biometric authentication that doesn't require shipping your biometric data to a third party'.

==Interesting distros/payloads==

'''System diagnostics / recovery'''
*[http://en.wikipedia.org/wiki/Memtest Memtest86+] Very small
*[http://www.ultimatebootcd.com/ The Ultimate Boot CD]
*[http://www.sysresccd.org/Main_Page System Rescue CD]
*[http://www.tux.org/pub/people/kent-robotti/looplinux/rip/ Recovery IS Possible] < 80 MB

'''General purpose'''
*[http://jinx-linux.sourceforge.net/ jinx-linux] < 50 MB
*[http://knoppix.com/ Knoppix] 700 MB CD-ROM
*[http://www.puppylinux.org PuppyLinux] < 100 MB
*[http://www.damnsmalllinux.org DSL] ~50 MB
*[http://gentoo-wiki.com/TinyGentoo#What_is_this_.3F TinyGentoo] 5 MB

'''Security-oriented''': pentesting, forensics, anonymous webbrowsing
*[http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-test-forensics-recovery/ List of Security distros]
*[http://www.remote-exploit.org/backtrack.html Backtrack Livecd]
*[http://en.wikipedia.org/wiki/Anonym.OS Anonym.OS: browse the web anonymously]
*[http://dban.sourceforge.net/ Darik's Boot and Nuke]: ''dangerous''
*[http://byzgl.sourceforge.net/wiki/index.php/Main_Page Byzantine OS] is a webkiosk appliance (32/48 MB)

The SabayonLinux distro offers a lot of [http://www.sabayonlinux.org/wiki/index.php?title=Sabayon_Linux#Boot_Methods_for_special_ways_of_using_Sabayon boot cheatcodes], such as booting onto GeexBox. We should take a look at the method used.

'''[http://www.livecdlist.com/ List of live distros @ livecdlist.com]''' ''Includes sizes''

'''[http://en.wikipedia.org/wiki/List_of_LiveDistros List of live distros @ wikipedia]'''

'''[http://distrowatch.com/dwres.php?resource=cd List of live distros @ distrowatch]'''

'''[http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB article]'''

Talk:E17

2007-08-25T00:48:07Z

Fthiery:

Are there some news on the developement of the E17 version for OpenMoko? --[[User:Denis std|denis_std]] 22:28, 30 June 2007 (CEST)

Yes, apparently the EFL are going to be used. Some video demos (ex kinetic scrolling) are around and seem to be made using EFL's, but nothing official. An e17 port is nowhere realistic in the contrary to EFL use (at least it won't before the hardware would take advantage of regular X apps -- ex. vga output + very good hardware).
--[[User:fthiery|flo]] 22:28, 30 June 2007 (CEST)

EDIT: i was wrong, kinetic scrolling only relies on GTK+Cairo --[[User:fthiery|flo]]

Wishlist/LiveUSB distro

2007-08-25T00:42:39Z

Fthiery: /* Putting a vnc server on openmoko */

{{wishlist}}

==Introduction==

NOTE: this page may be renamed: portable destkop

An openmoko device could act as '''the perfect geeky swiss knive''': go anywhere with your Linux desktop and tools.

YES, there are tons of liveCDs out there, but CDs/DVDs:
* get easily damaged
* don't fit in a pocket (except for businesscard-sized ones ; too exotic...)
* don't update or save settings/personal data without a supplementary USB device

YES, there are LiveUSB distros but:
* a Neo/OpenMoko device IS sort of an USB stick; one object is better than two
* you may carry my cellphone everywhere, what about usb sticks or cds ?

The biggest argument against these ones is that you'd have to carry an USB cable with you... But it's a standard one, which is good news :) An integrated zippable plug could be an interesting add-on...

Similar functionality can be found in [http://en.wikipedia.org/wiki/Wizpy the Wizpy portable media player], but relies on different methods for working under Windows (cd-rom emulation).

When the openmoko device is in mass storage mode, a host computer should be able to boot on it, presenting a grub menu offering to boot into several images / partitions (payloads) on the transflash: memtest, UBCD (the ultimate boot cd), a lightweight security oriented livecd distro, you name it...

It's sometimes called Live USB: from [http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB definition], ''A live USB is a USB flash drive containing a full operating system which can be booted from. Live USBs are closely related to Live CDs, and are sometimes used interchangeably. Like Live CDs, Live USBs can be used for system administration, data recovery, or the testing of operating system distributions without committing to a permanent installation on the local hard drive. Many of the smaller Linux distributions can also be used from a USB flash drive.''

Usually, all you need for this with an USB storage device (flash USB stick or external drive) is a partition, flagged "bootable" (see fdisk), containing a boot loader (say grub). But the problem is that specifications vary between motherboards, so there are variants and mandatory requirements to define (here would be a good idea), so that one can optimize/maximize the compatibility.

==The perfect geek knife==

We may want to start our own distro with the following constraints:
* Multiboot: small utilities/compilations + bull blown light OS
* Aiming at maximum hardware compatibility, possibly using liveCD technology (knoppix-based distros)
* Aiming at minimum resource+system footprint, and boot speed
* Being able to install software temporary (for the session) or permanently, upgradable
* Being able to use the phone features at the same time (access pim, data & calls)

Optimally, being able to charge the device while plugged in would be great too.

==About USB Booting==

From [http://www.damnsmalllinux.org/wiki/index.php/USB_Booting DSL Wiki]:

Older computer BIOS usually do not support direct booting from a USB device. Around 2001, PC motherboard manufacturers started to add USB boot support.

There are two common BIOS methods for direct USB booting:
* One method is called the "USBHDD" method and it is used to support the booting of standard USB mass storage devices that are configured like a normal PC hard drive.
* The other method is called the "USBZIP" method and it supports booting from a USB storage device that behaves like the original IOMEGA ZIP drive with USB support.

Most computers (just about all Dells, for example) made today have a BIOS that supports the USBHDD method so I expect that this will eventually become the standard way to boot a USB device. However, many motherboards will support BOTH methods, and many older motherboards have USBZIP support.

Some newer BIOSs which support USB 2.0 will not boot from an older pen drive. Using a USB 2.0 compliant one usually solves this problem. Also, some older BIOSs which only support USB 1.1 will not boot from newer drives which support USB 2.0!

Some USB keys don't boot. If this is the case, it may be possible to fix them by installing a new master boot record. (Most keys boot OK by default; some cannot be fixed even by doing this. However, it helps in some cases). Run the command:

''WARNING! May break your primary hard drive if it's named sda''

lilo -M /dev/sda

==USB Booting in Neo1973's kernel==

Although there is very little information about it, the g_storage kernel module is responsible for the Neo's mass storage mode. The boot-or-not criteria might rely on this very module.

==TODO==

Testing/further research has to be conducted to determine:
* if a bootable-flagged partition / grub doesn't mess openmoko up
* if the phone can still act as a phone, or if the booted os can use the gprs functions
* if the phone can recharge on mass storage mode (from the usb cable)
* if a dedicated partition for each payload is needed (see memtest example...)
* if multi-boot is possible: grub?

==Linux distro Howto's: liveUSB creation==

In short:
#Mount USB drive, with e.g. mount /dev/sda1 /flash - can be either FAT16 or FAT32!
#Mount ISO image, with e.g. mount /tmp/dsl-3.2.iso /tmp/iso -o loop
#Copy all contents from ISO to USB drive: cp -vr /tmp/iso/* /flash/
#Rename and move syslinux files to root directory: mv /flash/boot/isolinux/* /flash/
#Rename isolinux.cfg: mv /flash/isolinux.cfg /flash/syslinux.cfg
#Unmount USB drive: umount /flash
#Install syslinux: syslinux /dev/sda1 and eventually set the MBR boot flag for this partition (with fdisk).

*Fedora Core's [http://revisor.fedoraunity.org Revisor] enables easy GUI-driven ISO customization, allowing to choose the output medium (CD/DVD/USB)
*[http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive Installing DSL to USB device - using grub]
*[http://wiki.debian.org/BootUsb Debian BootUSB - using Lilo]
*[http://www.debuntu.org/how-to-install-ubuntu-linux-on-usb-bar Howto install ubuntu (or any other distro) on usb stick]
*[http://www.gentoo.org/doc/en/liveusb.xml Gentoo on usb stick]
*[http://pendrivelinux.com/ Windows & Linux tutorials & resources]:
* Flash installation via Windows: XUbuntu, DSL, Knoppix, Slax, MiniMe
* Flash installation via Linux: Ubuntu Edgy, Knoppix, PCLinuxOS

'''Transflash partitioning schema example'''

* 2 Gb: /dev/sd? (where N is the transflash's number)
* 700 Mb: /dev/sd?1 : containing bootable iso, FAT16
* 1300 Mb: /dev/sd?2 : openmoko & bootable os home partition, EXT3?
* + eventually a swap

The limitation of this method is that the booted OS is static (not-self modifiable). There's the option to install linux using the partition, but it's not really mobility-oriented (liveCDs are optimized for maximum autoconfiguration).

==Alternative methods==

===Booting ISO files ?===

* Grub? From [http://www.freesoftwaremagazine.com/articles/grub_intro#comment-12900 How to install grub on an usb pendrive]:
If you want to boot from a iso file on a harddisk, do something in menu.lst like
title Boot from iso on a harddisk
map (hdX,Y)/your.iso (hdZ)
map --rehook
chainloader (hdZ)+1
rootnoverify (hdZ)
boot

Grub & memdisk can boot floppy images, but (AFAIK) not isos. Example from the stock /boot/grub/menu.lst ubuntu:
title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin
quiet
boot

''Using the memdisk kernel from the syslinux package, you can load disk images and execute them in a non-emulated environment.''

* [http://www.knoppix.net/wiki/Cheat_Codes knoppix '''bootfrom''' cheatcode]

bootfrom=/dev/hda1/KNX.iso

Bootfrom needs access to a running Knoppix-System '''with the same Kernel as the Bootkernel''', before it is able to mount the partition / ISO-Image.

* Maybe there's a way to achieve a boot cd iso selecter see ([http://www.linuxquestions.org/questions/showthread.php?t=528840 this post])
* virtualization-oriented distro... ex [http://unit.aist.go.jp/itri/knoppix/xen/index-en.html Xenoppix] Cons: would work best only on modern VT-capable computers. What about a very minimal distribution (say DSL without X) kqemu-enblaed, booting into an ncurses iso-loader? The delicate part would be on the networking side.

===Zero-Install / Networked systems===

*[http://klik.atekon.de/wiki/index.php/Main_Page Klik] with more than 1000 packages ported. See [http://klik.atekon.de/wiki/index.php/Comparison comparison chart].
*[http://0install.net/index.html The Zero Install System], with automatic dependancies, upgrading. See [http://0install.net/matrix.html] comparison chart.
*Why not rolling a [http://wiki.rpath.com/wiki/Conary:Concepts Conary/RPath-based] distro, based on [http://www.rpath.com/rbuilder/project/rpath/ rPath linux] reference distribution ? Or choose within the available ones...
* The FUSE [http://httpfs.sourceforge.net/net_boot.htm httpfs] may allow pure http net booting, downloading files from a distant livecd only when needed (this image may be very minimal though).
* [http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&start=75&tstart=0 some people] are thinking of using Amazon S3 as virtual disk

===Using Qemu to boot Linux from a flash drive within Windows===

The following tutorial explains how to use Qemu to boot Linux from a portable USB flash device while still working within Windows. This Enables the user to have both systems running at the same time eliminating the need to restart the PC and set your BIOS options to boot Linux.

[http://www.pendrivelinux.com/2007/03/09/use-qemu-to-boot-linux-from-windows/ PendriveLinux article]

===Putting a vnc server on openmoko + IP over USB or BlueTooth/Wifi===

That's another alternative :) Will bring a VGA-sized capture screen. However, given the current animated behaviour in openmoko, performance will be poor.

===The BlackDog project===

"When plugged into the host computer’s USB port, BlackDog starts up, '''automatically launching the X Window system for Windows Xming and a software NAT router via an autorun application that the BlackDog presents to the host through a virtual USB CD-ROM. Once those applications are running, the virtual USB CD-ROM drive disconnects, and a virtual USB-to-Ethernet adapter is connected to provide the communications link.'''

The host machine’s monitor, keyboard, mouse, and Internet connection are used by the BlackDog for the duration of the session."

The [http://en.wikipedia.org/wiki/BlackDog BlackDog server] project has lots of interesting concepts (and software). Could [http://www.jasonkolb.com/weblog/2006/11/idea_7_personal.html personal servers] change the face of computing ? Together with biometry, ''they open up the possibility of carrying your personal server, or an extension of it, with you on your physical person. The reason this is so exciting is because it enables biometric authentication that doesn't require shipping your biometric data to a third party'.

==Interesting distros/payloads==

'''System diagnostics / recovery'''
*[http://en.wikipedia.org/wiki/Memtest Memtest86+] Very small
*[http://www.ultimatebootcd.com/ The Ultimate Boot CD]
*[http://www.sysresccd.org/Main_Page System Rescue CD]
*[http://www.tux.org/pub/people/kent-robotti/looplinux/rip/ Recovery IS Possible] < 80 MB

'''General purpose'''
*[http://jinx-linux.sourceforge.net/ jinx-linux] < 50 MB
*[http://knoppix.com/ Knoppix] 700 MB CD-ROM
*[http://www.puppylinux.org PuppyLinux] < 100 MB
*[http://www.damnsmalllinux.org DSL] ~50 MB
*[http://gentoo-wiki.com/TinyGentoo#What_is_this_.3F TinyGentoo] 5 MB

'''Security-oriented''': pentesting, forensics, anonymous webbrowsing
*[http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-test-forensics-recovery/ List of Security distros]
*[http://www.remote-exploit.org/backtrack.html Backtrack Livecd]
*[http://en.wikipedia.org/wiki/Anonym.OS Anonym.OS: browse the web anonymously]
*[http://dban.sourceforge.net/ Darik's Boot and Nuke]: ''dangerous''
*[http://byzgl.sourceforge.net/wiki/index.php/Main_Page Byzantine OS] is a webkiosk appliance (32/48 MB)

The SabayonLinux distro offers a lot of [http://www.sabayonlinux.org/wiki/index.php?title=Sabayon_Linux#Boot_Methods_for_special_ways_of_using_Sabayon boot cheatcodes], such as booting onto GeexBox. We should take a look at the method used.

'''[http://www.livecdlist.com/ List of live distros @ livecdlist.com]''' ''Includes sizes''

'''[http://en.wikipedia.org/wiki/List_of_LiveDistros List of live distros @ wikipedia]'''

'''[http://distrowatch.com/dwres.php?resource=cd List of live distros @ distrowatch]'''

'''[http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB article]'''

Wishlist/LiveUSB distro

2007-08-25T00:41:01Z

Fthiery: /* Zero-Install / Networked systems */

{{wishlist}}

==Introduction==

NOTE: this page may be renamed: portable destkop

An openmoko device could act as '''the perfect geeky swiss knive''': go anywhere with your Linux desktop and tools.

YES, there are tons of liveCDs out there, but CDs/DVDs:
* get easily damaged
* don't fit in a pocket (except for businesscard-sized ones ; too exotic...)
* don't update or save settings/personal data without a supplementary USB device

YES, there are LiveUSB distros but:
* a Neo/OpenMoko device IS sort of an USB stick; one object is better than two
* you may carry my cellphone everywhere, what about usb sticks or cds ?

The biggest argument against these ones is that you'd have to carry an USB cable with you... But it's a standard one, which is good news :) An integrated zippable plug could be an interesting add-on...

Similar functionality can be found in [http://en.wikipedia.org/wiki/Wizpy the Wizpy portable media player], but relies on different methods for working under Windows (cd-rom emulation).

When the openmoko device is in mass storage mode, a host computer should be able to boot on it, presenting a grub menu offering to boot into several images / partitions (payloads) on the transflash: memtest, UBCD (the ultimate boot cd), a lightweight security oriented livecd distro, you name it...

It's sometimes called Live USB: from [http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB definition], ''A live USB is a USB flash drive containing a full operating system which can be booted from. Live USBs are closely related to Live CDs, and are sometimes used interchangeably. Like Live CDs, Live USBs can be used for system administration, data recovery, or the testing of operating system distributions without committing to a permanent installation on the local hard drive. Many of the smaller Linux distributions can also be used from a USB flash drive.''

Usually, all you need for this with an USB storage device (flash USB stick or external drive) is a partition, flagged "bootable" (see fdisk), containing a boot loader (say grub). But the problem is that specifications vary between motherboards, so there are variants and mandatory requirements to define (here would be a good idea), so that one can optimize/maximize the compatibility.

==The perfect geek knife==

We may want to start our own distro with the following constraints:
* Multiboot: small utilities/compilations + bull blown light OS
* Aiming at maximum hardware compatibility, possibly using liveCD technology (knoppix-based distros)
* Aiming at minimum resource+system footprint, and boot speed
* Being able to install software temporary (for the session) or permanently, upgradable
* Being able to use the phone features at the same time (access pim, data & calls)

Optimally, being able to charge the device while plugged in would be great too.

==About USB Booting==

From [http://www.damnsmalllinux.org/wiki/index.php/USB_Booting DSL Wiki]:

Older computer BIOS usually do not support direct booting from a USB device. Around 2001, PC motherboard manufacturers started to add USB boot support.

There are two common BIOS methods for direct USB booting:
* One method is called the "USBHDD" method and it is used to support the booting of standard USB mass storage devices that are configured like a normal PC hard drive.
* The other method is called the "USBZIP" method and it supports booting from a USB storage device that behaves like the original IOMEGA ZIP drive with USB support.

Most computers (just about all Dells, for example) made today have a BIOS that supports the USBHDD method so I expect that this will eventually become the standard way to boot a USB device. However, many motherboards will support BOTH methods, and many older motherboards have USBZIP support.

Some newer BIOSs which support USB 2.0 will not boot from an older pen drive. Using a USB 2.0 compliant one usually solves this problem. Also, some older BIOSs which only support USB 1.1 will not boot from newer drives which support USB 2.0!

Some USB keys don't boot. If this is the case, it may be possible to fix them by installing a new master boot record. (Most keys boot OK by default; some cannot be fixed even by doing this. However, it helps in some cases). Run the command:

''WARNING! May break your primary hard drive if it's named sda''

lilo -M /dev/sda

==USB Booting in Neo1973's kernel==

Although there is very little information about it, the g_storage kernel module is responsible for the Neo's mass storage mode. The boot-or-not criteria might rely on this very module.

==TODO==

Testing/further research has to be conducted to determine:
* if a bootable-flagged partition / grub doesn't mess openmoko up
* if the phone can still act as a phone, or if the booted os can use the gprs functions
* if the phone can recharge on mass storage mode (from the usb cable)
* if a dedicated partition for each payload is needed (see memtest example...)
* if multi-boot is possible: grub?

==Linux distro Howto's: liveUSB creation==

In short:
#Mount USB drive, with e.g. mount /dev/sda1 /flash - can be either FAT16 or FAT32!
#Mount ISO image, with e.g. mount /tmp/dsl-3.2.iso /tmp/iso -o loop
#Copy all contents from ISO to USB drive: cp -vr /tmp/iso/* /flash/
#Rename and move syslinux files to root directory: mv /flash/boot/isolinux/* /flash/
#Rename isolinux.cfg: mv /flash/isolinux.cfg /flash/syslinux.cfg
#Unmount USB drive: umount /flash
#Install syslinux: syslinux /dev/sda1 and eventually set the MBR boot flag for this partition (with fdisk).

*Fedora Core's [http://revisor.fedoraunity.org Revisor] enables easy GUI-driven ISO customization, allowing to choose the output medium (CD/DVD/USB)
*[http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive Installing DSL to USB device - using grub]
*[http://wiki.debian.org/BootUsb Debian BootUSB - using Lilo]
*[http://www.debuntu.org/how-to-install-ubuntu-linux-on-usb-bar Howto install ubuntu (or any other distro) on usb stick]
*[http://www.gentoo.org/doc/en/liveusb.xml Gentoo on usb stick]
*[http://pendrivelinux.com/ Windows & Linux tutorials & resources]:
* Flash installation via Windows: XUbuntu, DSL, Knoppix, Slax, MiniMe
* Flash installation via Linux: Ubuntu Edgy, Knoppix, PCLinuxOS

'''Transflash partitioning schema example'''

* 2 Gb: /dev/sd? (where N is the transflash's number)
* 700 Mb: /dev/sd?1 : containing bootable iso, FAT16
* 1300 Mb: /dev/sd?2 : openmoko & bootable os home partition, EXT3?
* + eventually a swap

The limitation of this method is that the booted OS is static (not-self modifiable). There's the option to install linux using the partition, but it's not really mobility-oriented (liveCDs are optimized for maximum autoconfiguration).

==Alternative methods==

===Booting ISO files ?===

* Grub? From [http://www.freesoftwaremagazine.com/articles/grub_intro#comment-12900 How to install grub on an usb pendrive]:
If you want to boot from a iso file on a harddisk, do something in menu.lst like
title Boot from iso on a harddisk
map (hdX,Y)/your.iso (hdZ)
map --rehook
chainloader (hdZ)+1
rootnoverify (hdZ)
boot

Grub & memdisk can boot floppy images, but (AFAIK) not isos. Example from the stock /boot/grub/menu.lst ubuntu:
title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin
quiet
boot

''Using the memdisk kernel from the syslinux package, you can load disk images and execute them in a non-emulated environment.''

* [http://www.knoppix.net/wiki/Cheat_Codes knoppix '''bootfrom''' cheatcode]

bootfrom=/dev/hda1/KNX.iso

Bootfrom needs access to a running Knoppix-System '''with the same Kernel as the Bootkernel''', before it is able to mount the partition / ISO-Image.

* Maybe there's a way to achieve a boot cd iso selecter see ([http://www.linuxquestions.org/questions/showthread.php?t=528840 this post])
* virtualization-oriented distro... ex [http://unit.aist.go.jp/itri/knoppix/xen/index-en.html Xenoppix] Cons: would work best only on modern VT-capable computers. What about a very minimal distribution (say DSL without X) kqemu-enblaed, booting into an ncurses iso-loader? The delicate part would be on the networking side.

===Zero-Install / Networked systems===

*[http://klik.atekon.de/wiki/index.php/Main_Page Klik] with more than 1000 packages ported. See [http://klik.atekon.de/wiki/index.php/Comparison comparison chart].
*[http://0install.net/index.html The Zero Install System], with automatic dependancies, upgrading. See [http://0install.net/matrix.html] comparison chart.
*Why not rolling a [http://wiki.rpath.com/wiki/Conary:Concepts Conary/RPath-based] distro, based on [http://www.rpath.com/rbuilder/project/rpath/ rPath linux] reference distribution ? Or choose within the available ones...
* The FUSE [http://httpfs.sourceforge.net/net_boot.htm httpfs] may allow pure http net booting, downloading files from a distant livecd only when needed (this image may be very minimal though).
* [http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&start=75&tstart=0 some people] are thinking of using Amazon S3 as virtual disk

===Using Qemu to boot Linux from a flash drive within Windows===

The following tutorial explains how to use Qemu to boot Linux from a portable USB flash device while still working within Windows. This Enables the user to have both systems running at the same time eliminating the need to restart the PC and set your BIOS options to boot Linux.

[http://www.pendrivelinux.com/2007/03/09/use-qemu-to-boot-linux-from-windows/ PendriveLinux article]

===Putting a vnc server on openmoko===

That's another alternative :)

===The BlackDog project===

"When plugged into the host computer’s USB port, BlackDog starts up, '''automatically launching the X Window system for Windows Xming and a software NAT router via an autorun application that the BlackDog presents to the host through a virtual USB CD-ROM. Once those applications are running, the virtual USB CD-ROM drive disconnects, and a virtual USB-to-Ethernet adapter is connected to provide the communications link.'''

The host machine’s monitor, keyboard, mouse, and Internet connection are used by the BlackDog for the duration of the session."

The [http://en.wikipedia.org/wiki/BlackDog BlackDog server] project has lots of interesting concepts (and software). Could [http://www.jasonkolb.com/weblog/2006/11/idea_7_personal.html personal servers] change the face of computing ? Together with biometry, ''they open up the possibility of carrying your personal server, or an extension of it, with you on your physical person. The reason this is so exciting is because it enables biometric authentication that doesn't require shipping your biometric data to a third party'.

==Interesting distros/payloads==

'''System diagnostics / recovery'''
*[http://en.wikipedia.org/wiki/Memtest Memtest86+] Very small
*[http://www.ultimatebootcd.com/ The Ultimate Boot CD]
*[http://www.sysresccd.org/Main_Page System Rescue CD]
*[http://www.tux.org/pub/people/kent-robotti/looplinux/rip/ Recovery IS Possible] < 80 MB

'''General purpose'''
*[http://jinx-linux.sourceforge.net/ jinx-linux] < 50 MB
*[http://knoppix.com/ Knoppix] 700 MB CD-ROM
*[http://www.puppylinux.org PuppyLinux] < 100 MB
*[http://www.damnsmalllinux.org DSL] ~50 MB
*[http://gentoo-wiki.com/TinyGentoo#What_is_this_.3F TinyGentoo] 5 MB

'''Security-oriented''': pentesting, forensics, anonymous webbrowsing
*[http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-test-forensics-recovery/ List of Security distros]
*[http://www.remote-exploit.org/backtrack.html Backtrack Livecd]
*[http://en.wikipedia.org/wiki/Anonym.OS Anonym.OS: browse the web anonymously]
*[http://dban.sourceforge.net/ Darik's Boot and Nuke]: ''dangerous''
*[http://byzgl.sourceforge.net/wiki/index.php/Main_Page Byzantine OS] is a webkiosk appliance (32/48 MB)

The SabayonLinux distro offers a lot of [http://www.sabayonlinux.org/wiki/index.php?title=Sabayon_Linux#Boot_Methods_for_special_ways_of_using_Sabayon boot cheatcodes], such as booting onto GeexBox. We should take a look at the method used.

'''[http://www.livecdlist.com/ List of live distros @ livecdlist.com]''' ''Includes sizes''

'''[http://en.wikipedia.org/wiki/List_of_LiveDistros List of live distros @ wikipedia]'''

'''[http://distrowatch.com/dwres.php?resource=cd List of live distros @ distrowatch]'''

'''[http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB article]'''

Wishlist/LiveUSB distro

2007-08-25T00:40:23Z

Fthiery: /* Booting ISOs ? */

{{wishlist}}

==Introduction==

NOTE: this page may be renamed: portable destkop

An openmoko device could act as '''the perfect geeky swiss knive''': go anywhere with your Linux desktop and tools.

YES, there are tons of liveCDs out there, but CDs/DVDs:
* get easily damaged
* don't fit in a pocket (except for businesscard-sized ones ; too exotic...)
* don't update or save settings/personal data without a supplementary USB device

YES, there are LiveUSB distros but:
* a Neo/OpenMoko device IS sort of an USB stick; one object is better than two
* you may carry my cellphone everywhere, what about usb sticks or cds ?

The biggest argument against these ones is that you'd have to carry an USB cable with you... But it's a standard one, which is good news :) An integrated zippable plug could be an interesting add-on...

Similar functionality can be found in [http://en.wikipedia.org/wiki/Wizpy the Wizpy portable media player], but relies on different methods for working under Windows (cd-rom emulation).

When the openmoko device is in mass storage mode, a host computer should be able to boot on it, presenting a grub menu offering to boot into several images / partitions (payloads) on the transflash: memtest, UBCD (the ultimate boot cd), a lightweight security oriented livecd distro, you name it...

It's sometimes called Live USB: from [http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB definition], ''A live USB is a USB flash drive containing a full operating system which can be booted from. Live USBs are closely related to Live CDs, and are sometimes used interchangeably. Like Live CDs, Live USBs can be used for system administration, data recovery, or the testing of operating system distributions without committing to a permanent installation on the local hard drive. Many of the smaller Linux distributions can also be used from a USB flash drive.''

Usually, all you need for this with an USB storage device (flash USB stick or external drive) is a partition, flagged "bootable" (see fdisk), containing a boot loader (say grub). But the problem is that specifications vary between motherboards, so there are variants and mandatory requirements to define (here would be a good idea), so that one can optimize/maximize the compatibility.

==The perfect geek knife==

We may want to start our own distro with the following constraints:
* Multiboot: small utilities/compilations + bull blown light OS
* Aiming at maximum hardware compatibility, possibly using liveCD technology (knoppix-based distros)
* Aiming at minimum resource+system footprint, and boot speed
* Being able to install software temporary (for the session) or permanently, upgradable
* Being able to use the phone features at the same time (access pim, data & calls)

Optimally, being able to charge the device while plugged in would be great too.

==About USB Booting==

From [http://www.damnsmalllinux.org/wiki/index.php/USB_Booting DSL Wiki]:

Older computer BIOS usually do not support direct booting from a USB device. Around 2001, PC motherboard manufacturers started to add USB boot support.

There are two common BIOS methods for direct USB booting:
* One method is called the "USBHDD" method and it is used to support the booting of standard USB mass storage devices that are configured like a normal PC hard drive.
* The other method is called the "USBZIP" method and it supports booting from a USB storage device that behaves like the original IOMEGA ZIP drive with USB support.

Most computers (just about all Dells, for example) made today have a BIOS that supports the USBHDD method so I expect that this will eventually become the standard way to boot a USB device. However, many motherboards will support BOTH methods, and many older motherboards have USBZIP support.

Some newer BIOSs which support USB 2.0 will not boot from an older pen drive. Using a USB 2.0 compliant one usually solves this problem. Also, some older BIOSs which only support USB 1.1 will not boot from newer drives which support USB 2.0!

Some USB keys don't boot. If this is the case, it may be possible to fix them by installing a new master boot record. (Most keys boot OK by default; some cannot be fixed even by doing this. However, it helps in some cases). Run the command:

''WARNING! May break your primary hard drive if it's named sda''

lilo -M /dev/sda

==USB Booting in Neo1973's kernel==

Although there is very little information about it, the g_storage kernel module is responsible for the Neo's mass storage mode. The boot-or-not criteria might rely on this very module.

==TODO==

Testing/further research has to be conducted to determine:
* if a bootable-flagged partition / grub doesn't mess openmoko up
* if the phone can still act as a phone, or if the booted os can use the gprs functions
* if the phone can recharge on mass storage mode (from the usb cable)
* if a dedicated partition for each payload is needed (see memtest example...)
* if multi-boot is possible: grub?

==Linux distro Howto's: liveUSB creation==

In short:
#Mount USB drive, with e.g. mount /dev/sda1 /flash - can be either FAT16 or FAT32!
#Mount ISO image, with e.g. mount /tmp/dsl-3.2.iso /tmp/iso -o loop
#Copy all contents from ISO to USB drive: cp -vr /tmp/iso/* /flash/
#Rename and move syslinux files to root directory: mv /flash/boot/isolinux/* /flash/
#Rename isolinux.cfg: mv /flash/isolinux.cfg /flash/syslinux.cfg
#Unmount USB drive: umount /flash
#Install syslinux: syslinux /dev/sda1 and eventually set the MBR boot flag for this partition (with fdisk).

*Fedora Core's [http://revisor.fedoraunity.org Revisor] enables easy GUI-driven ISO customization, allowing to choose the output medium (CD/DVD/USB)
*[http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive Installing DSL to USB device - using grub]
*[http://wiki.debian.org/BootUsb Debian BootUSB - using Lilo]
*[http://www.debuntu.org/how-to-install-ubuntu-linux-on-usb-bar Howto install ubuntu (or any other distro) on usb stick]
*[http://www.gentoo.org/doc/en/liveusb.xml Gentoo on usb stick]
*[http://pendrivelinux.com/ Windows & Linux tutorials & resources]:
* Flash installation via Windows: XUbuntu, DSL, Knoppix, Slax, MiniMe
* Flash installation via Linux: Ubuntu Edgy, Knoppix, PCLinuxOS

'''Transflash partitioning schema example'''

* 2 Gb: /dev/sd? (where N is the transflash's number)
* 700 Mb: /dev/sd?1 : containing bootable iso, FAT16
* 1300 Mb: /dev/sd?2 : openmoko & bootable os home partition, EXT3?
* + eventually a swap

The limitation of this method is that the booted OS is static (not-self modifiable). There's the option to install linux using the partition, but it's not really mobility-oriented (liveCDs are optimized for maximum autoconfiguration).

==Alternative methods==

===Booting ISO files ?===

* Grub? From [http://www.freesoftwaremagazine.com/articles/grub_intro#comment-12900 How to install grub on an usb pendrive]:
If you want to boot from a iso file on a harddisk, do something in menu.lst like
title Boot from iso on a harddisk
map (hdX,Y)/your.iso (hdZ)
map --rehook
chainloader (hdZ)+1
rootnoverify (hdZ)
boot

Grub & memdisk can boot floppy images, but (AFAIK) not isos. Example from the stock /boot/grub/menu.lst ubuntu:
title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin
quiet
boot

''Using the memdisk kernel from the syslinux package, you can load disk images and execute them in a non-emulated environment.''

* [http://www.knoppix.net/wiki/Cheat_Codes knoppix '''bootfrom''' cheatcode]

bootfrom=/dev/hda1/KNX.iso

Bootfrom needs access to a running Knoppix-System '''with the same Kernel as the Bootkernel''', before it is able to mount the partition / ISO-Image.

* Maybe there's a way to achieve a boot cd iso selecter see ([http://www.linuxquestions.org/questions/showthread.php?t=528840 this post])
* virtualization-oriented distro... ex [http://unit.aist.go.jp/itri/knoppix/xen/index-en.html Xenoppix] Cons: would work best only on modern VT-capable computers. What about a very minimal distribution (say DSL without X) kqemu-enblaed, booting into an ncurses iso-loader? The delicate part would be on the networking side.

===Zero-Install / Networked systems===

*[http://klik.atekon.de/wiki/index.php/Main_Page Klik] with more than 1000 packages ported. See [http://klik.atekon.de/wiki/index.php/Comparison comparison chart].
*[http://0install.net/index.html The Zero Install System], with automatic dependancies, upgrading. See [http://0install.net/matrix.html] comparison chart.
*Why not rolling a [http://wiki.rpath.com/wiki/Conary:Concepts Conary/RPath-based] distro, based on [http://www.rpath.com/rbuilder/project/rpath/ rPath linux] reference distribution ? Or choose within the available ones...
* The FUSE [http://httpfs.sourceforge.net/net_boot.htm httpfs] may allow pure http net booting, downloading files from a distant livecd only when needed (this image may be very minimal though).
* [http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&start=75&tstart=0 some people] hare thinking of using Amazon S3 as virtual disk

===Using Qemu to boot Linux from a flash drive within Windows===

The following tutorial explains how to use Qemu to boot Linux from a portable USB flash device while still working within Windows. This Enables the user to have both systems running at the same time eliminating the need to restart the PC and set your BIOS options to boot Linux.

[http://www.pendrivelinux.com/2007/03/09/use-qemu-to-boot-linux-from-windows/ PendriveLinux article]

===Putting a vnc server on openmoko===

That's another alternative :)

===The BlackDog project===

"When plugged into the host computer’s USB port, BlackDog starts up, '''automatically launching the X Window system for Windows Xming and a software NAT router via an autorun application that the BlackDog presents to the host through a virtual USB CD-ROM. Once those applications are running, the virtual USB CD-ROM drive disconnects, and a virtual USB-to-Ethernet adapter is connected to provide the communications link.'''

The host machine’s monitor, keyboard, mouse, and Internet connection are used by the BlackDog for the duration of the session."

The [http://en.wikipedia.org/wiki/BlackDog BlackDog server] project has lots of interesting concepts (and software). Could [http://www.jasonkolb.com/weblog/2006/11/idea_7_personal.html personal servers] change the face of computing ? Together with biometry, ''they open up the possibility of carrying your personal server, or an extension of it, with you on your physical person. The reason this is so exciting is because it enables biometric authentication that doesn't require shipping your biometric data to a third party'.

==Interesting distros/payloads==

'''System diagnostics / recovery'''
*[http://en.wikipedia.org/wiki/Memtest Memtest86+] Very small
*[http://www.ultimatebootcd.com/ The Ultimate Boot CD]
*[http://www.sysresccd.org/Main_Page System Rescue CD]
*[http://www.tux.org/pub/people/kent-robotti/looplinux/rip/ Recovery IS Possible] < 80 MB

'''General purpose'''
*[http://jinx-linux.sourceforge.net/ jinx-linux] < 50 MB
*[http://knoppix.com/ Knoppix] 700 MB CD-ROM
*[http://www.puppylinux.org PuppyLinux] < 100 MB
*[http://www.damnsmalllinux.org DSL] ~50 MB
*[http://gentoo-wiki.com/TinyGentoo#What_is_this_.3F TinyGentoo] 5 MB

'''Security-oriented''': pentesting, forensics, anonymous webbrowsing
*[http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-test-forensics-recovery/ List of Security distros]
*[http://www.remote-exploit.org/backtrack.html Backtrack Livecd]
*[http://en.wikipedia.org/wiki/Anonym.OS Anonym.OS: browse the web anonymously]
*[http://dban.sourceforge.net/ Darik's Boot and Nuke]: ''dangerous''
*[http://byzgl.sourceforge.net/wiki/index.php/Main_Page Byzantine OS] is a webkiosk appliance (32/48 MB)

The SabayonLinux distro offers a lot of [http://www.sabayonlinux.org/wiki/index.php?title=Sabayon_Linux#Boot_Methods_for_special_ways_of_using_Sabayon boot cheatcodes], such as booting onto GeexBox. We should take a look at the method used.

'''[http://www.livecdlist.com/ List of live distros @ livecdlist.com]''' ''Includes sizes''

'''[http://en.wikipedia.org/wiki/List_of_LiveDistros List of live distros @ wikipedia]'''

'''[http://distrowatch.com/dwres.php?resource=cd List of live distros @ distrowatch]'''

'''[http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB article]'''

Wishlist/LiveUSB distro

2007-08-25T00:40:00Z

Fthiery: Fixed typo

{{wishlist}}

==Introduction==

NOTE: this page may be renamed: portable destkop

An openmoko device could act as '''the perfect geeky swiss knive''': go anywhere with your Linux desktop and tools.

YES, there are tons of liveCDs out there, but CDs/DVDs:
* get easily damaged
* don't fit in a pocket (except for businesscard-sized ones ; too exotic...)
* don't update or save settings/personal data without a supplementary USB device

YES, there are LiveUSB distros but:
* a Neo/OpenMoko device IS sort of an USB stick; one object is better than two
* you may carry my cellphone everywhere, what about usb sticks or cds ?

The biggest argument against these ones is that you'd have to carry an USB cable with you... But it's a standard one, which is good news :) An integrated zippable plug could be an interesting add-on...

Similar functionality can be found in [http://en.wikipedia.org/wiki/Wizpy the Wizpy portable media player], but relies on different methods for working under Windows (cd-rom emulation).

When the openmoko device is in mass storage mode, a host computer should be able to boot on it, presenting a grub menu offering to boot into several images / partitions (payloads) on the transflash: memtest, UBCD (the ultimate boot cd), a lightweight security oriented livecd distro, you name it...

It's sometimes called Live USB: from [http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB definition], ''A live USB is a USB flash drive containing a full operating system which can be booted from. Live USBs are closely related to Live CDs, and are sometimes used interchangeably. Like Live CDs, Live USBs can be used for system administration, data recovery, or the testing of operating system distributions without committing to a permanent installation on the local hard drive. Many of the smaller Linux distributions can also be used from a USB flash drive.''

Usually, all you need for this with an USB storage device (flash USB stick or external drive) is a partition, flagged "bootable" (see fdisk), containing a boot loader (say grub). But the problem is that specifications vary between motherboards, so there are variants and mandatory requirements to define (here would be a good idea), so that one can optimize/maximize the compatibility.

==The perfect geek knife==

We may want to start our own distro with the following constraints:
* Multiboot: small utilities/compilations + bull blown light OS
* Aiming at maximum hardware compatibility, possibly using liveCD technology (knoppix-based distros)
* Aiming at minimum resource+system footprint, and boot speed
* Being able to install software temporary (for the session) or permanently, upgradable
* Being able to use the phone features at the same time (access pim, data & calls)

Optimally, being able to charge the device while plugged in would be great too.

==About USB Booting==

From [http://www.damnsmalllinux.org/wiki/index.php/USB_Booting DSL Wiki]:

Older computer BIOS usually do not support direct booting from a USB device. Around 2001, PC motherboard manufacturers started to add USB boot support.

There are two common BIOS methods for direct USB booting:
* One method is called the "USBHDD" method and it is used to support the booting of standard USB mass storage devices that are configured like a normal PC hard drive.
* The other method is called the "USBZIP" method and it supports booting from a USB storage device that behaves like the original IOMEGA ZIP drive with USB support.

Most computers (just about all Dells, for example) made today have a BIOS that supports the USBHDD method so I expect that this will eventually become the standard way to boot a USB device. However, many motherboards will support BOTH methods, and many older motherboards have USBZIP support.

Some newer BIOSs which support USB 2.0 will not boot from an older pen drive. Using a USB 2.0 compliant one usually solves this problem. Also, some older BIOSs which only support USB 1.1 will not boot from newer drives which support USB 2.0!

Some USB keys don't boot. If this is the case, it may be possible to fix them by installing a new master boot record. (Most keys boot OK by default; some cannot be fixed even by doing this. However, it helps in some cases). Run the command:

''WARNING! May break your primary hard drive if it's named sda''

lilo -M /dev/sda

==USB Booting in Neo1973's kernel==

Although there is very little information about it, the g_storage kernel module is responsible for the Neo's mass storage mode. The boot-or-not criteria might rely on this very module.

==TODO==

Testing/further research has to be conducted to determine:
* if a bootable-flagged partition / grub doesn't mess openmoko up
* if the phone can still act as a phone, or if the booted os can use the gprs functions
* if the phone can recharge on mass storage mode (from the usb cable)
* if a dedicated partition for each payload is needed (see memtest example...)
* if multi-boot is possible: grub?

==Linux distro Howto's: liveUSB creation==

In short:
#Mount USB drive, with e.g. mount /dev/sda1 /flash - can be either FAT16 or FAT32!
#Mount ISO image, with e.g. mount /tmp/dsl-3.2.iso /tmp/iso -o loop
#Copy all contents from ISO to USB drive: cp -vr /tmp/iso/* /flash/
#Rename and move syslinux files to root directory: mv /flash/boot/isolinux/* /flash/
#Rename isolinux.cfg: mv /flash/isolinux.cfg /flash/syslinux.cfg
#Unmount USB drive: umount /flash
#Install syslinux: syslinux /dev/sda1 and eventually set the MBR boot flag for this partition (with fdisk).

*Fedora Core's [http://revisor.fedoraunity.org Revisor] enables easy GUI-driven ISO customization, allowing to choose the output medium (CD/DVD/USB)
*[http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive Installing DSL to USB device - using grub]
*[http://wiki.debian.org/BootUsb Debian BootUSB - using Lilo]
*[http://www.debuntu.org/how-to-install-ubuntu-linux-on-usb-bar Howto install ubuntu (or any other distro) on usb stick]
*[http://www.gentoo.org/doc/en/liveusb.xml Gentoo on usb stick]
*[http://pendrivelinux.com/ Windows & Linux tutorials & resources]:
* Flash installation via Windows: XUbuntu, DSL, Knoppix, Slax, MiniMe
* Flash installation via Linux: Ubuntu Edgy, Knoppix, PCLinuxOS

'''Transflash partitioning schema example'''

* 2 Gb: /dev/sd? (where N is the transflash's number)
* 700 Mb: /dev/sd?1 : containing bootable iso, FAT16
* 1300 Mb: /dev/sd?2 : openmoko & bootable os home partition, EXT3?
* + eventually a swap

The limitation of this method is that the booted OS is static (not-self modifiable). There's the option to install linux using the partition, but it's not really mobility-oriented (liveCDs are optimized for maximum autoconfiguration).

==Alternative methods==

===Booting ISOs ?===

* Grub? From [http://www.freesoftwaremagazine.com/articles/grub_intro#comment-12900 How to install grub on an usb pendrive]:
If you want to boot from a iso file on a harddisk, do something in menu.lst like
title Boot from iso on a harddisk
map (hdX,Y)/your.iso (hdZ)
map --rehook
chainloader (hdZ)+1
rootnoverify (hdZ)
boot

Grub & memdisk can boot floppy images, but (AFAIK) not isos. Example from the stock /boot/grub/menu.lst ubuntu:
title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin
quiet
boot

''Using the memdisk kernel from the syslinux package, you can load disk images and execute them in a non-emulated environment.''

* [http://www.knoppix.net/wiki/Cheat_Codes knoppix '''bootfrom''' cheatcode]

bootfrom=/dev/hda1/KNX.iso

Bootfrom needs access to a running Knoppix-System '''with the same Kernel as the Bootkernel''', before it is able to mount the partition / ISO-Image.

* Maybe there's a way to achieve a boot cd iso selecter see ([http://www.linuxquestions.org/questions/showthread.php?t=528840 this post])
* virtualization-oriented distro... ex [http://unit.aist.go.jp/itri/knoppix/xen/index-en.html Xenoppix] Cons: would work best only on modern VT-capable computers. What about a very minimal distribution (say DSL without X) kqemu-enblaed, booting into an ncurses iso-loader? The delicate part would be on the networking side.

===Zero-Install / Networked systems===

*[http://klik.atekon.de/wiki/index.php/Main_Page Klik] with more than 1000 packages ported. See [http://klik.atekon.de/wiki/index.php/Comparison comparison chart].
*[http://0install.net/index.html The Zero Install System], with automatic dependancies, upgrading. See [http://0install.net/matrix.html] comparison chart.
*Why not rolling a [http://wiki.rpath.com/wiki/Conary:Concepts Conary/RPath-based] distro, based on [http://www.rpath.com/rbuilder/project/rpath/ rPath linux] reference distribution ? Or choose within the available ones...
* The FUSE [http://httpfs.sourceforge.net/net_boot.htm httpfs] may allow pure http net booting, downloading files from a distant livecd only when needed (this image may be very minimal though).
* [http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&start=75&tstart=0 some people] hare thinking of using Amazon S3 as virtual disk

===Using Qemu to boot Linux from a flash drive within Windows===

The following tutorial explains how to use Qemu to boot Linux from a portable USB flash device while still working within Windows. This Enables the user to have both systems running at the same time eliminating the need to restart the PC and set your BIOS options to boot Linux.

[http://www.pendrivelinux.com/2007/03/09/use-qemu-to-boot-linux-from-windows/ PendriveLinux article]

===Putting a vnc server on openmoko===

That's another alternative :)

===The BlackDog project===

"When plugged into the host computer’s USB port, BlackDog starts up, '''automatically launching the X Window system for Windows Xming and a software NAT router via an autorun application that the BlackDog presents to the host through a virtual USB CD-ROM. Once those applications are running, the virtual USB CD-ROM drive disconnects, and a virtual USB-to-Ethernet adapter is connected to provide the communications link.'''

The host machine’s monitor, keyboard, mouse, and Internet connection are used by the BlackDog for the duration of the session."

The [http://en.wikipedia.org/wiki/BlackDog BlackDog server] project has lots of interesting concepts (and software). Could [http://www.jasonkolb.com/weblog/2006/11/idea_7_personal.html personal servers] change the face of computing ? Together with biometry, ''they open up the possibility of carrying your personal server, or an extension of it, with you on your physical person. The reason this is so exciting is because it enables biometric authentication that doesn't require shipping your biometric data to a third party'.

==Interesting distros/payloads==

'''System diagnostics / recovery'''
*[http://en.wikipedia.org/wiki/Memtest Memtest86+] Very small
*[http://www.ultimatebootcd.com/ The Ultimate Boot CD]
*[http://www.sysresccd.org/Main_Page System Rescue CD]
*[http://www.tux.org/pub/people/kent-robotti/looplinux/rip/ Recovery IS Possible] < 80 MB

'''General purpose'''
*[http://jinx-linux.sourceforge.net/ jinx-linux] < 50 MB
*[http://knoppix.com/ Knoppix] 700 MB CD-ROM
*[http://www.puppylinux.org PuppyLinux] < 100 MB
*[http://www.damnsmalllinux.org DSL] ~50 MB
*[http://gentoo-wiki.com/TinyGentoo#What_is_this_.3F TinyGentoo] 5 MB

'''Security-oriented''': pentesting, forensics, anonymous webbrowsing
*[http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-test-forensics-recovery/ List of Security distros]
*[http://www.remote-exploit.org/backtrack.html Backtrack Livecd]
*[http://en.wikipedia.org/wiki/Anonym.OS Anonym.OS: browse the web anonymously]
*[http://dban.sourceforge.net/ Darik's Boot and Nuke]: ''dangerous''
*[http://byzgl.sourceforge.net/wiki/index.php/Main_Page Byzantine OS] is a webkiosk appliance (32/48 MB)

The SabayonLinux distro offers a lot of [http://www.sabayonlinux.org/wiki/index.php?title=Sabayon_Linux#Boot_Methods_for_special_ways_of_using_Sabayon boot cheatcodes], such as booting onto GeexBox. We should take a look at the method used.

'''[http://www.livecdlist.com/ List of live distros @ livecdlist.com]''' ''Includes sizes''

'''[http://en.wikipedia.org/wiki/List_of_LiveDistros List of live distros @ wikipedia]'''

'''[http://distrowatch.com/dwres.php?resource=cd List of live distros @ distrowatch]'''

'''[http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB article]'''

Wishlist/LiveUSB distro

2007-08-25T00:39:28Z

Fthiery: Typo fix

{{wishlist}}

==Introduction==

NOTE: this page may be renamed: portable destkop

An openmoko device could act as '''the perfect geeky swiss knive''': go anywhere with your Linux desktop and tools.

YES, there are tons of liveCDs out there, but CDs/DVDs:
* get easily damaged
* don't fit in a pocket (except for businesscard-sized ones ; too exotic...)
* don't update or save settings/personal data without a supplementary USB device

YES, there are LiveUSB distros but:
* a Neo/OpenMoko device IS sort of an USB stick; one object is better than two
* you may carry my cellphone everywhere, what about usb sticks or cds ?

The biggest argument against these ones is that you'd have to carry an USB cable with you... But it's a standard one, which is good news :) An integrated zippable plug could be an interesting add-on...

Similar functionality can be found in [http://en.wikipedia.org/wiki/Wizpy the Wizpy portable media player], but relies on different methods for working under Windows (cd-rom emulation).

When the openmoko device is in mass storage mode, a host computer should be able to boot on it, presenting a grub menu offering to boot into several images / partitions (payloads) on the transflash: memtest, UBCD (the ultimate boot cd), a lightweight security oriented livecd distro, you name it...

It's sometimes called Live USB: from [http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB definition], ''A live USB is a USB flash drive containing a full operating system which can be booted from. Live USBs are closely related to Live CDs, and are sometimes used interchangeably. Like Live CDs, Live USBs can be used for system administration, data recovery, or the testing of operating system distributions without committing to a permanent installation on the local hard drive. Many of the smaller Linux distributions can also be used from a USB flash drive.''

Usually, all you need for this with an USB storage device (flash USB stick or external drive) is a partition, flagged "bootable" (see fdisk), containing a boot loader (say grub). But the problem is that specifications vary between motherboards, so there are variants and mandatory requirements to define (here would be a good idea), so that one can optimize/maximize the compatibility.

==The perfect geek knife==

We may want to start our own distro with the following constraints:
* Multiboot: small utilities/compilations + bull blown light OS
* Aiming at maximum hardware compatibility, possibly using liveCD technology (knoppix-based distros)
* Aiming at minimum resource+system footprint, and boot speed
* Being able to install software temporary (for the session) or permanently, upgradable
* Being able to use the phone features at the same time (access pim, data & calls)

Optimally, being able to charge the device while plugged in would be great too.

==About USB Booting==

From [http://www.damnsmalllinux.org/wiki/index.php/USB_Booting DSL Wiki]:

Older computer BIOS usually do not support direct booting from a USB device. Around 2001, PC motherboard manufacturers started to add USB boot support.

There are two common BIOS methods for direct USB booting:
* One method is called the "USBHDD" method and it is used to support the booting of standard USB mass storage devices that are configured like a normal PC hard drive.
* The other method is called the "USBZIP" method and it supports booting from a USB storage device that behaves like the original IOMEGA ZIP drive with USB support.

Most computers (just about all Dells, for example) made today have a BIOS that supports the USBHDD method so I expect that this will eventually become the standard way to boot a USB device. However, many motherboards will support BOTH methods, and many older motherboards have USBZIP support.

Some newer BIOSs which support USB 2.0 will not boot from an older pen drive. Using a USB 2.0 compliant one usually solves this problem. Also, some older BIOSs which only support USB 1.1 will not boot from newer drives which support USB 2.0!

Some USB keys don't boot. If this is the case, it may be possible to fix them by installing a new master boot record. (Most keys boot OK by default; some cannot be fixed even by doing this. However, it helps in some cases). Run the command:

''WARNING! May break your primary hard drive if it's named sda''

lilo -M /dev/sda

==USB Booting in Neo1973's kernel==

Although there is very little information about it, the g_storage kernel module is responsible for the Neo's mass storage mode. So the boot-or-not criteria might rely on this very module.

==TODO==

Testing/further research has to be conducted to determine:
* if a bootable-flagged partition / grub doesn't mess openmoko up
* if the phone can still act as a phone, or if the booted os can use the gprs functions
* if the phone can recharge on mass storage mode (from the usb cable)
* if a dedicated partition for each payload is needed (see memtest example...)
* if multi-boot is possible: grub?

==Linux distro Howto's: liveUSB creation==

In short:
#Mount USB drive, with e.g. mount /dev/sda1 /flash - can be either FAT16 or FAT32!
#Mount ISO image, with e.g. mount /tmp/dsl-3.2.iso /tmp/iso -o loop
#Copy all contents from ISO to USB drive: cp -vr /tmp/iso/* /flash/
#Rename and move syslinux files to root directory: mv /flash/boot/isolinux/* /flash/
#Rename isolinux.cfg: mv /flash/isolinux.cfg /flash/syslinux.cfg
#Unmount USB drive: umount /flash
#Install syslinux: syslinux /dev/sda1 and eventually set the MBR boot flag for this partition (with fdisk).

*Fedora Core's [http://revisor.fedoraunity.org Revisor] enables easy GUI-driven ISO customization, allowing to choose the output medium (CD/DVD/USB)
*[http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive Installing DSL to USB device - using grub]
*[http://wiki.debian.org/BootUsb Debian BootUSB - using Lilo]
*[http://www.debuntu.org/how-to-install-ubuntu-linux-on-usb-bar Howto install ubuntu (or any other distro) on usb stick]
*[http://www.gentoo.org/doc/en/liveusb.xml Gentoo on usb stick]
*[http://pendrivelinux.com/ Windows & Linux tutorials & resources]:
* Flash installation via Windows: XUbuntu, DSL, Knoppix, Slax, MiniMe
* Flash installation via Linux: Ubuntu Edgy, Knoppix, PCLinuxOS

'''Transflash partitioning schema example'''

* 2 Gb: /dev/sd? (where N is the transflash's number)
* 700 Mb: /dev/sd?1 : containing bootable iso, FAT16
* 1300 Mb: /dev/sd?2 : openmoko & bootable os home partition, EXT3?
* + eventually a swap

The limitation of this method is that the booted OS is static (not-self modifiable). There's the option to install linux using the partition, but it's not really mobility-oriented (liveCDs are optimized for maximum autoconfiguration).

==Alternative methods==

===Booting ISOs ?===

* Grub? From [http://www.freesoftwaremagazine.com/articles/grub_intro#comment-12900 How to install grub on an usb pendrive]:
If you want to boot from a iso file on a harddisk, do something in menu.lst like
title Boot from iso on a harddisk
map (hdX,Y)/your.iso (hdZ)
map --rehook
chainloader (hdZ)+1
rootnoverify (hdZ)
boot

Grub & memdisk can boot floppy images, but (AFAIK) not isos. Example from the stock /boot/grub/menu.lst ubuntu:
title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin
quiet
boot

''Using the memdisk kernel from the syslinux package, you can load disk images and execute them in a non-emulated environment.''

* [http://www.knoppix.net/wiki/Cheat_Codes knoppix '''bootfrom''' cheatcode]

bootfrom=/dev/hda1/KNX.iso

Bootfrom needs access to a running Knoppix-System '''with the same Kernel as the Bootkernel''', before it is able to mount the partition / ISO-Image.

* Maybe there's a way to achieve a boot cd iso selecter see ([http://www.linuxquestions.org/questions/showthread.php?t=528840 this post])
* virtualization-oriented distro... ex [http://unit.aist.go.jp/itri/knoppix/xen/index-en.html Xenoppix] Cons: would work best only on modern VT-capable computers. What about a very minimal distribution (say DSL without X) kqemu-enblaed, booting into an ncurses iso-loader? The delicate part would be on the networking side.

===Zero-Install / Networked systems===

*[http://klik.atekon.de/wiki/index.php/Main_Page Klik] with more than 1000 packages ported. See [http://klik.atekon.de/wiki/index.php/Comparison comparison chart].
*[http://0install.net/index.html The Zero Install System], with automatic dependancies, upgrading. See [http://0install.net/matrix.html] comparison chart.
*Why not rolling a [http://wiki.rpath.com/wiki/Conary:Concepts Conary/RPath-based] distro, based on [http://www.rpath.com/rbuilder/project/rpath/ rPath linux] reference distribution ? Or choose within the available ones...
* The FUSE [http://httpfs.sourceforge.net/net_boot.htm httpfs] may allow pure http net booting, downloading files from a distant livecd only when needed (this image may be very minimal though).
* [http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&start=75&tstart=0 some people] hare thinking of using Amazon S3 as virtual disk

===Using Qemu to boot Linux from a flash drive within Windows===

The following tutorial explains how to use Qemu to boot Linux from a portable USB flash device while still working within Windows. This Enables the user to have both systems running at the same time eliminating the need to restart the PC and set your BIOS options to boot Linux.

[http://www.pendrivelinux.com/2007/03/09/use-qemu-to-boot-linux-from-windows/ PendriveLinux article]

===Putting a vnc server on openmoko===

That's another alternative :)

===The BlackDog project===

"When plugged into the host computer’s USB port, BlackDog starts up, '''automatically launching the X Window system for Windows Xming and a software NAT router via an autorun application that the BlackDog presents to the host through a virtual USB CD-ROM. Once those applications are running, the virtual USB CD-ROM drive disconnects, and a virtual USB-to-Ethernet adapter is connected to provide the communications link.'''

The host machine’s monitor, keyboard, mouse, and Internet connection are used by the BlackDog for the duration of the session."

The [http://en.wikipedia.org/wiki/BlackDog BlackDog server] project has lots of interesting concepts (and software). Could [http://www.jasonkolb.com/weblog/2006/11/idea_7_personal.html personal servers] change the face of computing ? Together with biometry, ''they open up the possibility of carrying your personal server, or an extension of it, with you on your physical person. The reason this is so exciting is because it enables biometric authentication that doesn't require shipping your biometric data to a third party'.

==Interesting distros/payloads==

'''System diagnostics / recovery'''
*[http://en.wikipedia.org/wiki/Memtest Memtest86+] Very small
*[http://www.ultimatebootcd.com/ The Ultimate Boot CD]
*[http://www.sysresccd.org/Main_Page System Rescue CD]
*[http://www.tux.org/pub/people/kent-robotti/looplinux/rip/ Recovery IS Possible] < 80 MB

'''General purpose'''
*[http://jinx-linux.sourceforge.net/ jinx-linux] < 50 MB
*[http://knoppix.com/ Knoppix] 700 MB CD-ROM
*[http://www.puppylinux.org PuppyLinux] < 100 MB
*[http://www.damnsmalllinux.org DSL] ~50 MB
*[http://gentoo-wiki.com/TinyGentoo#What_is_this_.3F TinyGentoo] 5 MB

'''Security-oriented''': pentesting, forensics, anonymous webbrowsing
*[http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-test-forensics-recovery/ List of Security distros]
*[http://www.remote-exploit.org/backtrack.html Backtrack Livecd]
*[http://en.wikipedia.org/wiki/Anonym.OS Anonym.OS: browse the web anonymously]
*[http://dban.sourceforge.net/ Darik's Boot and Nuke]: ''dangerous''
*[http://byzgl.sourceforge.net/wiki/index.php/Main_Page Byzantine OS] is a webkiosk appliance (32/48 MB)

The SabayonLinux distro offers a lot of [http://www.sabayonlinux.org/wiki/index.php?title=Sabayon_Linux#Boot_Methods_for_special_ways_of_using_Sabayon boot cheatcodes], such as booting onto GeexBox. We should take a look at the method used.

'''[http://www.livecdlist.com/ List of live distros @ livecdlist.com]''' ''Includes sizes''

'''[http://en.wikipedia.org/wiki/List_of_LiveDistros List of live distros @ wikipedia]'''

'''[http://distrowatch.com/dwres.php?resource=cd List of live distros @ distrowatch]'''

'''[http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB article]'''

Wishlist/LiveUSB distro

2007-08-25T00:39:06Z

Fthiery: Added warning

{{wishlist}}

==Introduction==

NOTE: this page may be renamed: portable destkop

An openmoko device could act as '''the perfect geeky swiss knive''': go anywhere with your Linux desktop and tools.

YES, there are tons of liveCDs out there, but CDs/DVDs:
* get easily damaged
* don't fit in a pocket (except for businesscard-sized ones ; too exotic...)
* don't update or save settings/personal data without a supplementary USB device

YES, there are LiveUSB distros but:
* a Neo/OpenMoko device IS sort of an USB stick; one object is better than two
* you may carry my cellphone everywhere, what about usb sticks or cds ?

The biggest argument against these ones is that you'd have to carry an USB cable with you... But it's a standard one, which is good news :) An integrated zippable plug could be an interesting add-on...

Similar functionality can be found in [http://en.wikipedia.org/wiki/Wizpy the Wizpy portable media player], but relies on different methods for working under Windows (cd-rom emulation).

When the openmoko device is in mass storage mode, a host computer should be able to boot on it, presenting a grub menu offering to boot into several images / partitions (payloads) on the transflash: memtest, UBCD (the ultimate boot cd), a lightweight security oriented livecd distro, you name it...

It's sometimes called Live USB: from [http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB definition], ''A live USB is a USB flash drive containing a full operating system which can be booted from. Live USBs are closely related to Live CDs, and are sometimes used interchangeably. Like Live CDs, Live USBs can be used for system administration, data recovery, or the testing of operating system distributions without committing to a permanent installation on the local hard drive. Many of the smaller Linux distributions can also be used from a USB flash drive.''

Usually, all you need for this with an USB storage device (flash USB stick or external drive) is a partition, flagged "bootable" (see fdisk), containing a boot loader (say grub). But the problem is that specifications vary between motherboards, so there are variants and mandatory requirements to define (here would be a good idea), so that one can optimize/maximize the compatibility.

==The perfect geek knife==

We may want to start our own distro with the following constraints:
* Multiboot: small utilities/compilations + bull blown light OS
* Aiming at maximum hardware compatibility, possibly using liveCD technology (knoppix-based distros)
* Aiming at minimum resource+system footprint, and boot speed
* Being able to install software temporary (for the session) or permanently, upgradable
* Being able to use the phone features at the same time (access pim, data & calls)

Optimally, being able to charge the device while plugged in would be great too.

==About USB Booting==

From [http://www.damnsmalllinux.org/wiki/index.php/USB_Booting DSL Wiki]:

Older computer BIOS usually do not support direct booting from a USB device. Around 2001, PC motherboard manufacturers started to add USB boot support.

There are two common BIOS methods for direct USB booting:
* One method is called the "USBHDD" method and it is used to support the booting of standard USB mass storage devices that are configured like a normal PC hard drive.
* The other method is called the "USBZIP" method and it supports booting from a USB storage device that behaves like the original IOMEGA ZIP drive with USB support.

Most computers (just about all Dells, for example) made today have a BIOS that supports the USBHDD method so I expect that this will eventually become the standard way to boot a USB device. However, many motherboards will support BOTH methods, and many older motherboards have USBZIP support.

Some newer BIOSs which support USB 2.0 will not boot from an older pen drive. Using a USB 2.0 compliant one usually solves this problem. Also, some older BIOSs which only support USB 1.1 will not boot from newer drives which support USB 2.0!

Some USB keys don't boot. If this is the case, it may be possible to fix them by installing a new master boot record. (Most keys boot OK by default; some cannot be fixed even by doing this. However, it helps in some cases). Run the command:

''WARNING! May break your primary hard drive if it's named sda'' lilo -M /dev/sda

==USB Booting in Neo1973's kernel==

Although there is very little information about it, the g_storage kernel module is responsible for the Neo's mass storage mode. So the boot-or-not criteria might rely on this very module.

==TODO==

Testing/further research has to be conducted to determine:
* if a bootable-flagged partition / grub doesn't mess openmoko up
* if the phone can still act as a phone, or if the booted os can use the gprs functions
* if the phone can recharge on mass storage mode (from the usb cable)
* if a dedicated partition for each payload is needed (see memtest example...)
* if multi-boot is possible: grub?

==Linux distro Howto's: liveUSB creation==

In short:
#Mount USB drive, with e.g. mount /dev/sda1 /flash - can be either FAT16 or FAT32!
#Mount ISO image, with e.g. mount /tmp/dsl-3.2.iso /tmp/iso -o loop
#Copy all contents from ISO to USB drive: cp -vr /tmp/iso/* /flash/
#Rename and move syslinux files to root directory: mv /flash/boot/isolinux/* /flash/
#Rename isolinux.cfg: mv /flash/isolinux.cfg /flash/syslinux.cfg
#Unmount USB drive: umount /flash
#Install syslinux: syslinux /dev/sda1 and eventually set the MBR boot flag for this partition (with fdisk).

*Fedora Core's [http://revisor.fedoraunity.org Revisor] enables easy GUI-driven ISO customization, allowing to choose the output medium (CD/DVD/USB)
*[http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive Installing DSL to USB device - using grub]
*[http://wiki.debian.org/BootUsb Debian BootUSB - using Lilo]
*[http://www.debuntu.org/how-to-install-ubuntu-linux-on-usb-bar Howto install ubuntu (or any other distro) on usb stick]
*[http://www.gentoo.org/doc/en/liveusb.xml Gentoo on usb stick]
*[http://pendrivelinux.com/ Windows & Linux tutorials & resources]:
* Flash installation via Windows: XUbuntu, DSL, Knoppix, Slax, MiniMe
* Flash installation via Linux: Ubuntu Edgy, Knoppix, PCLinuxOS

'''Transflash partitioning schema example'''

* 2 Gb: /dev/sd? (where N is the transflash's number)
* 700 Mb: /dev/sd?1 : containing bootable iso, FAT16
* 1300 Mb: /dev/sd?2 : openmoko & bootable os home partition, EXT3?
* + eventually a swap

The limitation of this method is that the booted OS is static (not-self modifiable). There's the option to install linux using the partition, but it's not really mobility-oriented (liveCDs are optimized for maximum autoconfiguration).

==Alternative methods==

===Booting ISOs ?===

* Grub? From [http://www.freesoftwaremagazine.com/articles/grub_intro#comment-12900 How to install grub on an usb pendrive]:
If you want to boot from a iso file on a harddisk, do something in menu.lst like
title Boot from iso on a harddisk
map (hdX,Y)/your.iso (hdZ)
map --rehook
chainloader (hdZ)+1
rootnoverify (hdZ)
boot

Grub & memdisk can boot floppy images, but (AFAIK) not isos. Example from the stock /boot/grub/menu.lst ubuntu:
title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin
quiet
boot

''Using the memdisk kernel from the syslinux package, you can load disk images and execute them in a non-emulated environment.''

* [http://www.knoppix.net/wiki/Cheat_Codes knoppix '''bootfrom''' cheatcode]

bootfrom=/dev/hda1/KNX.iso

Bootfrom needs access to a running Knoppix-System '''with the same Kernel as the Bootkernel''', before it is able to mount the partition / ISO-Image.

* Maybe there's a way to achieve a boot cd iso selecter see ([http://www.linuxquestions.org/questions/showthread.php?t=528840 this post])
* virtualization-oriented distro... ex [http://unit.aist.go.jp/itri/knoppix/xen/index-en.html Xenoppix] Cons: would work best only on modern VT-capable computers. What about a very minimal distribution (say DSL without X) kqemu-enblaed, booting into an ncurses iso-loader? The delicate part would be on the networking side.

===Zero-Install / Networked systems===

*[http://klik.atekon.de/wiki/index.php/Main_Page Klik] with more than 1000 packages ported. See [http://klik.atekon.de/wiki/index.php/Comparison comparison chart].
*[http://0install.net/index.html The Zero Install System], with automatic dependancies, upgrading. See [http://0install.net/matrix.html] comparison chart.
*Why not rolling a [http://wiki.rpath.com/wiki/Conary:Concepts Conary/RPath-based] distro, based on [http://www.rpath.com/rbuilder/project/rpath/ rPath linux] reference distribution ? Or choose within the available ones...
* The FUSE [http://httpfs.sourceforge.net/net_boot.htm httpfs] may allow pure http net booting, downloading files from a distant livecd only when needed (this image may be very minimal though).
* [http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&start=75&tstart=0 some people] hare thinking of using Amazon S3 as virtual disk

===Using Qemu to boot Linux from a flash drive within Windows===

The following tutorial explains how to use Qemu to boot Linux from a portable USB flash device while still working within Windows. This Enables the user to have both systems running at the same time eliminating the need to restart the PC and set your BIOS options to boot Linux.

[http://www.pendrivelinux.com/2007/03/09/use-qemu-to-boot-linux-from-windows/ PendriveLinux article]

===Putting a vnc server on openmoko===

That's another alternative :)

===The BlackDog project===

"When plugged into the host computer’s USB port, BlackDog starts up, '''automatically launching the X Window system for Windows Xming and a software NAT router via an autorun application that the BlackDog presents to the host through a virtual USB CD-ROM. Once those applications are running, the virtual USB CD-ROM drive disconnects, and a virtual USB-to-Ethernet adapter is connected to provide the communications link.'''

The host machine’s monitor, keyboard, mouse, and Internet connection are used by the BlackDog for the duration of the session."

The [http://en.wikipedia.org/wiki/BlackDog BlackDog server] project has lots of interesting concepts (and software). Could [http://www.jasonkolb.com/weblog/2006/11/idea_7_personal.html personal servers] change the face of computing ? Together with biometry, ''they open up the possibility of carrying your personal server, or an extension of it, with you on your physical person. The reason this is so exciting is because it enables biometric authentication that doesn't require shipping your biometric data to a third party'.

==Interesting distros/payloads==

'''System diagnostics / recovery'''
*[http://en.wikipedia.org/wiki/Memtest Memtest86+] Very small
*[http://www.ultimatebootcd.com/ The Ultimate Boot CD]
*[http://www.sysresccd.org/Main_Page System Rescue CD]
*[http://www.tux.org/pub/people/kent-robotti/looplinux/rip/ Recovery IS Possible] < 80 MB

'''General purpose'''
*[http://jinx-linux.sourceforge.net/ jinx-linux] < 50 MB
*[http://knoppix.com/ Knoppix] 700 MB CD-ROM
*[http://www.puppylinux.org PuppyLinux] < 100 MB
*[http://www.damnsmalllinux.org DSL] ~50 MB
*[http://gentoo-wiki.com/TinyGentoo#What_is_this_.3F TinyGentoo] 5 MB

'''Security-oriented''': pentesting, forensics, anonymous webbrowsing
*[http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-test-forensics-recovery/ List of Security distros]
*[http://www.remote-exploit.org/backtrack.html Backtrack Livecd]
*[http://en.wikipedia.org/wiki/Anonym.OS Anonym.OS: browse the web anonymously]
*[http://dban.sourceforge.net/ Darik's Boot and Nuke]: ''dangerous''
*[http://byzgl.sourceforge.net/wiki/index.php/Main_Page Byzantine OS] is a webkiosk appliance (32/48 MB)

The SabayonLinux distro offers a lot of [http://www.sabayonlinux.org/wiki/index.php?title=Sabayon_Linux#Boot_Methods_for_special_ways_of_using_Sabayon boot cheatcodes], such as booting onto GeexBox. We should take a look at the method used.

'''[http://www.livecdlist.com/ List of live distros @ livecdlist.com]''' ''Includes sizes''

'''[http://en.wikipedia.org/wiki/List_of_LiveDistros List of live distros @ wikipedia]'''

'''[http://distrowatch.com/dwres.php?resource=cd List of live distros @ distrowatch]'''

'''[http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB article]'''

Wishlist/LiveUSB distro

2007-08-25T00:37:32Z

Fthiery: /* Introduction */

{{wishlist}}

==Introduction==

NOTE: this page may be renamed: portable destkop

An openmoko device could act as '''the perfect geeky swiss knive''': go anywhere with your Linux desktop and tools.

YES, there are tons of liveCDs out there, but CDs/DVDs:
* get easily damaged
* don't fit in a pocket (except for businesscard-sized ones ; too exotic...)
* don't update or save settings/personal data without a supplementary USB device

YES, there are LiveUSB distros but:
* a Neo/OpenMoko device IS sort of an USB stick; one object is better than two
* you may carry my cellphone everywhere, what about usb sticks or cds ?

The biggest argument against these ones is that you'd have to carry an USB cable with you... But it's a standard one, which is good news :) An integrated zippable plug could be an interesting add-on...

Similar functionality can be found in [http://en.wikipedia.org/wiki/Wizpy the Wizpy portable media player], but relies on different methods for working under Windows (cd-rom emulation).

When the openmoko device is in mass storage mode, a host computer should be able to boot on it, presenting a grub menu offering to boot into several images / partitions (payloads) on the transflash: memtest, UBCD (the ultimate boot cd), a lightweight security oriented livecd distro, you name it...

It's sometimes called Live USB: from [http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB definition], ''A live USB is a USB flash drive containing a full operating system which can be booted from. Live USBs are closely related to Live CDs, and are sometimes used interchangeably. Like Live CDs, Live USBs can be used for system administration, data recovery, or the testing of operating system distributions without committing to a permanent installation on the local hard drive. Many of the smaller Linux distributions can also be used from a USB flash drive.''

Usually, all you need for this with an USB storage device (flash USB stick or external drive) is a partition, flagged "bootable" (see fdisk), containing a boot loader (say grub). But the problem is that specifications vary between motherboards, so there are variants and mandatory requirements to define (here would be a good idea), so that one can optimize/maximize the compatibility.

==The perfect geek knife==

We may want to start our own distro with the following constraints:
* Multiboot: small utilities/compilations + bull blown light OS
* Aiming at maximum hardware compatibility, possibly using liveCD technology (knoppix-based distros)
* Aiming at minimum resource+system footprint, and boot speed
* Being able to install software temporary (for the session) or permanently, upgradable
* Being able to use the phone features at the same time (access pim, data & calls)

Optimally, being able to charge the device while plugged in would be great too.

==About USB Booting==

From [http://www.damnsmalllinux.org/wiki/index.php/USB_Booting DSL Wiki]:

Older computer BIOS usually do not support direct booting from a USB device. Around 2001, PC motherboard manufacturers started to add USB boot support.

There are two common BIOS methods for direct USB booting:
* One method is called the "USBHDD" method and it is used to support the booting of standard USB mass storage devices that are configured like a normal PC hard drive.
* The other method is called the "USBZIP" method and it supports booting from a USB storage device that behaves like the original IOMEGA ZIP drive with USB support.

Most computers (just about all Dells, for example) made today have a BIOS that supports the USBHDD method so I expect that this will eventually become the standard way to boot a USB device. However, many motherboards will support BOTH methods, and many older motherboards have USBZIP support.

Some newer BIOSs which support USB 2.0 will not boot from an older pen drive. Using a USB 2.0 compliant one usually solves this problem. Also, some older BIOSs which only support USB 1.1 will not boot from newer drives which support USB 2.0!

Some USB keys don't boot. If this is the case, it may be possible to fix them by installing a new master boot record. (Most keys boot OK by default; some cannot be fixed even by doing this. However, it helps in some cases). Run the command:

lilo -M /dev/sda

==USB Booting in Neo1973's kernel==

Although there is very little information about it, the g_storage kernel module is responsible for the Neo's mass storage mode. So the boot-or-not criteria might rely on this very module.

==TODO==

Testing/further research has to be conducted to determine:
* if a bootable-flagged partition / grub doesn't mess openmoko up
* if the phone can still act as a phone, or if the booted os can use the gprs functions
* if the phone can recharge on mass storage mode (from the usb cable)
* if a dedicated partition for each payload is needed (see memtest example...)
* if multi-boot is possible: grub?

==Linux distro Howto's: liveUSB creation==

In short:
#Mount USB drive, with e.g. mount /dev/sda1 /flash - can be either FAT16 or FAT32!
#Mount ISO image, with e.g. mount /tmp/dsl-3.2.iso /tmp/iso -o loop
#Copy all contents from ISO to USB drive: cp -vr /tmp/iso/* /flash/
#Rename and move syslinux files to root directory: mv /flash/boot/isolinux/* /flash/
#Rename isolinux.cfg: mv /flash/isolinux.cfg /flash/syslinux.cfg
#Unmount USB drive: umount /flash
#Install syslinux: syslinux /dev/sda1 and eventually set the MBR boot flag for this partition (with fdisk).

*Fedora Core's [http://revisor.fedoraunity.org Revisor] enables easy GUI-driven ISO customization, allowing to choose the output medium (CD/DVD/USB)
*[http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive Installing DSL to USB device - using grub]
*[http://wiki.debian.org/BootUsb Debian BootUSB - using Lilo]
*[http://www.debuntu.org/how-to-install-ubuntu-linux-on-usb-bar Howto install ubuntu (or any other distro) on usb stick]
*[http://www.gentoo.org/doc/en/liveusb.xml Gentoo on usb stick]
*[http://pendrivelinux.com/ Windows & Linux tutorials & resources]:
* Flash installation via Windows: XUbuntu, DSL, Knoppix, Slax, MiniMe
* Flash installation via Linux: Ubuntu Edgy, Knoppix, PCLinuxOS

'''Transflash partitioning schema example'''

* 2 Gb: /dev/sd? (where N is the transflash's number)
* 700 Mb: /dev/sd?1 : containing bootable iso, FAT16
* 1300 Mb: /dev/sd?2 : openmoko & bootable os home partition, EXT3?
* + eventually a swap

The limitation of this method is that the booted OS is static (not-self modifiable). There's the option to install linux using the partition, but it's not really mobility-oriented (liveCDs are optimized for maximum autoconfiguration).

==Alternative methods==

===Booting ISOs ?===

* Grub? From [http://www.freesoftwaremagazine.com/articles/grub_intro#comment-12900 How to install grub on an usb pendrive]:
If you want to boot from a iso file on a harddisk, do something in menu.lst like
title Boot from iso on a harddisk
map (hdX,Y)/your.iso (hdZ)
map --rehook
chainloader (hdZ)+1
rootnoverify (hdZ)
boot

Grub & memdisk can boot floppy images, but (AFAIK) not isos. Example from the stock /boot/grub/menu.lst ubuntu:
title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin
quiet
boot

''Using the memdisk kernel from the syslinux package, you can load disk images and execute them in a non-emulated environment.''

* [http://www.knoppix.net/wiki/Cheat_Codes knoppix '''bootfrom''' cheatcode]

bootfrom=/dev/hda1/KNX.iso

Bootfrom needs access to a running Knoppix-System '''with the same Kernel as the Bootkernel''', before it is able to mount the partition / ISO-Image.

* Maybe there's a way to achieve a boot cd iso selecter see ([http://www.linuxquestions.org/questions/showthread.php?t=528840 this post])
* virtualization-oriented distro... ex [http://unit.aist.go.jp/itri/knoppix/xen/index-en.html Xenoppix] Cons: would work best only on modern VT-capable computers. What about a very minimal distribution (say DSL without X) kqemu-enblaed, booting into an ncurses iso-loader? The delicate part would be on the networking side.

===Zero-Install / Networked systems===

*[http://klik.atekon.de/wiki/index.php/Main_Page Klik] with more than 1000 packages ported. See [http://klik.atekon.de/wiki/index.php/Comparison comparison chart].
*[http://0install.net/index.html The Zero Install System], with automatic dependancies, upgrading. See [http://0install.net/matrix.html] comparison chart.
*Why not rolling a [http://wiki.rpath.com/wiki/Conary:Concepts Conary/RPath-based] distro, based on [http://www.rpath.com/rbuilder/project/rpath/ rPath linux] reference distribution ? Or choose within the available ones...
* The FUSE [http://httpfs.sourceforge.net/net_boot.htm httpfs] may allow pure http net booting, downloading files from a distant livecd only when needed (this image may be very minimal though).
* [http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&start=75&tstart=0 some people] hare thinking of using Amazon S3 as virtual disk

===Using Qemu to boot Linux from a flash drive within Windows===

The following tutorial explains how to use Qemu to boot Linux from a portable USB flash device while still working within Windows. This Enables the user to have both systems running at the same time eliminating the need to restart the PC and set your BIOS options to boot Linux.

[http://www.pendrivelinux.com/2007/03/09/use-qemu-to-boot-linux-from-windows/ PendriveLinux article]

===Putting a vnc server on openmoko===

That's another alternative :)

===The BlackDog project===

"When plugged into the host computer’s USB port, BlackDog starts up, '''automatically launching the X Window system for Windows Xming and a software NAT router via an autorun application that the BlackDog presents to the host through a virtual USB CD-ROM. Once those applications are running, the virtual USB CD-ROM drive disconnects, and a virtual USB-to-Ethernet adapter is connected to provide the communications link.'''

The host machine’s monitor, keyboard, mouse, and Internet connection are used by the BlackDog for the duration of the session."

The [http://en.wikipedia.org/wiki/BlackDog BlackDog server] project has lots of interesting concepts (and software). Could [http://www.jasonkolb.com/weblog/2006/11/idea_7_personal.html personal servers] change the face of computing ? Together with biometry, ''they open up the possibility of carrying your personal server, or an extension of it, with you on your physical person. The reason this is so exciting is because it enables biometric authentication that doesn't require shipping your biometric data to a third party'.

==Interesting distros/payloads==

'''System diagnostics / recovery'''
*[http://en.wikipedia.org/wiki/Memtest Memtest86+] Very small
*[http://www.ultimatebootcd.com/ The Ultimate Boot CD]
*[http://www.sysresccd.org/Main_Page System Rescue CD]
*[http://www.tux.org/pub/people/kent-robotti/looplinux/rip/ Recovery IS Possible] < 80 MB

'''General purpose'''
*[http://jinx-linux.sourceforge.net/ jinx-linux] < 50 MB
*[http://knoppix.com/ Knoppix] 700 MB CD-ROM
*[http://www.puppylinux.org PuppyLinux] < 100 MB
*[http://www.damnsmalllinux.org DSL] ~50 MB
*[http://gentoo-wiki.com/TinyGentoo#What_is_this_.3F TinyGentoo] 5 MB

'''Security-oriented''': pentesting, forensics, anonymous webbrowsing
*[http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-test-forensics-recovery/ List of Security distros]
*[http://www.remote-exploit.org/backtrack.html Backtrack Livecd]
*[http://en.wikipedia.org/wiki/Anonym.OS Anonym.OS: browse the web anonymously]
*[http://dban.sourceforge.net/ Darik's Boot and Nuke]: ''dangerous''
*[http://byzgl.sourceforge.net/wiki/index.php/Main_Page Byzantine OS] is a webkiosk appliance (32/48 MB)

The SabayonLinux distro offers a lot of [http://www.sabayonlinux.org/wiki/index.php?title=Sabayon_Linux#Boot_Methods_for_special_ways_of_using_Sabayon boot cheatcodes], such as booting onto GeexBox. We should take a look at the method used.

'''[http://www.livecdlist.com/ List of live distros @ livecdlist.com]''' ''Includes sizes''

'''[http://en.wikipedia.org/wiki/List_of_LiveDistros List of live distros @ wikipedia]'''

'''[http://distrowatch.com/dwres.php?resource=cd List of live distros @ distrowatch]'''

'''[http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB article]'''

Talk:E17

2007-08-24T22:46:29Z

Fthiery:

Talk:E17

2007-08-24T22:46:04Z

Fthiery:

Talk:E17

2007-08-24T22:45:07Z

Fthiery:

UI Improvements

2007-08-04T12:25:15Z

Fthiery: /* Clutter Toolkit */

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]
* Multi-touch on Linux with [http://www.youtube.com/watch?v=olWjnfBoY8E MPX]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo
*[http://files.mdk.am/demos/graff-demo-3.avi Graff]'s inertia scroll list

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

* '''Clever hacks'''

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====The Enlightenment Foundation Libraries====

EFL's Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

Features:
* Core languages: C, OpenGL
* Bindings: Python, C#, Ruby
* Backends: GLX / SDL / EGL
* Media support: Gstreamer integration, Cairo graphics rendering, Pango fonts rendering
* Embedding: GTK embedding

Some experimental code in the clutter svn show that embedded devices are one of it's targets:
* [http://svn.o-hand.com/repos/clutter/trunk/toys/foofone/ Some bulk phone dialing interface]
* Make sure to check out the [http://www.clutter-project.org/blog/?p=25 Clutter toys videos] to get a feeling of what can be done using clutter

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core languages: C OpenGL
* Bindings: Python
* Backends: DirectFB OpenGL
* Media playback integration: using Gstreamer

====Choosing====

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===Kinetic scrolling===

====EDIT: demo now on the neo====

Apparently the topics discussed here have found an echo in moko devs :)

http://dominion.kabel.utwente.nl/koen/cms/kinetic-scrolling-on-the-neo1973

enjoy the video demo; everything seems to be rendered software fine. More details about the implementation to come.

====Description====

Graff's inertia scrolling list example: http://files.mdk.am/demos/graff-demo-3.avi

Because an example is worth 1000s words...

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

There is a fake keyboard module (for dev purposes) in the main kernel tree, which could be used to simulate keyboard presses (hence keeping keyboard-enabled apps unmodified).

====Full multi-touch emulation====

Doable, but tricky...

==Preparing the multi touch==

One day we might get multitouch devices. Let's get ready.

===MPX===

The Multi-Pointer X Server is a modification of the X server to support multiple mice and keyboards in X. It provides users with one cursor per device and one keyboard focus per keyboard. Each cursor can operate independently. MPX is the first multicursor windowing system and allows two-handed interaction with legacy applications, but also the creation of innovative applications and user interfaces.

[http://wearables.unisa.edu.au/mpx/ The multipoint X server project]

===MacSlow's Lowfat getting multitouched===

http://dlai.jafu.dk/?p=1

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-31T16:35:21Z

Fthiery: fix typo

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]
* Multi-touch on Linux with [http://www.youtube.com/watch?v=olWjnfBoY8E MPX]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo
*[http://files.mdk.am/demos/graff-demo-3.avi Graff]'s inertia scroll list

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

* '''Clever hacks'''

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====The Enlightenment Foundation Libraries====

EFL's Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

Features:
* Core languages: C, OpenGL
* Bindings: Python, C#, Ruby
* Backends: GLX / SDL / EGL
* Media support: Gstreamer integration, Cairo graphics rendering, Pango fonts rendering
* Embedding: GTK embedding

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core languages: C OpenGL
* Bindings: Python
* Backends: DirectFB OpenGL
* Media playback integration: using Gstreamer

====Choosing====

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===Kinetic scrolling===

====EDIT: demo now on the neo====

Apparently the topics discussed here have found an echo in moko devs :)

http://dominion.kabel.utwente.nl/koen/cms/kinetic-scrolling-on-the-neo1973

enjoy the video demo; everything seems to be rendered software fine. More details about the implementation to come.

====Description====

Graff's inertia scrolling list example: http://files.mdk.am/demos/graff-demo-3.avi

Because an example is worth 1000s words...

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

There is a fake keyboard module (for dev purposes) in the main kernel tree, which could be used to simulate keyboard presses (hence keeping keyboard-enabled apps unmodified).

====Full multi-touch emulation====

Doable, but tricky...

==Preparing the multi touch==

One day we might get multitouch devices. Let's get ready.

===MPX===

The Multi-Pointer X Server is a modification of the X server to support multiple mice and keyboards in X. It provides users with one cursor per device and one keyboard focus per keyboard. Each cursor can operate independently. MPX is the first multicursor windowing system and allows two-handed interaction with legacy applications, but also the creation of innovative applications and user interfaces.

[http://wearables.unisa.edu.au/mpx/ The multipoint X server project]

===MacSlow's Lowfat getting multitouched===

http://dlai.jafu.dk/?p=1

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-31T16:34:51Z

Fthiery: /* 1D list scrolling: looped physics-driven item list */

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]
* Multi-touch on Linux with [http://www.youtube.com/watch?v=olWjnfBoY8E MPX]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo
*[http://files.mdk.am/demos/graff-demo-3.avi Graff]'s inertia scroll list

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

* '''Clever hacks'''

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====The Enlightenment Foundation Libraries====

EFL's Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

Features:
* Core languages: C, OpenGL
* Bindings: Python, C#, Ruby
* Backends: GLX / SDL / EGL
* Media support: Gstreamer integration, Cairo graphics rendering, Pango fonts rendering
* Embedding: GTK embedding

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core languages: C OpenGL
* Bindings: Python
* Backends: DirectFB OpenGL
* Media playback integration: using Gstreamer

====Choosing====

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===kinetic scrolling===

====EDIT: demo now on the neo====

Apparently the topics discussed here have found an echo in moko devs :)

http://dominion.kabel.utwente.nl/koen/cms/kinetic-scrolling-on-the-neo1973

enjoy the video demo; everything seems to be rendered software fine. More details about the implementation to come.

====Description====

Graff's inertia scrolling list example: http://files.mdk.am/demos/graff-demo-3.avi

Because an example is worth 1000s words...

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

There is a fake keyboard module (for dev purposes) in the main kernel tree, which could be used to simulate keyboard presses (hence keeping keyboard-enabled apps unmodified).

====Full multi-touch emulation====

Doable, but tricky...

==Preparing the multi touch==

One day we might get multitouch devices. Let's get ready.

===MPX===

The Multi-Pointer X Server is a modification of the X server to support multiple mice and keyboards in X. It provides users with one cursor per device and one keyboard focus per keyboard. Each cursor can operate independently. MPX is the first multicursor windowing system and allows two-handed interaction with legacy applications, but also the creation of innovative applications and user interfaces.

[http://wearables.unisa.edu.au/mpx/ The multipoint X server project]

===MacSlow's Lowfat getting multitouched===

http://dlai.jafu.dk/?p=1

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-31T16:32:27Z

Fthiery: /* 1D list scrolling: looped physics-driven item list */

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]
* Multi-touch on Linux with [http://www.youtube.com/watch?v=olWjnfBoY8E MPX]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo
*[http://files.mdk.am/demos/graff-demo-3.avi Graff]'s inertia scroll list

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

* '''Clever hacks'''

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====The Enlightenment Foundation Libraries====

EFL's Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

Features:
* Core languages: C, OpenGL
* Bindings: Python, C#, Ruby
* Backends: GLX / SDL / EGL
* Media support: Gstreamer integration, Cairo graphics rendering, Pango fonts rendering
* Embedding: GTK embedding

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core languages: C OpenGL
* Bindings: Python
* Backends: DirectFB OpenGL
* Media playback integration: using Gstreamer

====Choosing====

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====EDIT: demo now on the neo====

Apparently the topics discussed here have found an echo in moko devs :)

http://dominion.kabel.utwente.nl/koen/cms/kinetic-scrolling-on-the-neo1973

enjoy the video demo; everything seems to be rendered software fine. More details about the implementation to come.

====Description====

[EDIT] Graff's inertia scrolling list example: http://files.mdk.am/demos/graff-demo-3.avi

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

There is a fake keyboard module (for dev purposes) in the main kernel tree, which could be used to simulate keyboard presses (hence keeping keyboard-enabled apps unmodified).

====Full multi-touch emulation====

Doable, but tricky...

==Preparing the multi touch==

One day we might get multitouch devices. Let's get ready.

===MPX===

The Multi-Pointer X Server is a modification of the X server to support multiple mice and keyboards in X. It provides users with one cursor per device and one keyboard focus per keyboard. Each cursor can operate independently. MPX is the first multicursor windowing system and allows two-handed interaction with legacy applications, but also the creation of innovative applications and user interfaces.

[http://wearables.unisa.edu.au/mpx/ The multipoint X server project]

===MacSlow's Lowfat getting multitouched===

http://dlai.jafu.dk/?p=1

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-27T17:20:59Z

Fthiery: /* Choosing */

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]
* Multi-touch on Linux with [http://www.youtube.com/watch?v=olWjnfBoY8E MPX]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo
*[http://files.mdk.am/demos/graff-demo-3.avi Graff]'s inertia scroll list

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

* '''Clever hacks'''

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====The Enlightenment Foundation Libraries====

EFL's Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

Features:
* Core languages: C, OpenGL
* Bindings: Python, C#, Ruby
* Backends: GLX / SDL / EGL
* Media support: Gstreamer integration, Cairo graphics rendering, Pango fonts rendering
* Embedding: GTK embedding

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core languages: C OpenGL
* Bindings: Python
* Backends: DirectFB OpenGL
* Media playback integration: using Gstreamer

====Choosing====

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====Description====

[EDIT] Graff's inertia scrolling list example: http://files.mdk.am/demos/graff-demo-3.avi

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

There is a fake keyboard module (for dev purposes) in the main kernel tree, which could be used to simulate keyboard presses (hence keeping keyboard-enabled apps unmodified).

====Full multi-touch emulation====

Doable, but tricky...

==Preparing the multi touch==

One day we might get multitouch devices. Let's get ready.

===MPX===

The Multi-Pointer X Server is a modification of the X server to support multiple mice and keyboards in X. It provides users with one cursor per device and one keyboard focus per keyboard. Each cursor can operate independently. MPX is the first multicursor windowing system and allows two-handed interaction with legacy applications, but also the creation of innovative applications and user interfaces.

[http://wearables.unisa.edu.au/mpx/ The multipoint X server project]

===MacSlow's Lowfat getting multitouched===

http://dlai.jafu.dk/?p=1

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-27T17:20:07Z

Fthiery: /* Extending the touchscreen capabilities and input methods */

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]
* Multi-touch on Linux with [http://www.youtube.com/watch?v=olWjnfBoY8E MPX]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo
*[http://files.mdk.am/demos/graff-demo-3.avi Graff]'s inertia scroll list

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

* '''Clever hacks'''

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====The Enlightenment Foundation Libraries====

EFL's Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

Features:
* Core languages: C, OpenGL
* Bindings: Python, C#, Ruby
* Backends: GLX / SDL / EGL
* Media support: Gstreamer integration, Cairo graphics rendering, Pango fonts rendering
* Embedding: GTK embedding

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core languages: C OpenGL
* Bindings: Python
* Backends: DirectFB OpenGL
* Media playback integration: using Gstreamer

===Choosing===

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====Description====

[EDIT] Graff's inertia scrolling list example: http://files.mdk.am/demos/graff-demo-3.avi

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

There is a fake keyboard module (for dev purposes) in the main kernel tree, which could be used to simulate keyboard presses (hence keeping keyboard-enabled apps unmodified).

====Full multi-touch emulation====

Doable, but tricky...

==Preparing the multi touch==

One day we might get multitouch devices. Let's get ready.

===MPX===

The Multi-Pointer X Server is a modification of the X server to support multiple mice and keyboards in X. It provides users with one cursor per device and one keyboard focus per keyboard. Each cursor can operate independently. MPX is the first multicursor windowing system and allows two-handed interaction with legacy applications, but also the creation of innovative applications and user interfaces.

[http://wearables.unisa.edu.au/mpx/ The multipoint X server project]

===MacSlow's Lowfat getting multitouched===

http://dlai.jafu.dk/?p=1

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-27T17:19:31Z

Fthiery: /* Clever hacks */

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]
* Multi-touch on Linux with [http://www.youtube.com/watch?v=olWjnfBoY8E MPX]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo
*[http://files.mdk.am/demos/graff-demo-3.avi Graff]'s inertia scroll list

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====The Enlightenment Foundation Libraries====

EFL's Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

Features:
* Core languages: C, OpenGL
* Bindings: Python, C#, Ruby
* Backends: GLX / SDL / EGL
* Media support: Gstreamer integration, Cairo graphics rendering, Pango fonts rendering
* Embedding: GTK embedding

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core languages: C OpenGL
* Bindings: Python
* Backends: DirectFB OpenGL
* Media playback integration: using Gstreamer

===Choosing===

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====Description====

[EDIT] Graff's inertia scrolling list example: http://files.mdk.am/demos/graff-demo-3.avi

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

There is a fake keyboard module (for dev purposes) in the main kernel tree, which could be used to simulate keyboard presses (hence keeping keyboard-enabled apps unmodified).

====Full multi-touch emulation====

Doable, but tricky...

==Preparing the multi touch==

One day we might get multitouch devices. Let's get ready.

===MPX===

The Multi-Pointer X Server is a modification of the X server to support multiple mice and keyboards in X. It provides users with one cursor per device and one keyboard focus per keyboard. Each cursor can operate independently. MPX is the first multicursor windowing system and allows two-handed interaction with legacy applications, but also the creation of innovative applications and user interfaces.

[http://wearables.unisa.edu.au/mpx/ The multipoint X server project]

===MacSlow's Lowfat getting multitouched===

http://dlai.jafu.dk/?p=1

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-27T17:19:04Z

Fthiery: /* Seen Live */

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]
* Multi-touch on Linux with [http://www.youtube.com/watch?v=olWjnfBoY8E MPX]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo
*[http://files.mdk.am/demos/graff-demo-3.avi Graff]'s inertia scroll list

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===Clever hacks===

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====The Enlightenment Foundation Libraries====

EFL's Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

Features:
* Core languages: C, OpenGL
* Bindings: Python, C#, Ruby
* Backends: GLX / SDL / EGL
* Media support: Gstreamer integration, Cairo graphics rendering, Pango fonts rendering
* Embedding: GTK embedding

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core languages: C OpenGL
* Bindings: Python
* Backends: DirectFB OpenGL
* Media playback integration: using Gstreamer

===Choosing===

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====Description====

[EDIT] Graff's inertia scrolling list example: http://files.mdk.am/demos/graff-demo-3.avi

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

There is a fake keyboard module (for dev purposes) in the main kernel tree, which could be used to simulate keyboard presses (hence keeping keyboard-enabled apps unmodified).

====Full multi-touch emulation====

Doable, but tricky...

==Preparing the multi touch==

One day we might get multitouch devices. Let's get ready.

===MPX===

The Multi-Pointer X Server is a modification of the X server to support multiple mice and keyboards in X. It provides users with one cursor per device and one keyboard focus per keyboard. Each cursor can operate independently. MPX is the first multicursor windowing system and allows two-handed interaction with legacy applications, but also the creation of innovative applications and user interfaces.

[http://wearables.unisa.edu.au/mpx/ The multipoint X server project]

===MacSlow's Lowfat getting multitouched===

http://dlai.jafu.dk/?p=1

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-27T17:14:15Z

Fthiery: /* Towards OpenGL compositing */

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo.

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===Clever hacks===

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====The Enlightenment Foundation Libraries====

EFL's Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

Features:
* Core languages: C, OpenGL
* Bindings: Python, C#, Ruby
* Backends: GLX / SDL / EGL
* Media support: Gstreamer integration, Cairo graphics rendering, Pango fonts rendering
* Embedding: GTK embedding

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core languages: C OpenGL
* Bindings: Python
* Backends: DirectFB OpenGL
* Media playback integration: using Gstreamer

===Choosing===

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====Description====

[EDIT] Graff's inertia scrolling list example: http://files.mdk.am/demos/graff-demo-3.avi

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

There is a fake keyboard module (for dev purposes) in the main kernel tree, which could be used to simulate keyboard presses (hence keeping keyboard-enabled apps unmodified).

====Full multi-touch emulation====

Doable, but tricky...

==Preparing the multi touch==

One day we might get multitouch devices. Let's get ready.

===MPX===

The Multi-Pointer X Server is a modification of the X server to support multiple mice and keyboards in X. It provides users with one cursor per device and one keyboard focus per keyboard. Each cursor can operate independently. MPX is the first multicursor windowing system and allows two-handed interaction with legacy applications, but also the creation of innovative applications and user interfaces.

[http://wearables.unisa.edu.au/mpx/ The multipoint X server project]

===MacSlow's Lowfat getting multitouched===

http://dlai.jafu.dk/?p=1

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-27T17:11:45Z

Fthiery: /* Description */

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo.

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===Clever hacks===

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====Switching to the Enlightenment Foundation Libraries====

Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

Features:
* Core languages: C, OpenGL
* Bindings: Python, C#, Ruby
* Backends: GLX / SDL / EGL
* Media support: Gstreamer integration, Cairo graphics rendering, Pango fonts rendering
* Embedding: GTK embedding

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core languages: C OpenGL
* Bindings: Python
* Backends: DirectFB OpenGL
* Media playback integration: using Gstreamer

===Choosing===

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====Description====

[EDIT] Graff's inertia scrolling list example: http://files.mdk.am/demos/graff-demo-3.avi

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

There is a fake keyboard module (for dev purposes) in the main kernel tree, which could be used to simulate keyboard presses (hence keeping keyboard-enabled apps unmodified).

====Full multi-touch emulation====

Doable, but tricky...

==Preparing the multi touch==

One day we might get multitouch devices. Let's get ready.

===MPX===

The Multi-Pointer X Server is a modification of the X server to support multiple mice and keyboards in X. It provides users with one cursor per device and one keyboard focus per keyboard. Each cursor can operate independently. MPX is the first multicursor windowing system and allows two-handed interaction with legacy applications, but also the creation of innovative applications and user interfaces.

[http://wearables.unisa.edu.au/mpx/ The multipoint X server project]

===MacSlow's Lowfat getting multitouched===

http://dlai.jafu.dk/?p=1

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-27T17:08:22Z

Fthiery: /* MPX */

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo.

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===Clever hacks===

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====Switching to the Enlightenment Foundation Libraries====

Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

Features:
* Core languages: C, OpenGL
* Bindings: Python, C#, Ruby
* Backends: GLX / SDL / EGL
* Media support: Gstreamer integration, Cairo graphics rendering, Pango fonts rendering
* Embedding: GTK embedding

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core languages: C OpenGL
* Bindings: Python
* Backends: DirectFB OpenGL
* Media playback integration: using Gstreamer

===Choosing===

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====Description====

[EDIT] Graff's inertia scrolling list example: http://files.mdk.am/demos/graff-demo-3.avi

Take an item list (ex: adress book), print it on a ribbon of paper, and glue it on a wheel (on the tire). You're looking in the front of it, so when you want to go from the A to Z, you touch the wheel and drag it up. When you let the wheel go, it goes furter, taken by it's inertia. Stop the wheel when you got your contact. Got the idea? That's why we may speak of an "infinite wheel", so that the surface is flat. For our case here, we always want to display square content, so the [http://en.wikipedia.org/wiki/Uniform_prism n-sided uniform prism] analogy is mathematically more exact.

Important features:
* '''weight''': the biggest the item list, the faster it scrolls; that way, you don't have to wait too long for big lists, and you don't miss your item on shorter lists
* '''friction''': there is friction where the wheel is fixed, so that the wheel doesn't turn infinitely; more friction on short lists, less on big ones
* the '''initial speed and acceleration vector''' you give it determines it's futher rotation
* it's "round"/cyclic, so you can '''browse the list in two directions'''
* we may want to add a "progression indicator", ex the current alphabetical letter, with a font size adequate to the proportion of the number of entries in the letter subcategory, or a reduced map of the distribution of the first letters...

We can add "parallel wheels", symbolizing different sorting methods. Slide long to the left / right to look at a different wheel = items organization.

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

There is a fake keyboard module (for dev purposes) in the main kernel tree, which could be used to simulate keyboard presses (hence keeping keyboard-enabled apps unmodified).

====Full multi-touch emulation====

Doable, but tricky...

==Preparing the multi touch==

One day we might get multitouch devices. Let's get ready.

===MPX===

The Multi-Pointer X Server is a modification of the X server to support multiple mice and keyboards in X. It provides users with one cursor per device and one keyboard focus per keyboard. Each cursor can operate independently. MPX is the first multicursor windowing system and allows two-handed interaction with legacy applications, but also the creation of innovative applications and user interfaces.

[http://wearables.unisa.edu.au/mpx/ The multipoint X server project]

===MacSlow's Lowfat getting multitouched===

http://dlai.jafu.dk/?p=1

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-27T17:08:05Z

Fthiery: /* Full multi-touch emulation */

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo.

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===Clever hacks===

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====Switching to the Enlightenment Foundation Libraries====

Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

Features:
* Core languages: C, OpenGL
* Bindings: Python, C#, Ruby
* Backends: GLX / SDL / EGL
* Media support: Gstreamer integration, Cairo graphics rendering, Pango fonts rendering
* Embedding: GTK embedding

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core languages: C OpenGL
* Bindings: Python
* Backends: DirectFB OpenGL
* Media playback integration: using Gstreamer

===Choosing===

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====Description====

[EDIT] Graff's inertia scrolling list example: http://files.mdk.am/demos/graff-demo-3.avi

Take an item list (ex: adress book), print it on a ribbon of paper, and glue it on a wheel (on the tire). You're looking in the front of it, so when you want to go from the A to Z, you touch the wheel and drag it up. When you let the wheel go, it goes furter, taken by it's inertia. Stop the wheel when you got your contact. Got the idea? That's why we may speak of an "infinite wheel", so that the surface is flat. For our case here, we always want to display square content, so the [http://en.wikipedia.org/wiki/Uniform_prism n-sided uniform prism] analogy is mathematically more exact.

Important features:
* '''weight''': the biggest the item list, the faster it scrolls; that way, you don't have to wait too long for big lists, and you don't miss your item on shorter lists
* '''friction''': there is friction where the wheel is fixed, so that the wheel doesn't turn infinitely; more friction on short lists, less on big ones
* the '''initial speed and acceleration vector''' you give it determines it's futher rotation
* it's "round"/cyclic, so you can '''browse the list in two directions'''
* we may want to add a "progression indicator", ex the current alphabetical letter, with a font size adequate to the proportion of the number of entries in the letter subcategory, or a reduced map of the distribution of the first letters...

We can add "parallel wheels", symbolizing different sorting methods. Slide long to the left / right to look at a different wheel = items organization.

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

There is a fake keyboard module (for dev purposes) in the main kernel tree, which could be used to simulate keyboard presses (hence keeping keyboard-enabled apps unmodified).

====Full multi-touch emulation====

Doable, but tricky...

==Preparing the multi touch==

One day we might get multitouch devices. Let's get ready.

===MPX===

The Multi-Pointer X Server is a modification of the X server to support multiple mice and keyboards in X. It provides users with one cursor per device and one keyboard focus per keyboard. Each cursor can operate independently. MPX is the first multicursor windowing system and allows two-handed interaction with legacy applications, but also the creation of innovative applications and user interfaces.

[http://wearables.unisa.edu.au/mpx/ The multipoint X server project]

===MacSlow's Lowfat getting multitouched===

http://dlai.jafu.dk/?p=1

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-27T17:04:26Z

Fthiery: /* What is to modify ? */

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo.

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===Clever hacks===

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====Switching to the Enlightenment Foundation Libraries====

Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

Features:
* Core languages: C, OpenGL
* Bindings: Python, C#, Ruby
* Backends: GLX / SDL / EGL
* Media support: Gstreamer integration, Cairo graphics rendering, Pango fonts rendering
* Embedding: GTK embedding

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core languages: C OpenGL
* Bindings: Python
* Backends: DirectFB OpenGL
* Media playback integration: using Gstreamer

===Choosing===

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====Description====

[EDIT] Graff's inertia scrolling list example: http://files.mdk.am/demos/graff-demo-3.avi

Take an item list (ex: adress book), print it on a ribbon of paper, and glue it on a wheel (on the tire). You're looking in the front of it, so when you want to go from the A to Z, you touch the wheel and drag it up. When you let the wheel go, it goes furter, taken by it's inertia. Stop the wheel when you got your contact. Got the idea? That's why we may speak of an "infinite wheel", so that the surface is flat. For our case here, we always want to display square content, so the [http://en.wikipedia.org/wiki/Uniform_prism n-sided uniform prism] analogy is mathematically more exact.

Important features:
* '''weight''': the biggest the item list, the faster it scrolls; that way, you don't have to wait too long for big lists, and you don't miss your item on shorter lists
* '''friction''': there is friction where the wheel is fixed, so that the wheel doesn't turn infinitely; more friction on short lists, less on big ones
* the '''initial speed and acceleration vector''' you give it determines it's futher rotation
* it's "round"/cyclic, so you can '''browse the list in two directions'''
* we may want to add a "progression indicator", ex the current alphabetical letter, with a font size adequate to the proportion of the number of entries in the letter subcategory, or a reduced map of the distribution of the first letters...

We can add "parallel wheels", symbolizing different sorting methods. Slide long to the left / right to look at a different wheel = items organization.

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

There is a fake keyboard module (for dev purposes) in the main kernel tree, which could be used to simulate keyboard presses (hence keeping keyboard-enabled apps unmodified).

====Full multi-touch emulation====

Doable, but tricky...

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-27T17:01:51Z

Fthiery: Added reference to graff's scrolling list

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo.

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===Clever hacks===

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====Switching to the Enlightenment Foundation Libraries====

Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

Features:
* Core languages: C, OpenGL
* Bindings: Python, C#, Ruby
* Backends: GLX / SDL / EGL
* Media support: Gstreamer integration, Cairo graphics rendering, Pango fonts rendering
* Embedding: GTK embedding

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core languages: C OpenGL
* Bindings: Python
* Backends: DirectFB OpenGL
* Media playback integration: using Gstreamer

===Choosing===

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====Description====

[EDIT] Graff's inertia scrolling list example: http://files.mdk.am/demos/graff-demo-3.avi

Take an item list (ex: adress book), print it on a ribbon of paper, and glue it on a wheel (on the tire). You're looking in the front of it, so when you want to go from the A to Z, you touch the wheel and drag it up. When you let the wheel go, it goes furter, taken by it's inertia. Stop the wheel when you got your contact. Got the idea? That's why we may speak of an "infinite wheel", so that the surface is flat. For our case here, we always want to display square content, so the [http://en.wikipedia.org/wiki/Uniform_prism n-sided uniform prism] analogy is mathematically more exact.

Important features:
* '''weight''': the biggest the item list, the faster it scrolls; that way, you don't have to wait too long for big lists, and you don't miss your item on shorter lists
* '''friction''': there is friction where the wheel is fixed, so that the wheel doesn't turn infinitely; more friction on short lists, less on big ones
* the '''initial speed and acceleration vector''' you give it determines it's futher rotation
* it's "round"/cyclic, so you can '''browse the list in two directions'''
* we may want to add a "progression indicator", ex the current alphabetical letter, with a font size adequate to the proportion of the number of entries in the letter subcategory, or a reduced map of the distribution of the first letters...

We can add "parallel wheels", symbolizing different sorting methods. Slide long to the left / right to look at a different wheel = items organization.

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

====Full multi-touch emulation====

Doable, but tricky...

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-27T17:00:27Z

Fthiery: /* Pigment API */

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo.

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===Clever hacks===

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====Switching to the Enlightenment Foundation Libraries====

Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

Features:
* Core languages: C, OpenGL
* Bindings: Python, C#, Ruby
* Backends: GLX / SDL / EGL
* Media support: Gstreamer integration, Cairo graphics rendering, Pango fonts rendering
* Embedding: GTK embedding

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core languages: C OpenGL
* Bindings: Python
* Backends: DirectFB OpenGL
* Media playback integration: using Gstreamer

===Choosing===

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====Description====

Take an item list (ex: adress book), print it on a ribbon of paper, and glue it on a wheel (on the tire). You're looking in the front of it, so when you want to go from the A to Z, you touch the wheel and drag it up. When you let the wheel go, it goes furter, taken by it's inertia. Stop the wheel when you got your contact. Got the idea? That's why we may speak of an "infinite wheel", so that the surface is flat. For our case here, we always want to display square content, so the [http://en.wikipedia.org/wiki/Uniform_prism n-sided uniform prism] analogy is mathematically more exact.

Important features:
* '''weight''': the biggest the item list, the faster it scrolls; that way, you don't have to wait too long for big lists, and you don't miss your item on shorter lists
* '''friction''': there is friction where the wheel is fixed, so that the wheel doesn't turn infinitely; more friction on short lists, less on big ones
* the '''initial speed and acceleration vector''' you give it determines it's futher rotation
* it's "round"/cyclic, so you can '''browse the list in two directions'''
* we may want to add a "progression indicator", ex the current alphabetical letter, with a font size adequate to the proportion of the number of entries in the letter subcategory, or a reduced map of the distribution of the first letters...

We can add "parallel wheels", symbolizing different sorting methods. Slide long to the left / right to look at a different wheel = items organization.

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

====Full multi-touch emulation====

Doable, but tricky...

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-27T17:00:02Z

Fthiery: /* Pigment API */

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo.

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===Clever hacks===

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====Switching to the Enlightenment Foundation Libraries====

Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

Features:
* Core languages: C, OpenGL
* Bindings: Python, C#, Ruby
* Backends: GLX / SDL / EGL
* Media support: Gstreamer integration, Cairo graphics rendering, Pango fonts rendering
* Embedding: GTK embedding

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core languages: C OpenGL Python
* Bindings: Python
* Backends: DirectFB OpenGL
* Media playback integration: using Gstreamer

===Choosing===

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====Description====

Take an item list (ex: adress book), print it on a ribbon of paper, and glue it on a wheel (on the tire). You're looking in the front of it, so when you want to go from the A to Z, you touch the wheel and drag it up. When you let the wheel go, it goes furter, taken by it's inertia. Stop the wheel when you got your contact. Got the idea? That's why we may speak of an "infinite wheel", so that the surface is flat. For our case here, we always want to display square content, so the [http://en.wikipedia.org/wiki/Uniform_prism n-sided uniform prism] analogy is mathematically more exact.

Important features:
* '''weight''': the biggest the item list, the faster it scrolls; that way, you don't have to wait too long for big lists, and you don't miss your item on shorter lists
* '''friction''': there is friction where the wheel is fixed, so that the wheel doesn't turn infinitely; more friction on short lists, less on big ones
* the '''initial speed and acceleration vector''' you give it determines it's futher rotation
* it's "round"/cyclic, so you can '''browse the list in two directions'''
* we may want to add a "progression indicator", ex the current alphabetical letter, with a font size adequate to the proportion of the number of entries in the letter subcategory, or a reduced map of the distribution of the first letters...

We can add "parallel wheels", symbolizing different sorting methods. Slide long to the left / right to look at a different wheel = items organization.

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

====Full multi-touch emulation====

Doable, but tricky...

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-27T16:59:44Z

Fthiery: /* Pigment API */

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo.

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===Clever hacks===

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====Switching to the Enlightenment Foundation Libraries====

Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

Features:
* Core languages: C, OpenGL
* Bindings: Python, C#, Ruby
* Backends: GLX / SDL / EGL
* Media support: Gstreamer integration, Cairo graphics rendering, Pango fonts rendering
* Embedding: GTK embedding

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core languages: C OpenGL Python
* Bindings: Python
* Backends: DirectFB OpenGL

===Choosing===

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====Description====

Take an item list (ex: adress book), print it on a ribbon of paper, and glue it on a wheel (on the tire). You're looking in the front of it, so when you want to go from the A to Z, you touch the wheel and drag it up. When you let the wheel go, it goes furter, taken by it's inertia. Stop the wheel when you got your contact. Got the idea? That's why we may speak of an "infinite wheel", so that the surface is flat. For our case here, we always want to display square content, so the [http://en.wikipedia.org/wiki/Uniform_prism n-sided uniform prism] analogy is mathematically more exact.

Important features:
* '''weight''': the biggest the item list, the faster it scrolls; that way, you don't have to wait too long for big lists, and you don't miss your item on shorter lists
* '''friction''': there is friction where the wheel is fixed, so that the wheel doesn't turn infinitely; more friction on short lists, less on big ones
* the '''initial speed and acceleration vector''' you give it determines it's futher rotation
* it's "round"/cyclic, so you can '''browse the list in two directions'''
* we may want to add a "progression indicator", ex the current alphabetical letter, with a font size adequate to the proportion of the number of entries in the letter subcategory, or a reduced map of the distribution of the first letters...

We can add "parallel wheels", symbolizing different sorting methods. Slide long to the left / right to look at a different wheel = items organization.

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

====Full multi-touch emulation====

Doable, but tricky...

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-27T16:59:06Z

Fthiery: /* Clutter Toolkit */

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo.

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===Clever hacks===

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====Switching to the Enlightenment Foundation Libraries====

Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

Features:
* Core languages: C, OpenGL
* Bindings: Python, C#, Ruby
* Backends: GLX / SDL / EGL
* Media support: Gstreamer integration, Cairo graphics rendering, Pango fonts rendering
* Embedding: GTK embedding

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core language: C/Python
* Bindings: Python
* Backends: DirectFB OpenGL

===Choosing===

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====Description====

Take an item list (ex: adress book), print it on a ribbon of paper, and glue it on a wheel (on the tire). You're looking in the front of it, so when you want to go from the A to Z, you touch the wheel and drag it up. When you let the wheel go, it goes furter, taken by it's inertia. Stop the wheel when you got your contact. Got the idea? That's why we may speak of an "infinite wheel", so that the surface is flat. For our case here, we always want to display square content, so the [http://en.wikipedia.org/wiki/Uniform_prism n-sided uniform prism] analogy is mathematically more exact.

Important features:
* '''weight''': the biggest the item list, the faster it scrolls; that way, you don't have to wait too long for big lists, and you don't miss your item on shorter lists
* '''friction''': there is friction where the wheel is fixed, so that the wheel doesn't turn infinitely; more friction on short lists, less on big ones
* the '''initial speed and acceleration vector''' you give it determines it's futher rotation
* it's "round"/cyclic, so you can '''browse the list in two directions'''
* we may want to add a "progression indicator", ex the current alphabetical letter, with a font size adequate to the proportion of the number of entries in the letter subcategory, or a reduced map of the distribution of the first letters...

We can add "parallel wheels", symbolizing different sorting methods. Slide long to the left / right to look at a different wheel = items organization.

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

====Full multi-touch emulation====

Doable, but tricky...

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-27T16:56:39Z

Fthiery: /* Clutter Toolkit */

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo.

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===Clever hacks===

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====Switching to the Enlightenment Foundation Libraries====

Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

Features:
* Core language: C
* Language bindings: Python, C#, Ruby
* Media support: Gstreamer integration
* Embedding: GTK embedding

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core language: C/Python
* Bindings: Python
* Backends: DirectFB OpenGL

===Choosing===

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====Description====

Take an item list (ex: adress book), print it on a ribbon of paper, and glue it on a wheel (on the tire). You're looking in the front of it, so when you want to go from the A to Z, you touch the wheel and drag it up. When you let the wheel go, it goes furter, taken by it's inertia. Stop the wheel when you got your contact. Got the idea? That's why we may speak of an "infinite wheel", so that the surface is flat. For our case here, we always want to display square content, so the [http://en.wikipedia.org/wiki/Uniform_prism n-sided uniform prism] analogy is mathematically more exact.

Important features:
* '''weight''': the biggest the item list, the faster it scrolls; that way, you don't have to wait too long for big lists, and you don't miss your item on shorter lists
* '''friction''': there is friction where the wheel is fixed, so that the wheel doesn't turn infinitely; more friction on short lists, less on big ones
* the '''initial speed and acceleration vector''' you give it determines it's futher rotation
* it's "round"/cyclic, so you can '''browse the list in two directions'''
* we may want to add a "progression indicator", ex the current alphabetical letter, with a font size adequate to the proportion of the number of entries in the letter subcategory, or a reduced map of the distribution of the first letters...

We can add "parallel wheels", symbolizing different sorting methods. Slide long to the left / right to look at a different wheel = items organization.

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

====Full multi-touch emulation====

Doable, but tricky...

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-27T16:55:04Z

Fthiery: /* Pigment API */

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo.

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===Clever hacks===

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====Switching to the Enlightenment Foundation Libraries====

Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Features:
* Core language: C/Python
* Bindings: Python
* Backends: DirectFB OpenGL

===Choosing===

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====Description====

Take an item list (ex: adress book), print it on a ribbon of paper, and glue it on a wheel (on the tire). You're looking in the front of it, so when you want to go from the A to Z, you touch the wheel and drag it up. When you let the wheel go, it goes furter, taken by it's inertia. Stop the wheel when you got your contact. Got the idea? That's why we may speak of an "infinite wheel", so that the surface is flat. For our case here, we always want to display square content, so the [http://en.wikipedia.org/wiki/Uniform_prism n-sided uniform prism] analogy is mathematically more exact.

Important features:
* '''weight''': the biggest the item list, the faster it scrolls; that way, you don't have to wait too long for big lists, and you don't miss your item on shorter lists
* '''friction''': there is friction where the wheel is fixed, so that the wheel doesn't turn infinitely; more friction on short lists, less on big ones
* the '''initial speed and acceleration vector''' you give it determines it's futher rotation
* it's "round"/cyclic, so you can '''browse the list in two directions'''
* we may want to add a "progression indicator", ex the current alphabetical letter, with a font size adequate to the proportion of the number of entries in the letter subcategory, or a reduced map of the distribution of the first letters...

We can add "parallel wheels", symbolizing different sorting methods. Slide long to the left / right to look at a different wheel = items organization.

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

====Full multi-touch emulation====

Doable, but tricky...

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-27T16:54:40Z

Fthiery: /* Pigment API */

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo.

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===Clever hacks===

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====Switching to the Enlightenment Foundation Libraries====

Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

Fluendo's (the Gstreamer guys) ''[https://core.fluendo.com/pigment/trac Pigment] is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

Core language: C/Python
Bindings: Python
Backends: DirectFB OpenGL

===Choosing===

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====Description====

Take an item list (ex: adress book), print it on a ribbon of paper, and glue it on a wheel (on the tire). You're looking in the front of it, so when you want to go from the A to Z, you touch the wheel and drag it up. When you let the wheel go, it goes furter, taken by it's inertia. Stop the wheel when you got your contact. Got the idea? That's why we may speak of an "infinite wheel", so that the surface is flat. For our case here, we always want to display square content, so the [http://en.wikipedia.org/wiki/Uniform_prism n-sided uniform prism] analogy is mathematically more exact.

Important features:
* '''weight''': the biggest the item list, the faster it scrolls; that way, you don't have to wait too long for big lists, and you don't miss your item on shorter lists
* '''friction''': there is friction where the wheel is fixed, so that the wheel doesn't turn infinitely; more friction on short lists, less on big ones
* the '''initial speed and acceleration vector''' you give it determines it's futher rotation
* it's "round"/cyclic, so you can '''browse the list in two directions'''
* we may want to add a "progression indicator", ex the current alphabetical letter, with a font size adequate to the proportion of the number of entries in the letter subcategory, or a reduced map of the distribution of the first letters...

We can add "parallel wheels", symbolizing different sorting methods. Slide long to the left / right to look at a different wheel = items organization.

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

====Full multi-touch emulation====

Doable, but tricky...

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

UI Improvements

2007-07-27T16:50:37Z

Fthiery: Added pigment/clutter/graff info

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo.

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===Clever hacks===

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that may be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strengths / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-clouds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Towards OpenGL compositing===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that could be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile Linux device has such a high DPI resolution. OpenGL ES compositing seems to have a bright future on embedded devices, because compositing seems to give natural zooming interfaces reality (at last!).

Considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware, for 100% fluid operation. Benchmarking will be needed to compare the different alternatives that are cited further.

====Switching to the Enlightenment Foundation Libraries====

Evas is a powerful and power sparing canvas drawing library. It can be OpenGL accelerated. Python/Ruby bindings are available in the "proto" e17 cvs folder.

''Moved [[E17|here]]''

====Clutter Toolkit====

Clutter, an [http://o-hand.com/ OpenedHand] project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

It does integrate gstreamer (for easy media playback, even camera or mic inputs), allows pango text rendering, cairo graphics rendering. Provided bindings are python, C# and Ruby.

GTK off screen rendering is supposed to be on it's way; once it is here, there will be a possibility of using GTK apps directly within OpenGL apps as textures, which would lead to the possibility of creating a full OpenGL "application manager" (as well as media consuming app) with ZUI features.

====Graff====

An early demonstration of Graff, which ''is a lighweight high-performance graphics rendering library.''
http://www.mdk.org.pl/articles/2007/04/23/chapter-1-in-which-we-meet-graff

Be sure to check out this demo (scrolling list with inertia scrolling)
http://files.mdk.am/demos/graff-demo-3.avi

Of course it will remind you of Apple iPhone's UI. But this one runs in software mode on Nokia N770&800 already. The most notable part of Graff seems to be the inertia and physics integration in general.

====Pigment API====

''Pigment is a Python library designed to easily build user interfaces with embedded multimedia. Its design allows to use it on several platforms, thanks to a plugin system allowing to choose the underlying graphical API. Pigment is the rendering engine of Elisa, the Fluendo Media Center project.''

===Choosing===

Benchmarking will be needed. We have therefore to define a std testing application that would allow to compare alternatives.

Some Clutter VS Pigment information: http://www.taimila.com/?q=node/14

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====Description====

Take an item list (ex: adress book), print it on a ribbon of paper, and glue it on a wheel (on the tire). You're looking in the front of it, so when you want to go from the A to Z, you touch the wheel and drag it up. When you let the wheel go, it goes furter, taken by it's inertia. Stop the wheel when you got your contact. Got the idea? That's why we may speak of an "infinite wheel", so that the surface is flat. For our case here, we always want to display square content, so the [http://en.wikipedia.org/wiki/Uniform_prism n-sided uniform prism] analogy is mathematically more exact.

Important features:
* '''weight''': the biggest the item list, the faster it scrolls; that way, you don't have to wait too long for big lists, and you don't miss your item on shorter lists
* '''friction''': there is friction where the wheel is fixed, so that the wheel doesn't turn infinitely; more friction on short lists, less on big ones
* the '''initial speed and acceleration vector''' you give it determines it's futher rotation
* it's "round"/cyclic, so you can '''browse the list in two directions'''
* we may want to add a "progression indicator", ex the current alphabetical letter, with a font size adequate to the proportion of the number of entries in the letter subcategory, or a reduced map of the distribution of the first letters...

We can add "parallel wheels", symbolizing different sorting methods. Slide long to the left / right to look at a different wheel = items organization.

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Left-handed UI Support===

====Description====
A discussion on the community list identified a desire to have the ability to switch the OpenMoko UI into "left-handed" mode.

The main problem is scrollbars, when they're on the right, dragging
the scrollbar left handed results in your hand covering the screen so
you can't see what you are doing. So having the option of scrollbars
on the left would be useful.

I don't think the whole screen should be mirrored! There are some elements
that should remain..like the main top bar with the status icons and such.
Scrollbars are the main thing I can think of right now.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the so-called "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multi-touchscreen mode"
* afterwards,
* slide your right-hand finger down, it scrolls up
* slide your right-hand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quarter circle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multi-touch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handed people?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

====Full multi-touch emulation====

Doable, but tricky...

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with [[GTA02#.22Phase_2.22_.28GTA02.2C_.22Mass_Market.22.29|GTA02]].
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

{{Languages|UI Improvements}}

[[Category:User Interfaces| ]]

User:Fthiery

2007-07-27T16:30:51Z

Fthiery:

http://wiki.openmoko.org/wiki/Wishlist:LiveUSB_distro
http://wiki.openmoko.org/wiki/UI_Improvements

Openmoko Local Groups: Paris

2007-07-27T16:29:35Z

Fthiery:

IDF - Paris

{|border=1
!Name
!Skills
!Interest
!Location
|-
|[[User:Fthiery|Florent Thiery]]
|Software architecture,Bash,Python
|Application development
|Evry(91)
|}

[[Category:Community]]

Openmoko Local Groups

2007-07-27T16:26:37Z

Fthiery: Paris local group

__TOC__
'''OpenMoko Local Groups...'''

* can be used for developers to meet and get to know each other
* can be used for developing
* may provide the room for real-life discussions away from mailinglists/wiki
* can support early support handling their GTA01Bv4 and exchange experiences
* fasten the community's cohesion
* give motivation

== EU ==

* Austria
** [[OpenMoko_Local_Groups: Vienna|Vienna]]

* Belgium
** [[OpenMoko_local_Groups: Antwerp|Antwerp]]

* Finland
** [[OpenMoko_Local_Groups: Helsinki|Helsinki]]

* France
** [[OpenMoko_Local_Groups: Paris|Paris]]
** [[OpenMoko_Local_Groups: Toulouse|Toulouse]]

* Germany
** [[OpenMoko_Local_Groups: Berlin|Berlin]]
** [[OpenMoko_Local_Groups: Braunschweig|Braunschweig]]
** [[OpenMoko_Local_Groups: Duesseldorf|Duesseldorf]]
** [[OpenMoko_Local_Groups: Frankfurt Main|Frankfurt Main]]
** [[OpenMoko_Local_Groups: OWL|OWL]]

* [[OpenMoko_Local_Groups:_Netherlands|Netherlands]]
** [[OpenMoko_Local_Groups: Amsterdam|Amsterdam]]
** [[OpenMoko_Local_Groups: Den Haag|Den Haag/The Hague]]
** [[OpenMoko_Local_Groups: Eindhoven|Eindhoven]]
** [[OpenMoko_Local_Groups: Ede|Ede]]

* Italy
** [[OpenMoko_Local_Groups: Milan|Milan]]

* [[OpenMoko_Local_Groups:_Norway|Norway]]
** [[OpenMoko_Local_Groups: Trondheim|Trondheim]]

* Poland
** [[OpenMoko_Local_Groups: Warsaw|Warsaw]]

* Portugal
** [[OpenMoko_Local_Groups: Braga|Braga]]

* Romania
** [[OpenMoko_Local_Groups: Bucharest|Bucharest]]

* Spain
** [[OpenMoko_Local_Groups: Madrid|Madrid]]

* Sweden
** [[OpenMoko_Local_Groups: Gothenburg|Gothenburg]]

* [[OpenMoko_Local_Groups: Switzerland|Switzerland]]
** [[OpenMoko_Local_Groups: Bern|Bern]]

* UK
** England
*** [[OpenMoko_Local_Groups: Birmingham|Birmingham]]
*** [[OpenMoko_Local_Groups: London|London]]
** Scotland
*** [[OpenMoko_Local_Groups: Fife|Fife]]
*** [[OpenMoko_Local_Groups: Edinburgh|Edinburgh]]

== Oceania ==

* Australia
** [[OpenMoko_Local_Groups: Adelaide|Adelaide]]
** [[OpenMoko_Local_Groups: Canberra|Canberra]]

* New Zealand
** [[OpenMoko_Local_Groups: Auckland|Auckland]]
** [[OpenMoko_Local_Groups: Christchurch|Christchurch]]

== Asia ==
* India
** [[OpenMoko_Local_Groups: Delhi|Delhi]]

== USA ==
* Arizona
** [[OpenMoko_Local_Groups: Arizona|Arizona]]

* California
** [[OpenMoko_Local_Groups: San Diego|San Diego]]
** [[OpenMoko_Local_Groups: San Francisco|San Francisco]]
** [[OpenMoko_Local_Groups: Sacramento|Sacramento]]

* Colorado
** [[OpenMoko_Local_Groups: Colorado Springs|Colorado Springs]]

* District of Columbia
** [[OpenMoko_Local_Groups: Washington DC Metro | Washington DC Metro]]

* Florida
** [[OpenMoko_Local_Groups: Central Florida | Central Florida]]

* Georgia
** [[OpenMoko_Local_Groups: Atlanta | Atlanta]]

* Illinois
** [[OpenMoko_Local_Groups: Chicago|Chicago]]

* Indiana
** [[OpenMoko_Local_Groups: Bloomington|Bloomington]]

* Iowa
** [[OpenMoko_Local_Groups: Iowa-Ames | Iowa-Ames]]

* Massachusetts
** [[OpenMoko_Local_groups: Boston|Boston]]

* Michigan
** [[OpenMoko_Local_Groups: Detroit|Detroit]]

* New Jersey
** [[OpenMoko_Local_Groups: Stevens Institute of Technology, Hoboken NJ | Stevens Institute of Technology, Hoboken NJ ]]

* New York
** [[OpenMoko_Local_Groups: NYC Metro | NYC Metro]]

* North Carolina
** [[OpenMoko Local_Groups: Charlotte | Charlotte]]

* Ohio
** [[OpenMoko_Local_Groups: Cleveland|Cleveland]]

* Oregon
** [[OpenMoko_Local_Groups: Eugene|Eugene]]
** [[OpenMoko_Local_Groups: Portland|Portland]]

* Texas
** [[OpenMoko_Local_Groups: North Texas|North Texas]]

* Utah
** [[OpenMoko_Local_Groups: Salt Lake|Salt Lake]]

* Virginia
** [[OpenMoko_Local_Groups: Virginia|Virginia]]

== Canada ==
* Alberta
** [[OpenMoko_Local_Groups: Edmonton|Edmonton]]
** [[OpenMoko_Local_Groups: Pincher Creek|Pincher Creek]]
* British Columbia
** [[OpenMoko_Local_Groups: Vancouver|Vancouver]]
* Ontario
** [[OpenMoko_Local_Groups: Ottawa|Ottawa]]
** [[OpenMoko_Local_Groups: Toronto|Toronto]]
* Quebec
** [[OpenMoko_Local_Groups: Montreal|Montreal]]

[[Category:Community]]
[[Category:Neo1973 Phase 1 related]]

UI Improvements

2007-07-10T14:13:54Z

Fthiery: Added digital phisics readwriteweb article

==Introduction==

Obviously the tools are in the wild to build interfaces that could rival
(or better IMO) anything Apple comes up with. We just need to organize
this stuff. This would need hardware that can support dynamic
interfaces. I can help here, too.
sean@openmoko.com

In fact, this place shall be dedicated to human-machine interactions improvements discussion;

Human-machine interaction can be separated into several aspects:
* the physical contact/input device: the mono-touch touchscreen
* the graphics:
** accelerated rendering can add more consistency to zooming user interfaces, which seem to be quite an interesting concept for embedded scrensize-limited devices
** adding physics "look and feel" to (ex: menu's) behaviours can add coherence too
* the input methods
** the virtual keyboard
** handgestures

==Finding inspiration ...==

===Seen Live===

'''Multi-touch'''
*[http://www.youtube.com/watch?v=89sz8ExZndc Multi-Touchscreen experiments video @youtube]
*[http://www.youtube.com/watch?v=nPqqfVLQ_qY iPhone UI features demo @youtube]

'''Zooming user interfaces'''
*[http://www.zenzui.com/products.html ZenZui], a [http://en.wikipedia.org/wiki/ZUI ZUI (zooming user interface)]
*[http://labs.live.com/Deepfish/videos.aspx Microsoft Bluefish's ZUI] (new mobile webbrowser)
*[http://googlesystem.blogspot.com/2007/04/opera-920-more-homepages-at-your.html Opera speed dial]

'''Graphics'''
*[http://www.rasterman.com/files/eem.avi EEM], Rasterman's EFL libs on handelds proof of concept (doesn't DO anything, just showing off the EFLs on a handeld)
*[http://www.youtube.com/watch?v=8kLFPfaxQ6U&eurl= Nvidia's cutting-edge techno], an [http://www.khronos.org/openkode/ Openkode] demo.

'''Science fiction'''
*[http://www.youtube.com/watch?v=G_FS2TiK3AI UPMC Future?]

'''Links:'''
*[http://elevate.sourceforge.net/links.html elevate project's links] sums up quite greatly the latest innovations in the desktop area
*[http://nooface.net/ Nooface] is a human-machine news site
*Asus-Intel's [http://www.hothardware.com/articles/Hands_on_with_the_ASUS_Eee/?page=2 Eee]'s interface

===Clever hacks===

It's been said that having no multitouch screen allows less freedom for innovation. Maybe we could get something out of our touchscreen drivers.

Why ? Think of apple's scroll up/down feature on macbooks touchpads (which '''aren't multi touch''', it's a [http://iscroll2.sourceforge.net/ clever driver hack, iscroll2]):

To scroll, just place two fingers on your trackpad instead of one. Both fingers
need to be placed next to each other horizontally (not vertically, the trackpad
cannot detect that). Some people get better results with their finger spaced a
little bit apart, while others prefer having the fingers right next to each other.

iScroll2 provides two scrolling modes: Linear and circular scrolling.

For linear scrolling, move the two fingers up/down or left/right in a straight
line, respectively, to scroll in that direction.

Circular scrolling works in a way similar to the iPod's scroll wheel: Move the two
fingers in a circle to scroll up or down, depending on whether you move in a
clockwise or counterclockwise direction.

Maybe we can port/adapt/get inspiration from this macintosh driver.

===n-D navigation: the polyhedra inspiration===

When we want to navigate files, mp3s in an mp3 player, etc... Every control that the application needs is a button. What about looking at the polyhedrons? We could find one for each usage, with as many surrounding subzones that way be used as controls. Ex: you need 5 buttons, let's take a pentagon with 5 surrounding zones all around. That way, it's always optimized...

http://en.wikipedia.org/wiki/Polyhedra
http://en.wikipedia.org/wiki/List_of_uniform_polyhedra

===Our weapons===

We can't improve the human-machine interface without knowing the strenghts / weaknesses of our hardware; some of the weaknesses might turn out as exploitable features, some strengths as limiting constraints.

====The touchscreen====

Question:

What exactly does the touchscreen see when you touch the screen with 2 fingers
at the same time, when you move them, when you move only one of the 2, etc. I'm
also interested in knowing how precise the touchscreen is (ex: refresh rate,
possible pressure indication, ...)?

Answer:

* The output is the center of the bounding box of the touched area
* The touch point skips instantly on double touch
* Pressure has almost no effect on a single touch, but not so on a double touch. The relative pressures will cause a significant skewing effect towards the harder touch. You can easily move the pointer along the line between your two fingers by changing the relative pressure.

Conclusions:
* we can detect double touch as jumps, and that's all
* no pressure
* This could be an interesting input method for games - e.g. holding the Neo in landscape view, letting each thumb rest on a specific input area; probably needs to be checked for usability with a real device

Question:

What does one see when sliding two fingers in parallel up(L,R)->down(L,R)?

Answer:

* In theory you see a slide along the center line between your two fingers. In practice, you can't keep the pressure equal, so you will see some kind of zig-zag line somewhere between the two pressure points in the direction of your slide.

Question:

What does one see when narrowing two fingers in slide (=zoom effect on iphone)?

Answer:

* In theory you see the pointer stay at the center of the zoom movement. In practice, you can't keep the pressure equal for both fingers, so the pointer will move towards one of the two pressure points.

====Graphics and computational capabilities====

It would be good to report what performance the current hardware allows:
* There was no pure X11 benchmarking done (AFAIK) (how many fps at full VGA scrolling, ex: 1024*480 image scrolling?)
* what about the lcd reactivity? What if we don't see anything but blur while moving items fast?

Please report here your impressions.

==Areas of improvement==

* OpenGL for fluid zooming interfaces (2D: the infinite sphere model, 1D: the infinite wheel of fortune/ribbon model, exposé)
* HandGestures
* Physics-model based improvements: inertia and friction
* multi touch screen for natural handgestures
* improved virtual keyboard
* switching to another GUI toolkit (EFLs)

===Physics-inspired animation a.k.a. "Digital Physics"===

If we want to add eye candy & usability to the UI (such as smooth realistic list scrolling, as seen in apple's
iphone demo on contacts lists for instance), we'll need a physics engine, so that moves & animations aren't all linear.

The following aticle explains the [http://www.readwriteweb.com/archives/the_physics_of_iphone.php Digital Physics] term from the iPhone example.

The most used technique for calculating trajectories and systems of related geometrical objects seems to be [http://en.wikipedia.org/wiki/Verlet_integration verlet integration] implementation; it is an alternative to Euler's integration method, using fast approximation.

We may have no need for such a mathematical method at first, but perhaps there are other use cases. For instance, it may be useful to gesture recognition (i'm not aware if existing gesture recognition engines measure speed, acceleration...).

====Open Dynamics Engine====

ODE is an open source, high performance library for simulating rigid body dynamics. It is fully featured, stable, mature and platform independent with an easy to use C/C++ API. It has advanced joint types and integrated collision detection with friction. ODE is useful for simulating vehicles, objects in virtual reality environments and virtual creatures. It is currently used in many computer games, 3D authoring tools and simulation tools.

http://www.ode.org/

====Libakamaru====

The [http://people.freedesktop.org/~krh/akamaru.git/ akamaru library] is the code behind [http://www.youtube.com/watch?v=VekgyKQoTeM kiba dock]'s fun and dynamic behaviour. It's dependencies are light (needs just GLib). It takes elasticity, friction, gravity into account.

If you want to take a quick look at the code:
svn co http://svn.kiba-dock.org/akamaru/ akamaru

The only (AFAIK) application using this library is kiba-dock, a *fun* app launcher, but we may find another use to it in the future.

As suggested on the mailing list, it is mostly overkill for the uses we intend to have, but this library may be optimized already, the API can spare some time for too. Furthermore, ''Qui peut le plus, peut le moins''.

====Verlet integration implementation from e17====

There's an undergoing verlet integration implementation into the e17 project (by rephorm) see http://rephorm.com/news/tag/physics , so we may see some UI physics integration into e17 someday.

====Robert Pernner's easy equations====

http://www.robertpenner.com/easing/

See the demo: implements non linear behaviour (actionscript), but may give inspiration

===Extending the touchscreen capabilities and input methods===

* '''Multitouchscreen emulation'''

If we got it right, when touching the screen on a second place, the cursor oscillates between the two points depending on relative pressure distribution. Using averaging algorithms, we may have the opportunity to detect peculiar behaviours.

We need raw data (x,y,t) from the real hardware for the following behaviours:
* slide two fingers in parallel - vertical up/down (scroll)
* turn the two fingers around (rotate)
* slide two fingers towards each other (zoom-)
* slide two fingers apart (zoom+)

When touching the screen with two fingers at the same time, we necessarily see the two points, or are able to extrapolate the position of the second one. This solution can add feature, but will probably be little erratic...

* '''Touchscreen kernel module hacking'''

We may correct the "half distance" phenomenon on double touch: if double touch is detected, then assimilate the cursor as twice further than the first touch. It would allow finer control, but higher instability.

The double touch detection may be implemented in the driver itself, as well as stabilization.

* '''Other detectable behaviours'''

The warping can be used in the 4 diagonals, plus the up/down/right/left cross:

---------------- ---------------- ---------------- ----------------
- 1 - - 1 2 - - 1 - - 2 -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- - - - - - - -
- 2- - - - 2 - - 1 -
---------------- ---------------- ---------------- ----------------

It's not double touch, but two sequential presses with a short time in between (~0.5 s)

===Improved virtual keyboard===

One nice idea for virtual input is [http://www.micropp.se/openmoko/ finger-splash]

Yet, optimization does not only apply to the plain one-letter-at-a-time input. We need some sort of T9 (dictionary-based input help). When typing a word, the first letter determines the next possible ones. Therefore, we may let disappear the less-probable following letters. Ex: type an L, there's no way an X follows...

Hints:
* ZIP's huffmann compression applied to SMSs/mails for detecting the most used characters/words/sentences.
* html tag-couds, one-letter tag clouds ; font size proportional to the probability of being used

The most critical point is the initial disposition of the letters, before any letter is typed. We may also want to use horizontal two-parts keyboard (with the neo in hands like a psp..)

The [http://www.strout.net/info/ideas/hexinput.html hexinput] concept is interesting. What about hiding the less probable letters and increasing the remaining ones during typing?

===Choosing the right GUI toolkit===

There are [http://www.hbmobile.org/wiki/index.php?title=GUI_Frameworks lots of possible GUI frameworks] with various software architectures that can be used for OpenMoko.

GTA01 hardware uses GTK+/matchbox without hardware acceleration, and it's not enough: this is a first that a mobile linux device has such a high DPI resolution.

Benchmarking will be needed to compare the different alternatives: qt, directfbgtk, ...

====Switching to the Enlightenment Foundation Libraries====

''Moved [[E17|here]]''

====Distant future: OpenGL compositing====

Compositing seems to give zooming interfaces reality (at last!).

Well, considering recent changes in destkop applications, opengl has a definite future. For instance, the expose (be it apple's or beryl's) is a very interesting and usable feature. Using compositing allows the physics metaphore: '''the human brain doesn't like "gaps"/jumps (for instance while scrolling a text), it needs continuity''', which can be provided by opengl. When you look at apple's iphone prototype, it's not just eye candy, it's maybe the most natural/human way of navigating, because it's sufficiently
realistic for the brain to forget the non-physical nature of what's inside.

So, opengl hardware will be needed in a more or less distant hardware,
for 100% fluid operation.

How cool will solid-based (polygons, as seen in beryl) interfaces be? :) Real ZUIs...

====Clutter Toolkit====

Clutter, an openedhand project, is an open source software library for creating fast, visually rich graphical user interfaces. The most obvious example of potential usage is in media center type applications. We hope however it can be used for a lot more.

http://clutter-project.org/

Clutter uses OpenGL (and optionally '''OpenGL ES''') for rendering but with an API which hides the underlying GL complexity from the developer. The Clutter API is intended to be easy to use, efficient and flexible.

From the [http://en.wikipedia.org/wiki/OpenGL_ES wikipedia article], OpenGL ES (OpenGL for Embedded Systems) is a subset of the OpenGL 3D graphics API designed for embedded devices such as mobile phones, PDAs, and video game consoles.

==Improvement ideas==

Please add here any idea that seems of relevance.

===1D list scrolling: looped physics-driven item list===

====Description====

Take an item list (ex: adress book), print it on a ribbon of paper, and glue it on a wheel (on the tire). You're looking in the front of it, so when you want to go from the A to Z, you touch the wheel and drag it up. When you let the wheel go, it goes furter, taken by it's inertia. Stop the wheel when you got your contact. Got the idea? That's why we may speak of an "infinite wheel", so that the surface is flat. For our case here, we always want to display square content, so the [http://en.wikipedia.org/wiki/Uniform_prism n-sided uniform prism] analogy is mathematically more exact.

Important features:
* '''weight''': the biggest the item list, the faster it scrolls; that way, you don't have to wait too long for big lists, and you don't miss your item on shorter lists
* '''friction''': there is friction where the wheel is fixed, so that the wheel doesn't turn infinitely; more friction on short lists, less on big ones
* the '''initial speed and acceleration vector''' you give it determines it's futher rotation
* it's "round"/cyclic, so you can '''browse the list in two directions'''
* we may want to add a "progression indicator", ex the current alphabetical letter, with a font size adequate to the proportion of the number of entries in the letter subcategory, or a reduced map of the distribution of the first letters...

We can add "parallel wheels", symbolizing different sorting methods. Slide long to the left / right to look at a different wheel = items organization.

====Controls====

* Sliding up/down = Single click + maintained for a minimal distance

Effect: scroll in an inverted/negated fashion (slide down = scroll up, slide up = scroll down)

When finger is released (i.e. touchscreen doesn't detect any press):
if (last_speed_seen > 0 ) then keep this speed and acceleration, with friction
else stop scrolling

Scrolling here is seen as unidimensional, but can apply to bidimensional situations (ex: zoomed image) too

* Action = quick double tap

* Details/select = short single tap

* Right click = long tap

* Sliding left/right: switch sorting method

====Parts to modify====

''Having a scroll that isn't a 1:1 map to the user's action isn't hard. It's just an extra calculation in the scroll code.''

<---- Where is the scroll code? :)

* libmokoui
** [[Stylus_Scrolling | Stylus scrolling widget]]
** [[Finger_Scrolling | Finger scrolling wheel widget]]
* gtk
** [http://www.gtk.org/api/2.6/gtk/GtkVScrollbar.html GtkVScrollbar]

The best would be to add the feature for both finger and stylus scrolling.

TODO:
* make the entire list a "scrolling zone", i.e. an overlay transparent scrolling widget?
* define controls
* add the inertia feature

===1D Scrolling: inertia friction integration into openmoko's finger wheel===

The same, but for the wheel. It can be very short to do: you don't have 1:1 anymore, but, for example, 1/4 wheel turn = 1 item. It's demultiplicated, but has inertia.

===Handgesture recognition proposals===

====Using simple, localized warp as modifier key====

As discussed on community list:

If you hold down one finger and tap the other one, the mouse pops over and back
again. If you keep your second finger touching, the cursor follows it. When you
release it, cursor goes back to first finger position. This could be a way to
set a bounding box or turn on the mode. So the second finger can do something
like rotating around the first, or increase or lower the distance to the first.

* the socalled "first touch" can be done on the mokowheel zone itself: put your left thumb on the black area; if you touch the screen with another finger, there is a '''warp'''; the warp is detectable and allows to enter "fake multitouchscreen mode"
* afterwards,
* slide your righthand finger down, it scrolls up
* slide your righthand finger up, it scrolls down
* slide it left, next page/item
* slide it right, previous page/item
* do a circle: rotation
* narrow towards the black circle: zoom -
* go away: zoom +
* if you had kept your first finger on the black quartercircle, you can continue issuing other gestures

The advantages of using simple origin-driven cursor warping as double touch detection criteria is that:
* you don't have to use the wheel as button, which would slow things down and generate errors (false button presses)
* simpler detection algorithms that can pass by the fluctuation due to pressure relative distribution
* the space taken by the wheel itself is "useless" (i.e. displays no information); using it as modifier allows to keep the screen clean for reading
* the origin of this zone lets use maximum surface of the screen, allowing more fine controlling

[[Image:Fake_multitouch_rotate_right.png]]

[[Image:Fake_multitouch_scroll_previous.png]]

[[Image:Fake_multitouch_scroll_down.png]]

[[Image:Fake_multitouch_zoom-.png]]

*Pros:
** who said we need multitouch hardware?
** this may be the easiest way (in terms of design/implementation complexity, reliability)
*Cons:
** no matter what we'll invent, we'll need two hands for on-the-move controlling
** what about left-handeds?

====What is to modify ?====

We need to emulate key presses. We need to work at a layer where we can get raw cursor coordinates. <---- X server layer?

====Full multi-touch emulation====

Doable, but tricky...

==Open questions==

* will the neo/openmoko graphics system be powerful enough for such uses? I apple uses an OpenGL ES acceleration on this device (as well as on recent iPods), which is on the way with GTA02
* how does the touchscreen behave? We need a detailed touchscreen wiki information page, with visual traces. How hardware-specific is it?
* is multi touch really that awesome? I guess not.

[[Category:User Interfaces]]

Wishlist/LiveUSB distro

2007-06-20T23:09:59Z

Fthiery: /* The BlackDog project */

{{wishlist}}

==Introduction==

NOTE: this page may be renamed: portable destkop

An openmoko device could act as '''the perfect geeky swiss knive''': go anywhere with your Linux desktop and tools.

YES, there are tons of liveCDs out there, but CDs/DVDs:
* get easily damaged
* don't fit in a pocket (except for businesscard-sized ones ; too exotic...)
* don't update or save settings/personal data without a supplementary USB device

YES, there are LiveUSB distros but:
* a Neo/OpenMoko device IS sort of an USB stick; one object is better than two
* i carry my cellphone everywhere, not my USB stick
* there is almost no security on USB sticks (nothing like cryptoloop devices, right?)

The biggest argument against these ones is that you'd have to carry an USB cable with you... But it's a standard one, which is good news :)

Similar functionality can be found in [http://en.wikipedia.org/wiki/Wizpy the Wizpy portable media player]

When the openmoko device is in mass storage mode, a host computer should be able to boot on it, presenting a grub menu offering to boot into several images / partitions (payloads) on the transflash: memtest, UBCD (the ultimate boot cd), a lightweight security oriented livecd distro, you name it...

It's sometimes called Live USB: from [http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB definition], ''A live USB is a USB flash drive containing a full operating system which can be booted from. Live USBs are closely related to Live CDs, and are sometimes used interchangeably. Like Live CDs, Live USBs can be used for system administration, data recovery, or the testing of operating system distributions without committing to a permanent installation on the local hard drive. Many of the smaller Linux distributions can also be used from a USB flash drive.''

Usually, all you need for this with an USB storage device (flash USB stick or external drive) is a partition, flagged "bootable" (see fdisk), containing a boot loader (say grub). But the problem is that specifications vary between motherboards, so there are variants and mandatory requirements to define (here would be a good idea), so that one can optimize/maximize the compatibility.

==The perfect geek knife==

We may want to start our own distro with the following constraints:
* Multiboot: small utilities/compilations + bull blown light OS
* Aiming at maximum hardware compatibility, possibly using liveCD technology (knoppix-based distros)
* Aiming at minimum resource+system footprint, and boot speed
* Being able to install software temporary (for the session) or permanently, upgradable
* Being able to use the phone features at the same time (access pim, data & calls)

Optimally, being able to charge the device while plugged in would be great too.

==About USB Booting==

From [http://www.damnsmalllinux.org/wiki/index.php/USB_Booting DSL Wiki]:

Older computer BIOS usually do not support direct booting from a USB device. Around 2001, PC motherboard manufacturers started to add USB boot support.

There are two common BIOS methods for direct USB booting:
* One method is called the "USBHDD" method and it is used to support the booting of standard USB mass storage devices that are configured like a normal PC hard drive.
* The other method is called the "USBZIP" method and it supports booting from a USB storage device that behaves like the original IOMEGA ZIP drive with USB support.

Most computers (just about all Dells, for example) made today have a BIOS that supports the USBHDD method so I expect that this will eventually become the standard way to boot a USB device. However, many motherboards will support BOTH methods, and many older motherboards have USBZIP support.

Some newer BIOSs which support USB 2.0 will not boot from an older pen drive. Using a USB 2.0 compliant one usually solves this problem. Also, some older BIOSs which only support USB 1.1 will not boot from newer drives which support USB 2.0!

Some USB keys don't boot. If this is the case, it may be possible to fix them by installing a new master boot record. (Most keys boot OK by default; some cannot be fixed even by doing this. However, it helps in some cases). Run the command:

lilo -M /dev/sda

==USB Booting in Neo1973's kernel==

Although there is very little information about it, the g_storage kernel module is responsible for the Neo's mass storage mode. So the boot-or-not criteria might rely on this very module.

==TODO==

Testing/further research has to be conducted to determine:
* if a bootable-flagged partition / grub doesn't mess openmoko up
* if the phone can still act as a phone, or if the booted os can use the gprs functions
* if the phone can recharge on mass storage mode (from the usb cable)
* if a dedicated partition for each payload is needed (see memtest example...)
* if multi-boot is possible: grub?

==Linux distro Howto's: liveUSB creation==

In short:
#Mount USB drive, with e.g. mount /dev/sda1 /flash - can be either FAT16 or FAT32!
#Mount ISO image, with e.g. mount /tmp/dsl-3.2.iso /tmp/iso -o loop
#Copy all contents from ISO to USB drive: cp -vr /tmp/iso/* /flash/
#Rename and move syslinux files to root directory: mv /flash/boot/isolinux/* /flash/
#Rename isolinux.cfg: mv /flash/isolinux.cfg /flash/syslinux.cfg
#Unmount USB drive: umount /flash
#Install syslinux: syslinux /dev/sda1 and eventually set the MBR boot flag for this partition (with fdisk).

*Fedora Core's [http://revisor.fedoraunity.org Revisor] enables easy GUI-driven ISO customization, allowing to choose the output medium (CD/DVD/USB)
*[http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive Installing DSL to USB device - using grub]
*[http://wiki.debian.org/BootUsb Debian BootUSB - using Lilo]
*[http://www.debuntu.org/how-to-install-ubuntu-linux-on-usb-bar Howto install ubuntu (or any other distro) on usb stick]
*[http://www.gentoo.org/doc/en/liveusb.xml Gentoo on usb stick]
*[http://pendrivelinux.com/ Windows & Linux tutorials & resources]:
* Flash installation via Windows: XUbuntu, DSL, Knoppix, Slax, MiniMe
* Flash installation via Linux: Ubuntu Edgy, Knoppix, PCLinuxOS

'''Transflash partitioning schema example'''

* 2 Gb: /dev/sd? (where N is the transflash's number)
* 700 Mb: /dev/sd?1 : containing bootable iso, FAT16
* 1300 Mb: /dev/sd?2 : openmoko & bootable os home partition, EXT3?
* + eventually a swap

The limitation of this method is that the booted OS is static (not-self modifiable). There's the option to install linux using the partition, but it's not really mobility-oriented (liveCDs are optimized for maximum autoconfiguration).

==Alternative methods==

===Booting ISOs ?===

* Grub? From [http://www.freesoftwaremagazine.com/articles/grub_intro#comment-12900 How to install grub on an usb pendrive]:
If you want to boot from a iso file on a harddisk, do something in menu.lst like
title Boot from iso on a harddisk
map (hdX,Y)/your.iso (hdZ)
map --rehook
chainloader (hdZ)+1
rootnoverify (hdZ)
boot

Grub & memdisk can boot floppy images, but (AFAIK) not isos. Example from the stock /boot/grub/menu.lst ubuntu:
title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin
quiet
boot

''Using the memdisk kernel from the syslinux package, you can load disk images and execute them in a non-emulated environment.''

* [http://www.knoppix.net/wiki/Cheat_Codes knoppix '''bootfrom''' cheatcode]

bootfrom=/dev/hda1/KNX.iso

Bootfrom needs access to a running Knoppix-System '''with the same Kernel as the Bootkernel''', before it is able to mount the partition / ISO-Image.

* Maybe there's a way to achieve a boot cd iso selecter see ([http://www.linuxquestions.org/questions/showthread.php?t=528840 this post])
* virtualization-oriented distro... ex [http://unit.aist.go.jp/itri/knoppix/xen/index-en.html Xenoppix] Cons: would work best only on modern VT-capable computers. What about a very minimal distribution (say DSL without X) kqemu-enblaed, booting into an ncurses iso-loader? The delicate part would be on the networking side.

===Zero-Install / Networked systems===

*[http://klik.atekon.de/wiki/index.php/Main_Page Klik] with more than 1000 packages ported. See [http://klik.atekon.de/wiki/index.php/Comparison comparison chart].
*[http://0install.net/index.html The Zero Install System], with automatic dependancies, upgrading. See [http://0install.net/matrix.html] comparison chart.
*Why not rolling a [http://wiki.rpath.com/wiki/Conary:Concepts Conary/RPath-based] distro, based on [http://www.rpath.com/rbuilder/project/rpath/ rPath linux] reference distribution ? Or choose within the available ones...
* The FUSE [http://httpfs.sourceforge.net/net_boot.htm httpfs] may allow pure http net booting, downloading files from a distant livecd only when needed (this image may be very minimal though).
* [http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&start=75&tstart=0 some people] hare thinking of using Amazon S3 as virtual disk

===Using Qemu to boot Linux from a flash drive within Windows===

The following tutorial explains how to use Qemu to boot Linux from a portable USB flash device while still working within Windows. This Enables the user to have both systems running at the same time eliminating the need to restart the PC and set your BIOS options to boot Linux.

[http://www.pendrivelinux.com/2007/03/09/use-qemu-to-boot-linux-from-windows/ PendriveLinux article]

===Putting a vnc server on openmoko===

That's another alternative :)

===The BlackDog project===

"When plugged into the host computer’s USB port, BlackDog starts up, '''automatically launching the X Window system for Windows Xming and a software NAT router via an autorun application that the BlackDog presents to the host through a virtual USB CD-ROM. Once those applications are running, the virtual USB CD-ROM drive disconnects, and a virtual USB-to-Ethernet adapter is connected to provide the communications link.'''

The host machine’s monitor, keyboard, mouse, and Internet connection are used by the BlackDog for the duration of the session."

The [http://en.wikipedia.org/wiki/BlackDog BlackDog server] project has lots of interesting concepts (and software). Could [http://www.jasonkolb.com/weblog/2006/11/idea_7_personal.html personal servers] change the face of computing ? Together with biometry, ''they open up the possibility of carrying your personal server, or an extension of it, with you on your physical person. The reason this is so exciting is because it enables biometric authentication that doesn't require shipping your biometric data to a third party'.

==Interesting distros/payloads==

'''System diagnostics / recovery'''
*[http://en.wikipedia.org/wiki/Memtest Memtest86+] Very small
*[http://www.ultimatebootcd.com/ The Ultimate Boot CD]
*[http://www.sysresccd.org/Main_Page System Rescue CD]
*[http://www.tux.org/pub/people/kent-robotti/looplinux/rip/ Recovery IS Possible] < 80 MB

'''General purpose'''
*[http://jinx-linux.sourceforge.net/ jinx-linux] < 50 MB
*[http://knoppix.com/ Knoppix] 700 MB CD-ROM
*[http://www.puppylinux.org PuppyLinux] < 100 MB
*[http://www.damnsmalllinux.org DSL] ~50 MB
*[http://gentoo-wiki.com/TinyGentoo#What_is_this_.3F TinyGentoo] 5 MB

'''Security-oriented''': pentesting, forensics, anonymous webbrowsing
*[http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-test-forensics-recovery/ List of Security distros]
*[http://www.remote-exploit.org/backtrack.html Backtrack Livecd]
*[http://en.wikipedia.org/wiki/Anonym.OS Anonym.OS: browse the web anonymously]
*[http://dban.sourceforge.net/ Darik's Boot and Nuke]: ''dangerous''
*[http://byzgl.sourceforge.net/wiki/index.php/Main_Page Byzantine OS] is a webkiosk appliance (32/48 MB)

The SabayonLinux distro offers a lot of [http://www.sabayonlinux.org/wiki/index.php?title=Sabayon_Linux#Boot_Methods_for_special_ways_of_using_Sabayon boot cheatcodes], such as booting onto GeexBox. We should take a look at the method used.

'''[http://www.livecdlist.com/ List of live distros @ livecdlist.com]''' ''Includes sizes''

'''[http://en.wikipedia.org/wiki/List_of_LiveDistros List of live distros @ wikipedia]'''

'''[http://distrowatch.com/dwres.php?resource=cd List of live distros @ distrowatch]'''

'''[http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB article]'''

[[Category:Ideas]]

Wishlist/LiveUSB distro

2007-06-20T23:08:58Z

Fthiery: /* The BlackDog project */

{{wishlist}}

==Introduction==

NOTE: this page may be renamed: portable destkop

An openmoko device could act as '''the perfect geeky swiss knive''': go anywhere with your Linux desktop and tools.

YES, there are tons of liveCDs out there, but CDs/DVDs:
* get easily damaged
* don't fit in a pocket (except for businesscard-sized ones ; too exotic...)
* don't update or save settings/personal data without a supplementary USB device

YES, there are LiveUSB distros but:
* a Neo/OpenMoko device IS sort of an USB stick; one object is better than two
* i carry my cellphone everywhere, not my USB stick
* there is almost no security on USB sticks (nothing like cryptoloop devices, right?)

The biggest argument against these ones is that you'd have to carry an USB cable with you... But it's a standard one, which is good news :)

Similar functionality can be found in [http://en.wikipedia.org/wiki/Wizpy the Wizpy portable media player]

When the openmoko device is in mass storage mode, a host computer should be able to boot on it, presenting a grub menu offering to boot into several images / partitions (payloads) on the transflash: memtest, UBCD (the ultimate boot cd), a lightweight security oriented livecd distro, you name it...

It's sometimes called Live USB: from [http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB definition], ''A live USB is a USB flash drive containing a full operating system which can be booted from. Live USBs are closely related to Live CDs, and are sometimes used interchangeably. Like Live CDs, Live USBs can be used for system administration, data recovery, or the testing of operating system distributions without committing to a permanent installation on the local hard drive. Many of the smaller Linux distributions can also be used from a USB flash drive.''

Usually, all you need for this with an USB storage device (flash USB stick or external drive) is a partition, flagged "bootable" (see fdisk), containing a boot loader (say grub). But the problem is that specifications vary between motherboards, so there are variants and mandatory requirements to define (here would be a good idea), so that one can optimize/maximize the compatibility.

==The perfect geek knife==

We may want to start our own distro with the following constraints:
* Multiboot: small utilities/compilations + bull blown light OS
* Aiming at maximum hardware compatibility, possibly using liveCD technology (knoppix-based distros)
* Aiming at minimum resource+system footprint, and boot speed
* Being able to install software temporary (for the session) or permanently, upgradable
* Being able to use the phone features at the same time (access pim, data & calls)

Optimally, being able to charge the device while plugged in would be great too.

==About USB Booting==

From [http://www.damnsmalllinux.org/wiki/index.php/USB_Booting DSL Wiki]:

Older computer BIOS usually do not support direct booting from a USB device. Around 2001, PC motherboard manufacturers started to add USB boot support.

There are two common BIOS methods for direct USB booting:
* One method is called the "USBHDD" method and it is used to support the booting of standard USB mass storage devices that are configured like a normal PC hard drive.
* The other method is called the "USBZIP" method and it supports booting from a USB storage device that behaves like the original IOMEGA ZIP drive with USB support.

Most computers (just about all Dells, for example) made today have a BIOS that supports the USBHDD method so I expect that this will eventually become the standard way to boot a USB device. However, many motherboards will support BOTH methods, and many older motherboards have USBZIP support.

Some newer BIOSs which support USB 2.0 will not boot from an older pen drive. Using a USB 2.0 compliant one usually solves this problem. Also, some older BIOSs which only support USB 1.1 will not boot from newer drives which support USB 2.0!

Some USB keys don't boot. If this is the case, it may be possible to fix them by installing a new master boot record. (Most keys boot OK by default; some cannot be fixed even by doing this. However, it helps in some cases). Run the command:

lilo -M /dev/sda

==USB Booting in Neo1973's kernel==

Although there is very little information about it, the g_storage kernel module is responsible for the Neo's mass storage mode. So the boot-or-not criteria might rely on this very module.

==TODO==

Testing/further research has to be conducted to determine:
* if a bootable-flagged partition / grub doesn't mess openmoko up
* if the phone can still act as a phone, or if the booted os can use the gprs functions
* if the phone can recharge on mass storage mode (from the usb cable)
* if a dedicated partition for each payload is needed (see memtest example...)
* if multi-boot is possible: grub?

==Linux distro Howto's: liveUSB creation==

In short:
#Mount USB drive, with e.g. mount /dev/sda1 /flash - can be either FAT16 or FAT32!
#Mount ISO image, with e.g. mount /tmp/dsl-3.2.iso /tmp/iso -o loop
#Copy all contents from ISO to USB drive: cp -vr /tmp/iso/* /flash/
#Rename and move syslinux files to root directory: mv /flash/boot/isolinux/* /flash/
#Rename isolinux.cfg: mv /flash/isolinux.cfg /flash/syslinux.cfg
#Unmount USB drive: umount /flash
#Install syslinux: syslinux /dev/sda1 and eventually set the MBR boot flag for this partition (with fdisk).

*Fedora Core's [http://revisor.fedoraunity.org Revisor] enables easy GUI-driven ISO customization, allowing to choose the output medium (CD/DVD/USB)
*[http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive Installing DSL to USB device - using grub]
*[http://wiki.debian.org/BootUsb Debian BootUSB - using Lilo]
*[http://www.debuntu.org/how-to-install-ubuntu-linux-on-usb-bar Howto install ubuntu (or any other distro) on usb stick]
*[http://www.gentoo.org/doc/en/liveusb.xml Gentoo on usb stick]
*[http://pendrivelinux.com/ Windows & Linux tutorials & resources]:
* Flash installation via Windows: XUbuntu, DSL, Knoppix, Slax, MiniMe
* Flash installation via Linux: Ubuntu Edgy, Knoppix, PCLinuxOS

'''Transflash partitioning schema example'''

* 2 Gb: /dev/sd? (where N is the transflash's number)
* 700 Mb: /dev/sd?1 : containing bootable iso, FAT16
* 1300 Mb: /dev/sd?2 : openmoko & bootable os home partition, EXT3?
* + eventually a swap

The limitation of this method is that the booted OS is static (not-self modifiable). There's the option to install linux using the partition, but it's not really mobility-oriented (liveCDs are optimized for maximum autoconfiguration).

==Alternative methods==

===Booting ISOs ?===

* Grub? From [http://www.freesoftwaremagazine.com/articles/grub_intro#comment-12900 How to install grub on an usb pendrive]:
If you want to boot from a iso file on a harddisk, do something in menu.lst like
title Boot from iso on a harddisk
map (hdX,Y)/your.iso (hdZ)
map --rehook
chainloader (hdZ)+1
rootnoverify (hdZ)
boot

Grub & memdisk can boot floppy images, but (AFAIK) not isos. Example from the stock /boot/grub/menu.lst ubuntu:
title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin
quiet
boot

''Using the memdisk kernel from the syslinux package, you can load disk images and execute them in a non-emulated environment.''

* [http://www.knoppix.net/wiki/Cheat_Codes knoppix '''bootfrom''' cheatcode]

bootfrom=/dev/hda1/KNX.iso

Bootfrom needs access to a running Knoppix-System '''with the same Kernel as the Bootkernel''', before it is able to mount the partition / ISO-Image.

* Maybe there's a way to achieve a boot cd iso selecter see ([http://www.linuxquestions.org/questions/showthread.php?t=528840 this post])
* virtualization-oriented distro... ex [http://unit.aist.go.jp/itri/knoppix/xen/index-en.html Xenoppix] Cons: would work best only on modern VT-capable computers. What about a very minimal distribution (say DSL without X) kqemu-enblaed, booting into an ncurses iso-loader? The delicate part would be on the networking side.

===Zero-Install / Networked systems===

*[http://klik.atekon.de/wiki/index.php/Main_Page Klik] with more than 1000 packages ported. See [http://klik.atekon.de/wiki/index.php/Comparison comparison chart].
*[http://0install.net/index.html The Zero Install System], with automatic dependancies, upgrading. See [http://0install.net/matrix.html] comparison chart.
*Why not rolling a [http://wiki.rpath.com/wiki/Conary:Concepts Conary/RPath-based] distro, based on [http://www.rpath.com/rbuilder/project/rpath/ rPath linux] reference distribution ? Or choose within the available ones...
* The FUSE [http://httpfs.sourceforge.net/net_boot.htm httpfs] may allow pure http net booting, downloading files from a distant livecd only when needed (this image may be very minimal though).
* [http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&start=75&tstart=0 some people] hare thinking of using Amazon S3 as virtual disk

===Using Qemu to boot Linux from a flash drive within Windows===

The following tutorial explains how to use Qemu to boot Linux from a portable USB flash device while still working within Windows. This Enables the user to have both systems running at the same time eliminating the need to restart the PC and set your BIOS options to boot Linux.

[http://www.pendrivelinux.com/2007/03/09/use-qemu-to-boot-linux-from-windows/ PendriveLinux article]

===Putting a vnc server on openmoko===

That's another alternative :)

===The BlackDog project===

"When plugged into the host computer’s USB port, BlackDog starts up, '''automatically launching the X Window system for Windows Xming and a software NAT router via an autorun application that the BlackDog presents to the host through a virtual USB CD-ROM. Once those applications are running, the virtual USB CD-ROM drive disconnects, and a virtual USB-to-Ethernet adapter is connected to provide the communications link.'''

The host machine’s monitor, keyboard, mouse, and Internet connection are used by the BlackDog for the duration of the session."

The [http://en.wikipedia.org/wiki/BlackDog BlackDog server] project has lots of interesting concepts (and software). Could [http://www.jasonkolb.com/weblog/2006/11/idea_7_personal.html Personal servers] change the face of computing ?

==Interesting distros/payloads==

'''System diagnostics / recovery'''
*[http://en.wikipedia.org/wiki/Memtest Memtest86+] Very small
*[http://www.ultimatebootcd.com/ The Ultimate Boot CD]
*[http://www.sysresccd.org/Main_Page System Rescue CD]
*[http://www.tux.org/pub/people/kent-robotti/looplinux/rip/ Recovery IS Possible] < 80 MB

'''General purpose'''
*[http://jinx-linux.sourceforge.net/ jinx-linux] < 50 MB
*[http://knoppix.com/ Knoppix] 700 MB CD-ROM
*[http://www.puppylinux.org PuppyLinux] < 100 MB
*[http://www.damnsmalllinux.org DSL] ~50 MB
*[http://gentoo-wiki.com/TinyGentoo#What_is_this_.3F TinyGentoo] 5 MB

'''Security-oriented''': pentesting, forensics, anonymous webbrowsing
*[http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-test-forensics-recovery/ List of Security distros]
*[http://www.remote-exploit.org/backtrack.html Backtrack Livecd]
*[http://en.wikipedia.org/wiki/Anonym.OS Anonym.OS: browse the web anonymously]
*[http://dban.sourceforge.net/ Darik's Boot and Nuke]: ''dangerous''
*[http://byzgl.sourceforge.net/wiki/index.php/Main_Page Byzantine OS] is a webkiosk appliance (32/48 MB)

The SabayonLinux distro offers a lot of [http://www.sabayonlinux.org/wiki/index.php?title=Sabayon_Linux#Boot_Methods_for_special_ways_of_using_Sabayon boot cheatcodes], such as booting onto GeexBox. We should take a look at the method used.

'''[http://www.livecdlist.com/ List of live distros @ livecdlist.com]''' ''Includes sizes''

'''[http://en.wikipedia.org/wiki/List_of_LiveDistros List of live distros @ wikipedia]'''

'''[http://distrowatch.com/dwres.php?resource=cd List of live distros @ distrowatch]'''

'''[http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB article]'''

[[Category:Ideas]]

Wishlist/LiveUSB distro

2007-06-20T22:52:42Z

Fthiery: /* The BlackDog project */

{{wishlist}}

==Introduction==

NOTE: this page may be renamed: portable destkop

An openmoko device could act as '''the perfect geeky swiss knive''': go anywhere with your Linux desktop and tools.

YES, there are tons of liveCDs out there, but CDs/DVDs:
* get easily damaged
* don't fit in a pocket (except for businesscard-sized ones ; too exotic...)
* don't update or save settings/personal data without a supplementary USB device

YES, there are LiveUSB distros but:
* a Neo/OpenMoko device IS sort of an USB stick; one object is better than two
* i carry my cellphone everywhere, not my USB stick
* there is almost no security on USB sticks (nothing like cryptoloop devices, right?)

The biggest argument against these ones is that you'd have to carry an USB cable with you... But it's a standard one, which is good news :)

Similar functionality can be found in [http://en.wikipedia.org/wiki/Wizpy the Wizpy portable media player]

When the openmoko device is in mass storage mode, a host computer should be able to boot on it, presenting a grub menu offering to boot into several images / partitions (payloads) on the transflash: memtest, UBCD (the ultimate boot cd), a lightweight security oriented livecd distro, you name it...

It's sometimes called Live USB: from [http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB definition], ''A live USB is a USB flash drive containing a full operating system which can be booted from. Live USBs are closely related to Live CDs, and are sometimes used interchangeably. Like Live CDs, Live USBs can be used for system administration, data recovery, or the testing of operating system distributions without committing to a permanent installation on the local hard drive. Many of the smaller Linux distributions can also be used from a USB flash drive.''

Usually, all you need for this with an USB storage device (flash USB stick or external drive) is a partition, flagged "bootable" (see fdisk), containing a boot loader (say grub). But the problem is that specifications vary between motherboards, so there are variants and mandatory requirements to define (here would be a good idea), so that one can optimize/maximize the compatibility.

==The perfect geek knife==

We may want to start our own distro with the following constraints:
* Multiboot: small utilities/compilations + bull blown light OS
* Aiming at maximum hardware compatibility, possibly using liveCD technology (knoppix-based distros)
* Aiming at minimum resource+system footprint, and boot speed
* Being able to install software temporary (for the session) or permanently, upgradable
* Being able to use the phone features at the same time (access pim, data & calls)

Optimally, being able to charge the device while plugged in would be great too.

==About USB Booting==

From [http://www.damnsmalllinux.org/wiki/index.php/USB_Booting DSL Wiki]:

Older computer BIOS usually do not support direct booting from a USB device. Around 2001, PC motherboard manufacturers started to add USB boot support.

There are two common BIOS methods for direct USB booting:
* One method is called the "USBHDD" method and it is used to support the booting of standard USB mass storage devices that are configured like a normal PC hard drive.
* The other method is called the "USBZIP" method and it supports booting from a USB storage device that behaves like the original IOMEGA ZIP drive with USB support.

Most computers (just about all Dells, for example) made today have a BIOS that supports the USBHDD method so I expect that this will eventually become the standard way to boot a USB device. However, many motherboards will support BOTH methods, and many older motherboards have USBZIP support.

Some newer BIOSs which support USB 2.0 will not boot from an older pen drive. Using a USB 2.0 compliant one usually solves this problem. Also, some older BIOSs which only support USB 1.1 will not boot from newer drives which support USB 2.0!

Some USB keys don't boot. If this is the case, it may be possible to fix them by installing a new master boot record. (Most keys boot OK by default; some cannot be fixed even by doing this. However, it helps in some cases). Run the command:

lilo -M /dev/sda

==USB Booting in Neo1973's kernel==

Although there is very little information about it, the g_storage kernel module is responsible for the Neo's mass storage mode. So the boot-or-not criteria might rely on this very module.

==TODO==

Testing/further research has to be conducted to determine:
* if a bootable-flagged partition / grub doesn't mess openmoko up
* if the phone can still act as a phone, or if the booted os can use the gprs functions
* if the phone can recharge on mass storage mode (from the usb cable)
* if a dedicated partition for each payload is needed (see memtest example...)
* if multi-boot is possible: grub?

==Linux distro Howto's: liveUSB creation==

In short:
#Mount USB drive, with e.g. mount /dev/sda1 /flash - can be either FAT16 or FAT32!
#Mount ISO image, with e.g. mount /tmp/dsl-3.2.iso /tmp/iso -o loop
#Copy all contents from ISO to USB drive: cp -vr /tmp/iso/* /flash/
#Rename and move syslinux files to root directory: mv /flash/boot/isolinux/* /flash/
#Rename isolinux.cfg: mv /flash/isolinux.cfg /flash/syslinux.cfg
#Unmount USB drive: umount /flash
#Install syslinux: syslinux /dev/sda1 and eventually set the MBR boot flag for this partition (with fdisk).

*Fedora Core's [http://revisor.fedoraunity.org Revisor] enables easy GUI-driven ISO customization, allowing to choose the output medium (CD/DVD/USB)
*[http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive Installing DSL to USB device - using grub]
*[http://wiki.debian.org/BootUsb Debian BootUSB - using Lilo]
*[http://www.debuntu.org/how-to-install-ubuntu-linux-on-usb-bar Howto install ubuntu (or any other distro) on usb stick]
*[http://www.gentoo.org/doc/en/liveusb.xml Gentoo on usb stick]
*[http://pendrivelinux.com/ Windows & Linux tutorials & resources]:
* Flash installation via Windows: XUbuntu, DSL, Knoppix, Slax, MiniMe
* Flash installation via Linux: Ubuntu Edgy, Knoppix, PCLinuxOS

'''Transflash partitioning schema example'''

* 2 Gb: /dev/sd? (where N is the transflash's number)
* 700 Mb: /dev/sd?1 : containing bootable iso, FAT16
* 1300 Mb: /dev/sd?2 : openmoko & bootable os home partition, EXT3?
* + eventually a swap

The limitation of this method is that the booted OS is static (not-self modifiable). There's the option to install linux using the partition, but it's not really mobility-oriented (liveCDs are optimized for maximum autoconfiguration).

==Alternative methods==

===Booting ISOs ?===

* Grub? From [http://www.freesoftwaremagazine.com/articles/grub_intro#comment-12900 How to install grub on an usb pendrive]:
If you want to boot from a iso file on a harddisk, do something in menu.lst like
title Boot from iso on a harddisk
map (hdX,Y)/your.iso (hdZ)
map --rehook
chainloader (hdZ)+1
rootnoverify (hdZ)
boot

Grub & memdisk can boot floppy images, but (AFAIK) not isos. Example from the stock /boot/grub/menu.lst ubuntu:
title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin
quiet
boot

''Using the memdisk kernel from the syslinux package, you can load disk images and execute them in a non-emulated environment.''

* [http://www.knoppix.net/wiki/Cheat_Codes knoppix '''bootfrom''' cheatcode]

bootfrom=/dev/hda1/KNX.iso

Bootfrom needs access to a running Knoppix-System '''with the same Kernel as the Bootkernel''', before it is able to mount the partition / ISO-Image.

* Maybe there's a way to achieve a boot cd iso selecter see ([http://www.linuxquestions.org/questions/showthread.php?t=528840 this post])
* virtualization-oriented distro... ex [http://unit.aist.go.jp/itri/knoppix/xen/index-en.html Xenoppix] Cons: would work best only on modern VT-capable computers. What about a very minimal distribution (say DSL without X) kqemu-enblaed, booting into an ncurses iso-loader? The delicate part would be on the networking side.

===Zero-Install / Networked systems===

*[http://klik.atekon.de/wiki/index.php/Main_Page Klik] with more than 1000 packages ported. See [http://klik.atekon.de/wiki/index.php/Comparison comparison chart].
*[http://0install.net/index.html The Zero Install System], with automatic dependancies, upgrading. See [http://0install.net/matrix.html] comparison chart.
*Why not rolling a [http://wiki.rpath.com/wiki/Conary:Concepts Conary/RPath-based] distro, based on [http://www.rpath.com/rbuilder/project/rpath/ rPath linux] reference distribution ? Or choose within the available ones...
* The FUSE [http://httpfs.sourceforge.net/net_boot.htm httpfs] may allow pure http net booting, downloading files from a distant livecd only when needed (this image may be very minimal though).
* [http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&start=75&tstart=0 some people] hare thinking of using Amazon S3 as virtual disk

===Using Qemu to boot Linux from a flash drive within Windows===

The following tutorial explains how to use Qemu to boot Linux from a portable USB flash device while still working within Windows. This Enables the user to have both systems running at the same time eliminating the need to restart the PC and set your BIOS options to boot Linux.

[http://www.pendrivelinux.com/2007/03/09/use-qemu-to-boot-linux-from-windows/ PendriveLinux article]

===Putting a vnc server on openmoko===

That's another alternative :)

===The BlackDog project===

"When plugged into the host computer’s USB port, BlackDog starts up, '''automatically launching the X Window system for Windows Xming and a software NAT router via an autorun application that the BlackDog presents to the host through a virtual USB CD-ROM. Once those applications are running, the virtual USB CD-ROM drive disconnects, and a virtual USB-to-Ethernet adapter is connected to provide the communications link.'''

The host machine’s monitor, keyboard, mouse, and Internet connection are used by the BlackDog for the duration of the session."

The [http://en.wikipedia.org/wiki/BlackDog BlackDog server] project has lots of interesting concepts (and software).

==Interesting distros/payloads==

'''System diagnostics / recovery'''
*[http://en.wikipedia.org/wiki/Memtest Memtest86+] Very small
*[http://www.ultimatebootcd.com/ The Ultimate Boot CD]
*[http://www.sysresccd.org/Main_Page System Rescue CD]
*[http://www.tux.org/pub/people/kent-robotti/looplinux/rip/ Recovery IS Possible] < 80 MB

'''General purpose'''
*[http://jinx-linux.sourceforge.net/ jinx-linux] < 50 MB
*[http://knoppix.com/ Knoppix] 700 MB CD-ROM
*[http://www.puppylinux.org PuppyLinux] < 100 MB
*[http://www.damnsmalllinux.org DSL] ~50 MB
*[http://gentoo-wiki.com/TinyGentoo#What_is_this_.3F TinyGentoo] 5 MB

'''Security-oriented''': pentesting, forensics, anonymous webbrowsing
*[http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-test-forensics-recovery/ List of Security distros]
*[http://www.remote-exploit.org/backtrack.html Backtrack Livecd]
*[http://en.wikipedia.org/wiki/Anonym.OS Anonym.OS: browse the web anonymously]
*[http://dban.sourceforge.net/ Darik's Boot and Nuke]: ''dangerous''
*[http://byzgl.sourceforge.net/wiki/index.php/Main_Page Byzantine OS] is a webkiosk appliance (32/48 MB)

The SabayonLinux distro offers a lot of [http://www.sabayonlinux.org/wiki/index.php?title=Sabayon_Linux#Boot_Methods_for_special_ways_of_using_Sabayon boot cheatcodes], such as booting onto GeexBox. We should take a look at the method used.

'''[http://www.livecdlist.com/ List of live distros @ livecdlist.com]''' ''Includes sizes''

'''[http://en.wikipedia.org/wiki/List_of_LiveDistros List of live distros @ wikipedia]'''

'''[http://distrowatch.com/dwres.php?resource=cd List of live distros @ distrowatch]'''

'''[http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB article]'''

[[Category:Ideas]]

Wishlist/LiveUSB distro

2007-06-20T22:51:40Z

Fthiery: /* Putting a vnc server on openmoko */

{{wishlist}}

==Introduction==

NOTE: this page may be renamed: portable destkop

An openmoko device could act as '''the perfect geeky swiss knive''': go anywhere with your Linux desktop and tools.

YES, there are tons of liveCDs out there, but CDs/DVDs:
* get easily damaged
* don't fit in a pocket (except for businesscard-sized ones ; too exotic...)
* don't update or save settings/personal data without a supplementary USB device

YES, there are LiveUSB distros but:
* a Neo/OpenMoko device IS sort of an USB stick; one object is better than two
* i carry my cellphone everywhere, not my USB stick
* there is almost no security on USB sticks (nothing like cryptoloop devices, right?)

The biggest argument against these ones is that you'd have to carry an USB cable with you... But it's a standard one, which is good news :)

Similar functionality can be found in [http://en.wikipedia.org/wiki/Wizpy the Wizpy portable media player]

When the openmoko device is in mass storage mode, a host computer should be able to boot on it, presenting a grub menu offering to boot into several images / partitions (payloads) on the transflash: memtest, UBCD (the ultimate boot cd), a lightweight security oriented livecd distro, you name it...

It's sometimes called Live USB: from [http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB definition], ''A live USB is a USB flash drive containing a full operating system which can be booted from. Live USBs are closely related to Live CDs, and are sometimes used interchangeably. Like Live CDs, Live USBs can be used for system administration, data recovery, or the testing of operating system distributions without committing to a permanent installation on the local hard drive. Many of the smaller Linux distributions can also be used from a USB flash drive.''

Usually, all you need for this with an USB storage device (flash USB stick or external drive) is a partition, flagged "bootable" (see fdisk), containing a boot loader (say grub). But the problem is that specifications vary between motherboards, so there are variants and mandatory requirements to define (here would be a good idea), so that one can optimize/maximize the compatibility.

==The perfect geek knife==

We may want to start our own distro with the following constraints:
* Multiboot: small utilities/compilations + bull blown light OS
* Aiming at maximum hardware compatibility, possibly using liveCD technology (knoppix-based distros)
* Aiming at minimum resource+system footprint, and boot speed
* Being able to install software temporary (for the session) or permanently, upgradable
* Being able to use the phone features at the same time (access pim, data & calls)

Optimally, being able to charge the device while plugged in would be great too.

==About USB Booting==

From [http://www.damnsmalllinux.org/wiki/index.php/USB_Booting DSL Wiki]:

Older computer BIOS usually do not support direct booting from a USB device. Around 2001, PC motherboard manufacturers started to add USB boot support.

There are two common BIOS methods for direct USB booting:
* One method is called the "USBHDD" method and it is used to support the booting of standard USB mass storage devices that are configured like a normal PC hard drive.
* The other method is called the "USBZIP" method and it supports booting from a USB storage device that behaves like the original IOMEGA ZIP drive with USB support.

Most computers (just about all Dells, for example) made today have a BIOS that supports the USBHDD method so I expect that this will eventually become the standard way to boot a USB device. However, many motherboards will support BOTH methods, and many older motherboards have USBZIP support.

Some newer BIOSs which support USB 2.0 will not boot from an older pen drive. Using a USB 2.0 compliant one usually solves this problem. Also, some older BIOSs which only support USB 1.1 will not boot from newer drives which support USB 2.0!

Some USB keys don't boot. If this is the case, it may be possible to fix them by installing a new master boot record. (Most keys boot OK by default; some cannot be fixed even by doing this. However, it helps in some cases). Run the command:

lilo -M /dev/sda

==USB Booting in Neo1973's kernel==

Although there is very little information about it, the g_storage kernel module is responsible for the Neo's mass storage mode. So the boot-or-not criteria might rely on this very module.

==TODO==

Testing/further research has to be conducted to determine:
* if a bootable-flagged partition / grub doesn't mess openmoko up
* if the phone can still act as a phone, or if the booted os can use the gprs functions
* if the phone can recharge on mass storage mode (from the usb cable)
* if a dedicated partition for each payload is needed (see memtest example...)
* if multi-boot is possible: grub?

==Linux distro Howto's: liveUSB creation==

In short:
#Mount USB drive, with e.g. mount /dev/sda1 /flash - can be either FAT16 or FAT32!
#Mount ISO image, with e.g. mount /tmp/dsl-3.2.iso /tmp/iso -o loop
#Copy all contents from ISO to USB drive: cp -vr /tmp/iso/* /flash/
#Rename and move syslinux files to root directory: mv /flash/boot/isolinux/* /flash/
#Rename isolinux.cfg: mv /flash/isolinux.cfg /flash/syslinux.cfg
#Unmount USB drive: umount /flash
#Install syslinux: syslinux /dev/sda1 and eventually set the MBR boot flag for this partition (with fdisk).

*Fedora Core's [http://revisor.fedoraunity.org Revisor] enables easy GUI-driven ISO customization, allowing to choose the output medium (CD/DVD/USB)
*[http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive Installing DSL to USB device - using grub]
*[http://wiki.debian.org/BootUsb Debian BootUSB - using Lilo]
*[http://www.debuntu.org/how-to-install-ubuntu-linux-on-usb-bar Howto install ubuntu (or any other distro) on usb stick]
*[http://www.gentoo.org/doc/en/liveusb.xml Gentoo on usb stick]
*[http://pendrivelinux.com/ Windows & Linux tutorials & resources]:
* Flash installation via Windows: XUbuntu, DSL, Knoppix, Slax, MiniMe
* Flash installation via Linux: Ubuntu Edgy, Knoppix, PCLinuxOS

'''Transflash partitioning schema example'''

* 2 Gb: /dev/sd? (where N is the transflash's number)
* 700 Mb: /dev/sd?1 : containing bootable iso, FAT16
* 1300 Mb: /dev/sd?2 : openmoko & bootable os home partition, EXT3?
* + eventually a swap

The limitation of this method is that the booted OS is static (not-self modifiable). There's the option to install linux using the partition, but it's not really mobility-oriented (liveCDs are optimized for maximum autoconfiguration).

==Alternative methods==

===Booting ISOs ?===

* Grub? From [http://www.freesoftwaremagazine.com/articles/grub_intro#comment-12900 How to install grub on an usb pendrive]:
If you want to boot from a iso file on a harddisk, do something in menu.lst like
title Boot from iso on a harddisk
map (hdX,Y)/your.iso (hdZ)
map --rehook
chainloader (hdZ)+1
rootnoverify (hdZ)
boot

Grub & memdisk can boot floppy images, but (AFAIK) not isos. Example from the stock /boot/grub/menu.lst ubuntu:
title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin
quiet
boot

''Using the memdisk kernel from the syslinux package, you can load disk images and execute them in a non-emulated environment.''

* [http://www.knoppix.net/wiki/Cheat_Codes knoppix '''bootfrom''' cheatcode]

bootfrom=/dev/hda1/KNX.iso

Bootfrom needs access to a running Knoppix-System '''with the same Kernel as the Bootkernel''', before it is able to mount the partition / ISO-Image.

* Maybe there's a way to achieve a boot cd iso selecter see ([http://www.linuxquestions.org/questions/showthread.php?t=528840 this post])
* virtualization-oriented distro... ex [http://unit.aist.go.jp/itri/knoppix/xen/index-en.html Xenoppix] Cons: would work best only on modern VT-capable computers. What about a very minimal distribution (say DSL without X) kqemu-enblaed, booting into an ncurses iso-loader? The delicate part would be on the networking side.

===Zero-Install / Networked systems===

*[http://klik.atekon.de/wiki/index.php/Main_Page Klik] with more than 1000 packages ported. See [http://klik.atekon.de/wiki/index.php/Comparison comparison chart].
*[http://0install.net/index.html The Zero Install System], with automatic dependancies, upgrading. See [http://0install.net/matrix.html] comparison chart.
*Why not rolling a [http://wiki.rpath.com/wiki/Conary:Concepts Conary/RPath-based] distro, based on [http://www.rpath.com/rbuilder/project/rpath/ rPath linux] reference distribution ? Or choose within the available ones...
* The FUSE [http://httpfs.sourceforge.net/net_boot.htm httpfs] may allow pure http net booting, downloading files from a distant livecd only when needed (this image may be very minimal though).
* [http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&start=75&tstart=0 some people] hare thinking of using Amazon S3 as virtual disk

===Using Qemu to boot Linux from a flash drive within Windows===

The following tutorial explains how to use Qemu to boot Linux from a portable USB flash device while still working within Windows. This Enables the user to have both systems running at the same time eliminating the need to restart the PC and set your BIOS options to boot Linux.

[http://www.pendrivelinux.com/2007/03/09/use-qemu-to-boot-linux-from-windows/ PendriveLinux article]

===Putting a vnc server on openmoko===

That's another alternative :)

===The BlackDog project===

"When plugged into the host computer’s USB port, BlackDog starts up, automatically launching the X Window system for Windows Xming and a software NAT router via an autorun application that the BlackDog presents to the host through a virtual USB CD-ROM. Once those applications are running, the virtual USB CD-ROM drive disconnects, and a virtual USB-to-Ethernet adapter is connected to provide the communications link.

The host machine’s monitor, keyboard, mouse, and Internet connection are used by the BlackDog for the duration of the session."

The [http://en.wikipedia.org/wiki/BlackDog BlackDog server] project has lots of interesting concepts (and software).

==Interesting distros/payloads==

'''System diagnostics / recovery'''
*[http://en.wikipedia.org/wiki/Memtest Memtest86+] Very small
*[http://www.ultimatebootcd.com/ The Ultimate Boot CD]
*[http://www.sysresccd.org/Main_Page System Rescue CD]
*[http://www.tux.org/pub/people/kent-robotti/looplinux/rip/ Recovery IS Possible] < 80 MB

'''General purpose'''
*[http://jinx-linux.sourceforge.net/ jinx-linux] < 50 MB
*[http://knoppix.com/ Knoppix] 700 MB CD-ROM
*[http://www.puppylinux.org PuppyLinux] < 100 MB
*[http://www.damnsmalllinux.org DSL] ~50 MB
*[http://gentoo-wiki.com/TinyGentoo#What_is_this_.3F TinyGentoo] 5 MB

'''Security-oriented''': pentesting, forensics, anonymous webbrowsing
*[http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-test-forensics-recovery/ List of Security distros]
*[http://www.remote-exploit.org/backtrack.html Backtrack Livecd]
*[http://en.wikipedia.org/wiki/Anonym.OS Anonym.OS: browse the web anonymously]
*[http://dban.sourceforge.net/ Darik's Boot and Nuke]: ''dangerous''
*[http://byzgl.sourceforge.net/wiki/index.php/Main_Page Byzantine OS] is a webkiosk appliance (32/48 MB)

The SabayonLinux distro offers a lot of [http://www.sabayonlinux.org/wiki/index.php?title=Sabayon_Linux#Boot_Methods_for_special_ways_of_using_Sabayon boot cheatcodes], such as booting onto GeexBox. We should take a look at the method used.

'''[http://www.livecdlist.com/ List of live distros @ livecdlist.com]''' ''Includes sizes''

'''[http://en.wikipedia.org/wiki/List_of_LiveDistros List of live distros @ wikipedia]'''

'''[http://distrowatch.com/dwres.php?resource=cd List of live distros @ distrowatch]'''

'''[http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB article]'''

[[Category:Ideas]]

Wishlist/LiveUSB distro

2007-06-20T22:51:23Z

Fthiery: /* The BlackDog project */

{{wishlist}}

==Introduction==

NOTE: this page may be renamed: portable destkop

An openmoko device could act as '''the perfect geeky swiss knive''': go anywhere with your Linux desktop and tools.

YES, there are tons of liveCDs out there, but CDs/DVDs:
* get easily damaged
* don't fit in a pocket (except for businesscard-sized ones ; too exotic...)
* don't update or save settings/personal data without a supplementary USB device

YES, there are LiveUSB distros but:
* a Neo/OpenMoko device IS sort of an USB stick; one object is better than two
* i carry my cellphone everywhere, not my USB stick
* there is almost no security on USB sticks (nothing like cryptoloop devices, right?)

The biggest argument against these ones is that you'd have to carry an USB cable with you... But it's a standard one, which is good news :)

Similar functionality can be found in [http://en.wikipedia.org/wiki/Wizpy the Wizpy portable media player]

When the openmoko device is in mass storage mode, a host computer should be able to boot on it, presenting a grub menu offering to boot into several images / partitions (payloads) on the transflash: memtest, UBCD (the ultimate boot cd), a lightweight security oriented livecd distro, you name it...

It's sometimes called Live USB: from [http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB definition], ''A live USB is a USB flash drive containing a full operating system which can be booted from. Live USBs are closely related to Live CDs, and are sometimes used interchangeably. Like Live CDs, Live USBs can be used for system administration, data recovery, or the testing of operating system distributions without committing to a permanent installation on the local hard drive. Many of the smaller Linux distributions can also be used from a USB flash drive.''

Usually, all you need for this with an USB storage device (flash USB stick or external drive) is a partition, flagged "bootable" (see fdisk), containing a boot loader (say grub). But the problem is that specifications vary between motherboards, so there are variants and mandatory requirements to define (here would be a good idea), so that one can optimize/maximize the compatibility.

==The perfect geek knife==

We may want to start our own distro with the following constraints:
* Multiboot: small utilities/compilations + bull blown light OS
* Aiming at maximum hardware compatibility, possibly using liveCD technology (knoppix-based distros)
* Aiming at minimum resource+system footprint, and boot speed
* Being able to install software temporary (for the session) or permanently, upgradable
* Being able to use the phone features at the same time (access pim, data & calls)

Optimally, being able to charge the device while plugged in would be great too.

==About USB Booting==

From [http://www.damnsmalllinux.org/wiki/index.php/USB_Booting DSL Wiki]:

Older computer BIOS usually do not support direct booting from a USB device. Around 2001, PC motherboard manufacturers started to add USB boot support.

There are two common BIOS methods for direct USB booting:
* One method is called the "USBHDD" method and it is used to support the booting of standard USB mass storage devices that are configured like a normal PC hard drive.
* The other method is called the "USBZIP" method and it supports booting from a USB storage device that behaves like the original IOMEGA ZIP drive with USB support.

Most computers (just about all Dells, for example) made today have a BIOS that supports the USBHDD method so I expect that this will eventually become the standard way to boot a USB device. However, many motherboards will support BOTH methods, and many older motherboards have USBZIP support.

Some newer BIOSs which support USB 2.0 will not boot from an older pen drive. Using a USB 2.0 compliant one usually solves this problem. Also, some older BIOSs which only support USB 1.1 will not boot from newer drives which support USB 2.0!

Some USB keys don't boot. If this is the case, it may be possible to fix them by installing a new master boot record. (Most keys boot OK by default; some cannot be fixed even by doing this. However, it helps in some cases). Run the command:

lilo -M /dev/sda

==USB Booting in Neo1973's kernel==

Although there is very little information about it, the g_storage kernel module is responsible for the Neo's mass storage mode. So the boot-or-not criteria might rely on this very module.

==TODO==

Testing/further research has to be conducted to determine:
* if a bootable-flagged partition / grub doesn't mess openmoko up
* if the phone can still act as a phone, or if the booted os can use the gprs functions
* if the phone can recharge on mass storage mode (from the usb cable)
* if a dedicated partition for each payload is needed (see memtest example...)
* if multi-boot is possible: grub?

==Linux distro Howto's: liveUSB creation==

In short:
#Mount USB drive, with e.g. mount /dev/sda1 /flash - can be either FAT16 or FAT32!
#Mount ISO image, with e.g. mount /tmp/dsl-3.2.iso /tmp/iso -o loop
#Copy all contents from ISO to USB drive: cp -vr /tmp/iso/* /flash/
#Rename and move syslinux files to root directory: mv /flash/boot/isolinux/* /flash/
#Rename isolinux.cfg: mv /flash/isolinux.cfg /flash/syslinux.cfg
#Unmount USB drive: umount /flash
#Install syslinux: syslinux /dev/sda1 and eventually set the MBR boot flag for this partition (with fdisk).

*Fedora Core's [http://revisor.fedoraunity.org Revisor] enables easy GUI-driven ISO customization, allowing to choose the output medium (CD/DVD/USB)
*[http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive Installing DSL to USB device - using grub]
*[http://wiki.debian.org/BootUsb Debian BootUSB - using Lilo]
*[http://www.debuntu.org/how-to-install-ubuntu-linux-on-usb-bar Howto install ubuntu (or any other distro) on usb stick]
*[http://www.gentoo.org/doc/en/liveusb.xml Gentoo on usb stick]
*[http://pendrivelinux.com/ Windows & Linux tutorials & resources]:
* Flash installation via Windows: XUbuntu, DSL, Knoppix, Slax, MiniMe
* Flash installation via Linux: Ubuntu Edgy, Knoppix, PCLinuxOS

'''Transflash partitioning schema example'''

* 2 Gb: /dev/sd? (where N is the transflash's number)
* 700 Mb: /dev/sd?1 : containing bootable iso, FAT16
* 1300 Mb: /dev/sd?2 : openmoko & bootable os home partition, EXT3?
* + eventually a swap

The limitation of this method is that the booted OS is static (not-self modifiable). There's the option to install linux using the partition, but it's not really mobility-oriented (liveCDs are optimized for maximum autoconfiguration).

==Alternative methods==

===Booting ISOs ?===

* Grub? From [http://www.freesoftwaremagazine.com/articles/grub_intro#comment-12900 How to install grub on an usb pendrive]:
If you want to boot from a iso file on a harddisk, do something in menu.lst like
title Boot from iso on a harddisk
map (hdX,Y)/your.iso (hdZ)
map --rehook
chainloader (hdZ)+1
rootnoverify (hdZ)
boot

Grub & memdisk can boot floppy images, but (AFAIK) not isos. Example from the stock /boot/grub/menu.lst ubuntu:
title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin
quiet
boot

''Using the memdisk kernel from the syslinux package, you can load disk images and execute them in a non-emulated environment.''

* [http://www.knoppix.net/wiki/Cheat_Codes knoppix '''bootfrom''' cheatcode]

bootfrom=/dev/hda1/KNX.iso

Bootfrom needs access to a running Knoppix-System '''with the same Kernel as the Bootkernel''', before it is able to mount the partition / ISO-Image.

* Maybe there's a way to achieve a boot cd iso selecter see ([http://www.linuxquestions.org/questions/showthread.php?t=528840 this post])
* virtualization-oriented distro... ex [http://unit.aist.go.jp/itri/knoppix/xen/index-en.html Xenoppix] Cons: would work best only on modern VT-capable computers. What about a very minimal distribution (say DSL without X) kqemu-enblaed, booting into an ncurses iso-loader? The delicate part would be on the networking side.

===Zero-Install / Networked systems===

*[http://klik.atekon.de/wiki/index.php/Main_Page Klik] with more than 1000 packages ported. See [http://klik.atekon.de/wiki/index.php/Comparison comparison chart].
*[http://0install.net/index.html The Zero Install System], with automatic dependancies, upgrading. See [http://0install.net/matrix.html] comparison chart.
*Why not rolling a [http://wiki.rpath.com/wiki/Conary:Concepts Conary/RPath-based] distro, based on [http://www.rpath.com/rbuilder/project/rpath/ rPath linux] reference distribution ? Or choose within the available ones...
* The FUSE [http://httpfs.sourceforge.net/net_boot.htm httpfs] may allow pure http net booting, downloading files from a distant livecd only when needed (this image may be very minimal though).
* [http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&start=75&tstart=0 some people] hare thinking of using Amazon S3 as virtual disk

===Using Qemu to boot Linux from a flash drive within Windows===

The following tutorial explains how to use Qemu to boot Linux from a portable USB flash device while still working within Windows. This Enables the user to have both systems running at the same time eliminating the need to restart the PC and set your BIOS options to boot Linux.

[http://www.pendrivelinux.com/2007/03/09/use-qemu-to-boot-linux-from-windows/ PendriveLinux article]

==Putting a vnc server on openmoko==

That's another alternative :)

===The BlackDog project===

"When plugged into the host computer’s USB port, BlackDog starts up, automatically launching the X Window system for Windows Xming and a software NAT router via an autorun application that the BlackDog presents to the host through a virtual USB CD-ROM. Once those applications are running, the virtual USB CD-ROM drive disconnects, and a virtual USB-to-Ethernet adapter is connected to provide the communications link.

The host machine’s monitor, keyboard, mouse, and Internet connection are used by the BlackDog for the duration of the session."

The [http://en.wikipedia.org/wiki/BlackDog BlackDog server] project has lots of interesting concepts (and software).

==Interesting distros/payloads==

'''System diagnostics / recovery'''
*[http://en.wikipedia.org/wiki/Memtest Memtest86+] Very small
*[http://www.ultimatebootcd.com/ The Ultimate Boot CD]
*[http://www.sysresccd.org/Main_Page System Rescue CD]
*[http://www.tux.org/pub/people/kent-robotti/looplinux/rip/ Recovery IS Possible] < 80 MB

'''General purpose'''
*[http://jinx-linux.sourceforge.net/ jinx-linux] < 50 MB
*[http://knoppix.com/ Knoppix] 700 MB CD-ROM
*[http://www.puppylinux.org PuppyLinux] < 100 MB
*[http://www.damnsmalllinux.org DSL] ~50 MB
*[http://gentoo-wiki.com/TinyGentoo#What_is_this_.3F TinyGentoo] 5 MB

'''Security-oriented''': pentesting, forensics, anonymous webbrowsing
*[http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-test-forensics-recovery/ List of Security distros]
*[http://www.remote-exploit.org/backtrack.html Backtrack Livecd]
*[http://en.wikipedia.org/wiki/Anonym.OS Anonym.OS: browse the web anonymously]
*[http://dban.sourceforge.net/ Darik's Boot and Nuke]: ''dangerous''
*[http://byzgl.sourceforge.net/wiki/index.php/Main_Page Byzantine OS] is a webkiosk appliance (32/48 MB)

The SabayonLinux distro offers a lot of [http://www.sabayonlinux.org/wiki/index.php?title=Sabayon_Linux#Boot_Methods_for_special_ways_of_using_Sabayon boot cheatcodes], such as booting onto GeexBox. We should take a look at the method used.

'''[http://www.livecdlist.com/ List of live distros @ livecdlist.com]''' ''Includes sizes''

'''[http://en.wikipedia.org/wiki/List_of_LiveDistros List of live distros @ wikipedia]'''

'''[http://distrowatch.com/dwres.php?resource=cd List of live distros @ distrowatch]'''

'''[http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB article]'''

[[Category:Ideas]]

Wishlist/LiveUSB distro

2007-06-20T22:51:15Z

Fthiery: /* Using Qemu to boot Linux from a flash drive within Windows */

{{wishlist}}

==Introduction==

NOTE: this page may be renamed: portable destkop

An openmoko device could act as '''the perfect geeky swiss knive''': go anywhere with your Linux desktop and tools.

YES, there are tons of liveCDs out there, but CDs/DVDs:
* get easily damaged
* don't fit in a pocket (except for businesscard-sized ones ; too exotic...)
* don't update or save settings/personal data without a supplementary USB device

YES, there are LiveUSB distros but:
* a Neo/OpenMoko device IS sort of an USB stick; one object is better than two
* i carry my cellphone everywhere, not my USB stick
* there is almost no security on USB sticks (nothing like cryptoloop devices, right?)

The biggest argument against these ones is that you'd have to carry an USB cable with you... But it's a standard one, which is good news :)

Similar functionality can be found in [http://en.wikipedia.org/wiki/Wizpy the Wizpy portable media player]

When the openmoko device is in mass storage mode, a host computer should be able to boot on it, presenting a grub menu offering to boot into several images / partitions (payloads) on the transflash: memtest, UBCD (the ultimate boot cd), a lightweight security oriented livecd distro, you name it...

It's sometimes called Live USB: from [http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB definition], ''A live USB is a USB flash drive containing a full operating system which can be booted from. Live USBs are closely related to Live CDs, and are sometimes used interchangeably. Like Live CDs, Live USBs can be used for system administration, data recovery, or the testing of operating system distributions without committing to a permanent installation on the local hard drive. Many of the smaller Linux distributions can also be used from a USB flash drive.''

Usually, all you need for this with an USB storage device (flash USB stick or external drive) is a partition, flagged "bootable" (see fdisk), containing a boot loader (say grub). But the problem is that specifications vary between motherboards, so there are variants and mandatory requirements to define (here would be a good idea), so that one can optimize/maximize the compatibility.

==The perfect geek knife==

We may want to start our own distro with the following constraints:
* Multiboot: small utilities/compilations + bull blown light OS
* Aiming at maximum hardware compatibility, possibly using liveCD technology (knoppix-based distros)
* Aiming at minimum resource+system footprint, and boot speed
* Being able to install software temporary (for the session) or permanently, upgradable
* Being able to use the phone features at the same time (access pim, data & calls)

Optimally, being able to charge the device while plugged in would be great too.

==About USB Booting==

From [http://www.damnsmalllinux.org/wiki/index.php/USB_Booting DSL Wiki]:

Older computer BIOS usually do not support direct booting from a USB device. Around 2001, PC motherboard manufacturers started to add USB boot support.

There are two common BIOS methods for direct USB booting:
* One method is called the "USBHDD" method and it is used to support the booting of standard USB mass storage devices that are configured like a normal PC hard drive.
* The other method is called the "USBZIP" method and it supports booting from a USB storage device that behaves like the original IOMEGA ZIP drive with USB support.

Most computers (just about all Dells, for example) made today have a BIOS that supports the USBHDD method so I expect that this will eventually become the standard way to boot a USB device. However, many motherboards will support BOTH methods, and many older motherboards have USBZIP support.

Some newer BIOSs which support USB 2.0 will not boot from an older pen drive. Using a USB 2.0 compliant one usually solves this problem. Also, some older BIOSs which only support USB 1.1 will not boot from newer drives which support USB 2.0!

Some USB keys don't boot. If this is the case, it may be possible to fix them by installing a new master boot record. (Most keys boot OK by default; some cannot be fixed even by doing this. However, it helps in some cases). Run the command:

lilo -M /dev/sda

==USB Booting in Neo1973's kernel==

Although there is very little information about it, the g_storage kernel module is responsible for the Neo's mass storage mode. So the boot-or-not criteria might rely on this very module.

==TODO==

Testing/further research has to be conducted to determine:
* if a bootable-flagged partition / grub doesn't mess openmoko up
* if the phone can still act as a phone, or if the booted os can use the gprs functions
* if the phone can recharge on mass storage mode (from the usb cable)
* if a dedicated partition for each payload is needed (see memtest example...)
* if multi-boot is possible: grub?

==Linux distro Howto's: liveUSB creation==

In short:
#Mount USB drive, with e.g. mount /dev/sda1 /flash - can be either FAT16 or FAT32!
#Mount ISO image, with e.g. mount /tmp/dsl-3.2.iso /tmp/iso -o loop
#Copy all contents from ISO to USB drive: cp -vr /tmp/iso/* /flash/
#Rename and move syslinux files to root directory: mv /flash/boot/isolinux/* /flash/
#Rename isolinux.cfg: mv /flash/isolinux.cfg /flash/syslinux.cfg
#Unmount USB drive: umount /flash
#Install syslinux: syslinux /dev/sda1 and eventually set the MBR boot flag for this partition (with fdisk).

*Fedora Core's [http://revisor.fedoraunity.org Revisor] enables easy GUI-driven ISO customization, allowing to choose the output medium (CD/DVD/USB)
*[http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive Installing DSL to USB device - using grub]
*[http://wiki.debian.org/BootUsb Debian BootUSB - using Lilo]
*[http://www.debuntu.org/how-to-install-ubuntu-linux-on-usb-bar Howto install ubuntu (or any other distro) on usb stick]
*[http://www.gentoo.org/doc/en/liveusb.xml Gentoo on usb stick]
*[http://pendrivelinux.com/ Windows & Linux tutorials & resources]:
* Flash installation via Windows: XUbuntu, DSL, Knoppix, Slax, MiniMe
* Flash installation via Linux: Ubuntu Edgy, Knoppix, PCLinuxOS

'''Transflash partitioning schema example'''

* 2 Gb: /dev/sd? (where N is the transflash's number)
* 700 Mb: /dev/sd?1 : containing bootable iso, FAT16
* 1300 Mb: /dev/sd?2 : openmoko & bootable os home partition, EXT3?
* + eventually a swap

The limitation of this method is that the booted OS is static (not-self modifiable). There's the option to install linux using the partition, but it's not really mobility-oriented (liveCDs are optimized for maximum autoconfiguration).

==Alternative methods==

===Booting ISOs ?===

* Grub? From [http://www.freesoftwaremagazine.com/articles/grub_intro#comment-12900 How to install grub on an usb pendrive]:
If you want to boot from a iso file on a harddisk, do something in menu.lst like
title Boot from iso on a harddisk
map (hdX,Y)/your.iso (hdZ)
map --rehook
chainloader (hdZ)+1
rootnoverify (hdZ)
boot

Grub & memdisk can boot floppy images, but (AFAIK) not isos. Example from the stock /boot/grub/menu.lst ubuntu:
title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin
quiet
boot

''Using the memdisk kernel from the syslinux package, you can load disk images and execute them in a non-emulated environment.''

* [http://www.knoppix.net/wiki/Cheat_Codes knoppix '''bootfrom''' cheatcode]

bootfrom=/dev/hda1/KNX.iso

Bootfrom needs access to a running Knoppix-System '''with the same Kernel as the Bootkernel''', before it is able to mount the partition / ISO-Image.

* Maybe there's a way to achieve a boot cd iso selecter see ([http://www.linuxquestions.org/questions/showthread.php?t=528840 this post])
* virtualization-oriented distro... ex [http://unit.aist.go.jp/itri/knoppix/xen/index-en.html Xenoppix] Cons: would work best only on modern VT-capable computers. What about a very minimal distribution (say DSL without X) kqemu-enblaed, booting into an ncurses iso-loader? The delicate part would be on the networking side.

===Zero-Install / Networked systems===

*[http://klik.atekon.de/wiki/index.php/Main_Page Klik] with more than 1000 packages ported. See [http://klik.atekon.de/wiki/index.php/Comparison comparison chart].
*[http://0install.net/index.html The Zero Install System], with automatic dependancies, upgrading. See [http://0install.net/matrix.html] comparison chart.
*Why not rolling a [http://wiki.rpath.com/wiki/Conary:Concepts Conary/RPath-based] distro, based on [http://www.rpath.com/rbuilder/project/rpath/ rPath linux] reference distribution ? Or choose within the available ones...
* The FUSE [http://httpfs.sourceforge.net/net_boot.htm httpfs] may allow pure http net booting, downloading files from a distant livecd only when needed (this image may be very minimal though).
* [http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&start=75&tstart=0 some people] hare thinking of using Amazon S3 as virtual disk

===Using Qemu to boot Linux from a flash drive within Windows===

The following tutorial explains how to use Qemu to boot Linux from a portable USB flash device while still working within Windows. This Enables the user to have both systems running at the same time eliminating the need to restart the PC and set your BIOS options to boot Linux.

[http://www.pendrivelinux.com/2007/03/09/use-qemu-to-boot-linux-from-windows/ PendriveLinux article]

==Putting a vnc server on openmoko==

That's another alternative :)

==The BlackDog project==

"When plugged into the host computer’s USB port, BlackDog starts up, automatically launching the X Window system for Windows Xming and a software NAT router via an autorun application that the BlackDog presents to the host through a virtual USB CD-ROM. Once those applications are running, the virtual USB CD-ROM drive disconnects, and a virtual USB-to-Ethernet adapter is connected to provide the communications link.

The host machine’s monitor, keyboard, mouse, and Internet connection are used by the BlackDog for the duration of the session."

The [http://en.wikipedia.org/wiki/BlackDog BlackDog server] project has lots of interesting concepts (and software).

==Interesting distros/payloads==

'''System diagnostics / recovery'''
*[http://en.wikipedia.org/wiki/Memtest Memtest86+] Very small
*[http://www.ultimatebootcd.com/ The Ultimate Boot CD]
*[http://www.sysresccd.org/Main_Page System Rescue CD]
*[http://www.tux.org/pub/people/kent-robotti/looplinux/rip/ Recovery IS Possible] < 80 MB

'''General purpose'''
*[http://jinx-linux.sourceforge.net/ jinx-linux] < 50 MB
*[http://knoppix.com/ Knoppix] 700 MB CD-ROM
*[http://www.puppylinux.org PuppyLinux] < 100 MB
*[http://www.damnsmalllinux.org DSL] ~50 MB
*[http://gentoo-wiki.com/TinyGentoo#What_is_this_.3F TinyGentoo] 5 MB

'''Security-oriented''': pentesting, forensics, anonymous webbrowsing
*[http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-test-forensics-recovery/ List of Security distros]
*[http://www.remote-exploit.org/backtrack.html Backtrack Livecd]
*[http://en.wikipedia.org/wiki/Anonym.OS Anonym.OS: browse the web anonymously]
*[http://dban.sourceforge.net/ Darik's Boot and Nuke]: ''dangerous''
*[http://byzgl.sourceforge.net/wiki/index.php/Main_Page Byzantine OS] is a webkiosk appliance (32/48 MB)

The SabayonLinux distro offers a lot of [http://www.sabayonlinux.org/wiki/index.php?title=Sabayon_Linux#Boot_Methods_for_special_ways_of_using_Sabayon boot cheatcodes], such as booting onto GeexBox. We should take a look at the method used.

'''[http://www.livecdlist.com/ List of live distros @ livecdlist.com]''' ''Includes sizes''

'''[http://en.wikipedia.org/wiki/List_of_LiveDistros List of live distros @ wikipedia]'''

'''[http://distrowatch.com/dwres.php?resource=cd List of live distros @ distrowatch]'''

'''[http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB article]'''

[[Category:Ideas]]

Wishlist/LiveUSB distro

2007-06-20T22:50:25Z

Fthiery: /* Zero-Install / Networked systems */

{{wishlist}}

==Introduction==

NOTE: this page may be renamed: portable destkop

An openmoko device could act as '''the perfect geeky swiss knive''': go anywhere with your Linux desktop and tools.

YES, there are tons of liveCDs out there, but CDs/DVDs:
* get easily damaged
* don't fit in a pocket (except for businesscard-sized ones ; too exotic...)
* don't update or save settings/personal data without a supplementary USB device

YES, there are LiveUSB distros but:
* a Neo/OpenMoko device IS sort of an USB stick; one object is better than two
* i carry my cellphone everywhere, not my USB stick
* there is almost no security on USB sticks (nothing like cryptoloop devices, right?)

The biggest argument against these ones is that you'd have to carry an USB cable with you... But it's a standard one, which is good news :)

Similar functionality can be found in [http://en.wikipedia.org/wiki/Wizpy the Wizpy portable media player]

When the openmoko device is in mass storage mode, a host computer should be able to boot on it, presenting a grub menu offering to boot into several images / partitions (payloads) on the transflash: memtest, UBCD (the ultimate boot cd), a lightweight security oriented livecd distro, you name it...

It's sometimes called Live USB: from [http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB definition], ''A live USB is a USB flash drive containing a full operating system which can be booted from. Live USBs are closely related to Live CDs, and are sometimes used interchangeably. Like Live CDs, Live USBs can be used for system administration, data recovery, or the testing of operating system distributions without committing to a permanent installation on the local hard drive. Many of the smaller Linux distributions can also be used from a USB flash drive.''

Usually, all you need for this with an USB storage device (flash USB stick or external drive) is a partition, flagged "bootable" (see fdisk), containing a boot loader (say grub). But the problem is that specifications vary between motherboards, so there are variants and mandatory requirements to define (here would be a good idea), so that one can optimize/maximize the compatibility.

==The perfect geek knife==

We may want to start our own distro with the following constraints:
* Multiboot: small utilities/compilations + bull blown light OS
* Aiming at maximum hardware compatibility, possibly using liveCD technology (knoppix-based distros)
* Aiming at minimum resource+system footprint, and boot speed
* Being able to install software temporary (for the session) or permanently, upgradable
* Being able to use the phone features at the same time (access pim, data & calls)

Optimally, being able to charge the device while plugged in would be great too.

==About USB Booting==

From [http://www.damnsmalllinux.org/wiki/index.php/USB_Booting DSL Wiki]:

Older computer BIOS usually do not support direct booting from a USB device. Around 2001, PC motherboard manufacturers started to add USB boot support.

There are two common BIOS methods for direct USB booting:
* One method is called the "USBHDD" method and it is used to support the booting of standard USB mass storage devices that are configured like a normal PC hard drive.
* The other method is called the "USBZIP" method and it supports booting from a USB storage device that behaves like the original IOMEGA ZIP drive with USB support.

Most computers (just about all Dells, for example) made today have a BIOS that supports the USBHDD method so I expect that this will eventually become the standard way to boot a USB device. However, many motherboards will support BOTH methods, and many older motherboards have USBZIP support.

Some newer BIOSs which support USB 2.0 will not boot from an older pen drive. Using a USB 2.0 compliant one usually solves this problem. Also, some older BIOSs which only support USB 1.1 will not boot from newer drives which support USB 2.0!

Some USB keys don't boot. If this is the case, it may be possible to fix them by installing a new master boot record. (Most keys boot OK by default; some cannot be fixed even by doing this. However, it helps in some cases). Run the command:

lilo -M /dev/sda

==USB Booting in Neo1973's kernel==

Although there is very little information about it, the g_storage kernel module is responsible for the Neo's mass storage mode. So the boot-or-not criteria might rely on this very module.

==TODO==

Testing/further research has to be conducted to determine:
* if a bootable-flagged partition / grub doesn't mess openmoko up
* if the phone can still act as a phone, or if the booted os can use the gprs functions
* if the phone can recharge on mass storage mode (from the usb cable)
* if a dedicated partition for each payload is needed (see memtest example...)
* if multi-boot is possible: grub?

==Linux distro Howto's: liveUSB creation==

In short:
#Mount USB drive, with e.g. mount /dev/sda1 /flash - can be either FAT16 or FAT32!
#Mount ISO image, with e.g. mount /tmp/dsl-3.2.iso /tmp/iso -o loop
#Copy all contents from ISO to USB drive: cp -vr /tmp/iso/* /flash/
#Rename and move syslinux files to root directory: mv /flash/boot/isolinux/* /flash/
#Rename isolinux.cfg: mv /flash/isolinux.cfg /flash/syslinux.cfg
#Unmount USB drive: umount /flash
#Install syslinux: syslinux /dev/sda1 and eventually set the MBR boot flag for this partition (with fdisk).

*Fedora Core's [http://revisor.fedoraunity.org Revisor] enables easy GUI-driven ISO customization, allowing to choose the output medium (CD/DVD/USB)
*[http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive Installing DSL to USB device - using grub]
*[http://wiki.debian.org/BootUsb Debian BootUSB - using Lilo]
*[http://www.debuntu.org/how-to-install-ubuntu-linux-on-usb-bar Howto install ubuntu (or any other distro) on usb stick]
*[http://www.gentoo.org/doc/en/liveusb.xml Gentoo on usb stick]
*[http://pendrivelinux.com/ Windows & Linux tutorials & resources]:
* Flash installation via Windows: XUbuntu, DSL, Knoppix, Slax, MiniMe
* Flash installation via Linux: Ubuntu Edgy, Knoppix, PCLinuxOS

'''Transflash partitioning schema example'''

* 2 Gb: /dev/sd? (where N is the transflash's number)
* 700 Mb: /dev/sd?1 : containing bootable iso, FAT16
* 1300 Mb: /dev/sd?2 : openmoko & bootable os home partition, EXT3?
* + eventually a swap

The limitation of this method is that the booted OS is static (not-self modifiable). There's the option to install linux using the partition, but it's not really mobility-oriented (liveCDs are optimized for maximum autoconfiguration).

==Alternative methods==

===Booting ISOs ?===

* Grub? From [http://www.freesoftwaremagazine.com/articles/grub_intro#comment-12900 How to install grub on an usb pendrive]:
If you want to boot from a iso file on a harddisk, do something in menu.lst like
title Boot from iso on a harddisk
map (hdX,Y)/your.iso (hdZ)
map --rehook
chainloader (hdZ)+1
rootnoverify (hdZ)
boot

Grub & memdisk can boot floppy images, but (AFAIK) not isos. Example from the stock /boot/grub/menu.lst ubuntu:
title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin
quiet
boot

''Using the memdisk kernel from the syslinux package, you can load disk images and execute them in a non-emulated environment.''

* [http://www.knoppix.net/wiki/Cheat_Codes knoppix '''bootfrom''' cheatcode]

bootfrom=/dev/hda1/KNX.iso

Bootfrom needs access to a running Knoppix-System '''with the same Kernel as the Bootkernel''', before it is able to mount the partition / ISO-Image.

* Maybe there's a way to achieve a boot cd iso selecter see ([http://www.linuxquestions.org/questions/showthread.php?t=528840 this post])
* virtualization-oriented distro... ex [http://unit.aist.go.jp/itri/knoppix/xen/index-en.html Xenoppix] Cons: would work best only on modern VT-capable computers. What about a very minimal distribution (say DSL without X) kqemu-enblaed, booting into an ncurses iso-loader? The delicate part would be on the networking side.

===Zero-Install / Networked systems===

*[http://klik.atekon.de/wiki/index.php/Main_Page Klik] with more than 1000 packages ported. See [http://klik.atekon.de/wiki/index.php/Comparison comparison chart].
*[http://0install.net/index.html The Zero Install System], with automatic dependancies, upgrading. See [http://0install.net/matrix.html] comparison chart.
*Why not rolling a [http://wiki.rpath.com/wiki/Conary:Concepts Conary/RPath-based] distro, based on [http://www.rpath.com/rbuilder/project/rpath/ rPath linux] reference distribution ? Or choose within the available ones...
* The FUSE [http://httpfs.sourceforge.net/net_boot.htm httpfs] may allow pure http net booting, downloading files from a distant livecd only when needed (this image may be very minimal though).
* [http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&start=75&tstart=0 some people] hare thinking of using Amazon S3 as virtual disk

==Using Qemu to boot Linux from a flash drive within Windows==

The following tutorial explains how to use Qemu to boot Linux from a portable USB flash device while still working within Windows. This Enables the user to have both systems running at the same time eliminating the need to restart the PC and set your BIOS options to boot Linux.

[http://www.pendrivelinux.com/2007/03/09/use-qemu-to-boot-linux-from-windows/ PendriveLinux article]

==Putting a vnc server on openmoko==

That's another alternative :)

==The BlackDog project==

"When plugged into the host computer’s USB port, BlackDog starts up, automatically launching the X Window system for Windows Xming and a software NAT router via an autorun application that the BlackDog presents to the host through a virtual USB CD-ROM. Once those applications are running, the virtual USB CD-ROM drive disconnects, and a virtual USB-to-Ethernet adapter is connected to provide the communications link.

The host machine’s monitor, keyboard, mouse, and Internet connection are used by the BlackDog for the duration of the session."

The [http://en.wikipedia.org/wiki/BlackDog BlackDog server] project has lots of interesting concepts (and software).

==Interesting distros/payloads==

'''System diagnostics / recovery'''
*[http://en.wikipedia.org/wiki/Memtest Memtest86+] Very small
*[http://www.ultimatebootcd.com/ The Ultimate Boot CD]
*[http://www.sysresccd.org/Main_Page System Rescue CD]
*[http://www.tux.org/pub/people/kent-robotti/looplinux/rip/ Recovery IS Possible] < 80 MB

'''General purpose'''
*[http://jinx-linux.sourceforge.net/ jinx-linux] < 50 MB
*[http://knoppix.com/ Knoppix] 700 MB CD-ROM
*[http://www.puppylinux.org PuppyLinux] < 100 MB
*[http://www.damnsmalllinux.org DSL] ~50 MB
*[http://gentoo-wiki.com/TinyGentoo#What_is_this_.3F TinyGentoo] 5 MB

'''Security-oriented''': pentesting, forensics, anonymous webbrowsing
*[http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-test-forensics-recovery/ List of Security distros]
*[http://www.remote-exploit.org/backtrack.html Backtrack Livecd]
*[http://en.wikipedia.org/wiki/Anonym.OS Anonym.OS: browse the web anonymously]
*[http://dban.sourceforge.net/ Darik's Boot and Nuke]: ''dangerous''
*[http://byzgl.sourceforge.net/wiki/index.php/Main_Page Byzantine OS] is a webkiosk appliance (32/48 MB)

The SabayonLinux distro offers a lot of [http://www.sabayonlinux.org/wiki/index.php?title=Sabayon_Linux#Boot_Methods_for_special_ways_of_using_Sabayon boot cheatcodes], such as booting onto GeexBox. We should take a look at the method used.

'''[http://www.livecdlist.com/ List of live distros @ livecdlist.com]''' ''Includes sizes''

'''[http://en.wikipedia.org/wiki/List_of_LiveDistros List of live distros @ wikipedia]'''

'''[http://distrowatch.com/dwres.php?resource=cd List of live distros @ distrowatch]'''

'''[http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB article]'''

[[Category:Ideas]]

Wishlist/LiveUSB distro

2007-06-20T22:49:43Z

Fthiery: /* Zero-Install systems */

{{wishlist}}

==Introduction==

NOTE: this page may be renamed: portable destkop

An openmoko device could act as '''the perfect geeky swiss knive''': go anywhere with your Linux desktop and tools.

YES, there are tons of liveCDs out there, but CDs/DVDs:
* get easily damaged
* don't fit in a pocket (except for businesscard-sized ones ; too exotic...)
* don't update or save settings/personal data without a supplementary USB device

YES, there are LiveUSB distros but:
* a Neo/OpenMoko device IS sort of an USB stick; one object is better than two
* i carry my cellphone everywhere, not my USB stick
* there is almost no security on USB sticks (nothing like cryptoloop devices, right?)

The biggest argument against these ones is that you'd have to carry an USB cable with you... But it's a standard one, which is good news :)

Similar functionality can be found in [http://en.wikipedia.org/wiki/Wizpy the Wizpy portable media player]

When the openmoko device is in mass storage mode, a host computer should be able to boot on it, presenting a grub menu offering to boot into several images / partitions (payloads) on the transflash: memtest, UBCD (the ultimate boot cd), a lightweight security oriented livecd distro, you name it...

It's sometimes called Live USB: from [http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB definition], ''A live USB is a USB flash drive containing a full operating system which can be booted from. Live USBs are closely related to Live CDs, and are sometimes used interchangeably. Like Live CDs, Live USBs can be used for system administration, data recovery, or the testing of operating system distributions without committing to a permanent installation on the local hard drive. Many of the smaller Linux distributions can also be used from a USB flash drive.''

Usually, all you need for this with an USB storage device (flash USB stick or external drive) is a partition, flagged "bootable" (see fdisk), containing a boot loader (say grub). But the problem is that specifications vary between motherboards, so there are variants and mandatory requirements to define (here would be a good idea), so that one can optimize/maximize the compatibility.

==The perfect geek knife==

We may want to start our own distro with the following constraints:
* Multiboot: small utilities/compilations + bull blown light OS
* Aiming at maximum hardware compatibility, possibly using liveCD technology (knoppix-based distros)
* Aiming at minimum resource+system footprint, and boot speed
* Being able to install software temporary (for the session) or permanently, upgradable
* Being able to use the phone features at the same time (access pim, data & calls)

Optimally, being able to charge the device while plugged in would be great too.

==About USB Booting==

From [http://www.damnsmalllinux.org/wiki/index.php/USB_Booting DSL Wiki]:

Older computer BIOS usually do not support direct booting from a USB device. Around 2001, PC motherboard manufacturers started to add USB boot support.

There are two common BIOS methods for direct USB booting:
* One method is called the "USBHDD" method and it is used to support the booting of standard USB mass storage devices that are configured like a normal PC hard drive.
* The other method is called the "USBZIP" method and it supports booting from a USB storage device that behaves like the original IOMEGA ZIP drive with USB support.

Most computers (just about all Dells, for example) made today have a BIOS that supports the USBHDD method so I expect that this will eventually become the standard way to boot a USB device. However, many motherboards will support BOTH methods, and many older motherboards have USBZIP support.

Some newer BIOSs which support USB 2.0 will not boot from an older pen drive. Using a USB 2.0 compliant one usually solves this problem. Also, some older BIOSs which only support USB 1.1 will not boot from newer drives which support USB 2.0!

Some USB keys don't boot. If this is the case, it may be possible to fix them by installing a new master boot record. (Most keys boot OK by default; some cannot be fixed even by doing this. However, it helps in some cases). Run the command:

lilo -M /dev/sda

==USB Booting in Neo1973's kernel==

Although there is very little information about it, the g_storage kernel module is responsible for the Neo's mass storage mode. So the boot-or-not criteria might rely on this very module.

==TODO==

Testing/further research has to be conducted to determine:
* if a bootable-flagged partition / grub doesn't mess openmoko up
* if the phone can still act as a phone, or if the booted os can use the gprs functions
* if the phone can recharge on mass storage mode (from the usb cable)
* if a dedicated partition for each payload is needed (see memtest example...)
* if multi-boot is possible: grub?

==Linux distro Howto's: liveUSB creation==

In short:
#Mount USB drive, with e.g. mount /dev/sda1 /flash - can be either FAT16 or FAT32!
#Mount ISO image, with e.g. mount /tmp/dsl-3.2.iso /tmp/iso -o loop
#Copy all contents from ISO to USB drive: cp -vr /tmp/iso/* /flash/
#Rename and move syslinux files to root directory: mv /flash/boot/isolinux/* /flash/
#Rename isolinux.cfg: mv /flash/isolinux.cfg /flash/syslinux.cfg
#Unmount USB drive: umount /flash
#Install syslinux: syslinux /dev/sda1 and eventually set the MBR boot flag for this partition (with fdisk).

*Fedora Core's [http://revisor.fedoraunity.org Revisor] enables easy GUI-driven ISO customization, allowing to choose the output medium (CD/DVD/USB)
*[http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive Installing DSL to USB device - using grub]
*[http://wiki.debian.org/BootUsb Debian BootUSB - using Lilo]
*[http://www.debuntu.org/how-to-install-ubuntu-linux-on-usb-bar Howto install ubuntu (or any other distro) on usb stick]
*[http://www.gentoo.org/doc/en/liveusb.xml Gentoo on usb stick]
*[http://pendrivelinux.com/ Windows & Linux tutorials & resources]:
* Flash installation via Windows: XUbuntu, DSL, Knoppix, Slax, MiniMe
* Flash installation via Linux: Ubuntu Edgy, Knoppix, PCLinuxOS

'''Transflash partitioning schema example'''

* 2 Gb: /dev/sd? (where N is the transflash's number)
* 700 Mb: /dev/sd?1 : containing bootable iso, FAT16
* 1300 Mb: /dev/sd?2 : openmoko & bootable os home partition, EXT3?
* + eventually a swap

The limitation of this method is that the booted OS is static (not-self modifiable). There's the option to install linux using the partition, but it's not really mobility-oriented (liveCDs are optimized for maximum autoconfiguration).

==Alternative methods==

===Booting ISOs ?===

* Grub? From [http://www.freesoftwaremagazine.com/articles/grub_intro#comment-12900 How to install grub on an usb pendrive]:
If you want to boot from a iso file on a harddisk, do something in menu.lst like
title Boot from iso on a harddisk
map (hdX,Y)/your.iso (hdZ)
map --rehook
chainloader (hdZ)+1
rootnoverify (hdZ)
boot

Grub & memdisk can boot floppy images, but (AFAIK) not isos. Example from the stock /boot/grub/menu.lst ubuntu:
title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin
quiet
boot

''Using the memdisk kernel from the syslinux package, you can load disk images and execute them in a non-emulated environment.''

* [http://www.knoppix.net/wiki/Cheat_Codes knoppix '''bootfrom''' cheatcode]

bootfrom=/dev/hda1/KNX.iso

Bootfrom needs access to a running Knoppix-System '''with the same Kernel as the Bootkernel''', before it is able to mount the partition / ISO-Image.

* Maybe there's a way to achieve a boot cd iso selecter see ([http://www.linuxquestions.org/questions/showthread.php?t=528840 this post])
* virtualization-oriented distro... ex [http://unit.aist.go.jp/itri/knoppix/xen/index-en.html Xenoppix] Cons: would work best only on modern VT-capable computers. What about a very minimal distribution (say DSL without X) kqemu-enblaed, booting into an ncurses iso-loader? The delicate part would be on the networking side.

===Zero-Install / Networked systems===

*[http://klik.atekon.de/wiki/index.php/Main_Page Klik] with more than 1000 packages ported. See [http://klik.atekon.de/wiki/index.php/Comparison comparison chart].
*[http://0install.net/index.html The Zero Install System], with automatic dependancies, upgrading. See [http://0install.net/matrix.html] comparison chart.
*Why not rolling a [http://wiki.rpath.com/wiki/Conary:Concepts Conary/RPath-based] distro, based on [http://www.rpath.com/rbuilder/project/rpath/ rPath linux] reference distribution ? Or choose within the available ones...
* The FUSE [http://httpfs.sourceforge.net/net_boot.htm httpfs] may allow pure http net booting, downloading files from a distant livecd only when needed (this image may be very minimal though).
* [http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&start=75&tstart=0 some people] hare thinking of using Amzon S3 as virtual disk

==Using Qemu to boot Linux from a flash drive within Windows==

The following tutorial explains how to use Qemu to boot Linux from a portable USB flash device while still working within Windows. This Enables the user to have both systems running at the same time eliminating the need to restart the PC and set your BIOS options to boot Linux.

[http://www.pendrivelinux.com/2007/03/09/use-qemu-to-boot-linux-from-windows/ PendriveLinux article]

==Putting a vnc server on openmoko==

That's another alternative :)

==The BlackDog project==

"When plugged into the host computer’s USB port, BlackDog starts up, automatically launching the X Window system for Windows Xming and a software NAT router via an autorun application that the BlackDog presents to the host through a virtual USB CD-ROM. Once those applications are running, the virtual USB CD-ROM drive disconnects, and a virtual USB-to-Ethernet adapter is connected to provide the communications link.

The host machine’s monitor, keyboard, mouse, and Internet connection are used by the BlackDog for the duration of the session."

The [http://en.wikipedia.org/wiki/BlackDog BlackDog server] project has lots of interesting concepts (and software).

==Interesting distros/payloads==

'''System diagnostics / recovery'''
*[http://en.wikipedia.org/wiki/Memtest Memtest86+] Very small
*[http://www.ultimatebootcd.com/ The Ultimate Boot CD]
*[http://www.sysresccd.org/Main_Page System Rescue CD]
*[http://www.tux.org/pub/people/kent-robotti/looplinux/rip/ Recovery IS Possible] < 80 MB

'''General purpose'''
*[http://jinx-linux.sourceforge.net/ jinx-linux] < 50 MB
*[http://knoppix.com/ Knoppix] 700 MB CD-ROM
*[http://www.puppylinux.org PuppyLinux] < 100 MB
*[http://www.damnsmalllinux.org DSL] ~50 MB
*[http://gentoo-wiki.com/TinyGentoo#What_is_this_.3F TinyGentoo] 5 MB

'''Security-oriented''': pentesting, forensics, anonymous webbrowsing
*[http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-test-forensics-recovery/ List of Security distros]
*[http://www.remote-exploit.org/backtrack.html Backtrack Livecd]
*[http://en.wikipedia.org/wiki/Anonym.OS Anonym.OS: browse the web anonymously]
*[http://dban.sourceforge.net/ Darik's Boot and Nuke]: ''dangerous''
*[http://byzgl.sourceforge.net/wiki/index.php/Main_Page Byzantine OS] is a webkiosk appliance (32/48 MB)

The SabayonLinux distro offers a lot of [http://www.sabayonlinux.org/wiki/index.php?title=Sabayon_Linux#Boot_Methods_for_special_ways_of_using_Sabayon boot cheatcodes], such as booting onto GeexBox. We should take a look at the method used.

'''[http://www.livecdlist.com/ List of live distros @ livecdlist.com]''' ''Includes sizes''

'''[http://en.wikipedia.org/wiki/List_of_LiveDistros List of live distros @ wikipedia]'''

'''[http://distrowatch.com/dwres.php?resource=cd List of live distros @ distrowatch]'''

'''[http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB article]'''

[[Category:Ideas]]

Wishlist/LiveUSB distro

2007-06-20T22:48:09Z

Fthiery: /* Zero-Install systems */

{{wishlist}}

==Introduction==

NOTE: this page may be renamed: portable destkop

An openmoko device could act as '''the perfect geeky swiss knive''': go anywhere with your Linux desktop and tools.

YES, there are tons of liveCDs out there, but CDs/DVDs:
* get easily damaged
* don't fit in a pocket (except for businesscard-sized ones ; too exotic...)
* don't update or save settings/personal data without a supplementary USB device

YES, there are LiveUSB distros but:
* a Neo/OpenMoko device IS sort of an USB stick; one object is better than two
* i carry my cellphone everywhere, not my USB stick
* there is almost no security on USB sticks (nothing like cryptoloop devices, right?)

The biggest argument against these ones is that you'd have to carry an USB cable with you... But it's a standard one, which is good news :)

Similar functionality can be found in [http://en.wikipedia.org/wiki/Wizpy the Wizpy portable media player]

When the openmoko device is in mass storage mode, a host computer should be able to boot on it, presenting a grub menu offering to boot into several images / partitions (payloads) on the transflash: memtest, UBCD (the ultimate boot cd), a lightweight security oriented livecd distro, you name it...

It's sometimes called Live USB: from [http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB definition], ''A live USB is a USB flash drive containing a full operating system which can be booted from. Live USBs are closely related to Live CDs, and are sometimes used interchangeably. Like Live CDs, Live USBs can be used for system administration, data recovery, or the testing of operating system distributions without committing to a permanent installation on the local hard drive. Many of the smaller Linux distributions can also be used from a USB flash drive.''

Usually, all you need for this with an USB storage device (flash USB stick or external drive) is a partition, flagged "bootable" (see fdisk), containing a boot loader (say grub). But the problem is that specifications vary between motherboards, so there are variants and mandatory requirements to define (here would be a good idea), so that one can optimize/maximize the compatibility.

==The perfect geek knife==

We may want to start our own distro with the following constraints:
* Multiboot: small utilities/compilations + bull blown light OS
* Aiming at maximum hardware compatibility, possibly using liveCD technology (knoppix-based distros)
* Aiming at minimum resource+system footprint, and boot speed
* Being able to install software temporary (for the session) or permanently, upgradable
* Being able to use the phone features at the same time (access pim, data & calls)

Optimally, being able to charge the device while plugged in would be great too.

==About USB Booting==

From [http://www.damnsmalllinux.org/wiki/index.php/USB_Booting DSL Wiki]:

Older computer BIOS usually do not support direct booting from a USB device. Around 2001, PC motherboard manufacturers started to add USB boot support.

There are two common BIOS methods for direct USB booting:
* One method is called the "USBHDD" method and it is used to support the booting of standard USB mass storage devices that are configured like a normal PC hard drive.
* The other method is called the "USBZIP" method and it supports booting from a USB storage device that behaves like the original IOMEGA ZIP drive with USB support.

Most computers (just about all Dells, for example) made today have a BIOS that supports the USBHDD method so I expect that this will eventually become the standard way to boot a USB device. However, many motherboards will support BOTH methods, and many older motherboards have USBZIP support.

Some newer BIOSs which support USB 2.0 will not boot from an older pen drive. Using a USB 2.0 compliant one usually solves this problem. Also, some older BIOSs which only support USB 1.1 will not boot from newer drives which support USB 2.0!

Some USB keys don't boot. If this is the case, it may be possible to fix them by installing a new master boot record. (Most keys boot OK by default; some cannot be fixed even by doing this. However, it helps in some cases). Run the command:

lilo -M /dev/sda

==USB Booting in Neo1973's kernel==

Although there is very little information about it, the g_storage kernel module is responsible for the Neo's mass storage mode. So the boot-or-not criteria might rely on this very module.

==TODO==

Testing/further research has to be conducted to determine:
* if a bootable-flagged partition / grub doesn't mess openmoko up
* if the phone can still act as a phone, or if the booted os can use the gprs functions
* if the phone can recharge on mass storage mode (from the usb cable)
* if a dedicated partition for each payload is needed (see memtest example...)
* if multi-boot is possible: grub?

==Linux distro Howto's: liveUSB creation==

In short:
#Mount USB drive, with e.g. mount /dev/sda1 /flash - can be either FAT16 or FAT32!
#Mount ISO image, with e.g. mount /tmp/dsl-3.2.iso /tmp/iso -o loop
#Copy all contents from ISO to USB drive: cp -vr /tmp/iso/* /flash/
#Rename and move syslinux files to root directory: mv /flash/boot/isolinux/* /flash/
#Rename isolinux.cfg: mv /flash/isolinux.cfg /flash/syslinux.cfg
#Unmount USB drive: umount /flash
#Install syslinux: syslinux /dev/sda1 and eventually set the MBR boot flag for this partition (with fdisk).

*Fedora Core's [http://revisor.fedoraunity.org Revisor] enables easy GUI-driven ISO customization, allowing to choose the output medium (CD/DVD/USB)
*[http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive Installing DSL to USB device - using grub]
*[http://wiki.debian.org/BootUsb Debian BootUSB - using Lilo]
*[http://www.debuntu.org/how-to-install-ubuntu-linux-on-usb-bar Howto install ubuntu (or any other distro) on usb stick]
*[http://www.gentoo.org/doc/en/liveusb.xml Gentoo on usb stick]
*[http://pendrivelinux.com/ Windows & Linux tutorials & resources]:
* Flash installation via Windows: XUbuntu, DSL, Knoppix, Slax, MiniMe
* Flash installation via Linux: Ubuntu Edgy, Knoppix, PCLinuxOS

'''Transflash partitioning schema example'''

* 2 Gb: /dev/sd? (where N is the transflash's number)
* 700 Mb: /dev/sd?1 : containing bootable iso, FAT16
* 1300 Mb: /dev/sd?2 : openmoko & bootable os home partition, EXT3?
* + eventually a swap

The limitation of this method is that the booted OS is static (not-self modifiable). There's the option to install linux using the partition, but it's not really mobility-oriented (liveCDs are optimized for maximum autoconfiguration).

==Alternative methods==

===Booting ISOs ?===

* Grub? From [http://www.freesoftwaremagazine.com/articles/grub_intro#comment-12900 How to install grub on an usb pendrive]:
If you want to boot from a iso file on a harddisk, do something in menu.lst like
title Boot from iso on a harddisk
map (hdX,Y)/your.iso (hdZ)
map --rehook
chainloader (hdZ)+1
rootnoverify (hdZ)
boot

Grub & memdisk can boot floppy images, but (AFAIK) not isos. Example from the stock /boot/grub/menu.lst ubuntu:
title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin
quiet
boot

''Using the memdisk kernel from the syslinux package, you can load disk images and execute them in a non-emulated environment.''

* [http://www.knoppix.net/wiki/Cheat_Codes knoppix '''bootfrom''' cheatcode]

bootfrom=/dev/hda1/KNX.iso

Bootfrom needs access to a running Knoppix-System '''with the same Kernel as the Bootkernel''', before it is able to mount the partition / ISO-Image.

* Maybe there's a way to achieve a boot cd iso selecter see ([http://www.linuxquestions.org/questions/showthread.php?t=528840 this post])
* virtualization-oriented distro... ex [http://unit.aist.go.jp/itri/knoppix/xen/index-en.html Xenoppix] Cons: would work best only on modern VT-capable computers. What about a very minimal distribution (say DSL without X) kqemu-enblaed, booting into an ncurses iso-loader? The delicate part would be on the networking side.

===Zero-Install systems===

For limited storage devices (such as a transflash device), zero-install systems can be very interesting: you can download and run software, without installing it.

The most interesting projects so far seem to be:
*[http://klik.atekon.de/wiki/index.php/Main_Page Klik] with more than 1000 packages ported. See [http://klik.atekon.de/wiki/index.php/Comparison comparison chart].
*[http://0install.net/index.html The Zero Install System], with automatic dependancies, upgrading. See [http://0install.net/matrix.html] comparison chart.
*Why not rolling a [http://wiki.rpath.com/wiki/Conary:Concepts Conary/RPath-based] distro, based on [http://www.rpath.com/rbuilder/project/rpath/ rPath linux] reference distribution ? Or choose within the available ones...
* The FUSE [http://httpfs.sourceforge.net/net_boot.htm httpfs] may allow pure http net booting, downloading files from a distant livecd only when needed (this image may be very minimal though).

One may choose/begin a zero-install-enabled distro for minimal footprint.

==Using Qemu to boot Linux from a flash drive within Windows==

The following tutorial explains how to use Qemu to boot Linux from a portable USB flash device while still working within Windows. This Enables the user to have both systems running at the same time eliminating the need to restart the PC and set your BIOS options to boot Linux.

[http://www.pendrivelinux.com/2007/03/09/use-qemu-to-boot-linux-from-windows/ PendriveLinux article]

==Putting a vnc server on openmoko==

That's another alternative :)

==The BlackDog project==

"When plugged into the host computer’s USB port, BlackDog starts up, automatically launching the X Window system for Windows Xming and a software NAT router via an autorun application that the BlackDog presents to the host through a virtual USB CD-ROM. Once those applications are running, the virtual USB CD-ROM drive disconnects, and a virtual USB-to-Ethernet adapter is connected to provide the communications link.

The host machine’s monitor, keyboard, mouse, and Internet connection are used by the BlackDog for the duration of the session."

The [http://en.wikipedia.org/wiki/BlackDog BlackDog server] project has lots of interesting concepts (and software).

==Interesting distros/payloads==

'''System diagnostics / recovery'''
*[http://en.wikipedia.org/wiki/Memtest Memtest86+] Very small
*[http://www.ultimatebootcd.com/ The Ultimate Boot CD]
*[http://www.sysresccd.org/Main_Page System Rescue CD]
*[http://www.tux.org/pub/people/kent-robotti/looplinux/rip/ Recovery IS Possible] < 80 MB

'''General purpose'''
*[http://jinx-linux.sourceforge.net/ jinx-linux] < 50 MB
*[http://knoppix.com/ Knoppix] 700 MB CD-ROM
*[http://www.puppylinux.org PuppyLinux] < 100 MB
*[http://www.damnsmalllinux.org DSL] ~50 MB
*[http://gentoo-wiki.com/TinyGentoo#What_is_this_.3F TinyGentoo] 5 MB

'''Security-oriented''': pentesting, forensics, anonymous webbrowsing
*[http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-test-forensics-recovery/ List of Security distros]
*[http://www.remote-exploit.org/backtrack.html Backtrack Livecd]
*[http://en.wikipedia.org/wiki/Anonym.OS Anonym.OS: browse the web anonymously]
*[http://dban.sourceforge.net/ Darik's Boot and Nuke]: ''dangerous''
*[http://byzgl.sourceforge.net/wiki/index.php/Main_Page Byzantine OS] is a webkiosk appliance (32/48 MB)

The SabayonLinux distro offers a lot of [http://www.sabayonlinux.org/wiki/index.php?title=Sabayon_Linux#Boot_Methods_for_special_ways_of_using_Sabayon boot cheatcodes], such as booting onto GeexBox. We should take a look at the method used.

'''[http://www.livecdlist.com/ List of live distros @ livecdlist.com]''' ''Includes sizes''

'''[http://en.wikipedia.org/wiki/List_of_LiveDistros List of live distros @ wikipedia]'''

'''[http://distrowatch.com/dwres.php?resource=cd List of live distros @ distrowatch]'''

'''[http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB article]'''

[[Category:Ideas]]

Wishlist/LiveUSB distro

2007-06-20T22:47:57Z

Fthiery: /* Alternative methods */

{{wishlist}}

==Introduction==

NOTE: this page may be renamed: portable destkop

An openmoko device could act as '''the perfect geeky swiss knive''': go anywhere with your Linux desktop and tools.

YES, there are tons of liveCDs out there, but CDs/DVDs:
* get easily damaged
* don't fit in a pocket (except for businesscard-sized ones ; too exotic...)
* don't update or save settings/personal data without a supplementary USB device

YES, there are LiveUSB distros but:
* a Neo/OpenMoko device IS sort of an USB stick; one object is better than two
* i carry my cellphone everywhere, not my USB stick
* there is almost no security on USB sticks (nothing like cryptoloop devices, right?)

The biggest argument against these ones is that you'd have to carry an USB cable with you... But it's a standard one, which is good news :)

Similar functionality can be found in [http://en.wikipedia.org/wiki/Wizpy the Wizpy portable media player]

When the openmoko device is in mass storage mode, a host computer should be able to boot on it, presenting a grub menu offering to boot into several images / partitions (payloads) on the transflash: memtest, UBCD (the ultimate boot cd), a lightweight security oriented livecd distro, you name it...

It's sometimes called Live USB: from [http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB definition], ''A live USB is a USB flash drive containing a full operating system which can be booted from. Live USBs are closely related to Live CDs, and are sometimes used interchangeably. Like Live CDs, Live USBs can be used for system administration, data recovery, or the testing of operating system distributions without committing to a permanent installation on the local hard drive. Many of the smaller Linux distributions can also be used from a USB flash drive.''

Usually, all you need for this with an USB storage device (flash USB stick or external drive) is a partition, flagged "bootable" (see fdisk), containing a boot loader (say grub). But the problem is that specifications vary between motherboards, so there are variants and mandatory requirements to define (here would be a good idea), so that one can optimize/maximize the compatibility.

==The perfect geek knife==

We may want to start our own distro with the following constraints:
* Multiboot: small utilities/compilations + bull blown light OS
* Aiming at maximum hardware compatibility, possibly using liveCD technology (knoppix-based distros)
* Aiming at minimum resource+system footprint, and boot speed
* Being able to install software temporary (for the session) or permanently, upgradable
* Being able to use the phone features at the same time (access pim, data & calls)

Optimally, being able to charge the device while plugged in would be great too.

==About USB Booting==

From [http://www.damnsmalllinux.org/wiki/index.php/USB_Booting DSL Wiki]:

Older computer BIOS usually do not support direct booting from a USB device. Around 2001, PC motherboard manufacturers started to add USB boot support.

There are two common BIOS methods for direct USB booting:
* One method is called the "USBHDD" method and it is used to support the booting of standard USB mass storage devices that are configured like a normal PC hard drive.
* The other method is called the "USBZIP" method and it supports booting from a USB storage device that behaves like the original IOMEGA ZIP drive with USB support.

Most computers (just about all Dells, for example) made today have a BIOS that supports the USBHDD method so I expect that this will eventually become the standard way to boot a USB device. However, many motherboards will support BOTH methods, and many older motherboards have USBZIP support.

Some newer BIOSs which support USB 2.0 will not boot from an older pen drive. Using a USB 2.0 compliant one usually solves this problem. Also, some older BIOSs which only support USB 1.1 will not boot from newer drives which support USB 2.0!

Some USB keys don't boot. If this is the case, it may be possible to fix them by installing a new master boot record. (Most keys boot OK by default; some cannot be fixed even by doing this. However, it helps in some cases). Run the command:

lilo -M /dev/sda

==USB Booting in Neo1973's kernel==

Although there is very little information about it, the g_storage kernel module is responsible for the Neo's mass storage mode. So the boot-or-not criteria might rely on this very module.

==TODO==

Testing/further research has to be conducted to determine:
* if a bootable-flagged partition / grub doesn't mess openmoko up
* if the phone can still act as a phone, or if the booted os can use the gprs functions
* if the phone can recharge on mass storage mode (from the usb cable)
* if a dedicated partition for each payload is needed (see memtest example...)
* if multi-boot is possible: grub?

==Linux distro Howto's: liveUSB creation==

In short:
#Mount USB drive, with e.g. mount /dev/sda1 /flash - can be either FAT16 or FAT32!
#Mount ISO image, with e.g. mount /tmp/dsl-3.2.iso /tmp/iso -o loop
#Copy all contents from ISO to USB drive: cp -vr /tmp/iso/* /flash/
#Rename and move syslinux files to root directory: mv /flash/boot/isolinux/* /flash/
#Rename isolinux.cfg: mv /flash/isolinux.cfg /flash/syslinux.cfg
#Unmount USB drive: umount /flash
#Install syslinux: syslinux /dev/sda1 and eventually set the MBR boot flag for this partition (with fdisk).

*Fedora Core's [http://revisor.fedoraunity.org Revisor] enables easy GUI-driven ISO customization, allowing to choose the output medium (CD/DVD/USB)
*[http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive Installing DSL to USB device - using grub]
*[http://wiki.debian.org/BootUsb Debian BootUSB - using Lilo]
*[http://www.debuntu.org/how-to-install-ubuntu-linux-on-usb-bar Howto install ubuntu (or any other distro) on usb stick]
*[http://www.gentoo.org/doc/en/liveusb.xml Gentoo on usb stick]
*[http://pendrivelinux.com/ Windows & Linux tutorials & resources]:
* Flash installation via Windows: XUbuntu, DSL, Knoppix, Slax, MiniMe
* Flash installation via Linux: Ubuntu Edgy, Knoppix, PCLinuxOS

'''Transflash partitioning schema example'''

* 2 Gb: /dev/sd? (where N is the transflash's number)
* 700 Mb: /dev/sd?1 : containing bootable iso, FAT16
* 1300 Mb: /dev/sd?2 : openmoko & bootable os home partition, EXT3?
* + eventually a swap

The limitation of this method is that the booted OS is static (not-self modifiable). There's the option to install linux using the partition, but it's not really mobility-oriented (liveCDs are optimized for maximum autoconfiguration).

==Alternative methods==

===Booting ISOs ?===

* Grub? From [http://www.freesoftwaremagazine.com/articles/grub_intro#comment-12900 How to install grub on an usb pendrive]:
If you want to boot from a iso file on a harddisk, do something in menu.lst like
title Boot from iso on a harddisk
map (hdX,Y)/your.iso (hdZ)
map --rehook
chainloader (hdZ)+1
rootnoverify (hdZ)
boot

Grub & memdisk can boot floppy images, but (AFAIK) not isos. Example from the stock /boot/grub/menu.lst ubuntu:
title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin
quiet
boot

''Using the memdisk kernel from the syslinux package, you can load disk images and execute them in a non-emulated environment.''

* [http://www.knoppix.net/wiki/Cheat_Codes knoppix '''bootfrom''' cheatcode]

bootfrom=/dev/hda1/KNX.iso

Bootfrom needs access to a running Knoppix-System '''with the same Kernel as the Bootkernel''', before it is able to mount the partition / ISO-Image.

* Maybe there's a way to achieve a boot cd iso selecter see ([http://www.linuxquestions.org/questions/showthread.php?t=528840 this post])
* virtualization-oriented distro... ex [http://unit.aist.go.jp/itri/knoppix/xen/index-en.html Xenoppix] Cons: would work best only on modern VT-capable computers. What about a very minimal distribution (say DSL without X) kqemu-enblaed, booting into an ncurses iso-loader? The delicate part would be on the networking side.

==Zero-Install systems==

For limited storage devices (such as a transflash device), zero-install systems can be very interesting: you can download and run software, without installing it.

The most interesting projects so far seem to be:
*[http://klik.atekon.de/wiki/index.php/Main_Page Klik] with more than 1000 packages ported. See [http://klik.atekon.de/wiki/index.php/Comparison comparison chart].
*[http://0install.net/index.html The Zero Install System], with automatic dependancies, upgrading. See [http://0install.net/matrix.html] comparison chart.
*Why not rolling a [http://wiki.rpath.com/wiki/Conary:Concepts Conary/RPath-based] distro, based on [http://www.rpath.com/rbuilder/project/rpath/ rPath linux] reference distribution ? Or choose within the available ones...
* The FUSE [http://httpfs.sourceforge.net/net_boot.htm httpfs] may allow pure http net booting, downloading files from a distant livecd only when needed (this image may be very minimal though).

One may choose/begin a zero-install-enabled distro for minimal footprint.

==Using Qemu to boot Linux from a flash drive within Windows==

The following tutorial explains how to use Qemu to boot Linux from a portable USB flash device while still working within Windows. This Enables the user to have both systems running at the same time eliminating the need to restart the PC and set your BIOS options to boot Linux.

[http://www.pendrivelinux.com/2007/03/09/use-qemu-to-boot-linux-from-windows/ PendriveLinux article]

==Putting a vnc server on openmoko==

That's another alternative :)

==The BlackDog project==

"When plugged into the host computer’s USB port, BlackDog starts up, automatically launching the X Window system for Windows Xming and a software NAT router via an autorun application that the BlackDog presents to the host through a virtual USB CD-ROM. Once those applications are running, the virtual USB CD-ROM drive disconnects, and a virtual USB-to-Ethernet adapter is connected to provide the communications link.

The host machine’s monitor, keyboard, mouse, and Internet connection are used by the BlackDog for the duration of the session."

The [http://en.wikipedia.org/wiki/BlackDog BlackDog server] project has lots of interesting concepts (and software).

==Interesting distros/payloads==

'''System diagnostics / recovery'''
*[http://en.wikipedia.org/wiki/Memtest Memtest86+] Very small
*[http://www.ultimatebootcd.com/ The Ultimate Boot CD]
*[http://www.sysresccd.org/Main_Page System Rescue CD]
*[http://www.tux.org/pub/people/kent-robotti/looplinux/rip/ Recovery IS Possible] < 80 MB

'''General purpose'''
*[http://jinx-linux.sourceforge.net/ jinx-linux] < 50 MB
*[http://knoppix.com/ Knoppix] 700 MB CD-ROM
*[http://www.puppylinux.org PuppyLinux] < 100 MB
*[http://www.damnsmalllinux.org DSL] ~50 MB
*[http://gentoo-wiki.com/TinyGentoo#What_is_this_.3F TinyGentoo] 5 MB

'''Security-oriented''': pentesting, forensics, anonymous webbrowsing
*[http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-test-forensics-recovery/ List of Security distros]
*[http://www.remote-exploit.org/backtrack.html Backtrack Livecd]
*[http://en.wikipedia.org/wiki/Anonym.OS Anonym.OS: browse the web anonymously]
*[http://dban.sourceforge.net/ Darik's Boot and Nuke]: ''dangerous''
*[http://byzgl.sourceforge.net/wiki/index.php/Main_Page Byzantine OS] is a webkiosk appliance (32/48 MB)

The SabayonLinux distro offers a lot of [http://www.sabayonlinux.org/wiki/index.php?title=Sabayon_Linux#Boot_Methods_for_special_ways_of_using_Sabayon boot cheatcodes], such as booting onto GeexBox. We should take a look at the method used.

'''[http://www.livecdlist.com/ List of live distros @ livecdlist.com]''' ''Includes sizes''

'''[http://en.wikipedia.org/wiki/List_of_LiveDistros List of live distros @ wikipedia]'''

'''[http://distrowatch.com/dwres.php?resource=cd List of live distros @ distrowatch]'''

'''[http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB article]'''

[[Category:Ideas]]

Wishlist/LiveUSB distro

2007-06-20T22:45:41Z

Fthiery: Project BlackDog addition

{{wishlist}}

==Introduction==

NOTE: this page may be renamed: portable destkop

An openmoko device could act as '''the perfect geeky swiss knive''': go anywhere with your Linux desktop and tools.

YES, there are tons of liveCDs out there, but CDs/DVDs:
* get easily damaged
* don't fit in a pocket (except for businesscard-sized ones ; too exotic...)
* don't update or save settings/personal data without a supplementary USB device

YES, there are LiveUSB distros but:
* a Neo/OpenMoko device IS sort of an USB stick; one object is better than two
* i carry my cellphone everywhere, not my USB stick
* there is almost no security on USB sticks (nothing like cryptoloop devices, right?)

The biggest argument against these ones is that you'd have to carry an USB cable with you... But it's a standard one, which is good news :)

Similar functionality can be found in [http://en.wikipedia.org/wiki/Wizpy the Wizpy portable media player]

When the openmoko device is in mass storage mode, a host computer should be able to boot on it, presenting a grub menu offering to boot into several images / partitions (payloads) on the transflash: memtest, UBCD (the ultimate boot cd), a lightweight security oriented livecd distro, you name it...

It's sometimes called Live USB: from [http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB definition], ''A live USB is a USB flash drive containing a full operating system which can be booted from. Live USBs are closely related to Live CDs, and are sometimes used interchangeably. Like Live CDs, Live USBs can be used for system administration, data recovery, or the testing of operating system distributions without committing to a permanent installation on the local hard drive. Many of the smaller Linux distributions can also be used from a USB flash drive.''

Usually, all you need for this with an USB storage device (flash USB stick or external drive) is a partition, flagged "bootable" (see fdisk), containing a boot loader (say grub). But the problem is that specifications vary between motherboards, so there are variants and mandatory requirements to define (here would be a good idea), so that one can optimize/maximize the compatibility.

==The perfect geek knife==

We may want to start our own distro with the following constraints:
* Multiboot: small utilities/compilations + bull blown light OS
* Aiming at maximum hardware compatibility, possibly using liveCD technology (knoppix-based distros)
* Aiming at minimum resource+system footprint, and boot speed
* Being able to install software temporary (for the session) or permanently, upgradable
* Being able to use the phone features at the same time (access pim, data & calls)

Optimally, being able to charge the device while plugged in would be great too.

==About USB Booting==

From [http://www.damnsmalllinux.org/wiki/index.php/USB_Booting DSL Wiki]:

Older computer BIOS usually do not support direct booting from a USB device. Around 2001, PC motherboard manufacturers started to add USB boot support.

There are two common BIOS methods for direct USB booting:
* One method is called the "USBHDD" method and it is used to support the booting of standard USB mass storage devices that are configured like a normal PC hard drive.
* The other method is called the "USBZIP" method and it supports booting from a USB storage device that behaves like the original IOMEGA ZIP drive with USB support.

Most computers (just about all Dells, for example) made today have a BIOS that supports the USBHDD method so I expect that this will eventually become the standard way to boot a USB device. However, many motherboards will support BOTH methods, and many older motherboards have USBZIP support.

Some newer BIOSs which support USB 2.0 will not boot from an older pen drive. Using a USB 2.0 compliant one usually solves this problem. Also, some older BIOSs which only support USB 1.1 will not boot from newer drives which support USB 2.0!

Some USB keys don't boot. If this is the case, it may be possible to fix them by installing a new master boot record. (Most keys boot OK by default; some cannot be fixed even by doing this. However, it helps in some cases). Run the command:

lilo -M /dev/sda

==USB Booting in Neo1973's kernel==

Although there is very little information about it, the g_storage kernel module is responsible for the Neo's mass storage mode. So the boot-or-not criteria might rely on this very module.

==TODO==

Testing/further research has to be conducted to determine:
* if a bootable-flagged partition / grub doesn't mess openmoko up
* if the phone can still act as a phone, or if the booted os can use the gprs functions
* if the phone can recharge on mass storage mode (from the usb cable)
* if a dedicated partition for each payload is needed (see memtest example...)
* if multi-boot is possible: grub?

==Linux distro Howto's: liveUSB creation==

In short:
#Mount USB drive, with e.g. mount /dev/sda1 /flash - can be either FAT16 or FAT32!
#Mount ISO image, with e.g. mount /tmp/dsl-3.2.iso /tmp/iso -o loop
#Copy all contents from ISO to USB drive: cp -vr /tmp/iso/* /flash/
#Rename and move syslinux files to root directory: mv /flash/boot/isolinux/* /flash/
#Rename isolinux.cfg: mv /flash/isolinux.cfg /flash/syslinux.cfg
#Unmount USB drive: umount /flash
#Install syslinux: syslinux /dev/sda1 and eventually set the MBR boot flag for this partition (with fdisk).

*Fedora Core's [http://revisor.fedoraunity.org Revisor] enables easy GUI-driven ISO customization, allowing to choose the output medium (CD/DVD/USB)
*[http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive Installing DSL to USB device - using grub]
*[http://wiki.debian.org/BootUsb Debian BootUSB - using Lilo]
*[http://www.debuntu.org/how-to-install-ubuntu-linux-on-usb-bar Howto install ubuntu (or any other distro) on usb stick]
*[http://www.gentoo.org/doc/en/liveusb.xml Gentoo on usb stick]
*[http://pendrivelinux.com/ Windows & Linux tutorials & resources]:
* Flash installation via Windows: XUbuntu, DSL, Knoppix, Slax, MiniMe
* Flash installation via Linux: Ubuntu Edgy, Knoppix, PCLinuxOS

'''Transflash partitioning schema example'''

* 2 Gb: /dev/sd? (where N is the transflash's number)
* 700 Mb: /dev/sd?1 : containing bootable iso, FAT16
* 1300 Mb: /dev/sd?2 : openmoko & bootable os home partition, EXT3?
* + eventually a swap

The limitation of this method is that the booted OS is static (not-self modifiable). There's the option to install linux using the partition, but it's not really mobility-oriented (liveCDs are optimized for maximum autoconfiguration).

==Alternative methods==

The ideal way of doing it would be to have a bootloader on the usb flash, which would offer booting directly from an iso (stored on the very same device). This way, just download the new iso, and it's updated !

Hints to explore:
* [http://www.911cd.net/forums//index.php?showtopic=8955&hl= Isoemu] ''By using it, we can boot our system from a local iso file, if this iso file is bootable'' (well, i'd like to see that :p)
* Grub? From [http://www.freesoftwaremagazine.com/articles/grub_intro#comment-12900 How to install grub on an usb pendrive]:
If you want to boot from a iso file on a harddisk, do something in menu.lst like
title Boot from iso on a harddisk
map (hdX,Y)/your.iso (hdZ)
map --rehook
chainloader (hdZ)+1
rootnoverify (hdZ)
boot

Grub & memdisk can boot floppy images, but (AFAIK) not isos. Example from the stock /boot/grub/menu.lst ubuntu:
title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin
quiet
boot

''Using the memdisk kernel from the syslinux package, you can load disk images and execute them in a non-emulated environment.''

* [http://www.knoppix.net/wiki/Cheat_Codes knoppix '''bootfrom''' cheatcode]

bootfrom=/dev/hda1/KNX.iso

Bootfrom needs access to a running Knoppix-System '''with the same Kernel as the Bootkernel''', before it is able to mount the partition / ISO-Image.

* Maybe there's a way to achieve a boot cd iso selecter see ([http://www.linuxquestions.org/questions/showthread.php?t=528840 this post])
* virtualization-oriented distro... ex [http://unit.aist.go.jp/itri/knoppix/xen/index-en.html Xenoppix] Cons: would work best only on modern VT-capable computers. What about a very minimal distribution (say DSL without X) kqemu-enblaed, booting into an ncurses iso-loader? The delicate part would be on the networking side.

==Zero-Install systems==

For limited storage devices (such as a transflash device), zero-install systems can be very interesting: you can download and run software, without installing it.

The most interesting projects so far seem to be:
*[http://klik.atekon.de/wiki/index.php/Main_Page Klik] with more than 1000 packages ported. See [http://klik.atekon.de/wiki/index.php/Comparison comparison chart].
*[http://0install.net/index.html The Zero Install System], with automatic dependancies, upgrading. See [http://0install.net/matrix.html] comparison chart.
*Why not rolling a [http://wiki.rpath.com/wiki/Conary:Concepts Conary/RPath-based] distro, based on [http://www.rpath.com/rbuilder/project/rpath/ rPath linux] reference distribution ? Or choose within the available ones...
* The FUSE [http://httpfs.sourceforge.net/net_boot.htm httpfs] may allow pure http net booting, downloading files from a distant livecd only when needed (this image may be very minimal though).

One may choose/begin a zero-install-enabled distro for minimal footprint.

==Using Qemu to boot Linux from a flash drive within Windows==

The following tutorial explains how to use Qemu to boot Linux from a portable USB flash device while still working within Windows. This Enables the user to have both systems running at the same time eliminating the need to restart the PC and set your BIOS options to boot Linux.

[http://www.pendrivelinux.com/2007/03/09/use-qemu-to-boot-linux-from-windows/ PendriveLinux article]

==Putting a vnc server on openmoko==

That's another alternative :)

==The BlackDog project==

"When plugged into the host computer’s USB port, BlackDog starts up, automatically launching the X Window system for Windows Xming and a software NAT router via an autorun application that the BlackDog presents to the host through a virtual USB CD-ROM. Once those applications are running, the virtual USB CD-ROM drive disconnects, and a virtual USB-to-Ethernet adapter is connected to provide the communications link.

The host machine’s monitor, keyboard, mouse, and Internet connection are used by the BlackDog for the duration of the session."

The [http://en.wikipedia.org/wiki/BlackDog BlackDog server] project has lots of interesting concepts (and software).

==Interesting distros/payloads==

'''System diagnostics / recovery'''
*[http://en.wikipedia.org/wiki/Memtest Memtest86+] Very small
*[http://www.ultimatebootcd.com/ The Ultimate Boot CD]
*[http://www.sysresccd.org/Main_Page System Rescue CD]
*[http://www.tux.org/pub/people/kent-robotti/looplinux/rip/ Recovery IS Possible] < 80 MB

'''General purpose'''
*[http://jinx-linux.sourceforge.net/ jinx-linux] < 50 MB
*[http://knoppix.com/ Knoppix] 700 MB CD-ROM
*[http://www.puppylinux.org PuppyLinux] < 100 MB
*[http://www.damnsmalllinux.org DSL] ~50 MB
*[http://gentoo-wiki.com/TinyGentoo#What_is_this_.3F TinyGentoo] 5 MB

'''Security-oriented''': pentesting, forensics, anonymous webbrowsing
*[http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-test-forensics-recovery/ List of Security distros]
*[http://www.remote-exploit.org/backtrack.html Backtrack Livecd]
*[http://en.wikipedia.org/wiki/Anonym.OS Anonym.OS: browse the web anonymously]
*[http://dban.sourceforge.net/ Darik's Boot and Nuke]: ''dangerous''
*[http://byzgl.sourceforge.net/wiki/index.php/Main_Page Byzantine OS] is a webkiosk appliance (32/48 MB)

The SabayonLinux distro offers a lot of [http://www.sabayonlinux.org/wiki/index.php?title=Sabayon_Linux#Boot_Methods_for_special_ways_of_using_Sabayon boot cheatcodes], such as booting onto GeexBox. We should take a look at the method used.

'''[http://www.livecdlist.com/ List of live distros @ livecdlist.com]''' ''Includes sizes''

'''[http://en.wikipedia.org/wiki/List_of_LiveDistros List of live distros @ wikipedia]'''

'''[http://distrowatch.com/dwres.php?resource=cd List of live distros @ distrowatch]'''

'''[http://en.wikipedia.org/wiki/Live_USB Wikipedia LiveUSB article]'''

[[Category:Ideas]]

Wishlist/Text Input

2007-06-15T09:53:14Z

Fthiery: Added blog post to smartpen

Current version supports input using [[Input Method|on screen keyboard]].
You can also use Bluetooth keyboards and with (battery) powered USB hub can also use USB keyboards.

Near computer can also use networking over Bluetooth or USB and then ssh into device or run X11 programs remotely and thus use whatever input possibilities other computer supports.

For further predictive text input information see: [http://en.wikipedia.org/wiki/Predictive_text Predictive text wikipedia]

{{Wishlist}}
[[Category:User Interfaces]]

===== New input methods =====
* [http://patrickbaudisch.com/publications/2007-Vogel-CHI07-Shift.pdf Shift]
* [http://www.inf.ufsc.br/~guy/text_input.html Another text input]
* [http://www.inference.phy.cam.ac.uk/dasher/ Dasher] ([http://en.wikipedia.org/wiki/Dasher Wikipedia link]): A side-scrolling probabalistic text entry method that's well suited to stylus use. One disadvantage is that it makes little use of muscle memory so you need to pay close attention to the screen while entering text. An andvatage is that it is not limeted to english text, but can be used with any language/alphabet. A video presentation is also [http://video.google.com/videoplay?docid=5078334075080674416 available]
* [http://www.micropp.se/openmoko/ Finger splash] (Idea presented on [http://lists.openmoko.org/pipermail/community/2007-March/003984.html OpenMoko community mailing list])
* [http://www.strout.net/info/ideas/hexinput.html HexInput]: A keyboard-style input method optimized for stylus use.
* [http://www.exideas.com/ME/faq.html MessagEaseST] ([http://www.youtube.com/watch?v=zFf9Mw3nlsY YouTube demo])
* [http://en.wikipedia.org/wiki/Morse_code Morse Code]
* [http://mrl.nyu.edu/projects/quikwriting/ Quikwriting]
* [http://www.almaden.ibm.com/u/zhai/shapewriter.htm Shape Writing]
* [http://lists.openmoko.org/pipermail/openmoko-devel/2007-May/000912.html Werner Almesberger's finger input suggestion] ([http://www.acc.umu.se/~cm/inw.pdf Draft layout as pdf] [http://www.almesberger.net/misc/openmoko/inwheel-1.tar.gz Prototype])
* [http://www.hellkvist.org/software/ XMerlin]

===== Patented input methods =====
* [http://depts.washington.edu/ewrite/ EdgeWrite], a unistroke character/word input method (reminiscent of Palm's Graffiti)
* [http://www.fitaly.com/wince/pocketpcfitaly.htm Fitaly Keyboard]
* [http://images.overstock.com/f/102/3117/8h/www.overstock.com/images/products/L10480944.jpg Input method used by Garmin] (Maybe patented?)
* [http://www.speedscript.biz/ SpeedScript]
* [http://en.wikipedia.org/wiki/T9_%28predictive_text%29 T9] See http://www.tegic.com/about/patent-list.asp for legally addability for users in some countries.
* [http://www.tengo.net/ TenGO]
* [http://www.agiletext.com/ AgileText]

===== Text input method surveys =====

Here are some surveys / overviews of text entry methods that I turned up using Google:
* [http://www.dcs.gla.ac.uk/~jhw/text.html Text entry] A web page with a survey of text entry methods by John Williamson
* Poika Isokoski, [http://www.cs.uta.fi/~poika/g/g.html A Minimal Device-Independent Text Input Method], has a chapter listing existing approaches ca. 1999

===== Other ways to enter text =====
* Once there is hardware with multi-touch screen support, [[Wishlist:Spell weaving|gesturing with 2-3 fingers]] might offer interesting possibilities.
* Use [[VoiceText|voice to dictate text]]
* Use [[Optical Character Recognition]] and [[Barcode Recognition]] on an image that exists on the file system or via a picture that has just been taken (even if it is a temporary picture only for this purpose).
* [http://en.wikipedia.org/wiki/Speedwords Dutton Speedwords]

===== Interesting hardware input devices =====
* [http://www.mobilitysite.com/2007/02/small_compact_bluetooth_keyboard.php Credit-card sized bluetooth keyboard] (Note: apparently supports only Bluetooth Serial Port Profile and not HID; supportable, of course, perhaps using existing user space daemon [http://handhelds.org/moin/moin.cgi/kbdd kbdd])
* [http://www.thinkgeek.com/computing/input/8193/ Bluetooth laser virtual keyboard] eventually could be built into the phone once more miniaturized.
* [http://www.thinkgeek.com/computing/input/6c82/ Frogpad]
* [http://www.spartechnik.de/start.htm?d_Keyb_Mini_Mini_Bluetooth_Keyboard_im_Scheckkartenformat.htm Freedom Mini], apparently not manufactured anymore but still sold and works out of the box with Bluez' hidd. Has a spring-loaded hinge for squeezing a phone/PDA against the keyboard; seems like a Neo could attach nicely but don't have one to actually test. --[[User:Mjr|Mjr]] 10:30, 15 May 2007 (CEST)
* [http://www.3pointd.com/20070518/finger-mounted-3d-mouse-from-undergrads/ experimental ring-mouse]
* Livescribe's [http://www.livescribe.com/sneakpeek/index.html smartpen]; could act as touchscreen pen + laser + regular pen + "intelligent" pen + OCR device. See [http://www.pikesoft.com/blog/index.php?itemid=189 this] blog post.

[[Category:Ideas]]

Wishlist/Text Input

2007-06-15T09:48:04Z

Fthiery: Added livescribe pen link