Touchscreen Filters

From Openmoko

(Difference between revisions)
Jump to: navigation, search
(Generic filters)
Line 106: Line 106:
 
filters have to implement. These are the operations that each filter
 
filters have to implement. These are the operations that each filter
 
can perform.
 
can perform.
 +
<pre>
  
 
struct ts_filter_api {
 
struct ts_filter_api {
Line 114: Line 115:
 
void (*scale)(struct ts_filter *filter, int *coords);
 
void (*scale)(struct ts_filter *filter, int *coords);
 
};
 
};
 +
</pre>
  
 
clear: We call this operation when we need the filter to be in a clean state.
 
clear: We call this operation when we need the filter to be in a clean state.
Line 121: Line 123:
 
sample to the filter, in our driver this happens when an A/D
 
sample to the filter, in our driver this happens when an A/D
 
conversion ends. After that one of three things
 
conversion ends. After that one of three things
can happen: 1) If the filter needs more samples to be gathered it returns a 0.
+
can happen:
2) If it is ready to return a sample and there are no filters after itself, it returns 1.
+
 
3) If it is ready to return a sample and there are filters after itself it will pass
+
# If the filter needs more samples to be gathered it returns a 0.
the resulting sample to the next filter and it's up to that filter to decide what
+
# If it is ready to return a sample and there are no filters after itself, it returns 1.
it returns. If a filter reaches an error it returns -1. In the GTA01/GTA02 this happens
+
# If it is ready to return a sample and there are filters after itself it will pass the resulting sample to the next filter and it's up to that filter to decide what it returns. If a filter reaches an error it returns -1. In the GTA01/GTA02 this happens when many inputs are discarded because they are not good samples and we decide to give up trying to get good samples.
when many inputs are discarded because they are not good samples and we decide to give
+
up trying to get good samples.
+
  
 
scale: This function is called for each filter after the filter chain returns 1.
 
scale: This function is called for each filter after the filter chain returns 1.

Revision as of 23:01, 26 December 2008

Contents


Work in progress.

TODO

  • Complete the document.
  • Improve videos (with no music, please).
  • Check for correct English

Intro

In this document we describe the algorithms we use to make the touchscreen behave well. By "well" we mean that we can:

  • Get reliable clicks with the stylus and with the finger
  • Avoid further filtering in user space
  • Avoid calibration in user-space. We believe X (and other programs) should be able to gather reliable data from /dev/input/eventX.

Openmoko developers and contributors have tried different approaches that have helped improve the touchscreen performance. As a result now we have a touchscreen filtering framework that we are using in Linux 2.6. We describe the current working solution as of December 2008 as it is in the andy-tracking branch of the Openmoko GIT repository.

If you think we can improve something please send us feedback.

We include videos showing how things improve as we add filters. In the videos we use a program that plots the points reported by the driver.

Our hardware

The FreeRunner (GTA02) uses the S3C2442 touchscreen controller, while the Neo1973 (GTA01) uses the S3C2410 one. For both devices we use the same driver.

The driver can be used for other devices:

We don't use pressure information

The only information we are using from the hardware is the reported X and Y coordinates. We are not using pressure information. We could get the area of the contact to the touchscreen and we could tell between stylus, fingernail or thumb. The area information is reliable but we didn't manage to normalize the results across the screen as they varied widely depending on which corner of the screen is pressed, in a non-linear way.

Why are we doing filtering in kernel space?

This is not new. An early driver for this device does some filtering (corrections, averaging, etc).

In the latest driver we decided to add a generic filtering framework that could be used by more touchscreen drivers.

We are aware of tslib and we use it with the current stable kernel (2.6.24) but we also think that doing filtering in the kernel is a good idea, and in the latest kernel (2.6.28 - next stable) we are doing it and as a side effect we no longer need to use tslib.

Let's say we would like to deliver a TS event to user space each 10 milliseconds. In the GTA02 with the current configuration the Analog/Digital conversion time of a sample is 0.4697 milliseconds, thus can get 18 samples for each event that we send to user-space. Sending the event (with 18 samples) takes us about 8.45 milliseconds. Sometimes we even decide that the event should not be sent to user-space (because the hardware didn't provide reliable data), and our tests show that it's the right thing to do. In previous versions of the driver light taps would confuse the driver (that would report bad clicks) and this is no longer an issue.

Briefly stated: We think it's better if the driver sends only good data to the applications. If there are multiple distributions for a device (there are quite a few for the GTA02) they can share a tuned in-kernel configuration. We also save a lot of kernel mode/user mode transitions and data transmission allowing us to consider more samples for each reported event.

Raw output

This is the raw output of the s3c2410 driver we are using (source).

In this video:

  • We draw a few numbers
  • We draw points while trying to keep the stylus still and we get stain because of the variance of the raw output data. We also try to use some pressure, we are not testing light pressure here.
  • We draw two lines, two curves
  • We draw points with the finger
  • We use the stylus again to draw face

Note that we paint the points after we have lifted the stylus/finger[3] but keep in mind that the kernel is sending events while we press the screen.

Video: http://www.youtube.com/watch?v=-ouBKOETiG8


Generic filters

Sources:

drivers/input/touchscreen/ts_filter.c include/linux/ts_filter.h

Generic filters are not useful by themselves, they define an API that that actual filters have to implement. These are the operations that each filter can perform.


	struct ts_filter_api {
		struct ts_filter * (*create)(void *config, int count_coords);
		void (*destroy)(struct ts_filter *filter);
		void (*clear)(struct ts_filter *filter);
		int (*process)(struct ts_filter *filter, int *coords);
		void (*scale)(struct ts_filter *filter, int *coords);
	};

clear: We call this operation when we need the filter to be in a clean state. For instance, when we initialize the filter or when we have a pen-up event.

process: This function is called when we need to send a sample to the filter, in our driver this happens when an A/D conversion ends. After that one of three things can happen:

  1. If the filter needs more samples to be gathered it returns a 0.
  2. If it is ready to return a sample and there are no filters after itself, it returns 1.
  3. If it is ready to return a sample and there are filters after itself it will pass the resulting sample to the next filter and it's up to that filter to decide what it returns. If a filter reaches an error it returns -1. In the GTA01/GTA02 this happens when many inputs are discarded because they are not good samples and we decide to give up trying to get good samples.

scale: This function is called for each filter after the filter chain returns 1. It is used to save expensive operations (such as divisions) in those cases when you can get away with doing them with aggregated data.

Group filter

Source: drivers/input/touchscreen/ts_filter_group.c

This filter is useful to reject samples that are not reliable. We consider that a sample is not reliable if it deviates form the Majority.

(1) First we collect a set of S samples. (2) We make a copy of the values of each coordinate and process each dimension separately. For each dimension (X,Y):

  • Sort the points
  • Points that are "close enough" are considered to be in the same set.
  • Choose the set with more elements. If more than "threshold" points are in this set we use the first and the last point of the set to define the valid range for this dimension [min, max], otherwise we discard all the points and go to step 1.
  • We consider the unsorted S samples and try to feed them to the next filter in the chain in the same order they arrived. If one of the points of each sample is not in the allowed range for its dimension, we discard the sample.

http://www.youtube.com/watch?v=17gempMRhBs

With this filter get rid of most of the noise and we make up for the absence of pressure information. Light pressure means bad data, and the bad data is discarded.

Median filter

...

Mean filter

...

Linear calibration filter

...

Final outcome

...

Personal tools

Contents


Work in progress.

TODO

  • Complete the document.
  • Improve videos (with no music, please).
  • Check for correct English

Intro

In this document we describe the algorithms we use to make the touchscreen behave well. By "well" we mean that we can:

  • Get reliable clicks with the stylus and with the finger
  • Avoid further filtering in user space
  • Avoid calibration in user-space. We believe X (and other programs) should be able to gather reliable data from /dev/input/eventX.

Openmoko developers and contributors have tried different approaches that have helped improve the touchscreen performance. As a result now we have a touchscreen filtering framework that we are using in Linux 2.6. We describe the current working solution as of December 2008 as it is in the andy-tracking branch of the Openmoko GIT repository.

If you think we can improve something please send us feedback.

We include videos showing how things improve as we add filters. In the videos we use a program that plots the points reported by the driver.

Our hardware

The FreeRunner (GTA02) uses the S3C2442 touchscreen controller, while the Neo1973 (GTA01) uses the S3C2410 one. For both devices we use the same driver.

The driver can be used for other devices:

We don't use pressure information

The only information we are using from the hardware is the reported X and Y coordinates. We are not using pressure information. We could get the area of the contact to the touchscreen and we could tell between stylus, fingernail or thumb. The area information is reliable but we didn't manage to normalize the results across the screen as they varied widely depending on which corner of the screen is pressed, in a non-linear way.

Why are we doing filtering in kernel space?

This is not new. An early driver for this device does some filtering (corrections, averaging, etc).

In the latest driver we decided to add a generic filtering framework that could be used by more touchscreen drivers.

We are aware of tslib and we use it with the current stable kernel (2.6.24) but we also think that doing filtering in the kernel is a good idea, and in the latest kernel (2.6.28 - next stable) we are doing it and as a side effect we no longer need to use tslib.

Let's say we would like to deliver a TS event to user space each 10 milliseconds. In the GTA02 with the current configuration the Analog/Digital conversion time of a sample is 0.4697 milliseconds, thus can get 18 samples for each event that we send to user-space. Sending the event (with 18 samples) takes us about 8.45 milliseconds. Sometimes we even decide that the event should not be sent to user-space (because the hardware didn't provide reliable data), and our tests show that it's the right thing to do. In previous versions of the driver light taps would confuse the driver (that would report bad clicks) and this is no longer an issue.

Briefly stated: We think it's better if the driver sends only good data to the applications. If there are multiple distributions for a device (there are quite a few for the GTA02) they can share a tuned in-kernel configuration. We also save a lot of kernel mode/user mode transitions and data transmission allowing us to consider more samples for each reported event.

Raw output

This is the raw output of the s3c2410 driver we are using (source).

In this video:

  • We draw a few numbers
  • We draw points while trying to keep the stylus still and we get stain because of the variance of the raw output data. We also try to use some pressure, we are not testing light pressure here.
  • We draw two lines, two curves
  • We draw points with the finger
  • We use the stylus again to draw face

Note that we paint the points after we have lifted the stylus/finger[3] but keep in mind that the kernel is sending events while we press the screen.

Video: http://www.youtube.com/watch?v=-ouBKOETiG8


Generic filters

Sources:

drivers/input/touchscreen/ts_filter.c include/linux/ts_filter.h

Generic filters are not useful by themselves, they define an API that that actual filters have to implement. These are the operations that each filter can perform.

struct ts_filter_api { struct ts_filter * (*create)(void *config, int count_coords); void (*destroy)(struct ts_filter *filter); void (*clear)(struct ts_filter *filter); int (*process)(struct ts_filter *filter, int *coords); void (*scale)(struct ts_filter *filter, int *coords); };

clear: We call this operation when we need the filter to be in a clean state. For instance, when we initialize the filter or when we have a pen-up event.

process: This function is called when we need to send a sample to the filter, in our driver this happens when an A/D conversion ends. After that one of three things can happen: 1) If the filter needs more samples to be gathered it returns a 0. 2) If it is ready to return a sample and there are no filters after itself, it returns 1. 3) If it is ready to return a sample and there are filters after itself it will pass the resulting sample to the next filter and it's up to that filter to decide what it returns. If a filter reaches an error it returns -1. In the GTA01/GTA02 this happens when many inputs are discarded because they are not good samples and we decide to give up trying to get good samples.

scale: This function is called for each filter after the filter chain returns 1. It is used to save expensive operations (such as divisions) in those cases when you can get away with doing them with aggregated data.

Group filter

Source: drivers/input/touchscreen/ts_filter_group.c

This filter is useful to reject samples that are not reliable. We consider that a sample is not reliable if it deviates form the Majority.

(1) First we collect a set of S samples. (2) We make a copy of the values of each coordinate and process each dimension separately. For each dimension (X,Y):

  • Sort the points
  • Points that are "close enough" are considered to be in the same set.
  • Choose the set with more elements. If more than "threshold" points are in this set we use the first and the last point of the set to define the valid range for this dimension [min, max], otherwise we discard all the points and go to step 1.
  • We consider the unsorted S samples and try to feed them to the next filter in the chain in the same order they arrived. If one of the points of each sample is not in the allowed range for its dimension, we discard the sample.

http://www.youtube.com/watch?v=17gempMRhBs

With this filter get rid of most of the noise and we make up for the absence of pressure information. Light pressure means bad data, and the bad data is discarded.

Median filter

...

Mean filter

...

Linear calibration filter

...

Final outcome

...