Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
  • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
  • G06F3/042—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
  • G06F3/0421—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means by interrupting or reflecting a light beam, e.g. optical touch-screen
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
  • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
  • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
  • G06F3/0416—Control or interface arrangements specially adapted for digitisers
  • G06F3/0418—Control or interface arrangements specially adapted for digitisers for error correction or compensation, e.g. based on parallax, calibration or alignment

Definitions

• Touch-sensitive devices may detect touch via several different mechanisms, including but not limited to optical, resistive, and capacitive mechanisms.
• Some optical touch-sensitive devices detect touch by capturing an image of a backside of a touch screen via an image sensor, and then processing the image to detect objects located on the screen.
• Such devices may include a light source within the device to illuminate the backside of the display screen such that objects on the screen reflect the incident light toward the image sensor, thereby allowing the object to be detected.
• one disclosed embodiment comprises integrating a first field of pixels in an image data frame for a different duration of ambient light exposure than a second field of pixels in the image data frame. Intensity data is read from the first field of pixels and the second field of pixels, and an ambient light value is determined for one or more pixels in the image data frame from the intensity data. The ambient light value then is used to adjusting one or more pixels of the data frame for ambient light.
• Figure 1 shows an embodiment of an optical touch-sensitive device.
• Figure 2 shows a process flow depicting an embodiment of a method of correcting for ambient light in an optical touch-sensitive device.
• Figure 3 shows a timing diagram depicting an embodiment of a method for integrating and reading an image sensor in an optical touch-sensitive device.
• Figure 4 shows a schematic depiction of intensity data of two fields of pixels in adjacent image frames captured according to the method of Figure 3.
• Figure 5 shows a schematic depiction of one embodiment of a method of determining an ambient light value from the intensity data of Figure 4.
• Figure 6 shows a schematic depiction of another embodiment of a method of determining an ambient light value from the intensity data of Figure 4.
• Figure 7 shows a schematic depiction of another embodiment of a method of determining an ambient light value from the intensity data of Figure 4.
• Figure 8 shows a schematic depiction of another embodiment of a method of determining an ambient light value from the intensity data of Figure 4.
• Figure 9 shows a schematic depiction of another embodiment of a method of determining an ambient light value from the intensity data of Figure 4.
• Figures 10A-D show a schematic depiction of another embodiment of a method of determining an ambient light value from the intensity data of Figure 4.
• Figure 11 shows a timing diagram of an ambient light frequency compared to a frame rate prior to correction of the frame rate.
• Figure 12 shows a timing diagram of an ambient light frequency compared to a frame rate after correction of the frame rate.
• Figure 13 shows a process flow depicting an embodiment of a method of correcting for ambient light in an optical touch-sensitive device.
• Figure 1 shows a schematic depiction of an embodiment of an interactive display device
• Interactive display device 100 utilizing an optical touch-sensing mechanism.
• Interactive display device 100 comprises a projection display system having an image source 102, and a display screen
• Image source 102 includes an optical or light source 108 such as a lamp
• Image source 102 also includes an image-producing element 110 such as the depicted LCD (liquid crystal display), an LCOS
• liquid crystal on silicon display
• DLP digital light processing
• Display screen 106 includes a clear, transparent portion 112, such as sheet of glass, and a diffuser screen layer 114 disposed on top of the clear, transparent portion 112.
• an additional transparent layer may be disposed over diffuser screen layer 114 to provide a smooth look and feel to the display surface. Further, in embodiments that utilize a LCD panel rather than a projection image source to display images on display screen 106, the diffuser screen layer 114 may be omitted.
• interactive display device 100 further includes an electronic controller 116 comprising memory 118 and a microprocessor 120. Controller 116 may further include a field programmable gate array (FPGA) 122, and/or any other suitable electronic components, including application-specific integrated circuits (ASICs) (not shown), digital signal processors (DSPs) (not shown), etc. configured to conduct one or more ambient light correction calculations, as described below.
• FPGA field programmable gate array
• FPGA 122 and/or other electronic components may also be provided as one or more separate devices in electrical communication with controller 116.
• memory 118 may comprise instructions stored thereon that are executable by the processor 120 to control the various parts of device 100 to effect the methods and processes described herein.
• the FPGA 122 also may be configured to perform one or more of the correction methods described in detail below.
• Diffuser screen layer 114 helps to avoid the imaging of objects that are not in contact with or positioned within a few millimeters of display screen 106, and therefore helps to ensure that only objects that are touching or in close proximity to display screen 106 are detected by image sensor 124.
• Image sensor 124 may include any suitable image sensing mechanism.
• Suitable image sensing mechanisms include but are not limited to CCD and CMOS image sensors. Further, the image sensing mechanisms may capture images of display screen 106 at a sufficient frequency to detect motion of an object across display screen 106. While the embodiment of Figure 1 shows one image sensor, it will be appreciated that more than one image sensor may be used to capture images of display screen 106.
• Image sensor 124 may be configured to detect reflected or emitted energy of any suitable wavelength, including but not limited to infrared and visible wavelengths. To assist in detecting objects placed on display screen 106, image sensor 124 may further include an illuminant 126 such as one or more light emitting diodes (LEDs) 126 configured to produce infrared or visible light. Light from illuminant 126 may be reflected by objects placed on display screen 106 and then detected by image sensor 124. The use of infrared LEDs as opposed to visible LEDs may help to avoid washing out the appearance of images projected on display screen 106.
• LEDs light emitting diodes
• Figure 1 also depicts an object 130 placed on display screen 106. Light from the illuminant 126 reflected by object 130 may be detected by image sensor 124, thereby allowing the object 130 to be detected on the screen.
• Object 130 represents any object that may be in contact with display screen 106, including but not limited to fingers, brushes, optically readable tags, etc.
• ambient light sources may emit light in the band passed by bandpass filter 127.
• the term "ambient light” is used herein to describe light other than light from the illuminant 126.
• Examples of such ambient light sources include but are not limited to broad-spectrum light sources such as sunlight, incandescent lamp light, etc. Such light may have a sufficient intensity at the bandpass frequencies that the ambient light is difficult to distinguish from reflected light from the illuminant 126. Therefore, such ambient may cause the interactive display device 100 to mistakenly identify ambient light as an object on the display screen 106.
• Various techniques may be used to cancel or otherwise correct for ambient light in an image captured by image sensor 124.
• the illuminant 126 which may be referred to as "local” light, could be strobed such that alternate frames are exposed to "ambient-only” and “ambient + local” light.
• This allows the ambient light intensity to be determined by subtracting the "ambient-only” frame from the "ambient + local” frame to correct for ambient.
• the local light is turned on only every other frame, this effectively cuts the frame rate of the device in half.
• Another potential technique is to utilize a separate sensor (possibly with an optical filter) configured to integrate only ambient light.
• an additional sensor may be expensive, and may be prone to errors due to the different positioning of the sensors in the device.
• Yet another potential technique may be to utilize an extremely bright local light source in combination with a band-pass filter to boost the intensity of reflected light relative to ambient light.
• this approach may be susceptible to failure where the ambient light exceeds some percentage of local light.
• Figure 2 shows a flow diagram of a method for correcting an image for ambient light that may help to avoid the problems found in the methods described above.
• Method 200 first comprises, at 202, integrating first and second fields of pixels in an image sensor for different durations of ambient light exposure. Generally, this also involves integrating the first and second fields of images for an equal (or approximately equal) duration of ambient + local exposure such that both fields have been exposed to a similar duration of local light but different durations of ambient light.
• the two fields may be interlaced fields (for example, odd/even rows or odd/even columns of pixels), or may have any other suitable spatial relationship. Furthermore, in some embodiments, three or more fields of pixels may be exposed to different intervals of ambient light.
• method 200 comprises, at 204, reading intensity data from the first and second fields of pixels in the image sensor, and determining, at 206, a measure of ambient light from the difference between the intensity data from the first and second fields of pixels.
• method 200 comprises, at 208, adjusting the image data to correct for the ambient light based upon the determined measure of ambient light.
• the image data may be adjusted only if it is first determined, at 210, if the ambient light measure is over a threshold value. Further, in some embodiments, it is first determined whether an object detected on the screen is in motion, as indicated at 212, and then an adjustment to make to the image data is selected at 214 based upon whether the object is determined to be in motion.
• method 200 allows a correction for ambient light to be made to image data without the use of an additional image sensor or other additional parts, and also without any loss of frame rate. Each of the processes shown in method 200 is described in more detail below.
• Figure 3 illustrates a timing diagram 300 showing one manner of operating the illuminant 126 (shown in Fig. 3 as an LED) and the image sensor 124 to accomplish this.
• the illuminant 126 is operated in an off/on strobe pattern shown at 302 such that it is in an "on" state for a portion of an image frame and an "off state for a portion of the image frame.
• the illuminant 126 is in an "on” state for somewhat less than one-half the period of one frame.
• the sensor is then globally exposed (i.e. all fields are exposed) such that it integrates each frame for a period tl during which the illuminant 126 is on (i.e. an "on” state interval) and a period t2 during which the illuminant 126 is off (i.e. an "off state interval).
• the exposure pattern is shown in Figure 3 at 304. [0033] Using the LED strobe pattern and image sensor integration pattern shown in
• Figure 3 the readout of the pixels of image data from the image sensor can be controlled such that different fields of pixels have different levels of ambient exposure in each frame.
• Figure 3 shows this readout pattern in the context of odd/even rows of pixels, but it will be understood that the fields of pixels may have any other suitable spatial relationship to one another. Further, while disclosed herein in the context of utilizing two fields of pixels, it will be appreciated that the concepts also may be utilized with three or more fields of pixels.
• the data for a first image data frame n is read out first from the odd frames, and then from the even frames.
• the image sensor pixels are reset to an unexposed state. Therefore, at the time the odd field of pixels of frame n is read out, the odd field pixels has integrated light for a period of tl (i.e. since the beginning of the last odd pixel readout shown in pattern 306).
• the even field of pixels of frame n has integrated light for a t2 period, then a tl period, and then another t2 period, for a total of tl+2(t2).
• the cumulative integration times for each frame are shown in Figure 3 at 310.
• the data for a next image data frame n+1 is read out first from the even frames, and then from the odd frames. Therefore, in frame n+1, the even field of pixels has integrated light for only a tl period at the time of readout, while the odd field of pixels has integrated light for a period of tl+2(t2). As can be seen in Figure 3, the odd fields have a greater ambient exposure than the even fields in frame n+1, while the even fields have a greater ambient exposure than the odd fields in frame n. It will further be noted that the image sensor integrated local light for a period of tl for each frame. Thus, each frame has image data that can be used to identify objects on display screen 106, thereby allowing the frame rate to be preserved.
• the periods tl and t2 may have any suitable lengths, and may be the same or different from one another. In some use environments, t2 may be shorter than tl to reduce the chance that the image sensor will saturate before reading, as saturation of the image sensor may lead to inaccurate calculations of ambient intensities. Further, where saturated pixels are detected, the length of tl and/or t2 may be modified to reduce the total integration time of a frame to avoid saturation in future pixels. Likewise, where intensities are low, tl and/or t2 may be increased to increase the amount of light integrated in future frames. Alternately or additionally, the gain on the image sensor may be adjusted dynamically to avoid saturation and/or to increase the response of the sensor to an amount of light exposure.
• tl and t2 may have similar lengths.
• ambient light is fluctuating in intensity (i.e. incandescent light fluctuating at twice a line frequency of 50 or 60 Hz)
• the average incident ambient light strength will be different during the t2 phases compared to the tl phases for at least some image frames (depending upon the frame rate compared to the fluctuation frequency).
• t2 may be adjusted to have an approximately equal length to tl when fluctuating ambient light is detected.
• each image frame is integrated for different periods of ambient light exposure. Further, each single field of pixels has different periods of ambient light exposure in adjacent image frames. These differences in intra-frame and inter-frame ambient light exposure may be utilized in various ways to correct an image frame for ambient light.
• Figure 4 illustrates how the readout from the sensor for the two image frames, which show a stationary scene, may appear when integrated and read according to the process shown in Figure 3.
• a simple stationary scene with no ambient light is shown at 402
• a 3x3 matrix of pixels from scene 402 is shown at 404.
• the images in Figure 4 have only three intensity levels, wherein the lightest pixels signify the most integrated light and the darkest pixels signify the least integrated light.
• the odd rows have a greater interval of ambient exposure than the even rows.
• the addition of this ambient pattern to the 3x3 scene yields the intensity data shown at 406.
• the even rows have a greater interval of ambient exposure than the odd rows.
• the addition of this ambient pattern to the 3x3 scene yields the intensity data shown at 408.
• the ambient light can be calculated for the odd rows by subtracting frame n from frame n-1 (as shown at 502), and for the even rows by subtracting frame n-1 from frame n (as shown at 504). Combining the ambient determined for the odd rows with the ambient determined for the even rows yields an overall ambient 506 for the 3x3 matrix.
• Figures 6-8 show examples of various methods that may be used to correct an image frame for ambient light with the image data shown in Figure 4. These figures are shown in the context of determining ambient for a single pixel at a time. This may allow different ambient calculation methods to be used for different pixels depending upon pixel- specific factors.
• an ambient light value at a pixel may be calculated as described above for Figure 5 by simply subtracting frame n-1 from frame n.
• ambient for pixels in the top and bottom rows of the 3x3 matrix may be determined simply by subtracting frame n from frame n-1.
• This method utilizes information from temporally adjacent frames but does not utilize information from spatially adjacent pixels. Therefore, the method illustrated in Figure 6 may be referred to herein as a "temporal-local" correction.
• the intensity at that pixel is the same as in an adjacent frame.
• the temporal-local correction may effectively halve the frame rate of the device. For this reason, this correction may be used for stationary objects.
• Figure 7 shows another example of a method for correcting an image frame for ambient light.
• the method shown in Figure 7 takes into account both temporal information (i.e. temporally adjacent image frames) and spatial information (i.e. spatially adjacent pixels) when calculating the ambient for a pixel. Therefore, the method shown in Figure 7 may be referred to as a "temporal-spatial" correction.
• the temporal-spatial correction shown in Figure 7 utilizes a weighted average intensity of the pixels in the sample matrix to determine an ambient value, wherein the center pixel is weighted more strongly (1/4) than the side pixels (1/8 each), which are in turn weighted more strongly than the corner pixels.
• the intensities of the pixels are multiplied by the shown weighting factors, the two frames are added, and then the value at each pixel in the matrix after the addition of the two frames is summed to yield the ambient intensity at the center pixel. Because spatial data is taken into account in addition to temporal data, the temporal-spatial correction allows a frame rate to be maintained.
• Figure 8 shows another example of a method for correcting an image frame for ambient light.
• the method of Figure 8 utilizes only spatial information, and not temporal information, in making the ambient correction.
• the correction is made entirely from a weighted average of intra- frame data, utilizing no inter- frame data. As depicted, this calculation may lead to slightly high values of ambient light, but can avoid calculation problems due to motion that may occur in methods that utilize temporal information.
• the touch-sensitive device may be able to detect objects without any problems caused by ambient. Therefore, before performing any of the above-described corrections (or any others), it may be determined whether there is any potentially problematic ambient by comparing the sum of the intensities in the first field in a frame to the sum of the intensities in the second field in the frame. Because the intensities in the two fields differ by the amount of ambient light integrated, if the sums are relatively close together, it can be determined that the ambient light levels are sufficiently low not to interfere with device operation, and correction for ambient may be omitted, as shown in Figure 9.
• Figures 10A-D illustrate another embodiment of a method for correcting for ambient.
• a 5x5 region of pixels in a current frame (frame n) and a single pixel in two prior frames (frames n-1, n-2) are considered for ambient correction.
• a 3x3 region of pixels, or any other suitable region of pixels, in a current frame may be considered in the ambient correction.
• a center of a current frame is compared to a pixel from frame n-2, which was read in the same field order. If the difference between these pixels exceeds a threshold amount, this indicates that motion may have occurred, and a "motion flag" for that pixel is set.
• the value of the motion flag is compared to motion flags for nearby pixels (for example, via a Boolean "OR" operation), and if the result is zero (i.e. frame n-2 and frame n look the same in a local region), then a temporal ambient correction is performed by determining difference between a current center pixel in frame n and the same pixel in frame n-1, as indicated in Figure 1OC.
• the determination of whether to utilize a 5x5 or a 3x3 pixel region for ambient correction may depend upon factors such as the resolution and stability of the image sensor. For example, a 3x3 region may yield a slightly noisier result, while a 5x5 region may blur the result slightly. Other region sizes may be used, including but not limited to a 1x3 region (which may be noisier than a 3x3 region).
• Some ambient sources may fluctuate in a periodic manner. For example, electric lighting generally fluctuates at a frequency of twice the line frequency, which may be either 50 or 60 Hz depending upon location. This is illustrated graphically in Figure 11. Where the frame rate of the device has a frequency other than the line frequency or 2x the line frequency, the ambient light detected by the image sensor will have a beat frequency detectable as a periodic variation in the ambient light level. Such variations in ambient light levels may cause problems when temporal information is used in correcting ambient. These problems may be pronounced where the ambient light fluctuations are large between adjacent frames, which may occur if a frame rate of 100 Hz is used in the presence of 120 Hz ambient, or vice versa.
• the frame rate of an optical touch-sensitive device may be set to equal the line frequency or 2x the line frequency.
• this setting may be stored in the system, derived from a local power source, or detected optically.
• the fluctuation may be detected optically by observing a beat frequency in the overall levels of detected light, or by monitoring frame- to-frame variation in the amount of total ambient light measured.
• Figure 13 shows a process flow of a method 1300 for performing an ambient light correction that takes into account the various factors described above.
• the method of Figure 13 may be performed on a pixel-by-pixel basis, or in any other suitable manner.
• Method 1300 first comprises, at 1302, acquiring image data frames, and then, at 1304, determining for an image data frame whether the global ambient is below a threshold value. This can be determined, for example, by subtracting the sum of the intensities of all pixels in a first field from the sum of all intensities of pixels in a second field, and determining if the result of the calculation is below a threshold value.
• method 1300 ends without performing any correction.
• method 1300 comprises, at 1306, determining whether any motion is perceived in the intensity data. This may be performed, for example, by subtracting the intensity value for the pixel in the current frame (frame n) from the intensity value for the same pixel in frame n-2 (as the same pixel in n-1 has a different ambient exposure time). If the difference between these intensity values is sufficiently small, then it can be determined that the intensity data contains no motion information. In this case, a temporal local correction that utilizes no spatial information may be performed, as indicated at 1308.
• the pixel contains motion data (as long as the frame rate has been corrected for any periodically fluctuating ambient light), and either a spatial or a temporal-spatial correction may be used, as indicated at 1310.
• a spatial correction may be used where all spatial variation in a frame can be corrected with other information in the frame.
• One example of a method for making this determination is as follows. First, if any pixels in row (i-1) of the sample matrix differ significantly from the pixels in the same column in row (i+1), there is spatial information that may be corrected via a temporal-spatial correction. Likewise, if any of the pixels in row (i) of the sample matrix minus the mean for row (i) differs significantly from the corresponding pixels in row (i-1) minus the mean for the pixels in row (i-1) then there is spatial information that may be corrected via a temporal- spatial correction.
• a spatial correction may be used.
• either a spatial or temporal-spatial may be used exclusive of the other where motion information is contained in a frame.
• the correction calculations and calculation selection routine described above may be performed in any suitable manner.
• an FPGA as shown at 122 in Fig. 1
• the best ambient value for each pixel in a frame may be selected based upon the specific temporal and local characteristics of that pixel, as described for method 1200.
• the best ambient calculation for a pixel may be determined before performing the correction, such that only one correction is performed for each pixel.
• the lengths of the integration periods tl and/or t2 may be adjusted to prevent saturation in future frames. Further, a frame in which saturation is detected may also be processed in a manner to correct for the saturation. As an example, if saturated pixels are observed, it can be assumed that the saturated pixels are directly exposed to ambient light (as reflected light from the illuminant is generally not sufficiently intense to cause saturation). Therefore, in this situation, all light in the saturated region can be deemed to be ambient. Where saturation exists, a noise margin may exist around the saturated pixels.
• a minimum ambient level may be determined for this region by setting one possible ambient using the near-saturation test, and another using a computed ambient as described above. The higher of these two values may then be used as the value to be subtracted from the image when correcting pixels in this region for ambient.

Landscapes

• Engineering & Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Human Computer Interaction (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Position Input By Displaying (AREA)
• Liquid Crystal (AREA)
• Image Input (AREA)

Priority Applications (5)

Applications Claiming Priority (2)

Publications (2)

Family

ID=40522867

Family Applications (1)

Country Status (7)

Cited By (1)

Families Citing this family (67)

Citations (3)

Family Cites Families (26)

• 2007
  • 2007-10-05 US US11/868,466 patent/US8004502B2/en not_active Expired - Fee Related
• 2008
  • 2008-10-01 CN CN2008801104246A patent/CN101815979B/zh not_active Expired - Fee Related
  • 2008-10-01 EP EP08836179.5A patent/EP2206033B1/en not_active Not-in-force
  • 2008-10-01 WO PCT/US2008/078515 patent/WO2009046154A2/en not_active Ceased
  • 2008-10-01 KR KR1020107007059A patent/KR101465835B1/ko not_active Expired - Fee Related
  • 2008-10-01 CA CA2698623A patent/CA2698623C/en not_active Expired - Fee Related
  • 2008-10-01 JP JP2010528117A patent/JP5134688B2/ja not_active Expired - Fee Related

Patent Citations (3)

Also Published As

Similar Documents

Legal Events