JP2010055578A - Image analyzing device, image analysis method, imaging apparatus, image analysis program and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately calculate the intensity of external light in the surroundings of a pointing object for pointing an imaging screen having a plurality of imaging sensors. <P>SOLUTION: A touch position detection device 10 includes an external light intensity calculation part 3 for analyzing an image captured by a photo-sensor 12 to calculate the external light intensity which is the intensity of light in the surroundings of the light made incident on a portion of the photo-sensor 12. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image analysis apparatus that analyzes a captured image of a pointing object that indicates an imaging screen having a plurality of imaging sensors, an imaging apparatus that includes the image analysis apparatus, and an image analysis method.

  2. Description of the Related Art In recent years, displays have been developed that incorporate a photosensor in each pixel of a display such as an LCD (Liquid Crystal Display) or an organic EL display, and that can simultaneously display as well as display. Technology development that uses this optical sensor built-in display to capture an image of a pointing member such as a user's finger or a touch pen that indicates an arbitrary position on the surface of the optical sensor built-in display, and realizes a touch panel based on the image It has also been made. For example, Patent Document 1 describes a technique for realizing a touch panel using an optical sensor built-in LCD. Hereinafter, the user's finger and the pointing member are collectively referred to as a pointing object.

  As described above, a technique for realizing a touch panel using an optical sensor built-in LCD has been developed. However, the image acquired by the optical sensor varies greatly depending on the intensity of external light that is light around the finger (or pointing member) touched on the touch panel and the direction in which the external light is incident. For this reason, in order to accurately distinguish between touch and non-touch from such a rapidly changing image, it is necessary to perform image analysis in consideration of the influence of external light.

  Therefore, in the invention described in Patent Document 2, the intensity of external light is acquired from a user input or an external light sensor, and the image processing method is switched based on whether the external light intensity exceeds a certain threshold. To adapt to changes in external light.

The invention described in Patent Document 3 employs a method of determining the intensity of external light by determining the ratio of a black image and a white image in the image and switching the image processing method.
JP 2006-244446 A (published September 14, 2006) JP 2007-183706 A (published July 19, 2007) JP 2004-318819 A (published on November 11, 2004)

  However, the configurations described in Patent Documents 2 and 3 have a problem that the intensity of external light cannot be accurately determined.

  Specifically, in the invention of Patent Document 2, since the external light sensor for detecting external light is located away from the image acquisition optical sensor, the intensity of external light incident on the image acquisition optical sensor. Cannot be calculated accurately.

  In the invention of Patent Document 3, the intensity of external light is only roughly determined from the ratio of the black image and the white image included in the captured image, and the intensity of external light cannot be accurately calculated. .

  Furthermore, Patent Documents 2 and 3 disclose a technique for processing a pointing object image that indicates a position on a touch panel using the calculated intensity of external light to improve touch and non-touch identification accuracy. Absent.

  The present invention has been made to solve the above-described problems, and an object thereof is to provide an image analysis apparatus, an imaging apparatus, and an image analysis method capable of accurately calculating the intensity of external light. . Another object of the present invention is to improve the identification accuracy of touch and non-touch by processing the image of the pointing object using the intensity of external light.

  In order to solve the above problems, an image analysis apparatus according to the present invention captures an image of a pointing object that is in contact with or not in contact with an imaging screen having a plurality of imaging sensors. An image analysis apparatus for analyzing a captured image, wherein an acquisition unit that acquires the captured image and a surrounding of the pointing object that is incident on a part of the imaging sensor by analyzing the captured image acquired by the acquisition unit And an external light intensity calculating means for calculating the external light intensity which is the intensity of the light.

  In order to solve the above-described problem, the image analysis method according to the present invention captures an image of a pointing object that is in contact with or not in contact with an imaging screen having a plurality of imaging sensors. An image analysis method in an image analysis apparatus for analyzing a captured image, wherein the captured image is acquired, and the captured image acquired in the acquisition step is analyzed to be incident on a part of the imaging sensor And an external light intensity calculating step of calculating an external light intensity that is an intensity of light around the pointing object.

  When determining whether the pointing object is in contact with the imaging screen using a captured image in which the pointing object image is darker than the background, the background pixel value, in other words, around the pointing object It is preferable to accurately calculate the external light intensity, which is the intensity of the light, and to analyze the captured image using the external light intensity.

  According to the above configuration, the acquisition unit acquires a captured image of the pointing object, and the external light intensity calculation unit analyzes the captured image, thereby obtaining light around the pointing object that is incident on a part of the imaging sensor. The intensity of external light, which is the intensity of That is, the external light intensity calculation means calculates the external light intensity using the captured image itself captured by the image sensor.

  Therefore, the external light intensity can be calculated more accurately with a simple configuration without providing a sensor for measuring the external light intensity.

  The external light intensity calculating unit selects at least a part of pixels from a plurality of pixels included in the captured image acquired by the acquiring unit and arranges the selected pixels in descending order of pixel values. It is preferable to set the pixel value of the pixels in the order of the above-mentioned external light intensity.

  In the case of a captured image in which the image of the pointing object is darker than the background, the external light intensity is preferably calculated from a group of pixels having a relatively high pixel value among the pixels of the captured image. However, when the highest pixel value is adopted as the external light intensity, it is likely to be affected by noise or the like, and there is a high possibility that the external light intensity cannot be accurately calculated.

  According to said structure, an external light intensity | strength calculation means selects at least one pixel from the some pixel contained in the captured image which the acquisition means acquired, and arranges the pixel value of the selected pixel in order of high order. Then, the external light intensity calculation means employs the pixel values of the pixels in a predetermined order (for example, 10th) from the top as the external light intensity.

  Therefore, the external light intensity can be appropriately calculated by appropriately setting the predetermined order.

  Moreover, it is preferable that the predetermined order corresponds to a number less than 10% of the total number of selected pixels.

  According to the above configuration, the external light intensity calculation means employs the pixel value of the pixel in the order corresponding to a number less than 10% of the total number of selected pixels as the external light intensity. For example, if the total number of selected pixels is 1000 and the predetermined rank is a rank corresponding to 2% of the total number of selected pixels, the predetermined rank is the 20th.

  Therefore, it is possible to appropriately select a pixel having an appropriate pixel value from the captured image in order to calculate the external light intensity.

  Further, the image analysis device deletes the image of the pointing object when the pointing object is not in contact with the imaging screen from the captured image, based on the external light intensity calculated by the external light intensity calculating unit. A reference value calculating means for calculating a reference value of the pixel value for the pixel, and a pixel having a pixel value equal to or greater than the reference value calculated by the reference value calculating means among the pixels included in the captured image acquired by the acquiring means It is preferable to further comprise image processing means for replacing the value with the reference value.

  According to the above configuration, the reference value calculation means calculates the reference value of the pixel value for deleting the image of the pointing object from the captured image when the pointing object is not in contact with the imaging screen. It calculates based on the external light intensity calculated by the calculating means. Then, the image processing means replaces a pixel value of a pixel having a pixel value equal to or higher than the reference value among the pixels included in the captured image with the reference value.

  Therefore, when the pointing object is not in contact with the imaging screen, the pixel value of the pixel forming the image of the pointing object and the pixel value of the pixel corresponding to the background are all reduced to the reference value, and the uniform background Is formed. Therefore, the image of the pointing object when the pointing object is not in contact with the imaging screen is deleted from the captured image.

  As a result, an image of the pointing object that is unnecessary for the recognition of the pointing object when the pointing object is not in contact with the imaging screen can be deleted from the captured image, and the recognition accuracy of the pointing object can be improved.

  Further, the image analysis device includes a feature region extraction unit that extracts a feature region indicating a feature of the image of the pointing object from a captured image acquired by the acquisition unit, and an external light intensity calculated by the external light intensity calculation unit. A reference value calculating means for calculating a reference value of a pixel value for determining whether or not the characteristic region is derived from an image of a contact portion of the pointing object that has touched the imaging screen; and the reference value calculating means An image of a contact portion of the pointing object that has touched the imaging screen from the feature area that has not been deleted by the deletion means that deletes the feature area derived from a pixel having a pixel value equal to or greater than the reference value calculated by It is preferable to further comprise a position calculating means for calculating the position of.

  According to the above configuration, the feature region extraction unit extracts a feature region indicating the feature of the image of the pointing object from the captured image. The reference value calculation means is a pixel for determining whether or not the feature region is derived from the image of the contact portion of the pointing object that has touched the imaging screen, based on the external light intensity calculated by the external light intensity calculation means Calculate the reference value. The deletion unit deletes a feature region derived from a pixel having a pixel value equal to or higher than the reference value from the feature region extracted by the feature region extraction unit, and the position calculation unit deletes the feature region that has not been deleted by the deletion unit. Then, the position of the image of the contact portion of the pointing object that has touched the imaging screen is calculated.

  Therefore, it is possible to delete a feature region that is unnecessary for recognition of the pointing object and is derived from the image of the pointing object when the pointing object is not in contact with the imaging screen, and the accuracy of recognition of the pointing object can be improved.

  Further, it is preferable that the reference value calculation means calculates the reference value using a predetermined mathematical expression, and uses a plurality of types of the predetermined mathematical expressions according to the external light intensity calculated by the external light intensity calculation means. .

  According to said structure, according to the change of external light intensity, the reference value suitable for the external light intensity can be calculated. For example, the reference value calculation means calculates the reference value using the first mathematical expression when the external light intensity is in the first range, and the second mathematical expression when the external light intensity is in the second range. Can be used to calculate the reference value.

  Further, an image analysis program for operating the image analysis apparatus, an image analysis program for causing a computer to function as each of the means, and a computer-readable recording medium on which the image analysis program is recorded are also included in the technical scope of the present invention. include.

  In order to solve the above-described problem, an imaging apparatus according to the present invention includes an imaging screen having a plurality of imaging sensors, and captures an image of a pointing object that is in contact with or not in contact with the imaging screen. An image pickup apparatus for picking up an image by the image pickup sensor, including the image analysis apparatus, wherein the acquisition unit acquires a picked-up image picked up by the image pickup sensor.

  With the above configuration, a captured image can be acquired by a plurality of imaging sensors, and the external light intensity can be calculated from the acquired captured image more easily and more accurately than in the past.

  The imaging apparatus preferably further includes sensitivity setting means for setting the sensitivity of the imaging sensor based on the external light intensity calculated by the external light intensity calculation means.

  With the above configuration, a captured image can be acquired with appropriate sensitivity, and a captured image suitable for recognition of the pointing object can be acquired.

  The sensitivity setting means preferably sets the sensitivity of the imaging sensor in a stepwise manner, and when the external light intensity becomes a predetermined reference value or lower, the sensitivity of the imaging sensor is preferably increased in multiple steps at a time.

  According to the above configuration, when the external light intensity becomes equal to or lower than the predetermined reference value, the sensitivity of the image sensor increases at a plurality of stages at a time. Therefore, appropriate imaging can be performed more quickly than when the sensitivity is gradually increased.

  It is preferable that the sensitivity setting unit sets the sensitivity of the imaging sensor so that the external light intensity calculated by the external light intensity calculation unit is not saturated.

  When the external light intensity is saturated, the accuracy of recognizing the image of the pointing object is significantly reduced. Note that the saturation of external light intensity means that the external light intensity exceeds the range of light intensity that can be quantitatively detected by the imaging sensor.

  With the above configuration, a captured image can be acquired with a sensitivity that does not saturate the external light intensity, and a captured image suitable for recognition of the pointing object can be acquired.

  The sensitivity setting means changes the sensitivity from the first sensitivity to the first sensitivity when the external light intensity reaches a first reference value in a state where the sensitivity of the imaging sensor is set to the first sensitivity. The sensitivity is lowered from the second sensitivity to the first sensitivity when the outside light intensity is lowered to the second reference value in a state where the sensitivity of the imaging sensor is set to the second sensitivity. The second reference value may be smaller than the external light intensity when the actual external light intensity corresponding to the first reference value is detected by the imaging sensor set to the second sensitivity. preferable.

  According to the above configuration, external light intensity (external light intensity calculated by the external light intensity calculating means) that increases the sensitivity of the image sensor to the first sensitivity in a state where the sensitivity of the image sensor is set to the second sensitivity. The second reference value, which is a reference of the external light intensity, is an external light intensity (external light intensity calculated by the external light intensity calculation means) that reduces the sensitivity of the image sensor to the second sensitivity in a state where the sensitivity of the image sensor is set to the first sensitivity. External light intensity (external light calculated by the external light intensity calculating means) when the actual external light intensity corresponding to the first reference value that is the reference of the light intensity is detected by the imaging sensor set to the second sensitivity Less than (strength).

  Therefore, when the sensitivity of the image sensor is decreased from the first sensitivity to the second sensitivity, the external light intensity calculated by the external light intensity calculation means immediately reaches the second reference value, and the sensitivity of the image sensor is reduced. The possibility of switching back to the first sensitivity is reduced. Accordingly, it is possible to prevent the sensitivity of the imaging sensor from frequently switching from the first sensitivity to the second sensitivity or from the second sensitivity to the first sensitivity, only by a slight change in the external light intensity.

  As described above, the image analysis apparatus according to the present invention includes an acquisition unit that acquires a captured image, and an analysis of the captured image that is acquired by the acquisition unit. It is a structure provided with the external light intensity | strength calculation means which calculates the external light intensity which is an intensity | strength of light.

  The image method according to the present invention is an acquisition step of acquiring a captured image, and the intensity of light around the pointing object incident on a part of the imaging sensor by analyzing the captured image acquired in the acquisition step. And an external light intensity calculation step for calculating the external light intensity.

  Therefore, there is an effect that the external light intensity can be calculated more accurately with a simple configuration.

[Embodiment 1]
An embodiment of the present invention will be described below with reference to FIGS. Here, as one of the embodiments of the present invention, an image of a pointing member (generally referred to as a pointing object) such as a user's finger or a touch pen that points to an arbitrary position on the touch panel is picked up, and an instruction is made based on the image. The touch position detection apparatus 10 that detects the position indicated by the object will be described. Note that the touch position detection device can also be expressed as a display device, an imaging device, an input device, or an electronic device.

(Configuration of Touch Position Detection Device 10)
FIG. 1 is a block diagram illustrating a configuration of a touch position detection apparatus 10 according to the present embodiment. As illustrated in FIG. 1, the touch position detection device (image analysis device, imaging device) 10 includes a touch panel unit (imaging unit) 1, an image analysis unit (image analysis device) 9, and an application execution unit 15.

  The image analysis unit 9 includes an image adjustment unit 2, an external light intensity calculation unit (external light intensity calculation unit) 3, an optimum sensitivity calculation unit (sensitivity setting unit) 4, and a touch / non-touch pixel value calculation unit (reference value calculation unit). ) 5, a recognition unnecessary information deletion unit (image processing unit) 6, a feature amount extraction unit (feature region extraction unit) 7, and a touch position detection unit (position calculation unit) 8.

  The touch panel unit 1 includes an optical sensor built-in LCD 11 including an optical sensor (imaging sensor) 12 as an imaging element, an AD (analog / digital) converter 13, and a sensitivity adjustment unit 14.

  Since the optical sensor built-in LCD (liquid crystal panel / display) 11 includes the optical sensor 12, not only display but also imaging can be performed. Therefore, the optical sensor built-in LCD 11 functions as an imaging screen that captures an image (hereinafter, referred to as a captured image (or sensor image)) including an image of the pointing object that touches the surface of the optical sensor built-in LCD 11 as a touch panel.

  One photosensor 12 is provided for each pixel of the LCD 11 with a built-in photosensor. In other words, the photosensors 12 are arranged in a matrix within the photosensor built-in LCD 11. However, the arrangement pattern and the number of the optical sensors 12 are not limited to those described above, and may be changed as appropriate.

  The signal acquired by the optical sensor 12 is digitized by the AD converter 13 and then output to the image adjustment unit 2.

  The image adjustment unit 2 performs a calibration process for adjusting the gain and offset of the captured image captured by the touch panel unit 1 and outputs the adjusted captured image to the external light intensity calculation unit 3 and the recognition unnecessary information deletion unit 6. . In the following description, it is assumed that a grayscale image is output with an accuracy of 8 bits and 256 gradations. The image adjustment unit 2 also functions as an acquisition unit that acquires a captured image from the touch panel unit 1. The image adjustment unit 2 may store the acquired captured image or the adjusted captured image in the storage unit 40.

  The external light intensity calculation unit 3 calculates the external light intensity from the captured image output from the image adjustment unit 2, and outputs the calculated external light intensity to the optimum sensitivity calculation unit 4 and the touch / non-touch pixel value calculation unit 5. To do. Details of the processing in the external light intensity calculation unit 3 will be described later. The external light intensity is the intensity of light around the pointing object (imaging target) incident on the optical sensor 12. The external light intensity calculation unit 3 calculates the pixel value (luminance value) of the pixel in the captured image reflecting the external light intensity as the external light intensity.

  The optimum sensitivity calculation unit 4 calculates the optimum sensitivity of the optical sensor 12 for recognizing the pointing object according to the outside light intensity calculated by the outside light intensity calculation unit 3, and outputs it to the sensitivity adjustment unit 14. Details of processing in the optimum sensitivity calculation unit 4 will be described later.

  The sensitivity adjustment unit 14 adjusts the sensitivity of each optical sensor 12 to the optimum sensitivity output from the optimum sensitivity calculation unit 4.

  The pixel value calculation unit 5 between touch and non-touch is a reference value (pixel value between touch and non-touch) for deleting information in the captured image unnecessary for recognition of the pointing object in the recognition-unnecessary information deletion unit 6. ) Is calculated. More specifically, the touch / non-touch pixel value calculation unit 5 uses the external light intensity calculated by the external light intensity calculation unit 3 to indicate the instruction when the pointing object is not in contact with the optical sensor built-in LCD 11. A pixel value between touch and non-touch, which is a pixel reference value for deleting the object image from the captured image, is calculated. Details of processing in the pixel value calculation unit 5 between touch and non-touch will be described later.

  The recognition-unnecessary information deletion unit 6 acquires the touch / non-touch pixel value calculated by the touch / non-touch pixel value calculation unit 5, and out of the pixels included in the captured image, the pixel value between the touch / non-touch pixel values is greater than or equal to. By replacing the pixel value of the pixel having the pixel value with the pixel value between touch and non-touch, information in the captured image that is unnecessary for the recognition of the pointing object is deleted.

  The feature amount extraction unit 7 obtains, for each pixel of the captured image, a feature amount (edge feature amount) indicating the feature of the pointing object by edge detection processing such as a Sobel filter from the captured image processed by the recognition unnecessary information deletion unit 6. Extract. In other words, the feature amount extraction unit 7 extracts a feature region indicating the feature of the image (shape) of the pointing object from the captured image. The feature amount extraction unit 7 extracts the feature amount of the pointing object, for example, as a feature amount including an 8-direction vector indicating the gradient (gradation) direction of the pixel value in the eight directions around the pixel of interest. The feature amount extraction processing in the feature amount extraction unit 7 is not particularly limited as long as it can detect the shape (particularly the edge) of the pointing object. The feature amount extraction unit 7 associates the extracted feature amount with the pixel (feature region) from which the feature amount has been extracted, and outputs it to the touch position detection unit 8.

  The touch position detection unit 8 specifies the touch position by pattern matching with the feature region indicating the feature amount extracted by the feature amount extraction unit 7. Any pattern matching method may be used as long as the position of the image of the pointing object can be appropriately specified. The touch position detection unit 8 outputs coordinates indicating the specified touch position to the application execution unit 15.

  The application execution unit 15 executes an application corresponding to the coordinates using the coordinates output by the touch position detection unit 8, or performs processing corresponding to the coordinates in a specific application. Any application may be executed in the application execution unit 15.

(Details of processing in the external light intensity calculation unit 3)
Next, the significance and contents of the processing in the external light intensity calculation unit 3 will be described in detail.

  First, the reason for calculating the external light intensity from the captured image captured by the optical sensor 12 of the optical sensor built-in LCD 11 will be described. For example, an illuminance sensor can be provided outside the optical sensor built-in LCD 11 to calculate the external light intensity. However, the reaction differs between the optical sensor 12 that obtains a captured image and the illuminance sensor depending on the wavelength of light and the like. Therefore, there is a possibility that the external light intensity calculated by the illuminance sensor and the external light intensity calculated by the optical sensor 12 may be greatly different.

  For example, the light for which the illuminance sensor indicates 1000 Lux for indoor light and the light for which the illuminance sensor indicates 1000 Lux for sunlight are the same 1000 Lux for the illuminance sensor. However, for the illuminance sensor, even if the light has the same 1000 Lux, the wavelength region included in the light is greatly different, and therefore the result is different from the case where the light intensity is calculated by the optical sensor 12. Therefore, in order to calculate the intensity of light incident on the optical sensor 12, it is preferable to calculate from the image acquired by the optical sensor 12.

  In addition, it is possible to equip the outside with a sensor having the same characteristics as the light sensor 12 instead of the illuminance sensor, but the place and environment such as the light hit condition slightly change between the light sensor 12 and the external sensor. Therefore, in order to calculate the external light intensity, it is most preferable to calculate the external light intensity with the optical sensor 12.

  Therefore, the external light intensity calculation unit 3 calculates the external light intensity from the captured image captured by the optical sensor 12 (more precisely, the captured image output from the image adjustment unit 2).

  2A and 2B are diagrams for explaining the processing contents of the external light intensity calculation unit 3. FIG. 2A is a diagram illustrating pixels used for calculating the external light intensity, and FIG. 2B is an external light intensity calculation unit. FIG. 3 is a diagram illustrating a relationship between a histogram created by No. 3 and external light intensity.

  The external light intensity calculation unit 3 selects at least some of the plurality of pixels included in the captured image output from the image adjustment unit 2 and sets the selected pixels in order from the highest pixel value. The pixel values of the pixels in the order of are considered as the external light intensity.

  That is, the external light intensity calculation unit 3 creates a histogram indicating the relationship between the pixel values arranged in descending order and the number of pixels having the pixel value for a plurality of pixels included in the captured image. This histogram is preferably created from the pixel values of all the pixels of the captured image. However, from the viewpoint of cost and processing speed, as shown in FIG. 2A, the captured image is not obtained from all the pixels constituting the captured image (in other words, output values from all the optical sensors 12). Pixels selected at regular intervals (pixels indicated by black dots denoted by reference numeral 25 in FIG. 2A) (sample points) may be used with a certain interval from the pixel values. Pixels denoted by reference numeral 26 are pixels that are not sampled.

  When the captured image is acquired with the finger placed on the touch panel unit 1 in an environment with different external light intensity, the brightness of the captured image is different as shown in FIG. In contrast, an image acquired in an environment with strong external light has a shape in which a region in which a histogram is distributed is pulled by a portion having a higher pixel value. In FIG. 2B, A is the external light intensity obtained from the sensor image (3), B is the external light intensity obtained from the sensor image (2), and C is obtained from the sensor image (1). Outside light intensity.

  Next, in the process of calculating brightness from the obtained histogram, the pixel value of the histogram is counted in order from the highest pixel value, and the pixel value when the count reaches a certain percentage of the number of sample points that created the histogram is calculated. Adopted as the value of external light intensity.

  The reason for calculating the pixel value of the upper few percent of the histogram as the external light intensity will be described. FIG. 3 is a diagram for explaining the significance of processing in the external light intensity calculation unit 3. For example, as shown in FIG. 3, a change occurs in an image acquired depending on how to place a finger or a hand even under the same external light intensity condition. The sensor image (1) in FIG. 3 is a captured image when a finger is placed from the left of the panel toward the left end of the panel, and the sensor image (2) in FIG. 3 is a finger from the left of the panel toward the right end of the panel. It is a picked-up image at the time of putting.

  In both cases, the intensity of the external light is the same, but since the image differs depending on where the finger is placed on the panel, the created histograms are different as shown in FIG. In this case, if the external light intensity is calculated accurately, for example, if the upper 5% portion of the histogram is the external light intensity, the difference between the sensor image (1) in FIG. 3 and the sensor image (2) in FIG. Although it hardly occurs, when the upper 50% portion is calculated as the external light intensity, there is a large difference in the external light intensity value calculated between the sensor image (1) in FIG. 3 and the sensor image (2) in FIG. Will occur.

  For the above reasons, when calculating the external light intensity from the histogram, it is possible to reduce the fluctuation of the calculated value due to the place where the finger or the hand is placed as much as possible by calculating the external light intensity from the upper few%.

  However, for example, calculation from an extremely high value of the histogram such as the upper 0.1% causes a decrease in accuracy due to a defect in the pixel value, etc., so it is preferable to calculate the external light intensity from about one digit%. . That is, when the pixels selected from the captured image are arranged in descending order of the pixel values, the rank of the pixels having the pixel value adopted as the external light intensity is a number that is less than 10% of the total number of the selected pixels. It is preferable to correspond to these.

  FIG. 4 is a diagram for explaining a modification example of processing in the external light intensity calculation unit 3.

  The method of obtaining the external light intensity is not limited to the histogram, but is limited to a region (marked with reference numerals 31 to 35) where sample points are taken in the captured image, for example, as shown in FIG. Alternatively, the average value of the pixel values of the pixels (sample points) in each of the regions 31 to 35 may be obtained, and the value having the highest average value may be adopted as the external light intensity. Note that white pixels indicated by reference numeral 36 are non-sample points.

(Details of processing in the touch / non-touch pixel value calculation unit 5)
Before explaining the pixel value calculation unit 5 between touch and non-touch, what kind of image the captured image of touch and non-touch acquired by the optical sensor 12 will be described with reference to FIG. (A), (c), (e), and (g) of FIG. 5 are diagrams showing ambient brightness when the pointing object is imaged. (B), (d), (f), and (f) h) is a diagram illustrating an example of a captured image.

  In a conventional optical sensor panel, light from a backlight is reflected on the finger pad at a portion where the panel is touched with a finger, and reflected light is incident on the sensor. As shown in FIG. 5B, when the external light is weaker than the reflected light of the finger pad, the finger pad becomes a brighter white circle than the background, and the reflected light of the finger pad is shown in FIG. When the outside light is stronger, the finger pad becomes darker and darker than the background. The same applies to a pen.

  As another method, by changing the configuration of the touch position detection device 10, the background pixel value is always brighter than the pixel value of the pixel forming the finger pad image (hereinafter referred to as the finger pad pixel value). You can create a situation. FIG. 6 is a cross-sectional view showing a modified example of the touch panel unit 1. As shown in FIG. 6, a transparent substrate 16 and an elastic film 17 may be provided on the surface of the LCD 11 with a built-in optical sensor, and a backlight 19 may be provided on the opposite side.

  Since the elastic film 17 has the protrusions 17 a, an air layer 18 is formed between the transparent substrate 16 and the elastic film 17. The air layer 18 reflects light from the backlight 19 when no pressure is applied to the surface of the transparent substrate 16. On the other hand, when pressure is applied, the reflection at the air layer 18 is lost, so the reflectance is low. With this method, the pixel value (the pixel value under the finger pad) of the part always touched with the finger can be made darker than the background pixel value.

  An example of a captured image when the elastic film 17 is provided is shown in FIG. Due to the influence of the elastic film 17, even when the surroundings are in a completely dark state, the portion pressed with the finger becomes darker than the background portion, so that the portion pressed with the finger can always be kept dark. Similarly, when the outside light becomes brighter, the pressed part is always darker. The same applies to a pen.

  In the following description, a case in which the pixel value under the finger pad is smaller than the background pixel value in the touch panel unit 1 having the elastic film 17 will be described.

  FIG. 8 shows what the touched and non-touch captured images acquired by the optical sensor 12 are. When external light directly enters the optical sensor 12 with no finger or the like placed thereon, an image 41 (background only image) that does not include a finger image is obtained as in condition (1) in FIG. It is done. As shown in condition (2) of FIG. 8, the finger is brought close to the optical sensor built-in LCD 11, but when not touched, the image 42 is like a shadow 44 of the finger being reflected lightly. In a state where the finger is completely pressed against the optical sensor built-in LCD 11 as in condition (3) in FIG. 8, an image 43 in which the shadow 45 portion is darker than the image 42 is acquired.

  Next, the pixel value of the image 41 in FIG. 8 (equivalent to the external light intensity obtained by the external light intensity calculation unit 3), the pixel value under the finger pad when not touching in the image 42 in FIG. 8, and the image in FIG. FIG. 9 shows the relationship between the pixel value under the finger pad at the time of touch at 43. As shown in FIG. 9 (a), the pixel value of the background pixel (indicated by reference numeral 51), the lower fingertip pixel value at non-touch (indicated by reference numeral 52), and the lower fingertip pixel value at the time of touching (indicated by reference numeral 53). (Shown) increases as the external light intensity increases (becomes brighter). The captured images at that time are shown in FIG.

  Since the pixel value of the background pixel is always higher than the pixel value under the finger pad when not touching, and the pixel value under the finger pad when not touching is always higher than the pixel value under the finger pad when touching. There is always a gap in pixel value difference between the pixel value under the finger pad when not touching and the pixel value under the finger pad when touching.

  If this relationship holds, as shown in FIG. 10, the pixel value under the finger pad when not touched (indicated by reference numeral 52) and the pixel value under the finger pad when touched (indicated by reference numeral 53). If a threshold value (indicated by reference numeral 54) can be set for the pixel value, pixel values equal to or greater than the threshold value can be deleted as information unnecessary for recognition, and recognition accuracy can be improved.

  Therefore, the touch / non-touch pixel value calculation unit 5 is a touch that is a pixel value between the fingertip lower pixel value at the time of non-touch and the finger lower pixel value at the time of touch according to a change in external light intensity. -Calculate the non-touch pixel value dynamically.

  However, at the time of online processing (when the user actually touches the LCD 11 with a built-in optical sensor), it is impossible to obtain the finger pad pixel value at the time of touch and the finger pad pixel value at the time of non-touch. Preset a formula that indicates the relationship between the external light intensity that can be obtained in the field and the pixel value between touch and non-touch, and calculate the pixel value between touch and non-touch by substituting the external light intensity into this formula. To do.

  This formula is shown in the following formula (1), and the pixel value (T) between touch and non-touch is calculated by substituting the external light intensity (A) calculated by the external light intensity calculation unit 3 into this formula. be able to.

T = AX (1)
In the above equation (1), X is a constant determined in advance. In order to obtain this X, first, a value of N satisfying the following equation (2) is set.

T = (B + C) / N (2)
In the above equation (2), B is the pixel value under the finger pad when not touching, and C is the pixel value under the finger pad when touching. N is an arbitrary numerical value such that T is between B and C.

  Then, X satisfying the following (3) is calculated based on the above equation (2).

T = AX = (B + C) / N (3)
During online processing, the pixel value calculation unit 5 between touch and non-touch calculates T by substituting A obtained by the external light intensity calculation unit 3 for each frame into the equation (1).

  In addition, the said (1) Formula should just be stored in the memory | storage part (For example, the memory | storage part 40) which can use the pixel value calculation part 5 between touch and non-touch.

  FIGS. 11A and 11B are graphs respectively showing other examples of changes in the pixel values under the finger pad when touching and when not touching, corresponding to changes in environmental illuminance.

  When obtaining the pixel value between touch and non-touch, when the characteristics of the touch and non-touch pixel values change at the branch point as shown in FIGS. 11A and 11B, the branch point (outside The formula for calculating the pixel value between touch and non-touch may be changed at the point where the light intensity reaches a certain pixel value).

  That is, two different mathematical formulas for obtaining the pixel value between touch and non-touch are stored in the storage unit 40, and the pixel value calculation unit 5 between touch and non-touch is an external value calculated by the external light intensity calculation unit 3. Two different mathematical expressions may be used before and after the light intensity reaches a predetermined value. In other words, the touch / non-touch pixel value calculation unit 5 uses a plurality of types of mathematical formulas for obtaining the touch / non-touch pixel value according to the external light intensity calculated by the external light intensity calculation unit 3. Also good.

  The two different types of mathematical formulas are, for example, mathematical formulas having different values of the constant X in the formula (1).

(Details of processing in the recognition unnecessary information deletion unit 6)
The pixel value between touch and non-touch determined in this way is output to the recognition unnecessary information deletion unit 6. The recognition-unnecessary information deleting unit 6 converts the pixel value of the captured image pixel having a pixel value equal to or larger than the touch / non-touch pixel value obtained by the touch / non-touch pixel value calculation unit 5 to the touch / non-touch pixel value. By replacing, information unnecessary for recognition of the pointing object is deleted.

  FIG. 12 is a diagram for explaining processing in the recognition unnecessary information deleting unit 6. At the bottom of the figure, the relationship between the background pixel value and the finger pad pixel value is shown.

  That is, since a pixel having a pixel value higher than the pixel value between touch and non-touch is considered not to be involved in the formation of the image of the pointing object touching the photosensor built-in LCD 11, as shown in FIG. By replacing the pixel value of a pixel having a pixel value equal to or greater than the non-touch pixel value with the touch / non-touch pixel value, an unnecessary image existing in the background of the pointing object can be deleted.

(Details of processing in the optimum sensitivity calculation unit 4)
FIG. 13 is a diagram for explaining a problem when the external light intensity is saturated.

  The touch position can be detected by performing the above processing for each frame. However, as shown in FIG. 13 (a), when the external light intensity greatly increases and the calculated external light intensity reaches the saturation pixel value, the external light intensity is increased no matter how much the external light becomes stronger. Cannot be calculated.

  As a result, the pixel value between touch and non-touch calculated from the external light intensity cannot be calculated accurately, and in the worst case, all pixels are saturated even when the finger is placed on the panel. , It becomes a white image. In FIG. 13A, the touch / non-touch pixel value calculated by the touch / non-touch pixel value calculation unit 5 when the external light intensity reaches the saturated pixel value is indicated by reference numeral 54, and the external light intensity is The (actual) touch / non-touch pixel value when the saturation pixel value is not reached is indicated by reference numeral 55.

  In order to solve this problem, it is necessary to reduce the sensitivity of the optical sensor 12 so that the external light intensity is not saturated as shown in FIG. Since the process of reducing the sensitivity does not saturate the external light intensity, the pixel value between touch and non-touch can be accurately calculated. The timing of switching the sensitivity of the optical sensor 12 is the time when the external light intensity is saturated (the time indicated by reference numeral 56 in FIG. 13A) or just before that.

  The optimum sensitivity calculation unit 4 calculates the optimum sensitivity for the recognition of the pointing object in accordance with the outside light intensity calculated by the outside light intensity calculation unit 3 so that an optimum captured image can be acquired through the sensitivity adjustment unit 14. The sensitivity of the optical sensor 12 is adjusted.

  An example of a captured image with and without sensitivity switching is shown in FIG. The upper part of FIG. 14 shows a case where the sensitivity is not switched. If the sensitivity is not switched, as the ambient light intensity increases, the background pixel value and the light transmitted through the finger also increase the lower pixel value of the finger pad. Finally, all the pixels are saturated and become completely white. Become an image. It is impossible to accurately detect the touch position from such an image.

  On the other hand, as shown in the lower part of FIG. 14, when the sensitivity is switched, the background pixel value and the lower finger pad pixel value are not saturated because the sensitivity is reduced even at the same external light intensity as when the sensitivity is not switched. It can be seen that an image state in which the touch position can be detected is maintained.

  Here, as an example of the process of the optimum sensitivity calculation unit 4, a process of switching the sensitivity of the optical sensor 12 stepwise from sensitivity 1/1 to sensitivity 1/4 will be described with reference to FIG. FIG. 15 is a diagram for explaining an example of sensitivity switching processing in the optimum sensitivity calculation unit 4.

  First, an example in which sensitivity is lowered will be described. When the external light intensity reaches 255, which is the saturation pixel value, with sensitivity 1/1, sensitivity DOWN (decrease) processing is performed and the sensitivity becomes 1/2. At this time, when the external light intensity is 255 at the sensitivity of 1/1, the external light intensity is calculated to be 128, which is half of the same external light, because the sensitivity becomes 1/2. Similarly, when the external light intensity reaches 255, which is the saturation pixel value, with sensitivity 1/2, the sensitivity is reduced by the sensitivity DOWN process, and the external light intensity, which is 255 at sensitivity 1/2, is the sensitivity 1/4. It becomes 128.

  Next, an example of increasing sensitivity will be described. When the external light intensity becomes less than 64, which is about ¼ of the saturation pixel value “255”, with sensitivity 1/4, the sensitivity is returned to 1/2 by sensitivity UP (increase) processing. The external light intensity that is 64 at the sensitivity 1/4 is calculated as 128 when the sensitivity becomes 1/2. When the external light intensity becomes 64, which is about 1/4 of the saturation pixel value, with sensitivity 1/2, the sensitivity of the optical sensor 12 returns to sensitivity 1/1 by the sensitivity increase processing.

  Since the external light intensity is saturated at 255, when the external light intensity is higher than that, it is not possible to calculate how much the external light intensity has increased. Therefore, in the case of sensitivity DOWN, it is preferable to decrease the sensitivity in order of sensitivity 1/1, 1/2, 1/4. However, since the external light intensity does not saturate when the sensitivity is increased, the sensitivity can be jumped from 1/4 to 1/1. For example, if the external light intensity suddenly decreases from near 128 to 32 or less when the sensitivity is 1/4 in FIG. 15, the sensitivity is not increased to 1/2, but the sensitivity is increased to 1/1. May be.

  That is, the optimum sensitivity calculation unit 4 sets the sensitivity of the optical sensor 12 in a stepwise manner according to the external light intensity, and when the external light intensity becomes a predetermined reference value or less, the sensitivity of the optical sensor 12 is set at a time. Raise multiple stages. Note that the sensitivity setting stage is not limited to the above three stages, and may be two stages or four or more stages.

  Here, in order to prevent the sensitivity UP / DOWN from switching frequently due to slight changes in external light intensity, the sensitivity DOWN point is 255 (becomes 128 after sensitivity down), and the sensitivity UP point is 64 (after sensitivity up). Is set to 128), and hysteresis is provided. In other words, the optimum sensitivity calculation unit 4 has the external light intensity reaches the first reference value (for example, 255) in a state where the sensitivity of the optical sensor 12 is set to the first sensitivity (for example, sensitivity 1/1). The sensitivity of the optical sensor 12 is reduced from the first sensitivity to a second sensitivity (for example, sensitivity 1/2) lower than the first sensitivity, and the sensitivity of the optical sensor 12 is set to the second sensitivity. When the external light intensity decreases to the second reference value (for example, 64), the sensitivity of the optical sensor 12 is increased from the second sensitivity to the first sensitivity. The second reference value is a predetermined value higher than the external light intensity (for example, 128) when the actual external light intensity corresponding to the first reference value is detected by the optical sensor 12 set to the second sensitivity. Small is preferable. This predetermined value may be appropriately set by those skilled in the art.

  The first and second reference values may be stored in a storage unit that can be used by the optimum sensitivity calculation unit 4.

  As described above, by performing the sensitivity UP / DOWN process of the optical sensor 12 according to the external light intensity, the dynamic range of the image can be adjusted to an optimal value, and the recognition process can be performed with the optimal image. .

(Processing flow in the touch position detection apparatus 10)
Next, an example of the flow of touch position detection processing in the touch position detection device 10 will be described with reference to FIG. FIG. 16 is a flowchart illustrating an example of the flow of touch position detection processing in the touch position detection device 10.

  First, a captured image of the pointing object is captured by the optical sensor 12 of the LCD 11 with a built-in optical sensor. The captured image acquired by the optical sensor 12 is output to the image adjustment unit 2 via the AD converter 13 (S1).

  When the image adjustment unit 2 receives the captured image (acquisition process), the image adjustment unit 2 performs a calibration process for adjusting the gain and offset of the captured image, and the adjusted captured image is subjected to the external light intensity calculation unit 3 and the recognition unnecessary information deletion unit. 6 (S2).

  When the external light intensity calculation unit 3 receives the captured image, the external light intensity is calculated from the captured image as described above (external light intensity calculation step), and the calculated external light intensity is calculated using the optimum sensitivity calculation unit 4 and the touch sensitivity. It outputs to the non-touch pixel value calculation part 5 (S3).

  The optimum sensitivity calculation unit 4 calculates the optimum sensitivity for recognizing the pointing object in accordance with the outside light intensity calculated by the outside light intensity calculation unit 3, and outputs it to the sensitivity adjustment unit 14 (S4). The sensitivity adjustment unit 14 adjusts the sensitivity of each optical sensor 12 so as to correspond to the optimum sensitivity output from the optimum sensitivity calculation unit 4.

  Here, when the external light intensity is saturated and when the external light intensity is lower than a predetermined value, the sensitivity of the optical sensor 12 is adjusted by the sensitivity adjustment unit 14. This sensitivity adjustment is reflected in the captured image of the next frame.

  Next, as described above, the touch / non-touch pixel value calculation unit 5 calculates the touch / non-touch pixel value from the external light intensity calculated by the external light intensity calculation unit 3, and calculates the calculated touch / non-touch. The inter-pixel value is output to the recognition unnecessary information deleting unit 6 (S5).

  When the recognition unnecessary information deletion unit 6 receives the pixel value between touch and non-touch, the pixel value of a pixel having a pixel value greater than or equal to the pixel value between touch and non-touch among the pixels included in the captured image is touched and non-touched. By replacing with the inter-pixel value, information in the captured image that is unnecessary for the recognition of the pointing object (that is, information on the background of the pointing object) is deleted (S6). The recognition unnecessary information deletion unit 6 outputs the processed captured image to the feature amount extraction unit 7.

  When the feature amount extraction unit 7 receives the captured image from the recognition unnecessary information deletion unit 6, the feature amount extraction unit 7 extracts a feature amount (edge feature amount) indicating the feature of the pointing object for each pixel of the captured image by the edge detection process. The position information (pixel coordinates) of the amount and the pixel (feature area) indicating the feature amount is output to the touch position detection unit 8 (S7).

  When the touch position detection unit 8 receives the feature amount and the position information of the feature area, the touch position detection unit 8 calculates the touch position by pattern matching with the feature area (S8). The touch position detection unit 8 outputs coordinates indicating the calculated touch position to the application execution unit 15.

  When the image adjustment unit 2 stores the adjusted captured image in the storage unit 40, the external light intensity calculation unit 3 and the recognition unnecessary information deletion unit 6 may acquire the captured image from the storage unit 40.

[Embodiment 2]
The following will describe another embodiment of the present invention with reference to FIGS. In addition, about the member similar to Embodiment 1, the same code | symbol is attached | subjected and the description is abbreviate | omitted.

(Configuration of Touch Position Detection Device 20)
FIG. 17 is a block diagram illustrating a configuration of the touch position detection device 20 according to the present embodiment. As shown in FIG. 17, the touch position detection device 20, unlike the touch position detection device 10, includes a feature amount extraction unit (feature area extraction unit) 21 and a recognition unnecessary information deletion unit (deletion unit) 22.

  The feature amount extraction unit 21 extracts a feature amount indicating the feature of the image of the pointing object included in the captured image output from the image adjustment unit 2 and outputs the feature amount to the recognition unnecessary information deletion unit 22. The process in the feature quantity extraction unit 21 is the same as the process in the feature quantity extraction unit 7 except that the processing target and the output destination of the feature quantity are different.

  The recognition unnecessary information deletion unit 22 deletes at least a part of the feature region indicating the feature amount extracted by the feature amount extraction unit 21 based on the external light intensity calculated by the external light intensity calculation unit 3. More specifically, the recognition unnecessary information deletion unit 22 calculates a feature amount (feature region) derived from a pixel having a pixel value equal to or greater than the touch / non-touch pixel value calculated by the touch / non-touch pixel value calculation unit 5. delete. If the feature amount associated with the pixel is deleted, the information on the feature region (the pixel indicating the feature amount) is deleted. Therefore, the deletion of the feature amount and the deletion of the feature region are almost synonymous.

  Then, the touch position detection unit 8 specifies a touch position by pattern matching with the feature amount (feature region) from which noise is removed by the recognition unnecessary information deletion unit 22.

  FIG. 18 is a diagram for explaining recognition unnecessary information deletion processing in the recognition unnecessary information deleting unit 22. As shown in FIG. 18, the characteristics of an image of a pointing object (a pixel having a pixel value greater than or equal to the pixel value between touch and non-touch) that is not in contact with the optical sensor built-in LCD 11 included in the captured image at the time of non-touch. The amount is deleted by the recognition unnecessary information deleting unit 22. Therefore, the feature amount (annular region) seen in the image “before unnecessary part deletion” in FIG. 18 is deleted in the captured image at the time of non-touch, but is not deleted in the captured image at the time of touch.

  In the touch position detection apparatus 10 according to the first embodiment, after changing the relationship between the background pixel value and the lower finger pad pixel value as illustrated in FIG. 12 (the difference between the background pixel value and the lower finger pad pixel value is reduced). In order to extract the feature value from the captured image after deleting the unnecessary portion, it is necessary to change (relax) the threshold value for extracting the edge feature value.

  On the other hand, like the touch position detection device 20 of the present embodiment, after performing feature amount extraction, when deleting feature amounts corresponding to pixels having pixel values greater than or equal to the touch / non-touch pixel value, This method is more effective because there is no need to change parameters when extracting feature values.

  For this reason, the present embodiment employs noise removal processing using a touch / non-touch pixel value after extracting a feature amount from a captured image.

(Processing flow in the touch position detection device 20)
Next, an example of the flow of touch position detection processing in the touch position detection device 20 will be described with reference to FIG. FIG. 19 is a flowchart illustrating an example of a flow of touch position detection processing in the touch position detection device 20. Steps S11 to S15 shown in FIG. 19 are the same as steps S1 to S5 shown in FIG.

  In step S <b> 15, the touch / non-touch pixel value calculation unit 5 outputs the calculated touch / non-touch pixel value to the recognition unnecessary information deletion unit 22.

  In step S <b> 16, the feature amount extraction unit 21 extracts a feature amount indicating the feature of the image of the pointing object from the captured image output from the image adjustment unit 2, and the extracted feature amount and the position of the feature region indicating the feature amount The feature area data including information is output to the recognition unnecessary information deleting unit 22 together with the captured image.

  When the recognition unnecessary information deletion unit 22 receives the touch / non-touch pixel value from the touch / non-touch pixel value calculation unit 5 and the captured image and the feature area data from the feature amount extraction unit 21, the recognition / non-touch pixel value is received. A feature amount derived from a pixel having a pixel value equal to or greater than the value is deleted (S17). More specifically, the recognition unnecessary information deleting unit 22 acquires the pixel value of the pixel (feature region) in the captured image associated with the feature amount indicated by the feature region data by referring to the captured image. If the pixel value is equal to or greater than the pixel value between touch and non-touch, the feature amount is deleted from the feature area data. The recognition unnecessary information deleting unit 22 performs this process for each feature amount indicated by the feature area data. The recognition unnecessary information deletion unit 22 outputs the processed feature area data to the touch position detection unit 8.

  Upon receiving the feature area data processed by the recognition unnecessary information deletion unit 22, the touch position detection unit 8 calculates a touch position by pattern matching with the feature area indicated by the feature area data (S18). The touch position detection unit 8 outputs coordinates indicating the calculated touch position to the application execution unit 15.

(Example of change)
The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

  In addition, each block of the touch position detection device 10 and the touch position detection device 20 described above, in particular, the external light intensity calculation unit 3, the optimum sensitivity calculation unit 4, the touch / non-touch pixel value calculation unit 5, and the recognition unnecessary information deletion unit 6 And the recognition unnecessary information deletion part 22 may be comprised by a hardware logic, and may be implement | achieved by software using CPU as follows.

  That is, the touch position detection device 10 and the touch position detection device 20 develop a CPU (central processing unit) that executes instructions of a control program that realizes each function, a ROM (read only memory) that stores the program, and the program. A random access memory (RAM), and a storage device (recording medium) such as a memory for storing the program and various data. An object of the present invention is to provide program codes (execution format program, intermediate code program, source program) of a control program (image analysis program) for the touch position detection device 10 and the touch position detection device 20 which are software for realizing the functions described above. ) Is supplied to the touch position detecting device 10 and the touch position detecting device 20, and the computer (or CPU or MPU) reads out and executes the program code recorded on the recording medium. This can also be achieved.

  Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy (registered trademark) disks / hard disks, and disks including optical disks such as CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

  The touch position detection device 10 and the touch position detection device 20 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Also, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The present invention can also be expressed as follows.

  That is, the display device of the present invention is a display device with a built-in optical sensor capable of acquiring an image, and the display device detects external light that detects the intensity of light incident on the optical sensor from an image acquired by the optical sensor. It is characterized by comprising intensity calculation means.

  Further, it is preferable that the external light intensity calculation means includes image histogram calculation means for calculating a histogram of an image acquired by the optical sensor, and calculates light intensity from the histogram calculated by the image histogram calculation means.

  The image histogram calculation means preferably calculates a histogram from arbitrary sample points in all pixels.

  In the histogram calculated by the image histogram calculation means, the external light intensity calculation means adds the frequencies in order from the highest pixel value, and compares the total with the total value obtained by adding the frequencies of the entire histogram. When the ratio becomes higher than a preset ratio, the pixel value is preferably calculated as the external light intensity.

  It is preferable that the display device performs image processing using the external light intensity calculated by the external light intensity calculation unit in the previous period.

  It is possible to accurately calculate the ambient light intensity around the pointing object of the pointing object indicating the imaging screen, and to increase the recognition accuracy of the pointing object by processing the captured image of the pointing object based on the calculated external light intensity. Therefore, the present invention can be applied to a position detection device and an input device having a touch panel.

It is a block diagram which shows the structure of the touch position detection apparatus which concerns on one Embodiment of this invention. (A) is a figure which shows the pixel used for calculation of external light intensity, (b) is a figure which shows the relationship between the histogram which an external light intensity calculation part produces, and external light intensity. It is a figure for demonstrating the significance of the process in an external light intensity calculation part. It is a figure for demonstrating the example of a change of the process in an external light intensity calculation part. It is a figure which shows the surrounding brightness and the example of a captured image when an indication object is imaged. It is sectional drawing which shows the example of a change of a touchscreen part. It is a figure which shows the example of the picked-up image when the surrounding brightness at the time of imaging a pointing object and an elastic film are provided. It is a figure which shows the example of the captured image of a touch and a non-touch. (A) is a graph which shows the relationship between environmental illumination intensity and the pixel value of a captured image, (b) is a figure which shows the example of the captured image under different environmental illumination intensity. It is a graph for demonstrating the pixel value between touch and non-touch. (A) And (b) is a graph which shows another example of the change of the pixel value under a finger pad at the time of a touch at the time of non-touch corresponding to the change of environmental illumination intensity, respectively. It is a figure for demonstrating the process in a recognition unnecessary information deletion part. (A) And (b) is a figure for demonstrating a problem when external light intensity | strength is saturated. It is a figure which shows the example of the captured image with the case where sensitivity switching is carried out, and the case where it does not. It is a figure for demonstrating the example of the sensitivity switching process in an optimal sensitivity calculation part. It is a flowchart which shows an example of the flow of the touch position detection process in the said touch position detection apparatus. It is a block diagram which shows the structure of the touch position detection apparatus which concerns on another embodiment of this invention. It is a figure for demonstrating the process in a recognition unnecessary information deletion part. It is a flowchart which shows an example of the flow of the touch position detection process in the said touch position detection apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Touch panel part 2 Image adjustment part (acquisition means)
3 External light intensity calculation unit (external light intensity calculation means)
4 Optimum sensitivity calculator (sensitivity setting means)
5 Touch / non-touch pixel value calculation unit (reference value calculation means)
6 Unrecognized information deletion unit (image processing means)
7 Feature extraction unit (feature area extraction means)
8 Touch position detection unit (position calculation means)
9 Image analysis unit (image analysis device)
10 Touch position detection device (image analysis device, imaging device)
12 Optical sensor (imaging sensor)
20 Touch position detection device (image analysis device, imaging device)
21 Feature extraction unit (feature region extraction means)
22 Recognition unnecessary information deletion part (deletion means)
41 images (captured images)
42 images (captured images)
43 images (captured images)

Claims (14)

An image analysis apparatus that analyzes a captured image of an indication object that is in contact with or not in contact with an imaging screen having a plurality of imaging sensors, and is captured by the imaging sensor,
Acquisition means for acquiring the captured image;
An external light intensity calculating means for calculating an external light intensity that is an intensity of light around the pointing object incident on a part of the imaging sensor by analyzing the captured image acquired by the acquiring means. A characteristic image analysis apparatus.   The external light intensity calculation means selects at least some pixels from a plurality of pixels included in the captured image acquired by the acquisition means, and has a predetermined order when the selected pixels are arranged in descending order of the pixel values. The image analysis apparatus according to claim 1, wherein a pixel value of a pixel in the area is set as the external light intensity.   The image analysis apparatus according to claim 2, wherein the predetermined order corresponds to a number less than 10% of the total number of selected pixels. Based on the external light intensity calculated by the external light intensity calculating means, a reference value of a pixel value for deleting an image of the pointing object when the pointing object is not in contact with the imaging screen from the captured image is obtained. A reference value calculating means for calculating;
Image processing means for replacing a pixel value of a pixel having a pixel value greater than or equal to the reference value calculated by the reference value calculation means among the pixels included in the captured image acquired by the acquisition means with the reference value; The image analysis apparatus according to claim 1, wherein the image analysis apparatus is characterized. Feature region extraction means for extracting a feature region indicating the characteristics of the image of the pointing object from the captured image acquired by the acquisition means;
A reference value of a pixel value for determining whether or not the characteristic region is derived from the image of the contact portion of the pointing object that has touched the imaging screen, based on the external light intensity calculated by the external light intensity calculation unit. A reference value calculating means for calculating;
Deleting means for deleting the feature region derived from a pixel having a pixel value equal to or higher than the reference value calculated by the reference value calculating means;
The position calculating means for calculating the position of the image of the contact portion of the pointing object that has touched the imaging screen from the feature area that has not been deleted by the deleting means. The image analysis apparatus according to item 1.   The reference value calculating means calculates the reference value using a predetermined mathematical expression, and uses a plurality of types of the predetermined mathematical expressions according to the external light intensity calculated by the external light intensity calculating means. The image analysis apparatus according to claim 5.   An image analysis program for operating the image analysis apparatus according to any one of claims 1 to 6, wherein the image analysis program causes a computer to function as each of the means.   A computer-readable recording medium on which the image analysis program according to claim 7 is recorded. An image analysis method in an image analysis apparatus for analyzing a captured image of an indication object that is in contact with or not in contact with an imaging screen having a plurality of imaging sensors and is captured by the imaging sensor,
An acquisition step of acquiring the captured image;
An external light intensity calculating step of calculating an external light intensity that is an intensity of light around the pointing object incident on a part of the imaging sensor by analyzing the captured image acquired in the acquiring step. An image analysis method characterized by the above. An imaging apparatus that includes an imaging screen having a plurality of imaging sensors, and that captures an imaging image of a pointing object that is in contact with or not in contact with the imaging screen by the imaging sensor,
The image analysis apparatus according to any one of claims 1 to 6, comprising:
The imaging device according to claim 1, wherein the acquisition unit acquires a captured image captured by the imaging sensor.   The imaging apparatus according to claim 10, further comprising sensitivity setting means for setting the sensitivity of the imaging sensor based on the external light intensity calculated by the external light intensity calculation means.   The sensitivity setting means sets the sensitivity of the imaging sensor in a stepwise manner, and increases the sensitivity of the imaging sensor at a plurality of levels at a time when the external light intensity becomes a predetermined reference value or less. The imaging device according to claim 11.   The imaging apparatus according to claim 11 or 12, wherein the sensitivity setting unit sets the sensitivity of the imaging sensor so that the external light intensity calculated by the external light intensity calculation unit is not saturated. The sensitivity setting means changes the sensitivity from the first sensitivity to the first sensitivity when the external light intensity reaches a first reference value in a state where the sensitivity of the imaging sensor is set to the first sensitivity. The sensitivity is lowered from the second sensitivity to the first sensitivity when the outside light intensity is lowered to the second reference value in a state where the sensitivity of the imaging sensor is set to the second sensitivity. Increase to sensitivity,
The second reference value is smaller than an external light intensity when an actual external light intensity corresponding to the first reference value is detected by an imaging sensor set to the second sensitivity. The imaging device according to any one of 11 to 13.

Images