JP2010055575A - Image processor, image processing program, computer-readable recording medium, electronic device, and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor capable of specifying an instruction position on a pick up image by an object for imaging with high accuracy using image data of the pick up image which is picked up. <P>SOLUTION: The image processor 1 is provided with: vertical and horizontal gradient amount calculation parts 3a, 3b which calculate feature quantity for every pixel on the image data; a first edge extraction part 4 which extracts first edge pixels whose feature quantities are equal to or more than a first threshold; a score calculation part 10 which calculates a matching degree indicating a degree of matching between a collation area and a model pattern based on the specification results of the first edge pixels; a second edge extraction part 15 which extracts second edge pixels whose feature quantities are equal to or more than a second threshold which is larger than the first threshold; and a position specific part 11 which specifies the instruction position on the pick up image by the object for imaging from a position of the collation area in which ratio of the number of second edge pixels to the number of first edge pixels exceeds a predetermined value and the matching degree is the maximum. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus having a function of specifying an instruction position on a captured image by an imaging target using image data of a captured image.

  As is well known, image display devices (hereinafter referred to as “liquid crystal display devices”) having a liquid crystal display as an image display unit are widely used in various devices such as mobile phones and PDAs (Personal Digital Assistants). It is. In particular, PDAs have been equipped with a touch sensor for a long time, enabling touch input to input information by directly bringing a finger or the like into contact with a liquid crystal display. In addition, liquid crystal display devices including a touch sensor are expected to spread in mobile phones and other devices.

  As an example of a liquid crystal display device including such a touch sensor, there is a technique disclosed in Patent Document 1.

  This conventional liquid crystal display device mainly includes an edge detection circuit, a contact determination circuit, and a coordinate calculation circuit. The edge detection circuit detects an edge of the photographed image and obtains an edge image.

  Further, the contact determination circuit determines whether or not the object has touched the display screen using the edge image obtained by the edge detection circuit. The contact determination circuit checks the moving direction (time change of edge coordinates) for each edge, and determines that the object has touched the display screen when there is an edge moving in the opposite direction. This is based on the principle that the edges do not move in opposite directions unless the object is in contact. Specifically, the determination accuracy is improved by determining that the contact is made when the amount of movement in the reverse direction is equal to or greater than a predetermined threshold.

Further, the coordinate calculation circuit calculates the center of gravity of the edge as the coordinate position of the object when it is determined that the object has touched. This makes it possible to improve the accuracy of position calculation by not calculating the coordinate position before the object comes into contact.
JP 2006-244446 A (published September 14, 2006)

  However, in the conventional liquid crystal display device, since the contact determination circuit performs contact determination on the entire display screen, when there is noise in the edge image, pixel defects on the display screen, screen scratches, etc., the contact determination circuit performs this determination. There is a problem that an edge movement is erroneously recognized due to the occurrence of such noise, the presence of a pixel defect, and the like, and as a result, the contact determination accuracy by the contact determination circuit is lowered.

  Also, if there is noise such as the above, it may be determined that the center of gravity of the edge and the position of the object are different, so even if you try to determine that multiple objects have touched the display screen Further, there is a problem that it is not possible to accurately associate the center of gravity of each edge with the coordinate position of each object.

  In view of the above-described problems, the present invention provides an image processing apparatus, an image processing program, and a computer-readable recording medium that can specify an instruction position on a captured image by an imaging target with high accuracy using image data of a captured image. An object is to provide an electronic device and an image processing method.

  In order to achieve the above object, an image processing apparatus according to the present invention is an image processing apparatus having a function of specifying an instruction position on a captured image by an imaging target using image data of a captured image. Then, for each pixel on the image data, a feature amount calculation unit that calculates a feature amount according to a pixel value difference with an adjacent pixel, and the feature amount calculated by the feature amount calculation unit is a first threshold value or more. A first edge specifying means for specifying a certain first edge pixel, a collation area included in a partial area on the image data based on a result of specifying the first edge pixel, a predetermined model pattern, And a degree-of-matching calculating means for calculating a degree of matching indicating the degree of matching between the matching area and the model pattern, and the feature amount of the first edge pixel is less than the first threshold value. Based on a second edge specifying means for specifying a second edge pixel that is equal to or greater than a threshold second threshold, a ratio between the number of first edge pixels and the number of second edge pixels, and the degree of matching calculated by the degree of matching calculating means. And position specifying means for specifying the indicated position on the captured image by the imaging target.

  The position specifying unit, for example, has a ratio of the number of second edge pixels to the number of first edge pixels exceeding a predetermined value, and the degree of coincidence calculated by the degree of coincidence calculating unit is the image. The designated position on the captured image by the imaging target is specified from the position of the collation area that is the maximum in the partial area on the data.

  In the above image processing device, matching between the matching region and the model pattern is performed based on the identification result of the first edge pixel on the image data, and a second threshold value larger than the first threshold value for identifying the first edge pixel is set. Since the second edge pixel is specified by using, the ratio of the number of second edge pixels to the number of first edge pixels exceeds a predetermined value, and imaging is performed from the position of the matching region where the degree of matching is the maximum. The designated position on the captured image by the object can be specified.

  That is, in the above-described image processing apparatus, matching with the model pattern is performed based on the identification result of the first edge pixel, and the ratio of the second edge pixel number among the first edge pixel number is determined in advance for the matching region to be matched. The indicated position on the captured image by the imaging target can be specified using a value exceeding the obtained value.

  For this reason, it becomes possible to specify the designated position on the captured image by the imaging target with high accuracy using the image data of the captured image.

  It is preferable that the image processing apparatus further includes a threshold adjustment unit that adjusts the second threshold according to a change in brightness in a use space of the image processing apparatus.

  In this case, even when the feature amount for each pixel on the image data changes due to the change in brightness in the usage space of the image processing apparatus, the second threshold is adjusted according to the change in brightness in the usage space, The second edge pixel can be specified with high accuracy.

  The threshold adjustment means includes a plurality of second threshold values stored in advance in a table format for each pixel value that can be taken by the background image on the captured image in accordance with a change in brightness in the usage space of the image processing apparatus. It is preferable to select a second threshold value corresponding to the pixel value of the background image on the captured image.

  In this case, when the second threshold value is adjusted according to the change in brightness in the usage space, the second threshold value may be selected from a plurality of second threshold values, and the second threshold value can be adjusted efficiently.

  The plurality of second threshold values stored in the table format are used when the sensitivity of an imaging sensor that captures image data of a captured image has the first sensitivity, and the threshold adjustment unit is configured to capture the imaging When the sensitivity of the sensor changes from the first sensitivity to a second sensitivity different from the first sensitivity, and the second threshold to be used when the sensitivity of the imaging sensor has the second sensitivity, In the case where it is assumed that the pixel value difference between the designated position on the captured image at the first sensitivity and the background image is the pixel value difference between the designated position on the captured image at the second sensitivity and the background image. It is preferable to calculate a pixel value to be taken by the background image on the captured image with one sensitivity and select a second threshold value corresponding to the calculated pixel value of the background image on the captured image.

  In this case, even when the sensitivity of the imaging sensor changes from the first sensitivity to the second sensitivity, it is used when the imaging sensor has the second threshold and the second sensitivity that are used when the sensitivity of the imaging sensor has the first sensitivity. Since it is not necessary to store the second threshold separately, the capacity of a storage device or the like for storing the second threshold can be reduced.

  The feature amount is preferably a vector amount having a size and a direction.

  In this case, the amount of information is larger than in the case of pattern matching with a scalar amount, and pattern matching can be performed more accurately.

  The feature amount is preferably a gradient amount of pixel luminance.

  In this case, the accuracy of position detection can be improved by performing pattern matching based on the gradient amount of the luminance of the pixel.

  An electronic apparatus according to the present invention includes the above-described image processing apparatus.

  In the above electronic device, an electronic device including the above image processing apparatus is realized.

  An image processing method according to the present invention is an image processing method for specifying an instruction position on a captured image by an imaging target using image data of a captured image that is adjacent to each pixel on the image data. A feature amount calculation step of calculating a feature amount according to a pixel value difference from the pixel to be performed, and a first edge specification that specifies a first edge pixel whose feature amount calculated in the feature amount calculation step is equal to or greater than a first threshold value A verification area included in a partial area on the image data and a predetermined model pattern based on a step and the identification result of the first edge pixel, and the verification area and the model pattern A degree-of-match calculation step of calculating a degree of matching indicating the degree of matching of the first edge pixel, and among the first edge pixels, the second edge image having the feature amount equal to or greater than a second threshold value greater than the first threshold value. Based on the ratio of the first edge pixel number to the second edge pixel number and the degree of coincidence calculated in the degree of coincidence calculation step, the designated position on the captured image by the imaging target And a position specifying step for specifying.

  In the position specifying step, for example, the ratio of the number of second edge pixels to the number of first edge pixels exceeds a predetermined value, and the degree of coincidence calculated by the degree of coincidence calculating means is The designated position on the captured image by the imaging target is specified from the position of the collation area that is the maximum in the partial area on the image data.

  In the above image processing method, matching between the matching region and the model pattern is performed based on the identification result of the first edge pixel on the image data, and a second threshold value larger than the first threshold value for identifying the first edge pixel is set. Since the second edge pixel is specified by using, the ratio of the number of second edge pixels to the number of first edge pixels exceeds a predetermined value, and imaging is performed from the position of the matching region where the degree of matching is the maximum. The designated position on the captured image by the object can be specified.

  That is, in the above image processing method, matching with the model pattern is performed based on the identification result of the first edge pixel, and the proportion of the second edge pixel number in the first edge pixel number is determined in advance for the matching region to be matched. The indicated position on the captured image by the imaging target can be specified using a value exceeding the obtained value.

  For this reason, it becomes possible to specify the designated position on the captured image by the imaging target with high accuracy using the image data of the captured image.

  An image processing apparatus according to the present invention is an image processing apparatus having a function of specifying an instruction position on a captured image by an imaging target using image data of a captured image. A matching degree calculation means for matching a matching area included in a part area with a predetermined model pattern and calculating a matching degree indicating a degree of matching between the matching area and the model pattern; and the degree of matching calculation A feature amount calculating unit that calculates a feature amount according to a pixel value difference with an adjacent pixel for each pixel that matches the model pattern in the matching by the unit, and the feature amount calculated by the feature amount calculating unit is a predetermined threshold value The edge specifying means for specifying the edge pixels as described above, the ratio between the number of matched pixels and the number of edge pixels, and the matching degree calculated by the matching degree calculating means Zui and, and a position specifying means for specifying an indication position on the image captured by the imaging object.

  Then, the position specifying unit, for example, the ratio of the number of edge pixels to the number of matched pixels exceeds a predetermined value, and the degree of match calculated by the degree of match calculating unit is on the image data The designated position on the captured image by the imaging target is specified from the position of the collation area that is the maximum in the partial area.

  In the above image processing apparatus, matching is performed between the matching region and the model pattern, the ratio of the number of edge pixels to the number of pixels matching the model pattern exceeds a predetermined value, and the degree of matching is maximum. The designated position on the captured image by the imaging target can be specified from the position of the collation area.

  That is, in the image processing method described above, the indicated position on the captured image by the imaging target is determined by using the number of matched pixels in the matching region and the ratio of the number of edge pixels exceeding a predetermined value. Can be identified.

  For this reason, it becomes possible to specify the designated position on the captured image by the imaging target with high accuracy using the image data of the captured image.

  The image processing apparatus may be realized by a computer. In this case, an image processing program for causing the image processing apparatus to be realized by a computer by causing the computer to operate as the above-described means, and A recorded computer-readable recording medium also falls within the scope of the present invention.

  As described above, the image processing apparatus of the present invention specifies, from the first edge pixels, the second edge specifying means for specifying the second edge pixels having the feature amount equal to or larger than the second threshold larger than the first threshold. And position specifying means for specifying the indicated position on the captured image by the imaging target based on the ratio between the number of first edge pixels and the number of second edge pixels and the degree of coincidence calculated by the degree of coincidence calculating means. I have.

  Therefore, there is an effect that it is possible to provide an image processing apparatus that can specify the designated position on the captured image by the imaging target with high accuracy using the image data of the captured image.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, the same code | symbol is attached | subjected to the same part, and what attached the same code | symbol in drawing may abbreviate | omit description. In the present embodiment, a case where a liquid crystal display is employed as an example of the image display unit will be described. However, a case where an image display unit other than the liquid crystal display is employed is also included in the scope of the present invention.

(1. Configuration of image processing apparatus)
FIG. 1 is a block diagram showing a configuration of an image processing apparatus 1 according to an embodiment of the present invention. First, the configuration of the image processing apparatus 1 according to the embodiment of the present invention and an example of the captured image will be described. Hereinafter, for convenience, the image processing apparatus 1 will be described. However, as shown in FIG. 1, as long as the electronic apparatus 20 requires the function of the image processing apparatus 1 in the embodiment of the present invention, the general electronic apparatus is used. It is applicable to.

  First, the outline of the configuration of the image processing apparatus 1 and the imaging principle of the image processing apparatus 1 will be described. The image processing apparatus 1 has a display function, and includes a liquid crystal display (display) composed of a plurality of pixels and a backlight for irradiating light to the liquid crystal display in the same manner as a normal liquid crystal display. It is.

  The liquid crystal display in the image processing apparatus 1 includes a photosensor (imaging sensor) in each pixel, and images an external object or the like (imaging target) that has approached the display screen of the liquid crystal display by the photosensor, It differs from a normal liquid crystal display in that it can be captured as image data (image data captured by an image sensor).

  Note that the liquid crystal display may have a built-in photosensor for each of a predetermined number of pixels, but from the viewpoint of the resolution of an image captured by the photosensor, the photosensor is provided for all pixels. It is preferably built in.

  The liquid crystal display of the image processing apparatus 1 is wired so that a plurality of scanning lines and a plurality of signal lines intersect with each other, as in a normal liquid crystal display, and pixels are arranged at each intersection, A display portion including pixels having various capacitors, a driving circuit for driving a scanning line, and a driving circuit for driving a signal line are provided.

  Furthermore, the liquid crystal display of the image processing apparatus 1 has a configuration in which, for example, a photodiode is built in each pixel as an optical sensor. A capacitor is connected to the photodiode, and the charge amount of the capacitor is changed in accordance with a change in the amount of light incident on the photodiode from the display screen. Image data is generated by detecting the voltage across the capacitor, and an image is captured (captured). This is the imaging principle of the image processing apparatus 1 using the liquid crystal display.

  Note that the optical sensor is not limited to a photodiode, and any optical sensor may be used as long as it can be built in each pixel of a liquid crystal display or the like based on the photoelectric effect.

  With the above configuration, the image processing apparatus 1 includes a display function for displaying an original image of the liquid crystal display and an imaging function for capturing an image of an external object (imaging target) that has come close to the display screen. Yes. Therefore, it is possible to adopt a configuration that allows touch input on the display screen of the display.

  Here, based on FIGS. 2A to 2H, an imaging target to be imaged by a photodiode built in each pixel of the liquid crystal display of the image processing apparatus 1 will be described. Here, an example of “finger belly (hereinafter also referred to as“ finger belly ”)” and “pen tip” will be given and an outline of the characteristics of each captured image (or image data) will be described.

  FIG. 2A is a schematic diagram illustrating a captured image of the finger pad when the surrounding is dark, and FIG. 2B is a schematic diagram illustrating the characteristics of the captured image of the finger pad when the surrounding is dark. is there. As shown in FIG. 2A, a case is considered where the user touches the display screen 65 of the liquid crystal display in the dark room with the forefinger of the index finger.

  In this case, the captured image 61 in FIG. 2B is an image obtained by reflecting the backlight to the imaging target (finger pad), and is an image in which the white circular shape is blurred. Note that the gradient direction in each pixel has a tendency toward the center of the region surrounded by the edge portion from the edge portion in the captured image (here, the gradient direction is a direction from the dark portion to the bright portion). It is positive.)

  Next, FIG. 2C is a schematic diagram illustrating a captured image of the finger pad when the surrounding is bright, and FIG. 2D illustrates the characteristics of the captured image of the finger pad when the surrounding is bright. FIG. As shown in FIG. 2C, consider a case where the user touches the display finger 65 of the index finger in the bright room in a bright room.

  In this case, the captured image 62 in FIG. 2D is an image obtained by the external light 66 entering the display screen 65 of the liquid crystal display (when touched, the reflected light from the backlight is also mixed). It consists of the shadow of the finger caused by the external light being blocked by the index finger, and the portion where the white circular shape formed by the reflection of the light from the backlight on the belly of the finger in contact with the display screen of the liquid crystal display is blurred. Among these, the gradient direction in the white circular portion shows the same tendency as the case where the belly of the finger is brought into contact in the dark room described above, but the shadow surrounding the periphery is dark and the surrounding is bright with the external light 66. The gradient direction in each pixel shows a tendency opposite to the gradient direction in the white circular portion.

  FIG. 2E is a schematic diagram illustrating a captured image of the pen tip when the surrounding is dark, and FIG. 2F is a schematic diagram illustrating characteristics of the captured image of the pen tip when the periphery is dark. is there. As shown in FIG. 2E, consider a case where the user touches the display screen 65 of the liquid crystal display with a pen tip in a dark room.

  In this case, the captured image 63 in FIG. 2F is an image obtained by reflecting the backlight to the imaging target (pen tip), and is an image in which a small white circular shape is blurred. The gradient direction in each pixel shows a tendency toward the vicinity of the center of the area surrounded by the edge portion from the edge portion in the captured image.

  Next, FIG. 2G is a schematic diagram showing a state of a picked-up image of the pen tip when the surrounding is bright, and FIG. 2H shows the characteristics of the picked-up image of the pen tip when the surrounding is bright. FIG. As shown in FIG. 2G, consider a case where the user touches the display screen 65 of the liquid crystal display with a pen tip in a bright room.

  In this case, the captured image 64 in FIG. 2H is an image obtained when the external light 66 is incident on the display screen 65 of the liquid crystal display (when touched, the reflected light from the backlight is also mixed). It consists of the shadow of the pen caused by the outside light being blocked by the pen and the portion where the small white circular shape formed by the reflection of the light of the backlight reflected on the pen tip in contact with the display screen 65 of the liquid crystal display is blurred. Among these, the gradient direction in the small white circular portion shows the same tendency as when the pen tip is contacted in the dark room described above, but the shadow surrounding it is dark and the surrounding is bright with external light. The gradient direction in each pixel shows a tendency opposite to the gradient direction in the small white circular portion.

  The distribution in each gradient direction as described above is, for example, when the surface is soft like a finger and the contact surface becomes circular by touching the surface, or even when the surface is hard like a pen with a round tip When the surface is circular, the tendency is either from the edge part in the captured image to the vicinity of the center of the region surrounded by the edge part, or from the vicinity of the center to the edge part radially. Show.

  Moreover, even if the contact surface has other shapes, it goes from the edge part in the captured image to the area surrounded by the edge part, or from the area surrounded by the edge part to the outside of the area. One of the trends. Further, these tendencies do not change greatly depending on the situation of the imaging target. Therefore, the gradient direction is an amount suitable for pattern matching.

  Next, details of the configuration of the image processing apparatus 1 according to the present embodiment will be described with reference to FIG.

  As shown in FIG. 1, the image processing apparatus 1 has a function of specifying an instruction position on a captured image by an imaging target using image data of a captured image. A vertical gradient amount calculating unit (feature amount calculating unit) 3a, a lateral gradient amount calculating unit (feature amount calculating unit) 3b, a first edge extracting unit (first edge specifying unit) 4, and a direction specifying unit 5 , Efficiency improvement unit 6, matching pixel number calculation unit 7, model pattern / comparison matching pattern storage unit 8, pattern matching degree calculation unit 9, score calculation unit (matching degree calculation means) 10, position specifying unit (Position specifying unit) 11, second edge extracting unit (second edge specifying unit) 15, threshold adjusting unit (threshold adjusting unit) 16, and threshold storing unit 17.

  The resolution reduction unit 2 reduces the resolution of image data of a captured image captured using an optical sensor built in the liquid crystal display.

  The vertical gradient amount calculation unit 3a and the horizontal gradient amount calculation unit 3b, for each pixel on the image data, calculate the vertical gradient amount and the horizontal direction of the pixel value of the target pixel from the pixel value of the target pixel and the pixel values of a plurality of adjacent pixels. The amount of directional gradient is calculated. Specifically, a known edge extraction operator such as a Sobel operator or a Prewit operator may be used.

  For example, the Sobel operator will be described. The local vertical gradient Sy and the horizontal gradient Sx at the pixel position x (i, j) of each pixel are obtained by the following equation (1).

S _x = x _{i + 1j−1} −x _i−1j−1 + 2x _{i + 1j} −2x _i−1j + x _{i + 1j + 1} −x _{i−1j + 1}
S _y = x _{i−1j + 1} −x _i−1j−1 + 2x _{ij + 1} −2x _ij−1 + x _{i + 1j + 1} −x _{i + 1j−1}
... (1)
Here, x _ij represents the pixel value at the pixel position x (i, j), i represents the position of the pixel along the horizontal direction, and j represents the position of the pixel along the vertical direction. Here, i and j are positive integers.

  Here, the expression (1) is a 3 × 3 Sobel operator (matrix operators Sx and Sy) of the following expressions (2) and (3), and the pixel position x (i, j) is the pixel of interest 3 × 3. This is equivalent to applying to 3 pixels.

  Here, the gradient magnitude ABS (S) and gradient direction ANG (S) at the pixel position x (i, j) are given as follows based on the vertical gradient Sy and the horizontal gradient Sx. In the following description, for the sake of convenience, the vertical gradient amount Sy and the horizontal gradient amount obtained by applying the vertical gradient Sy and the horizontal gradient Sx as the operators to the respective pixels will be referred to as the vertical gradient amount Sy and the horizontal gradient amount, respectively. It may be described as a lateral gradient amount Sx.

ABS (S) = (Sx ² + Sy ² ) ^1/2 (4)
ANG (S) = tan ⁻¹ (Sy / Sx) (5)
The first edge extraction unit 4 calculates the pixels of the edge portion of the captured image from the calculation results of the vertical gradient amount Sy and the horizontal gradient amount Sx of each pixel calculated by the vertical gradient amount calculation unit 3a and the horizontal gradient amount calculation unit 3b. The first edge pixel is extracted (specified).

  Here, a 1st edge pixel is a pixel in the part (edge) where brightness changes rapidly among each pixel which comprises image data. Specifically, it is a pixel in which each of the vertical gradient amount Sy and the horizontal gradient amount Sx, or the gradient magnitude ABS (S) is equal to or greater than a predetermined first threshold value.

  The purpose of extracting the first edge pixels is that the direction specifying unit 5 specifies the gradient direction for the plurality of extracted first edge pixels and regards the pixels other than the first edge pixels as non-uniform directions. It is in the point to hesitate and specify.

  The important information in pattern matching is the gradient direction in the first edge pixel of the edge portion.

  Therefore, the efficiency of pattern matching can be further improved by regarding the gradient direction in the less important pixels as a uniform direction. In addition, it is possible to reduce the memory capacity when detecting the indicated position on the captured image by the imaging target described below, shorten the processing time, and further reduce the cost of the indicated position detection process. it can.

  As described above, the direction specifying unit 5 determines the gradient direction ANG (S) for each pixel from the vertical gradient amount Sy and the horizontal gradient amount Sx calculated by the vertical gradient amount calculation unit 3a and the horizontal gradient amount calculation unit 3b. Each of the vertical gradient amount Sy and the horizontal gradient amount Sx, or the non-direction in which the gradient magnitude ABS (S) is less than the first threshold value is specified.

  Here, the non-direction is defined as being less than the first threshold, but may be defined as being equal to or less than the first threshold.

  By setting the non-direction in advance, it is possible to suppress the occurrence of many unnecessary gradient directions due to noise or the like. In addition, it is possible to narrow down the object to be collated in the gradient direction in the vicinity of the edge, and the efficiency of collation can be improved.

  The direction specifying unit 5 may specify the gradient direction of the plurality of first edge pixels specified by the first edge extracting unit 4, and specify the pixels other than the first edge pixel as non-directional. preferable. It can be said that the important information in the pattern matching is the gradient direction in the first edge pixel of the edge portion.

  Therefore, the efficiency of pattern matching can be further improved by regarding the gradient direction in pixels that are not so important in pattern matching as uniform directions.

  Here, since the gradient direction ANG (S) is a continuous amount that changes in the range of 0 rad to 2π rad, in the present embodiment, for example, the quantized direction is used in pattern matching as a gradient direction. Characteristic amount (hereinafter, also referred to as “feature amount”).

  In order to perform pattern matching with higher accuracy, it may be quantized into 16 directions. A detailed procedure of quantization in a specific direction will be described later. In addition, direction quantization means that a direction in which the gradient direction ANG (S) is within a predetermined range is treated as a uniform gradient direction in a specific direction.

  The efficiency improving unit 6 performs matching between a matching area that includes a predetermined number of pixels around the target pixel and a predetermined model pattern (hereinafter, also referred to as “pattern matching”). In addition, the collation between the collation area and the model pattern is made efficient.

  For example, the matching pixel number calculation unit 7 performs matching between the matching region and the model pattern, and the number of pixels in which the gradient direction included in the matching region matches the gradient direction included in the model pattern (hereinafter referred to as “matching”). Called the “number of pixels”).

  The model pattern / comparison match pattern storage unit 8 stores a model pattern and a comparison match pattern. The comparison matching pattern is a pattern determined in advance by analyzing a matching pattern between the gradient direction for each pixel included in the matching region and the gradient direction for each pixel included in the model pattern. The model pattern / comparison pattern storage unit 8 for comparison includes, for example, a tape system such as a magnetic tape and a cassette tape, a magnetic disk such as a floppy (registered trademark) disk / hard disk, and a compact disk-ROM / MO / MD / digital video disk / A disk system including an optical disk such as a compact disk-R, a card system such as an IC card (including a memory card) / optical card, or a semiconductor memory system such as a mask ROM / EPROM / EEPROM / flash ROM can be used.

  The pattern matching degree calculation unit 9 indicates the degree of similarity between the matching pattern between the gradient direction for each pixel included in the matching region and the gradient direction for each pixel included in the model pattern, and a predetermined comparison matching pattern. The pattern matching degree is calculated.

  The gradient of the pixel value is a vector quantity, and has a magnitude (gradient magnitude ABS (S)) and a direction (gradient direction ANG (S)). Here, in particular, the gradient direction (orientation) can be discretized from 8 possible states (e.g., 9 if no direction is included) to a very small state by quantizing in 8 directions, for example. In addition, each state can have a feature that allows easy identification of different directions.

  The gradient direction tends to be as described above. Also, these tendencies do not change greatly depending on the situation of the imaging target. Therefore, the gradient direction is an amount suitable for pattern matching.

  The score calculation unit 10 calculates a matching degree indicating the degree of matching between the matching region and the model pattern from the matching pixel number calculated by the matching pixel number calculation unit 7 and the pattern matching degree calculated by the pattern matching degree calculation unit 9. To do. Note that the score calculation unit 10 may be configured to use any one of the matching pixel number calculated by the matching pixel number calculation unit 7 and the pattern matching degree calculated by the pattern matching degree calculation unit 9.

  In addition, the score calculation unit 10 may calculate the degree of matching when the number of types of gradient directions that match in the collation region is equal to or greater than a predetermined number of types.

  As described above, the gradient direction tends to be approximate. Also, these tendencies do not change greatly depending on the situation of the imaging target. Therefore, for example, when the number of types of gradient directions is 8, the number of types of gradient directions that match in pattern matching should be close to 8. Therefore, if the matching degree is calculated when the number of types of gradient directions that match in the matching area is equal to or greater than a predetermined number of types, the memory capacity of the indicated position detection process can be reduced. And the processing time can be shortened, and the cost of the indicated position detection process can be further reduced.

  Further, when the degree of coincidence calculated by the score calculation unit 10 is equal to or greater than a predetermined value, the score calculation unit 10 extracts a first edge from a second edge extraction unit 15 described later for a matching region corresponding to the degree of coincidence. The first edge pixel number extracted by the unit 4 and the second edge pixel number extracted by the second edge extraction unit 15 are acquired. Then, the score calculation unit 10 determines whether or not the ratio of the second edge pixel among the first edge pixels exceeds a predetermined value, and the match calculated by itself based on the determination result that it exceeds The degree is output to the position specifying unit 11. On the other hand, the score calculation unit 10 avoids using the degree of coincidence calculated by the score calculation unit 10 based on the determination that it has not exceeded the position specifying unit 11.

  The point is that the score calculation unit 10 specifies the position only when the proportion of the first edge pixel that the second edge pixel occupies exceeds a predetermined value corresponding to the degree of match calculated by itself. What is necessary is just to be comprised so that it may output to the part 11. FIG.

  The position specifying unit 11 specifies the indicated position on the captured image by the imaging target from the position of the pixel (hereinafter referred to as “peak pixel”) having the highest degree of matching calculated by the score calculation unit 10. The position specifying unit 11 includes a peak searching unit (peak pixel specifying unit, position specifying unit) 12, a coordinate calculation determining unit (coordinate calculation determining unit, position specifying unit) 13, and a coordinate calculating unit (coordinate calculating unit, position specifying unit). 14).

  The peak search unit 12 has a maximum matching value calculated by the score calculation unit 10 in a search region including a predetermined number of pixels around the target pixel (hereinafter, sometimes referred to as “first region”). A peak pixel which is a pixel to be taken is searched.

  The coordinate calculation determination unit 13 has a predetermined number of pixels smaller than the number of pixels in the search area, and is within a small area (hereinafter sometimes referred to as “second area”) that is completely included in the search area. In addition, when it is determined that the peak pixel discovered by the peak search unit 12 exists, the coordinate calculation unit 14 is caused to calculate the indicated position on the captured image by the imaging target.

  The coordinate calculation unit 14 uses the degree of match for each pixel in the peak pixel region, which is a region including a predetermined number of pixels centered on the peak pixel found by the peak search unit 12, on the captured image by the imaging target. The indicated position is calculated.

  The second edge extraction unit 15 calculates the pixel of the edge portion of the captured image from the calculation result of the vertical gradient amount Sy and the horizontal gradient amount Sx of each pixel calculated by the vertical gradient amount calculation unit 3a and the horizontal gradient amount calculation unit 3b. The second edge pixel is extracted (specified).

  Here, like the first edge pixel extracted by the first edge extraction unit 4, the second edge pixel is a pixel in a portion (edge) where the brightness changes abruptly among the pixels constituting the image data. is there. The second edge pixel is the first edge pixel previously extracted by the first edge extraction unit 4, and each of the vertical gradient amount Sy and the horizontal gradient amount Sx or the gradient magnitude ABS ( S) is a pixel that is greater than or equal to a second threshold that is greater than the first threshold.

  When a finger pad, pen, or the like, which is an external object that comes close to the display screen of the image processing apparatus 1 actually touches the display screen, the brightness change width at the edge that directly touches the display screen, that is, the edge The strength tends to increase compared to an edge that is close to the display screen but not in direct contact. For example, the strength of the edge of the fingertip, that is, the pixel value difference between the fingertip and the background is different between the touch finger touching the display screen and the floating non-touch finger. Since the captured image of the touch finger is clear, the edge is strong, but the non-touch finger has a blurred edge because the captured image is blurred.

  Therefore, the second edge extraction unit 15 has a vertical gradient amount Sy and a horizontal gradient amount that are equal to or larger than a second threshold value larger than the first threshold value among the first edge pixels extracted by the first edge extraction unit 4. A second edge pixel having each of Sx or gradient magnitude ABS (S) is extracted.

  If the ratio of the second edge pixel to the first edge pixel exceeds a predetermined value, it is assumed that an external object close to the display screen is actually in contact with the display screen.

  This is the purpose of extracting the second edge pixel.

  The threshold adjustment unit 16 adjusts the second threshold used when the second edge extraction unit 15 extracts the second edge pixel.

  As described above, the edge strength of an external object that comes close to the display screen of the image processing apparatus 1 changes according to the brightness of the usage space of the image processing apparatus 1. Therefore, the threshold adjustment unit 16 acquires the brightness of the usage space of the image processing apparatus 1 and adjusts the second threshold according to the brightness of the usage space.

  In addition, when the threshold adjustment unit 16 acquires the brightness of the usage space of the image processing device 1, for example, the threshold adjustment unit 16 acquires the brightness of the usage space from the measurement result of the illuminometer arranged in the usage space of the image processing device 1. That's fine. Alternatively, a dedicated optical sensor for detecting the brightness of the usage space may be arranged on the display screen, and the brightness of the usage space may be acquired from the detection result of the optical sensor.

  The threshold storage unit 17 stores in advance a second threshold to be used by the second edge extraction unit 15 in a table format according to the brightness of the usage space of the image processing apparatus 1. The threshold storage unit 17 stores, for example, a plurality of second threshold values corresponding to the pixel values of the background image on the captured image on a one-to-one basis. If the captured image is displayed with 256 gradations, the threshold storage unit 17 stores the second threshold for each case where the background pixel has 0 to 255 gradations.

  The threshold storage unit 17 is, for example, a tape system such as a magnetic tape or a cassette tape, a magnetic disk such as a floppy (registered trademark) disk / hard disk, or a compact disk-ROM / MO / MD / digital video disk / compact disk-R. A disk system including an optical disk such as an IC card, a card system such as an IC card (including a memory card) / optical card, or a semiconductor memory system such as a mask ROM / EPROM / EEPROM / flash ROM can be used.

(2. Outline of operation of image processing apparatus)
Next, an outline of the operation of the image processing apparatus 1 will be described.

  FIG. 3 is a flowchart showing an outline of the overall operation of the image processing apparatus 1. In step S101, the resolution reduction unit 2 in FIG. 1 reduces the resolution of the image data, and the process proceeds to S102. For example, in the case of image data of 320 × 240 pixels, bilinear reduction is performed such as 160 × 120 pixels (lower resolution ratio 1/2) or 80 × 60 pixels (lower resolution ratio 1/4). Here, bilinear reduction is, for example, by taking an average value of pixel values of each 2 × 2 pixel and replacing 2 × 2 pixel data with 1 × 1 pixel data having the average value. This means that 1/4 information compression is performed as a whole.

  From the viewpoint of high-speed processing, it is preferable to reduce the resolution as much as possible. However, in order to obtain necessary edge information and the like, for example, in the case of image data of 320 × 240 pixels (150 dpi), 80 × 60. It is preferable to set the pixel (lower resolution ratio 1/4) as the limit of lower resolution. From the viewpoint of high-precision processing, it is preferable that the resolution is not reduced at all or even if the resolution is reduced, it is limited to 160 × 120 pixels (reduction ratio 1/2).

  By reducing the resolution of the image data as described above, it is possible to reduce the pattern matching processing cost, the memory capacity, and the processing time.

  In S102, the vertical gradient amount calculation unit 3a and the horizontal gradient amount calculation unit 3b calculate the vertical gradient amount Sy and the horizontal gradient amount Sx for each pixel on the image data, and then the direction specifying unit 5 After passing through a process (gradient direction / non-direction specifying process) until specifying any one of the gradient direction and the non-direction, the process proceeds to S103.

  In S103, when the collation area and the model pattern are collated by the efficiency improving unit 6, whether or not the collation area and the model pattern should be collated efficiently, that is, whether or not the collation efficiency is required is selected. To do. When the collation efficiency is to be improved (Yes), the process proceeds to S104, the collation efficiency is increased by the efficiency improvement unit 6, and then the process proceeds to S105. When the collation efficiency is not to be performed (No), the process proceeds to S108, and the efficiency The conversion unit 6 does nothing, and proceeds to S105 with the information as it is (image data, but if the resolution is reduced by the resolution reduction unit 2, the image data after the resolution is reduced).

  In S105, the matching pixel number calculation unit 7 performs matching between the matching region and the model pattern to calculate the number of matching pixels, and the pattern matching degree calculation unit 9 calculates the pattern matching degree, and then the score calculation unit. 10 through the process (pattern matching process) until the matching degree is calculated from the matching pixel number calculated by the matching pixel number calculating unit 7 and the pattern matching degree calculated by the pattern matching degree calculating unit 9. The process proceeds to S106.

  In S <b> 106, a process (contact / non-contact determination) from when the second edge extraction unit 15 extracts the second edge pixel to when the score calculation unit 10 outputs the degree of coincidence calculated by itself to the position specifying unit 11. The process proceeds to step S107.

  In S <b> 107, the position specifying unit 11 specifies the designated position on the captured image by the imaging target from the position of the pixel (hereinafter referred to as “peak pixel”) having the highest degree of matching calculated by the score calculation unit 10. "End" (pointing position identification process).

  The above is the overall operation overview of the image processing apparatus 1. In the following, each operation of the image processing apparatus 1 in the gradient direction / non-direction specifying process, collation efficiency improvement, pattern matching process, contact / non-contact specifying process, and pointing position specifying process will be described.

(3. Gradient direction / non-directional identification process)
First, the operation of the image processing apparatus 1 in the gradient direction / non-direction specifying process will be described with reference to FIGS. 1, 4, 5A and 5B.

  FIG. 4 is a flowchart showing the operation of the gradient direction / non-direction specifying process among the operations of the image processing apparatus 1. FIG. 5A is an example of a table referred to in the gradient direction / non-direction specifying process, and FIG. 5B is another example of a table referred to in the gradient direction / non-direction specifying process.

  In the flowchart of FIG. 4, the process starts after the vertical gradient amount calculation unit 3a and the horizontal gradient amount calculation unit 3b calculate the vertical gradient amount Sy and the horizontal gradient amount Sx, respectively.

  In S201, the first edge extraction unit 4 determines whether or not the gradient magnitude ABS (S) in each pixel is equal to or greater than the first threshold. If ABS (S) ≧ first threshold (Yes), the process proceeds to S202. If ABS (S) <the first threshold (No), the process proceeds to S210. In the present embodiment, calculation is made as ABS (S) = Sx × Sx + Sy × Sy, but this amount is different from the magnitude of the gradient in the strict sense in the above equation (4). However, in terms of implementation, there is no problem even if the magnitude of the gradient is defined in this way.

  When the process proceeds to S210, the direction specifying unit 5 sets (specifies) the target pixel, that is, the pixel other than the first edge pixel as non-directional, proceeds to the next pixel, and returns to S201. . If the process proceeds to S202, the target pixel is, of course, the first edge pixel.

  In S202, the direction specifying unit 5 determines whether or not the lateral gradient amount Sx is 0. If Sx ≠ 0, the process proceeds to S203 (Yes), and if Sx = 0, the process proceeds to S206 (No). .

  In S203, the direction specifying unit 5 determines whether or not the lateral gradient amount Sx takes a positive value. If Sx> 0, the process proceeds to S204 (Yes), and the direction specifying unit 5 ), The gradient direction quantized according to the gradient direction ANG (S) is set for the corresponding pixel.

  On the other hand, if Sx <0, the process proceeds to S205, and the direction specifying unit 5 sets the gradient direction quantized according to the gradient direction ANG (S) for the corresponding pixel according to the table of FIG. 5B. .

  Next, in S206, the direction identifying unit 5 determines whether or not the vertical gradient amount Sy is 0. If Sy ≠ 0, the process proceeds to S207 (Yes), and if Sy = 0, the process proceeds to S210 (No). Proceed, set the corresponding pixel as non-directional, proceed to the next pixel, and return to S201.

  In S207, the direction specifying unit 5 determines whether or not the vertical gradient amount Sy takes a positive value. If Sy> 0, the process proceeds to S208 (Yes) and sets the gradient direction of the corresponding pixel upward. Then, the process returns to S201.

  On the other hand, if Sy <0, the process proceeds to S209 (No), the gradient direction of the corresponding pixel is set downward, the process proceeds to the next pixel, and the process returns to S201.

  The above process is repeated until the setting of the gradient direction / non-direction of all pixels is completed.

  The important information in pattern matching is the gradient direction in the first edge pixel that is the pixel of the edge portion.

  Therefore, by considering the gradient direction in the pixels other than the first edge pixel that is not so important as a uniform direction by the above operation, the efficiency of pattern matching can be further improved. In addition, it is possible to detect the designated position on the picked-up image by the imaging target, to reduce the memory capacity and the processing time, and to further reduce the cost of the designated position detection process.

(4. Verification efficiency improvement)
Next, collation efficiency improvement in the image processing apparatus 1 will be described.

  The efficiency improving unit 6 shown in FIG. 1 divides the collation area into a plurality of divided areas having the same number of pixels and divides the gradient direction and non-directional information for each pixel included in the divided area for each divided area. By replacing the gradient direction included in the region and the non-directional information, the collation region and the model pattern are efficiently collated.

  In addition, the score calculation unit 10 performs collation between the collation region and the model pattern that have been collated efficiently by the efficiency unit 6, and the gradient direction included in each divided region in the collation region and the model pattern The number of matches with the gradient direction is calculated as the degree of match.

  As described above, the gradient direction tends to be approximate. Also, these tendencies do not change greatly depending on the situation of the imaging target. Therefore, unless the number of pixels in the divided region is increased too much, the position of the pixel in the gradient direction in the divided region is not very important information in pattern matching using the gradient direction.

  Therefore, for each divided region, by replacing the gradient direction and non-directional information for each pixel included in the divided region with the gradient direction and non-directional information included in the divided region, while maintaining the matching accuracy of pattern matching Thus, collation efficiency can be improved. Further, with this efficiency improvement, it is possible to reduce the cost required for the detection processing of the designated position on the captured image by the imaging target.

  As described above, it is possible to provide the image processing apparatus 1 that can improve the matching efficiency while maintaining the matching accuracy of the pattern matching and can reduce the cost required for the detection process of the designated position on the captured image by the imaging target. it can.

  Here, based on FIG. 6A and FIG. 6B, a specific example of collation efficiency improvement in the image processing apparatus 1 will be described.

  First, as shown in FIG. 6A, the characteristic of the distribution in the gradient direction of each pixel of the image data in the case where the surrounding is dark is a non-directional pixel region having a substantially circular shape at the center. A large number of pixels having a gradient direction directed toward the non-directional region are distributed.

  Next, FIG. 6B is a schematic diagram showing a state after the collation efficiency is improved for the image data of FIG.

  As shown in FIG. 6A, a 14 × 14 pixel region (collation region) is divided into a plurality of 2 × 2 pixel regions (divided regions) and 2 × 2 pixel regions are divided into 2 × 2 pixels. By replacing the gradient direction and non-direction information for each pixel included in the 2-pixel area with the gradient direction and non-direction information included in the 2 × 2-pixel area, the 14 × 14-pixel area and the model pattern ( An example of the model pattern will be described in detail later).

  For example, among the plurality of 2 × 2 pixel regions obtained by dividing the 14 × 14 pixel region shown in FIG. 6A, the upper left pixel is the second 2 × 2 pixel region from the top and the first 2 × 2 pixel region from the left. Is “no direction”, the upper right pixel is “right downward (gradient direction)”, the lower left pixel is “rightward (gradient direction)”, and the lower right pixel is “right downward (gradient direction)”. In this 2 × 2 pixel region, information regarding the existence position in each gradient direction is omitted. The second block from the top in FIG. 6B and the first block from the left (hereinafter, referred to as “pixel” for convenience). Yes.) Other blocks can be generated in the same manner. As a result, the 14 × 14 pixel region shown in FIG. 6A is divided into a total of 7 × 7 = 49 2 × 2 pixel regions.

  Next, a specific example of the model pattern to be collated with the collation area will be described based on FIGS. 7 (a) to 9 (b).

  FIG. 7A is a schematic diagram showing an example of a model pattern that is collated with image data before collation efficiency improvement when the surroundings are dark. In addition, the model pattern in FIG. 7A is assumed to perform pattern matching with the 14 × 14 pixel region shown in FIG. In addition, a finger pad is assumed as an imaging target.

  By the way, the model pattern of FIG. 7A is a 13 × 13 pixel model pattern. In this case, although the total number of pixels is different from the 14 × 14 pixel area shown in FIG. 6A, the number of pixels in the collation area and the model pattern does not necessarily match as in this example.

  The reason why the odd number × odd number (13 × 13) is set is that the center pixel is one. Pattern matching is performed by shifting the center pixel by one pixel in accordance with the pixel of interest on the image data.

  Since the number of matching pixels in this case is collated and calculated for each pixel, it is necessary to collate 13 × 13 = 169 pixels.

  On the other hand, FIG. 7B is a schematic diagram illustrating an example of a model pattern that is collated with image data before collation efficiency improvement when the surroundings are bright. Compared with the model pattern of FIG. 7A, the gradient direction of each pixel is reversed. FIG. 7A assumes image data picked up based on the reflected light of the backlight, and shows a state of becoming brighter toward the center. On the other hand, FIG. 7B assumes image data captured on the basis of external light, and shows a state in which the image data becomes brighter toward the edge portion of the image.

  Next, FIG. 8A is a schematic diagram illustrating an example of a model pattern that is collated with image data after collation efficiency in the case where the surrounding is dark. The model pattern in FIG. 8A assumes pattern matching with the collation area after the collation efficiency in FIG. 6B. As in this example, the collation area, the model pattern, and the data format are not necessarily the same. In this example, a 2 × 2 pixel region is regarded as one pixel (one gradient direction is given), and the model pattern is simplified to further improve the efficiency of matching.

  FIG. 8B is a schematic diagram illustrating an example of a model pattern that is collated with the image data after the collation efficiency is improved when the surroundings are bright. FIG. 8A assumes image data picked up based on the reflected light of the backlight, and shows a state of becoming brighter toward the center. On the other hand, FIG. 8B assumes image data captured on the basis of external light, and shows a state in which the image is brightened toward the edge portion of the image.

  FIG. 9A is a schematic diagram illustrating another example of a model pattern that is collated with image data after collation efficiency improvement when the surrounding is dark. The difference from the model pattern in FIG. 8A is that 2 × 2 pixels are grouped in one area, but each one area gives two degrees of freedom in the gradient direction (or no direction). This is the point. By devising such a model pattern, it is possible to improve collation accuracy while improving collation efficiency.

  FIG. 9B is a schematic diagram showing another example of a model pattern that is collated with the image data after the collation efficiency is improved when the surroundings are bright. FIG. 9A assumes image data picked up based on the reflected light of the backlight, and shows a state where the image data becomes brighter toward the center. On the other hand, FIG. 9B assumes image data picked up based on outside light, and shows a state in which the image data becomes brighter toward the edge portion of the image.

(5. Pattern matching process)
Next, a pattern matching process in the image processing apparatus 1 will be described.

  FIG. 10 is a flowchart showing the operation of the pattern matching process among the operations of the image processing apparatus 1 shown in FIG.

  In S301, the matching pixel number calculation unit 7 performs matching between the matching region and the model pattern, and the number of pixels (matching pixel number) in which the gradient direction included in the matching region matches the gradient direction included in the model pattern. Is calculated and the process proceeds to S302.

  In S <b> 302, if the efficiency specifying unit 6 is not provided, the direction specifying unit 5 determines whether the efficiency determining unit 6 also calculates the pattern matching degree in the gradient direction if the efficiency specifying unit 6 is provided. When the calculation of the pattern matching degree is determined, the pattern matching degree calculation unit 9 is notified and the process proceeds to S303 (Yes). On the other hand, if it is determined not to calculate the pattern matching degree, the score calculation unit 10 is notified and the process proceeds to S304.

  Although the case where the number of matching pixels is always calculated has been described here, the present invention is not limited to such a case, and a configuration for calculating only the pattern matching degree may be employed.

  The pattern matching degree is determined in advance in the matching pattern between the gradient direction of each pixel included in the matching region and the gradient direction of each pixel included in the model pattern, and is stored in the model pattern / comparison matching pattern storage unit 8. This is an amount indicating the degree of similarity with the recorded comparison matching pattern.

  In S303, the pattern matching degree calculation unit 9 receives a notification that the calculation of the pattern matching degree has been determined from the direction specifying unit 5 or the efficiency improving unit 6, calculates the pattern matching degree, and proceeds to S304.

  In S304, when the pattern matching degree calculation unit 9 has not calculated the pattern matching degree, the matching pixel number calculated by the matching pixel number calculating unit 7 is used as the matching degree between the matching region and the model pattern. Calculated as the degree of match shown. On the other hand, when the pattern matching degree calculation unit 9 is calculating the pattern matching degree, the matching pixel number calculated by the matching pixel number calculating unit 7 and the pattern matching degree calculated by the pattern matching degree calculating unit 9 are The total amount is calculated as the degree of matching indicating the degree of matching between the matching area and the model pattern.

  Next, a specific score (matching degree) calculation method of the score calculation unit 10 in the pattern matching process will be described.

  FIG. 11A is a schematic diagram for explaining pattern matching between a matching region and a model pattern in the case where the surroundings before darkening efficiency is dark, and FIG. It is a figure which shows an example.

  FIG. 11A shows the result of pattern matching between the collation region of FIG. 6A and the model pattern of FIG. 7A. Here, 7 rows and 7 columns in the center of the figure (hereinafter, pixels arranged in the horizontal direction are called “columns”, and pixels arranged in the vertical direction are called “rows”. Further, the rows are counted from the top and the columns from the left). 1 × 1 pixel is the position of the target pixel to which a score is assigned. The shaded portion indicates pixels in which the gradient directions match in the matching region and the model pattern.

  The matching pattern shown in FIG. 11B shows a table when the number of types in the matching direction is viewed. In this example, it is shown that there is a matching pixel in all eight directions.

  Next, the number of matching pixels shown in FIG. 11B is calculated by calculating the number of matching pixels from the pixel in the upper left 1 row 1 column to the pixel in the lower right 13 row 13 column in the shaded portion. An example of the method of doing is shown. Here, a pixel that has a gradient direction and that matches the gradient direction of the model pattern is calculated as “1”, and a pixel that does not match the gradient direction of the model pattern and the model pattern is calculated as “0”. Yes. Note that the pixel determined to be non-directional may not be included in the calculation from the beginning. Here, the calculation result that the number of matching pixels indicated by shading is “85” is obtained. The number of matching pixels may be used as a score (matching degree), but a normalized matching pixel number (matching degree) as described below may be used.

  Next, the number of normalized matching pixels shown in FIG. The number of normalized matching pixels is, for example, when pattern matching is performed by preparing a plurality of model patterns in order to improve matching accuracy (for example, three types of model patterns of 21 × 21, 13 × 13, and 7 × 7 pixels). ) Normalizes the number of matched pixels as an amount independent of the size of the model pattern.

  Here, the number of normalized matching pixels is defined by the following equation (6).

Number of normalized matching pixels = appropriate constant × (number of matching pixels / number of elements having a direction component in the model) (6)
The “appropriate constant” may be appropriately determined in consideration of the convenience of calculation. Here, an appropriate constant = 10 is set so that the number of normalized matching pixels is in the range of 0-10. In the pattern matching example described below, the number of normalized matching pixels is used, but the description thereof is omitted.

  From the equation (6), the normalized coincidence pixel number in the case of FIG. 11A is obtained as follows.

Number of normalized matching pixels = 10 × (85/136) = 6.25≈6
Next, FIG. 12A is a schematic diagram for explaining pattern matching between a matching region and a model pattern in the case where the surroundings after darkening the matching efficiency are dark, and FIG. It is a figure which shows an example of the calculation method.

  FIG. 12A shows a result of pattern matching between the collation area after collation efficiency in FIG. 6B and the model pattern in FIG. 8A. Here, 1 × 1 pixel (corresponding to 2 × 2 pixels, but referred to as “pixel” for convenience) in 4 rows and 4 columns in the center of the figure is the position of the target pixel to which a score is assigned. is there. The shaded portion indicates pixels that may have the same gradient direction in the matching region and the model pattern.

  The matching pattern shown in FIG. 12B shows a table when the number of types in the matching direction is viewed. In this example, it is shown that there is a matching pixel in all eight directions.

  Next, the number of matching pixels shown in FIG. 12B is calculated by calculating the number of matching pixels from the pixel in the upper left 1 row 1 column to the pixel in the lower right 7 row 7 column in the shaded portion. An example of the method of doing is shown. Here, for example, in the first row and the second column, there are “three” pixels having the gradient direction “lower right” in the matching region, while the gradient direction of the model pattern is one, "Lower right". Therefore, the number of matching pixels in this case is calculated as “3”.

  In another example, in 2 rows and 1 column, in the matching region, there are “two” pixels having a gradient direction “lower right” and “one” “right”, while the gradient of the model pattern is There is one direction, but “bottom right”. As for the gradient direction, “bottom right” matches “two”, but “right” does not match. Therefore, the number of matching pixels in this case is calculated as “2”. Here, the pixel determined to be non-directional is not included in the calculation from the beginning.

  If the above calculation is performed for all the pixels, a calculation result is obtained that the number of matching pixels in the shaded portion is “91”. The number of matching pixels may be used as a score (matching degree), but a normalized matching pixel number as described below may be used.

  Here, the number of normalized matching pixels is defined by the following equation (7).

Number of normalized matching pixels = appropriate constant × (number of matching pixels / value obtained by multiplying the number of elements having a direction component in the model by four) (7)
Appropriate constant = 10 is set.

  From the equation (7), the number of normalized matching pixels in the case of FIG. 11A is obtained as follows.

Number of normalized matching pixels = 10 × (91/176) = 5.17≈5
Next, FIG. 13A is a schematic diagram for explaining pattern matching between a matching region and a model pattern in the case where the surroundings after darkening is dark, and FIG. 13B shows the degree of matching. It is a figure which shows an example of the calculation method.

  FIG. 13A shows the result of pattern matching between the collation area after collation efficiency in FIG. 6B and the model pattern in FIG. 9A. Here, 1 × 1 pixel (corresponding to 2 × 2 pixels, but referred to as “pixel” for convenience) in 4 rows and 4 columns in the center of the figure is the position of the target pixel to which a score is assigned. is there. The shaded portion indicates pixels that may have the same gradient direction in the matching region and the model pattern.

  The matching pattern shown in FIG. 13B is a table when the number of types in the matching direction is viewed. In this example, it is shown that there is a matching pixel in all eight directions.

  Next, the number of matching pixels shown in FIG. 13B is calculated by calculating the number of matching pixels from the pixel in the upper left 1 row 1 column to the pixel in the lower right 7 row 7 column in the shaded portion. An example of the method is shown. Here, for example, in the first row and the second column, there are “three” pixels having the gradient direction “lower right” in the matching region, while the model pattern has two gradient directions, “right “Lower” is “one” and “Lower” is “one”. Therefore, since “lower right” matches, the number of matching pixels in this case is calculated as “3”.

  In another example, in 2 rows and 1 column, in the matching region, there are “two” pixels having a gradient direction “lower right” and “one” “right”, while the gradient of the model pattern is There are two directions, but “right” is “one” and “lower right” is “one”. Regarding the gradient direction, “one” for “right” and “two” for “lower right” coincide. Therefore, the number of matching pixels in this case is calculated as “3”. Here, the pixel determined to be non-directional is not included in the calculation from the beginning.

  Here, numbers underlined and numbers not underlined are described, but the numbers underlined are those in which the number of matching pixels is increased compared to the case of FIG. Is shown.

  As a result, pattern matching that is more resistant to deformation (higher robustness against distortion from a circle) can be performed when the model pattern of FIG. 9A is used than the model pattern of FIG. 8A. It is shown that.

  If the above calculation is performed for all the pixels, a calculation result is obtained that the number of matching pixels in the shaded portion is “119”. The number of matching pixels may be used as a score (matching degree), but a normalized matching pixel number as described below may be used.

  From the equation (7), the number of normalized coincidence pixels in the case of FIG. 13A is obtained as follows.

Number of normalized matching pixels = 10 × (119/176) = 6.76≈7
Next, a case where the score calculation unit 10 in FIG. 1 calculates a score (matching degree) by using both the number of matching pixels and the matching pattern will be described with reference to FIGS.

  FIG. 14 is a flowchart for explaining a case where the number of matching pixels and the degree of pattern matching are used together in pattern matching in the image processing apparatus 1.

  In S401 of FIG. 14, the coincidence pixel number calculation unit 7 initializes the coincidence pixel number, and proceeds to S402. In S402, the pattern matching degree calculation unit 9 initializes the matching pattern and proceeds to S403. Here, a state in which the number of types in the gradient direction is initialized is shown. This is indicated by the fact that all gradient directions are displayed as “none”.

  In S403, the coincidence pixel number calculation unit 7 and the pattern coincidence degree calculation unit 9 perform collation in the gradient direction for each pixel (including pixels after collation efficiency improvement), and the process proceeds to S404.

  In S404, if the directions match (Yes), the process proceeds to S405, and the matching pixel number calculation unit 7 adds the number of elements in the direction matching the number of matching pixels (“1” in the case of no efficiency process) and proceeds to S406. On the other hand, if there are no pixels whose directions match (No), the process returns to S401.

  In S406, the pattern coincidence degree calculation unit 9 updates the matched gradient direction as “present”, and the process proceeds to S407.

  In S407, when the matching pixel number calculation unit 7 and the pattern matching degree calculation unit 9 complete collation for all the elements (pixels) of the model pattern (Yes), the process proceeds to S408, and in the case where the collation has not ended. , (No) Return to S403.

  In S408, the pattern matching degree calculation unit 9 checks the matching pattern, and the process proceeds to S409. The details of checking the matching pattern will be described later.

  In S409, the pattern matching degree calculation unit 9 refers to the model pattern / comparison matching pattern storage unit 8 to determine whether or not it corresponds to the “permitted pattern”. ) Go to S410. On the other hand, when it does not correspond to a permission pattern, it returns to (No) S404.

  Note that the pattern matching degree calculation unit 9 sets the pattern matching degree to “1” when it corresponds to “permitted pattern”, and sets the pattern matching degree to “0” when it does not correspond to “permitted pattern”. The unit 10 may be configured to multiply the number of coincidence pixels calculated by the coincidence pixel number calculation unit 7 by these numerical values.

  In S410, the score calculation unit 10 calculates the normalized matching pixel number from the matching pixel number calculated by the matching pixel number calculation unit 7 to obtain a pattern matching score (matching degree).

  Next, an example of matching pattern check in the pattern matching will be described with reference to FIGS. 15 (a) and 15 (b).

  FIG. 15A is a flowchart showing an example of the pattern matching degree calculation process, and FIG. 15B is a flowchart showing another example of the pattern matching degree calculation process.

  Here, there are eight types of gradient directions, and a case where the threshold value (DN) of the number of types of gradient directions is set to 5 will be described.

  As shown in FIG. 15A, in S501, when the number of “matched” in the matching pattern is 5 or more, the pattern matching degree calculation unit 9 proceeds to S502 and permits the matching pattern.

  On the other hand, if the number of matching patterns “exist” (number of types in the gradient direction) is less than 5, the pattern matching degree calculation unit 9 proceeds to S503 and rejects the matching pattern.

  Similarly, in FIG. 15B, the pattern matching degree calculation unit 9 calculates the maximum number of consecutive connections (number of consecutive matches) in the matching pattern, and a threshold ( The flow from S601 to S603 is the same as the flow from S501 to S503 in FIG. 15A except that the case where DN) is set to 5 which is the same as the above, and the description thereof is omitted here.

  Next, based on FIG. 16A to FIG. 16C, a first example in the matching pattern check will be described.

  FIG. 16A is a schematic diagram showing an example of the pattern matching degree calculation process, FIG. 16B is a schematic diagram showing another example of the pattern matching degree calculation process, and FIG. FIG. 10 is a schematic diagram showing still another example of a pattern matching degree calculation process.

  In FIG. 16A, the number of matching pixels is calculated as “24”. Further, since all the “8” directions exist in the matching pattern in the gradient direction, the threshold value exceeds 5, and is determined as the “permitted pattern” in the flow of FIG. 15A. On the other hand, the maximum number of concatenations in the matching pattern, the number of “existing” concatenated, is “8”, which exceeds the threshold value of 5, and is determined as “permitted pattern” in the flow of FIG. Is done. Accordingly, in the case of FIG. 16A, the pattern matching degree calculation unit 9 calculates “1” as the pattern matching degree, and the score calculation unit 10 calculates the number of matching pixels calculated by the matching pixel number calculation unit 7. After multiplying “24” and “1”, the number of normalized matching pixels is calculated as a score.

  In FIG. 16B, the number of matching pixels is calculated as “24”. Further, since the “6” direction exists in the matching pattern in the gradient direction, the threshold value exceeds 5, and is determined as the “permitted pattern” in the flow of FIG. 15A. On the other hand, the maximum number of concatenations in the matching pattern, the number of “existing” concatenated, is “6”, which exceeds the threshold value of 5, and is determined as “permitted pattern” in the flow of FIG. Is done. Therefore, in the case of FIG. 16B, the pattern matching degree calculation unit 9 calculates “1” as the pattern matching degree, and the score calculation unit 10 calculates the number of matching pixels calculated by the matching pixel number calculation unit 7. After multiplying “24” and “1”, the number of normalized matching pixels is calculated as a score.

  In FIG. 16C, the number of matching pixels is calculated as “24”. Further, since the “6” direction exists in the matching pattern in the gradient direction, the threshold value exceeds 5, and is determined as the “permitted pattern” in the flow of FIG. 15A. On the other hand, the maximum number of concatenations in the matching pattern, the number of “existing” concatenated, is “6”, which exceeds the threshold value of 5, and is determined as “permitted pattern” in the flow of FIG. Is done. As in this example, assuming that the left end and the right end of the matching pattern table are connected (periodic boundary condition), the maximum number of connections in the matching pattern is calculated.

  As a result of the above, in the case of FIG. 16C, the pattern matching degree calculation unit 9 calculates “1” as the pattern matching degree, and the score calculation unit 10 calculates the coincidence calculated by the matching pixel number calculation unit 7. After multiplying the number of pixels “24” and “1”, the number of normalized matching pixels is calculated and used as a score.

  Next, a second example in the matching pattern check will be described with reference to FIGS.

  17A is a schematic diagram showing still another example of the pattern matching degree calculation process, and FIG. 17B is a schematic diagram showing still another example of the pattern matching degree calculation process. (C) is a schematic diagram showing still another example of the pattern matching degree calculation process.

  In FIG. 17A, the number of matching pixels is calculated as “24”. Further, since the “6” direction exists in the matching pattern in the gradient direction, the threshold value exceeds 5, and is determined as the “permitted pattern” in the flow of FIG. 15A. On the other hand, the maximum number of concatenations in the matching pattern, the number of “existing” concatenated is “4”, which is 5 or less of the threshold value, and is determined as “non-permitted pattern” in the flow of FIG. Is done.

  Therefore, in the case of FIG. 17A, when using the flow of FIG. 15A, the pattern matching degree calculation unit 9 calculates “1” as the pattern matching degree, and the score calculation unit 10 After multiplying the number of matching pixels “24” and “1” calculated by the pixel number calculation unit 7, the number of normalized matching pixels is calculated and used as a score.

  On the other hand, when the flow of FIG. 15B is used, the pattern matching degree calculation unit 9 calculates “0” as the pattern matching degree, and the score calculation unit 10 calculates the coincidence calculated by the matching pixel number calculation unit 7. The number of pixels “24” is multiplied by “0” to obtain “0” as a score.

  In FIG. 17B, the number of matching pixels is calculated as “22”. The matching pattern in the gradient direction is equal to or less than the threshold value 5 because the “4” direction exists, and is determined as a “non-permitted pattern” in the flow of FIG. On the other hand, the maximum number of concatenations in the matching pattern, the number of “existing” concatenated, is “2”, which is 5 or less of the threshold, and is determined as “non-permitted pattern” in the flow of FIG. Is done.

  Accordingly, in the case of FIG. 17B, the pattern matching degree calculation unit 9 calculates “0” as the pattern matching degree, and the score calculation unit 10 calculates the number of matching pixels calculated by the matching pixel number calculation unit 7. Multiplying “22” and “0” gives “0” as the score.

  In FIG. 17C, the number of matching pixels is calculated as “22”. The matching pattern in the gradient direction is equal to or less than the threshold value 5 because the “4” direction exists, and is determined as a “non-permitted pattern” in the flow of FIG. On the other hand, the maximum number of concatenations in the matching pattern, the number of “existing” concatenated is “4”, which exceeds the threshold value of 5, and is also “non-permitted pattern” in the flow of FIG. Determined.

  As a result of the above, in the case of FIG. 17C, the pattern matching degree calculation unit 9 calculates “0” as the pattern matching degree, and the score calculation unit 10 matches the matching pixel number calculation unit 7. The number of pixels “22” is multiplied by “0” to obtain “0” as a score.

  As described above, the score calculation unit 10 performs matching between the matching area and the model pattern, and the number of pixels (the number of matching pixels) in which the gradient direction included in the matching area matches the gradient direction included in the model pattern. ), And the pattern matching degree indicating the degree of similarity between the matching pattern between the gradient direction for each pixel included in the matching region and the gradient direction for each pixel included in the model pattern, and a predetermined comparison matching pattern. The score (degree of match) is calculated.

(6. Contact / non-contact identification process)
Next, a contact / non-contact identification process in the image processing apparatus 1 will be described.

  FIG. 18 is a flowchart showing the operation of the contact / non-contact identification process among the operations of the image processing apparatus 1.

  In FIG. 18, in S701, the score calculation unit 10 proceeds to S702 when the degree of match calculated by itself is equal to or greater than a predetermined value (Yes). On the other hand, when the degree of coincidence calculated by itself is not equal to or greater than the predetermined value (No), the operation of the contact / non-contact identification process is ended.

  In S702, the second edge extraction unit 15 extracts the second edge pixel based on the determination of the score calculation unit 10 that the degree of match is equal to or greater than a predetermined value, and the process proceeds to S703.

  In S703, the score calculation unit 10 acquires the first edge pixel number and the second edge pixel number from the second edge extraction unit 15, and the proportion of the first edge pixels occupied by the second edge pixels exceeds a predetermined value. Judge whether or not. If it exceeds the predetermined value (Yes), the process proceeds to S704. On the other hand, if the predetermined value is not exceeded (No), the operation of the present contact / non-contact identification process ends.

  In S <b> 704, the score calculation unit 10 calculates the degree of coincidence calculated by itself this time based on its own determination that the proportion of the first edge pixels occupied by the second edge pixels exceeds a predetermined value. 11 is output.

  In this way, the operation of the contact / non-contact identification process shown in FIG. 18 is executed.

  Here, the effect of this operation will be described using a specific example. As shown in FIGS. 2C and 2D, when the surrounding brightness becomes a certain level or more, the pixel values of the portions other than the finger that are close to the display screen become brighter than the pixel values of the finger. When the finger is in contact with the display screen, for example, a captured image as shown in FIG. 19 is obtained.

  In this case, the gradient directions of the first edge pixels extracted by the first edge extraction unit 4 are arranged as shown in FIG. Then, pattern matching is performed at the portion A in the figure.

  Further, the second edge pixels extracted by the second edge extraction unit 15 are arranged as shown in FIG.

  On the other hand, FIG. 22 is a captured image when the finger is approaching the display screen but is not in contact. And the gradient direction of the 1st edge pixel extracted by the 1st edge extraction part 4 may become arrangement | positioning as shown in FIG. In this case, there is a possibility that pattern matching will be performed at the portion B in the figure.

  However, in the captured image shown in FIG. 22, the second edge extraction unit 15 does not extract the second edge pixel as shown in FIG. 24. This is because there is no second edge pixel in the first edge pixel in which the vertical gradient amount Sy and the horizontal gradient amount Sx or the gradient magnitude ABS (S) is equal to or greater than the second threshold value. .

  Therefore, even if pattern matching is performed on image data of a non-contact finger, it is possible to specify contact / non-contact with higher accuracy by extracting the second edge pixel.

  Next, a case where the brightness of the usage space of the image processing apparatus 1 changes will be described. As described above, when the surrounding brightness becomes a certain level or higher, the pixel values of portions other than the finger that are close to the display screen become brighter than the pixel value of the finger. When the surroundings become bright, the difference between the pixel value of the background portion other than the finger and the pixel value of the finger increases, that is, the edge strength of the fingertip changes according to the surrounding brightness.

  Here, the second threshold value used by the second edge extraction unit 15 is the difference between the pixel value of the target pixel and the pixel value of the adjacent pixel, that is, the difference between the pixel value of the background portion and the pixel value of the finger. Is set. For this reason, when the edge strength of the fingertip changes according to the surrounding brightness, if the second threshold value is set uniformly, the second edge pixel cannot be extracted with high accuracy.

  Therefore, when the brightness of the usage space of the image processing apparatus 1 changes, the second threshold value is adjusted according to the surrounding brightness, so that the second edge pixel is extracted even if the surrounding brightness changes. Is executed with high accuracy.

  Specifically, in the image processing apparatus 1 according to the present embodiment, as illustrated in FIG. 1, the threshold adjustment unit 16 acquires the brightness of the usage space, and adjusts the second threshold according to the brightness. And output to the second edge extraction unit 15.

  For example, the threshold adjustment unit 16 acquires the pixel value of the background image in the captured image. For example, a histogram of the captured image is generated, and a pixel value of a bright part on the histogram is set as a pixel value of the background image.

  Further, the maximum value of the average pixel value in the rectangle when a rectangle having a predetermined size is scanned on the captured image may be used as the pixel value of the background image.

  Furthermore, when collating with the model pattern, a histogram may be generated using a plurality of pixels outside the model pattern, and the pixel value of a bright part on the histogram may be used as the pixel value of the background image.

  Then, the threshold adjustment unit 16 acquires a second threshold corresponding to the pixel value of the background image acquired in this way from the threshold storage unit 17 and outputs the second threshold to the second edge extraction unit 15.

  In this way, by adjusting the second threshold according to the change in ambient brightness, the second edge pixel can be extracted with high accuracy even when the ambient brightness changes.

  Next, a case where the use space of the image processing apparatus 1 becomes too bright will be described. If the surroundings become too bright, the pixel value of the finger will increase due to the influence of light wrapping around the finger. As a result, if the sensitivity of the photosensor is not sufficient to detect the pixel value of the finger, the photosensor may not be able to recognize the finger image.

  In order to recognize a finger in such a case, it is necessary to reduce the sensitivity of the optical sensor. However, if the sensitivity of the optical sensor is changed, the difference between the pixel value of the background image and the pixel value of the finger pad also changes at the same time. For this reason, similarly to the case where the ambient brightness changes, the second threshold value must be adjusted in accordance with the change even when the sensitivity of the light sensor is changed. Hereinafter, the adjustment of the second threshold value in this case will be described.

  FIG. 25 shows the relationship between the pixel values of the background image and the finger pad image and the illuminance of the brightness of the usage space. In FIG. 25, when the sensitivity of the optical sensor is the standard sensitivity, the pixel value of the background image and the pixel value of the finger pad image, and the sensitivity of the optical sensor is 1/2 the standard sensitivity (hereinafter referred to as “1/2”). In this case, the pixel value of the background image and the pixel value of the finger pad image are shown.

  As shown in FIG. 25, when the sensitivity of the optical sensor is reduced from the standard sensitivity to ½ sensitivity, the inclination of the pixel value with respect to the illuminance in the usage space is the light in both the background image and the finger pad image. The sensor sensitivity is ½ of the standard sensitivity. Further, the pixel value of the finger pad image is ½ of the case where the illuminance of the use space is 0, that is, the pixel value at 0 Lux, when the optical sensor sensitivity is the standard sensitivity.

  Here, the pixel value of the background image at the standard sensitivity and the pixel value of the finger pad image, and the pixel value of the background image at the sensitivity of 1 / n of the standard sensitivity (hereinafter referred to as “1 / n sensitivity”). Each of the pixel values of the finger pad image is represented by the following equation with respect to illuminance.

Pixel value of background image at standard sensitivity:
vb ₁ = a _vb x (8)
x: Illuminance, a _vb : Constant Pixel value of the background image at 1 / n sensitivity:
vb _n = (a _vb / n) x (9)
n: Sensitivity Pixel value of finger pad image at standard sensitivity:
vf ₁ = a _vf x + b (10)
a _vf : Constant, b: Constant Pixel value of finger pad image at 1 / n sensitivity:
vf _n = (a _vf / n) x + (1 / n) b (11)
The threshold adjustment unit 16 holds in advance each constant used in the above formulas (8) to (11). In addition, the threshold storage unit 17 stores a plurality of second threshold values corresponding to the pixel values of the background image on the captured image on a one-to-one basis. The second threshold stored in the threshold storage unit 17 is set in advance on the assumption that the sensitivity of the optical sensor takes the standard sensitivity described above.

  When the sensitivity of the optical sensor changes from the standard sensitivity to the 1 / n sensitivity, the threshold adjustment unit 16 acquires the pixel value of the background image and the sensitivity of the optical sensor, and adjusts the second threshold using them.

  Specifically, when the threshold adjustment unit 16 acquires the brightness of the usage space, the threshold value adjustment unit 16 specifies the pixel value of the current background image from the brightness. Then, the threshold adjustment unit 16 obtains a difference between the pixel value of the finger pad image and the pixel value of the background image, so that the same difference is obtained when the sensitivity of the optical sensor is the standard sensitivity. Pixel values are calculated using the above equations (8) to (11).

  The pixel value of the standard-sensitivity background image is calculated from the current background pixel value by converting Equations (8) to (11) as follows.

  That is, the following equations are given from equations (8) and (10).

vf ₁ = (a _vf / a _vb ) vb ₁ + b (12)
Moreover, the following formulas are given from formulas (9) and (11).

vf _n = (a _vf / a _vb ) vb _n + (1 / n) b
vb _n −vf _n = (1− (a _vf / a _vb )) vb _n − (1 / n) b (13)
In order to obtain the difference between the pixel values of the current background image and the finger pad image and calculate the pixel value of the background image with the standard sensitivity so that the same difference is obtained with the standard sensitivity, the above equations (12) and (13) are used. And perform the conversion in the following order:

(A) vb ₁ −vf ₁ = vb _n −vf _n
(B) vb ₁ = vb _n −vf _n + vf _{1
(C) vb 1 = (1-} (a vf / a vb)) vb n - (1 / n) b + (a vf / a vb) vb 1 + b
(D) (1- (a _vf / a _vb )) vb ₁ = (1- (a _vf / a _vb )) vb _n + ((n−1) / n) b
(E) vb ₁ = vb _n + ((n−1) / n) b / (1− (a _vf / a _vb ))
In this way, the threshold adjustment unit 16 calculates the pixel value of the standard sensitivity background image, acquires the second threshold corresponding to the pixel value from the threshold storage unit 17, and outputs the second threshold to the second edge extraction unit 15. And since the 2nd edge extraction part 15 can acquire the 2nd threshold value adjusted according to the change of the sensitivity of an optical sensor from the threshold value adjustment part 16, it extracts a 2nd edge pixel with high precision. Can do.

  By the way, as shown in FIGS. 2C and 2D, the effect of the above-described operation of FIG. 18 is that the pixel values of the portions other than the finger that are close to the display screen when the surrounding brightness becomes a certain level or more. Is effective only when it becomes brighter than the pixel value of the finger. Therefore, as shown in FIGS. 2A and 2B, the above operation cannot be used when the pixel value of the portion other than the finger close to the display screen is darker than the finger pixel value.

  Therefore, in such a case, it is possible to always create a situation in which the pixel value of the background image is brighter than the pixel value of the finger pad image by devising the configuration of the liquid crystal display of the image processing apparatus 1.

  For example, a transparent substrate and an elastic film may be provided on the surface of the liquid crystal display. The air layer between the transparent substrate and the elastic film reflects light when no pressure is applied to the surface of the liquid crystal display. On the other hand, when the pressure is applied, since the reflection in the air layer is lost, the reflectance becomes low.

  FIG. 26 shows the relationship between the pixel value of each of the background image and finger pad image and the illuminance of the brightness of the use space when a transparent substrate and an elastic film are provided on the surface of the liquid crystal display. In FIG. 26, the pixel value of the background image and the finger pad image when the sensitivity of the optical sensor is the standard sensitivity, and the pixel value of the background image and the finger when the sensitivity of the optical sensor is ½ sensitivity. A pixel value of the abdomen image is shown.

  In this case, as shown in FIG. 26, a situation where the pixel value of the background image is always brighter than the pixel value of the finger pad image regardless of whether the sensitivity of the photosensor is the standard sensitivity or the half sensitivity. Can do.

  Therefore, even when the pixel value of the portion other than the finger close to the display screen becomes darker than the finger pixel value as shown in FIGS. 2A and 2B, the effect of the operation of FIG. 18 is effective. Can be. As a result, even in such a case, the second edge extraction unit 15 can acquire the second threshold value adjusted according to the change in the sensitivity of the optical sensor from the threshold value adjustment unit 16, so that the second Edge pixels can be extracted with high accuracy.

(7. Pointing position identification process)
Next, a pointing position specifying process in the image processing apparatus 1 will be described.

  FIG. 27 is a flowchart showing the operation of the pointing position coordinate calculation process among the operations of the image processing apparatus 1.

  In S <b> 801, the peak search unit 12 selects a peak pixel that is a pixel having a maximum matching value calculated by the score calculation unit 10 in a first region (search region) including a predetermined number of pixels around the target pixel. If a peak pixel is found by searching, the process proceeds to S802. Although not shown, if the peak searching unit 12 cannot find the peak pixel, the pixel of interest has a predetermined number of pixels (for example, the shortest course from the pixel of interest in the first region to the end pixel (in the second region). The size of one side)) is shifted and the process returns to S801.

  In S802, the coordinate calculation determination unit 13 has a pixel of interest common to the first area, has a predetermined number of pixels smaller than the number of pixels in the first area, and is completely included in the first area. If it is determined that the peak pixel discovered by the peak search unit 12 exists in (small region), the process proceeds to S803, and the coordinate calculation determination unit 13 determines that “there is a peak pixel” and proceeds to S804. . On the other hand, when the coordinate calculation determination unit 13 does not include the peak pixel found by the peak search unit 12 in the second region (small region), the process proceeds to S805, where the coordinate calculation determination unit 13 It is determined that there is no pixel, and the target pixel is shifted by a predetermined number (for example, the shortest course (the size of one side of the second region) from the target pixel to the end pixel in the first region), and the process returns to S801.

  The above procedure is repeated until the coordinate calculation unit 14 calculates a pointing (interpolation) position.

  In S804, the coordinate calculation unit 14 uses the score for each pixel in the peak pixel region, which is a region including a predetermined number of pixels centered on the peak pixel found by the peak search unit 12, and the captured image by the imaging target The upper indicated position is calculated to be “END”.

  In the above description, the case where the processing is repeated until the coordinate calculation unit 14 calculates the pointing (interpolation) position has been described. However, a configuration in which a plurality of pointing (interpolation) positions can be calculated may be used. It is only necessary to move the first and second areas and execute the processing of the flowchart shown in FIG.

  Next, determination of the presence / absence of a peak pixel will be described with reference to FIGS. 28 (a) and 28 (b).

  FIG. 28A is a schematic diagram for explaining a case where the coordinate calculation determination unit 13 in the image processing apparatus 1 determines that there is no peak pixel, and FIG. It is a schematic diagram for demonstrating the case where it determines with having a pixel.

  The solid line shown in FIG. 28A is the first area, and the broken line is the second area. The number of pixels in the first region is 9 × 9 pixels, and the number of pixels in the second region is 5 × 5 pixels. The reason why each of them is odd number × odd number is to make the center pixel of interest one pixel.

  In the example of FIG. 28A, the peak pixel “9” exists in the first region, but the peak pixel does not exist in the second region. Therefore, in this case, the coordinate calculation determination unit 13 determines “no peak pixel”.

  On the other hand, in the example of FIG. 28B, the peak pixel “9” exists in the first region and also exists in the second region. Therefore, in this case, the coordinate calculation determination unit 13 determines that “there is a peak pixel”.

  In the above example, when there is a peak pixel in the first region, and there is no peak pixel in the second region, the shortest course (the first course) from the target pixel to the end pixel in the first region. If the first region and the second region are moved by “5 pixels” which is the size of one side of the two regions), the first region, the second region, The difference from the number of pixels is set.

  Next, a method for calculating the pointing (interpolation) coordinates (indicated position on the captured image by the imaging target) of the coordinate calculation unit 14 will be described with reference to FIGS.

  FIG. 29A is a schematic diagram for explaining the peak pixel region used for calculating the designated position on the captured image by the imaging target in the image processing apparatus 1, and FIG. FIG. 3 is a schematic diagram for explaining a coordinate calculation method of pointing (interpolation) coordinates in the processing device 1;

  FIG. 29A illustrates a case where the coordinate calculation determination unit 13 determines that “there are peak coordinates”, which is the same as in FIG. 28B.

  In FIG. 29A, both the first area and the second area are indicated by broken line areas. On the other hand, a 5 × 5 pixel region indicated by a solid line is a peak pixel region that is a region including a predetermined number of pixels centered on the peak pixel.

  In the example shown in FIG. 29A, this peak pixel region is also completely included in the first region, like the second region. In this case, there is no need to examine the score in the peak pixel region again. Thus, it is preferable that the peak pixel region is included in the first region even when the peak pixel exists at the end of the second region.

  Next, a pointing coordinate calculation method of the coordinate calculation unit 14 will be described with reference to FIG.

  In this example, when the number of pixels of the image data is 320 × 240 pixels, the resolution reducing unit 2 in FIG. 1 performs bilinear reduction twice, and the efficiency improving unit 6 performs the processing for 2 × 2 pixels. It is assumed that collation efficiency is improved and the size of the score image (score data assigned to each pixel) is 40 × 30 pixels.

  Accordingly, the entire area of the score image enlarged by 8 times corresponds to the entire area of the image data. Therefore, the interpolation amount (scale enlargement amount) = 8.

  The specific calculation method is as follows. First, the sum of scores for each row in the peak pixel region is calculated (19, 28, 33, 24, and 11 in FIG. 29B).

  Next, the sum of scores for each column in the peak pixel region is calculated (16, 24, 28, 26, and 21 in FIG. 29B). Also, the sum of the scores in the peak pixel region (5 × 5 pixels) is obtained (115 in FIG. 29B).

  Next, assuming that the score value in the peak pixel region corresponds to the mass distribution, the center-of-gravity coordinates in the entire region of the score image are obtained, and when the scale is expanded eight times, the coordinates of the following equations (14) and (15) are obtained. Get.

  Next, when the scale position is adjusted in consideration of the pixel size, the pointing coordinates (X, Y) are expressed by the following equation (16).

  According to the above, since the peak search unit 12 searches in the first area (search area), the processing cost is reduced and the memory capacity is reduced compared to searching for the peak pixel in the image data area including the total number of pixels. Can be made small.

  For example, the fact that the number of pixels in the first region is small means that it is not necessary to store the score of the entire data image (score image) (= all pixels) in the buffer, and it is necessary for the first region to execute the peak search. This means that it is sufficient to have a memory capacity (for example, in the case of the first area of 9 × 9 pixels, a line buffer for 9 lines).

  In this way, the effect of reducing the memory capacity by implementing with a line buffer is not limited to peak search, but when transferring to subsequent processing such as temporary storage of vertical and horizontal gradient amounts, temporary storage of gradient directions, etc. This is the same in all cases where the implementation includes a buffer memory.

  Furthermore, the coordinate calculation unit 14 calculates a pointing position using a score for each pixel in a peak pixel region that is a region including a predetermined number of pixels centered on the peak pixel found by the peak search unit 12. For example, when the pointing position is obtained from the position of the center of gravity using an edge image, it is considered that the calculation becomes more difficult as the captured image is deformed into a distorted shape.

  However, the image processing apparatus 1 calculates the pointing position using a score obtained by pattern matching for each pixel in the peak pixel region. In the vicinity of the maximum value of the score in this pattern matching, even if the captured image is deformed into a distorted shape, the degree of coincidence decreases radially, with the vicinity of the maximum value being the center. It is thought that it shows.

  Therefore, regardless of whether or not the captured image is deformed into a distorted shape, the pointing position can be calculated by a predetermined procedure (for example, calculating the center of gravity of the score in the peak pixel region). Therefore, it is possible to reduce the image processing amount for calculating the pointing position, to reduce the processing cost, and to reduce the memory capacity while achieving both the maintenance of the coordinate position detection accuracy.

  As described above, by performing pattern matching using only one frame of image data regardless of detection of touch / non-touch on the captured image of the imaging target, the pointing position can be detected, the memory capacity can be reduced, and To provide an image processing apparatus 1 capable of realizing both reduction in image processing amount and maintenance of pointing position detection accuracy and reduction in memory capacity in calculation of a pointing position while reducing processing time. Can do.

  In addition, the coordinate calculation determination unit 13 has a target pixel common to the first region, has a predetermined number of pixels smaller than the number of pixels in the first region, and is a second region that is completely included in the first region. When it is determined that the peak pixel discovered by the peak search unit 12 exists in (small region), it is preferable to cause the coordinate calculation unit 14 to calculate the pointing position.

  Since the peak pixel area is an area spreading around the peak pixel existing in the second area as a target pixel, there are many pixels in common with the first area. Further, since the score of the common pixel between the peak pixel region and the first region has already been calculated, the pointing position can be calculated by the coordinate calculation unit 14 by examining the score of the non-common pixel.

  Further, the peak pixel region can be included in the first region by adjusting the number of pixels of the peak pixel region and the first region. In this case, since the score of each pixel in the peak pixel region is already known, there is no need to examine the score of each pixel that is not yet known for the calculation of the pointing position.

  As described above, in the calculation of the pointing position, it is possible to further reduce the image processing amount and the memory capacity. In addition, when the peak coordinates are determined, the score to be referred to when pipeline processing is performed for each processing module in response to a case where the increase in score is contiguous toward the outside of the first region, and hardware implementation, etc. The holding buffer capacity can be reduced (for example, only 9 lines, not the entire image).

  In addition, each block of the image processing apparatus 1, in particular, the resolution reduction unit 2, the vertical gradient amount calculation unit 3 a, the lateral gradient amount calculation unit 3 b, the first edge extraction unit 4, the direction identification unit 5, the efficiency improvement unit 6, and the matching pixel The number calculating unit 7, the pattern matching degree calculating unit 9, the score calculating unit 10, the position specifying unit 11, the second edge extracting unit 15, and the threshold adjusting unit 16 may be configured by hardware logic, Thus, it may be realized by software using a CPU.

  That is, the image processing apparatus 1 includes a CPU (central processing unit) that executes instructions of a control program that realizes each function, a ROM (read only memory) that stores the program, and a RAM (random access memory) that expands the program. And a storage device (recording medium) such as a memory for storing the program and various data. An object of the present invention is a recording medium on which a program code (execution format program, intermediate code program, source program) of a control program of the image processing apparatus 1 which is software that realizes the above-described functions is recorded so as to be readable by a computer. This can also be achieved by supplying the image processing apparatus 1 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

  Examples of the recording medium include a tape system such as a magnetic tape and a cassette tape, a magnetic disk such as a floppy (registered trademark) disk / hard disk, and a compact disk-ROM / MO / MD / digital video disk / compact disk-R. A disk system including an optical disk, a card system such as an IC card (including a memory card) / optical card, or a semiconductor memory system such as a mask ROM / EPROM / EEPROM / flash ROM can be used.

  The image processing apparatus 1 may be configured to be connectable to a communication network, and the program code may be supplied to the image processing apparatus 1 via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Also, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope shown in the claims. That is, embodiments obtained by combining technical means appropriately modified within the scope of the claims are also included in the technical scope of the present invention.

  The image processing apparatus of the present invention can be applied to an apparatus that performs operations and instructions by touching a display of a display device such as a liquid crystal display, such as a mobile phone or a PDA. Specifically, as a display device, for example, it can be used for an active matrix liquid crystal display device, and an electrophoretic display, a twist ball display, a reflective display using a fine prism film, a digital mirror device, etc. In addition to displays using light modulation elements, organic EL light-emitting elements, inorganic EL light-emitting elements, displays using variable-light-emitting elements such as LEDs (Light Emitting Diodes), and field emission displays (FED) It can also be used for plasma displays.

It is a block diagram which shows the structure of the image processing apparatus in embodiment of this invention. (A) is a schematic diagram showing the state of a captured image of the finger pad when the surrounding is dark, (b) is a schematic diagram showing the characteristics of the captured image of the finger pad when the surrounding is dark, and (c) is the surrounding FIG. 4D is a schematic diagram showing a captured image of the finger pad when the surrounding is bright, FIG. 4D is a schematic diagram showing the characteristics of the captured image of the finger pad when the surrounding is bright, and FIG. (F) is a schematic diagram illustrating the characteristics of a captured image of the pen tip when the surrounding is dark, and (g) is a captured image of the pen tip when the surrounding is bright. (H) is a schematic diagram illustrating characteristics of a captured image of the pen tip when the surroundings are bright. It is a flowchart which shows the outline | summary of the whole operation | movement of an image processing apparatus. It is a flowchart which shows the operation | movement of a gradient direction / non-direction specific process among operation | movement of an image processing apparatus. (A) is an example of the table referred in the gradient direction / non-direction specifying process, and (b) is another example of the table referred in the gradient direction / non-direction specifying process. (A) is a schematic diagram showing the characteristics of the gradient direction of image data when the surroundings are dark, and (b) is a schematic diagram showing a state after performing collation efficiency improvement for the same figure. (A) is a schematic diagram showing an example of a model pattern before collation efficiency improvement when the surroundings are dark, and (b) is a schematic diagram showing an example of a model pattern before collation efficiency improvement when the surroundings are bright. is there. (A) is a schematic diagram showing an example of a model pattern after collation efficiency improvement when the surroundings are dark, and (b) is a schematic diagram showing an example of a model pattern after collation efficiency improvement when the surroundings are bright. is there. (A) is a schematic diagram showing another example of a model pattern after collation efficiency improvement when the surroundings are dark, and (b) is another example of a model pattern after collation efficiency improvement when the surroundings are bright. FIG. It is a flowchart which shows operation | movement of a pattern matching process among operation | movement of an image processing apparatus. (A) is a schematic diagram for explaining pattern matching between a matching region and a model pattern in the case where the surroundings before darkening is dark, and (b) is a diagram showing an example of the matching degree calculation method It is. (A) is a schematic diagram for explaining an example of pattern matching between a matching area and a model pattern in the case where the surrounding after darkening is dark, and (b) is an example of the matching degree calculation method. FIG. (A) is a schematic diagram for explaining another example of pattern matching between a matching region and a model pattern in the case where the surroundings after darkening the matching efficiency are dark, and (b) It is a figure which shows an example. It is a flowchart for demonstrating the case where a matching pixel number and a pattern matching degree are used together in the pattern matching in an image processing apparatus. (A) is a flowchart which shows an example of a pattern matching degree calculation process, (b) is a flowchart which shows the other example of a pattern matching degree calculation process. (A) is a schematic diagram showing an example of a pattern matching degree calculation process, (b) is a schematic diagram showing another example of the pattern matching degree calculation process, and (c) is a pattern matching degree calculation process. It is a schematic diagram which shows another example of these. (A) is a schematic diagram showing still another example of the pattern matching degree calculation process, (b) is a schematic diagram showing still another example of the pattern matching degree calculation process, and (c) is a pattern. It is a schematic diagram which shows the further another example of a coincidence degree calculation process. It is a flowchart which shows operation | movement of a contact / non-contact specific process among operation | movement of an image processing apparatus. It is a schematic diagram which shows the mode of the captured image of a finger pad when the circumference | surroundings are bright (at the time of contact). It is a schematic diagram which shows the gradient direction of the 1st edge pixel about the captured image of the finger pad when the surrounding is bright (at the time of contact). It is a schematic diagram which shows the mode of the 2nd edge pixel about the captured image of the finger pad when the circumference | surroundings are bright (at the time of contact). It is a schematic diagram which shows the mode of the captured image of a finger pad when the circumference | surroundings are bright (at the time of non-contact). It is a schematic diagram which shows the gradient direction of the 1st edge pixel about the captured image of the finger pad when the periphery is bright (at the time of non-contact). It is a schematic diagram which shows the mode of the 2nd edge pixel about the captured image of the finger pad when the circumference | surroundings are bright (at the time of non-contact). It is a graph which shows the relationship between each pixel value of a background image and a finger pad image, and the illumination intensity of the brightness of use space. It is a graph which shows the other relationship between each pixel value of a background image and a finger pad image, and the illumination intensity of the brightness of use space. It is a flowchart which shows operation | movement of the pointing position coordinate calculation process among operation | movement of an image processing apparatus. (A) is a schematic diagram for explaining a case where it is determined that there is no peak pixel of the coordinate calculation determination unit in the image processing apparatus, and (b) is a peak pixel of the coordinate calculation determination unit in the image processing apparatus. It is a schematic diagram for demonstrating the case where it determines with. (A) is a schematic diagram for explaining a peak pixel region used for calculating an instruction position on a captured image by an imaging target in the image processing apparatus, and (b) is an instruction for the instruction in the image processing apparatus. It is a schematic diagram for demonstrating the coordinate calculation method of a position.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 Resolution reduction part 3a Vertical gradient amount calculation part (feature amount calculation means)
3b Lateral gradient amount calculation unit (feature amount calculation means)
4 1st edge extraction part (1st edge specific means)
DESCRIPTION OF SYMBOLS 5 Direction identification part 6 Efficiency improvement part 7 Matching pixel number calculation part 8 Model pattern and comparison matching pattern storage part 9 Pattern matching degree calculation part 10 Score calculation part (matching degree calculation means)
11 position specifying part (position specifying means)
12 peak search unit 13 coordinate calculation determination unit 14 coordinate calculation unit 15 second edge extraction unit (second edge specifying means)
16 Threshold adjustment unit (threshold adjustment means)
17 Threshold storage unit 20 Electronic device 61, 62, 63, 64 Captured image 65 Display screen 66 External light

Claims (13)

An image processing apparatus having a function of specifying an instruction position on the captured image by an imaging target using image data of a captured image,
For each pixel on the image data, a feature amount calculating means for calculating a feature amount according to a pixel value difference with an adjacent pixel;
First edge specifying means for specifying a first edge pixel whose feature value calculated by the feature value calculating means is not less than a first threshold;
Based on the identification result of the first edge pixel, a matching area included in a part of the image data is matched with a predetermined model pattern, and matching between the matching area and the model pattern is performed. A degree-of-match calculation means for calculating a degree of match indicating the degree;
Among the first edge pixels, a second edge specifying means for specifying a second edge pixel whose feature value is equal to or greater than a second threshold value that is larger than the first threshold value;
Position specifying means for specifying the indicated position on the captured image by the imaging target based on the ratio between the first edge pixel number and the second edge pixel number and the matching degree calculated by the matching degree calculating means. An image processing apparatus.   The position specifying means has a ratio of the number of second edge pixels to the number of first edge pixels exceeding a predetermined value, and the degree of match calculated by the degree of match calculation means is a portion on the image data. The image processing apparatus according to claim 1, wherein an instruction position on the captured image by an imaging target is specified from a position of a collation area that is the largest in the area.   The image processing apparatus according to claim 2, further comprising a threshold adjustment unit that adjusts the second threshold according to a change in brightness in a use space of the image processing apparatus.   The threshold adjustment means includes a plurality of second threshold values stored in advance in a table format for each pixel value that can be taken by the background image on the captured image in accordance with a change in brightness in the usage space of the image processing apparatus. The image processing apparatus according to claim 3, wherein a second threshold value corresponding to a pixel value of a background image on the captured image is selected. The plurality of second threshold values stored in the table format is used when the sensitivity of the image sensor that captures image data of a captured image has the first sensitivity.
The second threshold value adjusting means is to be used when the sensitivity of the imaging sensor changes from the first sensitivity to a second sensitivity different from the first sensitivity, and the sensitivity of the imaging sensor has the second sensitivity. When selecting a threshold,
In the case where it is assumed that the pixel value difference between the designated position on the captured image at the first sensitivity and the background image is the pixel value difference between the designated position on the captured image at the second sensitivity and the background image. Calculate the pixel value to be taken by the background image on the captured image with one sensitivity,
The image processing apparatus according to claim 4, wherein a second threshold value corresponding to the pixel value of the background image on the calculated captured image is selected.   The image processing apparatus according to claim 1, wherein the feature amount is a vector amount having a size and a direction.   The image processing apparatus according to claim 1, wherein the feature amount is a gradient amount of luminance of a pixel.   The image processing program for operating a computer as each means in the image processing apparatus of any one of Claims 1-7.   A computer-readable recording medium on which the image processing program according to claim 8 is recorded.   An electronic apparatus comprising the image processing apparatus according to claim 1. An image processing method for specifying an instruction position on the captured image by an imaging target using image data of a captured image,
For each pixel on the image data, a feature amount calculating step for calculating a feature amount according to a pixel value difference with an adjacent pixel;
A first edge specifying step of specifying a first edge pixel whose feature amount calculated in the feature amount calculating step is equal to or greater than a first threshold;
Based on the identification result of the first edge pixel, a matching area included in a part of the image data is matched with a predetermined model pattern, and matching between the matching area and the model pattern is performed. A degree-of-match calculation step for calculating a degree of match indicating the degree;
A second edge specifying step of specifying, from among the first edge pixels, a second edge pixel whose feature value is equal to or greater than a second threshold value that is greater than the first threshold value;
A position specifying step of specifying the indicated position on the captured image by the imaging target based on the ratio between the first edge pixel number and the second edge pixel number and the matching degree calculated in the matching degree calculating step. An image processing method characterized by comprising: An image processing apparatus having a function of specifying an instruction position on the captured image by an imaging target using image data of a captured image,
Matching degree calculation means for matching a matching area included in a partial area on the image data with a predetermined model pattern and calculating a matching degree indicating a degree of matching between the matching area and the model pattern When,
A feature amount calculating unit that calculates a feature amount according to a pixel value difference with an adjacent pixel for each pixel that matches a model pattern in the matching by the matching degree calculating unit;
Edge specifying means for specifying an edge pixel whose feature value calculated by the feature value calculating means is equal to or greater than a predetermined threshold;
Position specifying means for specifying the indicated position on the captured image by the imaging target based on the ratio between the number of matched pixels and the number of edge pixels and the matching degree calculated by the matching degree calculating means. A featured image processing apparatus.   In the position specifying means, the ratio of the number of edge pixels to the number of matched pixels exceeds a predetermined value, and the degree of matching calculated by the degree of matching calculating means is within a partial area on the image data. The image processing apparatus according to claim 12, wherein an instruction position on the captured image by the imaging target is specified from the position of the collation area that is the maximum.

Images