JP2011113133A - Image processing apparatus, control program, computer readable recording medium, and control method of image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus or the like capable of stably detecting an indication direction of an object to be photographed. <P>SOLUTION: The apparatus includes a gradient feature amount calculating unit 2 for calculating a gradient S from image data of a photographed picked-up image, an edge extracting unit 3 for specifying an edge pixel whose magnitude POW(S) of the gradient calculated by the gradient feature amount calculating unit 2 is not smaller than a predetermined threshold, a matching pixel number calculating unit 5 for calculating the matching pixel number indicating the number of pixels in which a gradient direction ANG(S) included in a checking area and a gradient direction ANG(S) included in a model pattern match after checking the checking area having a predetermined magnitude set on the image data and the previously determined model pattern, a matching area specifying unit 7 for specifying the checking area in which the matching pixel number calculated by the matching pixel number calculating unit 5 becomes to the maximum as the matching area, and an object direction specifying unit 9 for specifying an indication direction of the object on the picked-up image from the distribution of the gradient S in each edge pixel included in the matching area specified by the matching area specifying unit 7. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus having a function of recognizing an object (for example, a finger) in a captured image, a control program for the image processing apparatus, a computer-readable recording medium on which the control program is recorded, and The present invention relates to a method for controlling the image processing apparatus.

  In general, there is known an apparatus that recognizes an indicated position or the like of a predetermined object in a captured image captured using a camera or the like using an image processing technique and performs processing according to the recognition result. For example, there is known an image processing apparatus that recognizes a part such as a human hand or a finger in a captured image, specifies its position, and performs data input or operation according to the specified position.

  As described above, there is an operation input device disclosed in Patent Document 1 as a technique for recognizing an object in a captured image using an image processing technique and specifying an instruction position or the like.

  The operation input device disclosed in Patent Document 1 mainly uses a photographing means for photographing from the back side of a semi-transparent screen, and image data acquired by the photographing means, and a display surface based on the luminance of the image displayed on the semi-transparent screen. Operating position detecting means for detecting the operating position at.

  The operation position detection means detects a plurality of areas in the image data whose luminance is equal to or greater than a threshold as contact areas where the user is touching the display surface, and controls the operation direction when the user selects and operates the display object. The contact area located at the forefront of the operation direction is detected as the operation position.

  Furthermore, the operation position detecting means has a value that is greater than or equal to a second threshold value whose luminance is lower than the threshold value and lower than the threshold value in each of the first region and the second region that are the partitioned areas on the translucent screen. And the center of gravity of the image is calculated for each of the first operation part image displayed in the first area and the second operation part image displayed in the second area. A tip direction vector that passes through the center of gravity and extends from the center of gravity of the second operation part image to the center of gravity of the first operation part image is calculated as the operation direction.

  As described above, the operation input device specifies the operation position and the operation direction using the image input from the photographing unit.

JP 2007-80187 A (published March 29, 2007)

  However, the conventional operation input device disclosed in Patent Document 1 has the following problems.

  First, in this operation input device, since the operation direction is calculated using the image of the operation part recognized in each of the first area and the second area which are the divided areas on the translucent screen, the detection object is the first object. There is a problem in that the operation direction cannot be detected when it does not extend over two areas of the first area and the second area (exists only in one of the two areas).

  In addition, since the operation input device recognizes a plurality of regions whose luminance is equal to or higher than a threshold in the image data as the operation part image, and calculates the operation direction based on the center of gravity of the plurality of operation part images. For example, in the case of a human hand or finger, the brightness on the non-tip side (the base side in the case of a finger) of the contacted part is linked to the change in the angle of the hand or finger with respect to the contact surface. There is also a problem that it is difficult to stably calculate the operation direction because the area and shape of the operation site image change according to the fluctuation.

  The present invention has been made in view of the above-described conventional problems, and an object of the present invention is to provide an image processing apparatus and the like that can stably detect the pointing direction of an imaging target.

  In order to solve the above problems, the image processing apparatus of the present invention includes a gradient feature amount calculating unit that calculates a pixel value gradient feature amount from image data of a captured image, and the gradient feature amount calculating unit Edge pixel specifying means for specifying an edge pixel having a pixel value gradient feature amount equal to or greater than a predetermined threshold, a matching area of a predetermined size set on the image data, a predetermined model pattern, A matching degree calculation means for calculating a matching degree indicating a degree of matching between the matching area and the model pattern, and a matching area where the matching degree calculated by the matching degree calculation means is maximized. From the distribution of the pixel value gradient feature quantity for each edge pixel included in the match area specified by the match area specifying means and the match area specifying means specified as Characterized in that it comprises a direction specifying means for specifying a pointing direction of the object.

  In addition, in order to solve the above-described problem, the image processing apparatus control method of the present invention has a function of specifying an instruction direction of an object on the captured image using image data of the captured image. A control method for an image processing device, comprising: a gradient feature amount calculating step for calculating a pixel value gradient feature amount from the image data; and a magnitude of the pixel value gradient feature amount calculated in the gradient feature amount calculation step is a predetermined value. An edge pixel specifying step for specifying an edge pixel that is equal to or greater than a threshold value, a matching area of a predetermined size set on the image data, and a predetermined model pattern are matched, and the matching area and the model pattern A matching degree calculating step for calculating a matching degree indicating a degree of matching with the matching degree, and the matching region calculated by the matching degree calculating step having the maximum matching degree as a matching region. Specifying the specified direction of the target on the captured image from the matching area specifying step to be specified and the distribution of the pixel value gradient feature quantity for each edge pixel included in the matching area specified in the matching area specifying step And a direction specifying step.

  According to the configuration and method, in the gradient feature amount calculating unit or the gradient feature amount calculating step, the pixel value gradient feature amount is calculated from the image data of the captured image.

  Next, in the edge pixel specifying means or the edge pixel specifying step, an edge pixel having a pixel value gradient feature amount greater than or equal to a predetermined threshold is specified.

  As a result, out of all the pixels on the image data, edge pixels whose pixel value gradient feature value is greater than or equal to a predetermined threshold value are other than edge pixels whose pixel value gradient feature value value is less than the predetermined threshold value. Can be selected from the pixels.

  Next, in the degree-of-match calculation means or the degree-of-match calculation step, a matching area of a predetermined size set on the image data is compared with a predetermined model pattern, and matching between the matching area and the model pattern is performed. The degree of coincidence indicating the degree is calculated.

  Thereby, the degree of matching between the matching area and the model pattern can be recognized from the degree of matching calculated by the degree-of-matching calculation means or the degree-of-matching calculation step.

  Next, in the matching area specifying means or the matching area specifying step, the matching area having the maximum matching degree calculated by the matching degree calculating means or the matching degree calculation step is specified as the matching area.

  As a result, it is possible to identify a matching area indicating the area where the object exists in the image data.

  Next, in the direction specifying means or the direction specifying step, the indication direction of the object on the captured image is specified from the distribution of the pixel value gradient feature value for each edge pixel included in the specified matching region.

  Here, a scalar quantity such as a pixel value (density value or luminance value) is also conceivable as an amount used for collation between the collation area and a predetermined model pattern (hereinafter referred to as “pattern matching”). . However, even if such a scalar quantity is quantized, it can always change according to the condition of the object, and so it is difficult to set a model pattern in advance. Note that quantization means that a quantity within a predetermined range is treated as a constant quantity.

  On the other hand, the pixel value gradient feature amount has at least a degree of freedom in the direction of the gradient direction of the pixel value. The gradient of the pixel value is a vector amount obtained by taking a difference in pixel value between a specific pixel and an adjacent pixel and adding these differences for all adjacent pixels. Has a gradient direction.

  Here, for example, the gradient direction can be limited to an angle of 0 rad or more and less than 2π rad, and by dividing an angle range of 0 rad or more and less than 2π rad equally by an appropriate upper limit angle, the gradient direction having an infinite degree of freedom can be obtained. It is possible to quantize in a finite number of directions.

  In addition, for example, by quantizing the gradient direction into eight directions, the state that can be taken by the gradient direction calculated for one pixel can be discretized into an extremely small number of 8 (9 including no direction). In addition, each state can have a feature that allows easy identification of different directions. Therefore, the pixel value gradient feature amount is an amount suitable for the above-described pattern matching.

  Furthermore, the distribution of each gradient direction is, for example, when the surface is soft like a finger and the contact surface becomes circular by touching the surface, or from the edge part toward the center of the area surrounded by the edge part, Alternatively, it shows any tendency from the vicinity of the center toward the edge portion. In addition, when the object is a finger, the magnitude of the gradient (density difference) at each pixel is small at the base side of the finger regardless of the change in the angle of the finger with respect to the contact surface, and increases toward the tip of the finger. Show the trend. These tendencies do not change greatly according to fluctuations in the angle of the finger. When the object is a finger, the indication direction is usually along the direction from the base of the finger toward the tip of the finger.

  Therefore, if the pixel value gradient feature amount is used, the above-described tendency indicated by the distribution of the pixel value gradient feature amount for each edge pixel among the pixels included in the match region can be used. Therefore, for each edge pixel included in the match region, From the distribution of the pixel value gradient feature amount, the indication direction of the object on the captured image can be specified.

  Therefore, since the pointing direction can be specified using only a single matching area, the object does not span the two areas of the first area and the second area as in the operation input device disclosed in Patent Document 1. In addition, there is no problem that the operation direction cannot be detected. Therefore, it is possible to specify a stable pointing direction.

  Further, since the pixel value gradient feature amount is the difference between the pixel values as described above, for example, even if the pixel value on the base side of the finger changes in conjunction with the change in the angle of the finger, the pixel value gradient The tendency of the feature amount to be smaller at the base of the finger and larger toward the tip of the finger and the gradient direction thereof do not vary so much, and the operation amount of the finger is different from that of the operation input device disclosed in Patent Document 1. There is no problem that the operation direction cannot be stably calculated due to a change in the pixel value on the base side. Therefore, it is possible to specify a stable pointing direction.

  As described above, according to the image processing apparatus and the control method of the present invention, it is possible to stably detect the indication direction of the imaging target.

  Here, the “image data” may be either analog data or digital data. The “pixel value” is a luminance value or a density value. In the case of analog data, each divided area obtained by dividing image data into a plurality of areas is called a pixel, and an average density value in each divided area is called a pixel value. When the “pixel value” is a density value, the “pixel value gradient feature value” may be referred to as a “density gradient feature value”.

  By the way, when there is noise or a pixel defect that occurs during imaging (hereinafter, such a pixel in which such a problem has occurred is referred to as an “error pixel”), the error pixel affects the pixel value gradient feature amount and indicates The direction identification accuracy may be reduced.

  Therefore, in order to solve such a problem, in addition to the above configuration, in the image processing apparatus of the present invention, the matching degree calculation unit includes the edge specified by the edge pixel specifying unit on the collation area. A region including pixels may be compared with the model pattern.

  As a result, pixels other than the less important edge pixels are excluded, and pattern matching is performed only on the edge pixels, so that most of the error pixels described above can be excluded. As a result, it is possible to improve the accuracy of specifying the pointing direction, so that the pointing direction can be detected stably.

  Further, in the image processing apparatus of the present invention, in addition to the above configuration, the coincidence degree calculating unit adjusts the matching area while matching the target pixel included in the matching area with all the pixels included in the image data. A plurality of collation areas are set by scanning one pixel at a time, and the degree of match calculated for each collation area is calculated for each target pixel corresponding to each collation area. The degree of coincidence may be calculated for each of the pixels, and a position specifying unit may be provided for specifying the indicated position of the object on the captured image from the distribution of the degree of match for each pixel included in the match area.

  According to the above-described configuration, the matching degree calculation unit sets a plurality of matching areas by scanning the matching area pixel by pixel while matching the target pixel included in the matching area with all the pixels included in the image data. At the same time, the degree of coincidence calculated for each matching area is calculated for each pixel of interest corresponding to each matching area, thereby calculating the degree of matching for all pixels included in the image data.

  Here, for the sake of simplicity, it is assumed that each of the captured image, the collation region, and the model pattern is a rectangle, but each shape is not limited to a rectangle. In addition, a single pixel to be noticed is determined in advance from the pixels included in the collation area, and the single pixel is referred to as a noticed pixel.

  Here, for example, pattern matching starts from a state in which the pixel in the upper left corner of the image data is set to the position of the target pixel in the matching area, and the matching area is scanned pixel by pixel from the upper left corner to the lower right corner of the image data. However, matching with the model pattern is performed.

  Note that when the pixel at the end (outermost circumference) of the image data is set to the position of the target pixel in the collation area, there may be a case where no pixel exists in a part of the collation area. However, in the case where such a boundary problem occurs, pattern matching may be performed by excluding the pixel in the portion where the pixel in the matching area does not exist. Further, pattern matching may be performed by setting a periodic boundary condition or the like and replacing a pixel in a region where a pixel in the matching region on the matching region does not exist with a model having an appropriate pixel value gradient feature amount.

  Further, the size and shape of the collation region may be set as appropriate according to the size and shape of the model pattern.

  By the pattern matching as described above, there are a plurality of matching areas scanned for each pixel, and the matching degree is calculated for each matching area scanned for each pixel in the matching degree calculation means or the matching degree calculation step. become. That is, the degree of match is calculated for each of all the pixels on the image data.

  Next, the position specifying unit specifies the designated position of the object on the captured image from the distribution of the degree of matching calculated for each pixel included in the matching area.

  Therefore, it is possible to specify the designated position of the object on the captured image from the distribution of the degree of matching for each pixel included in the matching area.

  For example, the position of the pixel having the highest matching degree among the matching degrees for each pixel included in the matching area may be set as the target position of the object, or weighting with the matching degree for each pixel included in the matching area as a weight. The position obtained by averaging may be used as the designated position of the object.

  Further, in the image processing apparatus of the present invention, in addition to the above configuration, the predetermined threshold includes at least a predetermined first threshold and a second threshold larger than the first threshold, The edge pixel specifying means includes a first edge pixel in which the magnitude of the pixel value gradient feature quantity calculated by the gradient feature quantity calculation means is greater than or equal to the first threshold value, and a second edge pixel that is greater than or equal to the second threshold value. Is determined as the edge pixel, and the matching degree calculation means calculates the first matching degree and the matching for the second edge pixel from the result of matching the matching area and the model pattern for the first edge pixel. The second matching degree is calculated as the matching degree from the result of performing the above, and the direction specifying means determines the indicated direction from the distribution of the pixel value gradient feature quantity for each of the second edge pixels included in the matching area. specific , Wherein the position specifying means, the first match degree for each pixel included in the matching region, and may identify the indicated position from at least one of the distribution of the second matching degree.

  According to the above configuration, the predetermined threshold includes at least a predetermined first threshold and a second threshold that is larger than the first threshold.

  Further, the matching degree calculation means calculates the first matching degree (matching degree) from the result of matching between the matching region and the model pattern for the first edge pixel, and the second matching from the result of matching for the second edge pixel. The degree (degree of match) is calculated.

  Here, as described above, the magnitude of the gradient near the tip of the finger tends to be relatively large, and the magnitude of the gradient on the base side of the finger tends to be small.

  Therefore, for example, if the threshold value for detecting the pixel value gradient feature value near the tip of the finger is set as the second threshold value, and the threshold value for detecting the pixel value gradient feature value at least at the base of the finger is set as the first threshold value. The edge of the entire finger (first edge pixel) can be detected by a pixel equal to or higher than the first threshold, and the edge (second edge pixel) near the tip of the finger can be detected by a pixel equal to or higher than the second threshold. Can do.

  The first matching degree is a matching degree related to the edge of the entire finger, and the second matching degree is a matching degree related to the edge near the tip of the finger.

  On the other hand, the position specifying means specifies the indicated position from at least one of the first match degree and the second match degree calculated for each pixel included in the match area.

  Here, as described above, for example, the threshold value for detecting the pixel value gradient feature value near the tip of the finger is set as the second threshold value, and the threshold value for detecting at least the pixel value gradient feature value at the base side of the finger is set as the second threshold value. If the threshold value is 1, the distribution of the first matching degree on the matching area is a guideline for calculating the center of the portion surrounded by the edge, that is, the vicinity of the center of the belly of the finger. The distribution is a guide for calculating a position near the tip of the finger (hereinafter simply referred to as “tip position”).

  Therefore, the indicated position specified from the distribution of the first match degree is close to the center of the belly of the finger, and the specified position specified from the distribution of the second match degree is close to the tip position.

  In addition, the indicated position specified from the distribution of the first match degree and the second match degree is close to an intermediate position between the center of the finger belly and the tip position.

  Therefore, if the specified position is specified from the distribution of at least one of the first match degree and the second match degree on the match area, a position close to any of the center of the finger, the tip position, or the intermediate position is indicated. It can be specified as a position.

  In addition, by selecting one of the first match degree, the second match degree, or a combination of these match degrees, a position close to the center of the finger belly, the tip position, or the middle position according to the user's preference It is possible to select any of these and specify it as the target position of the object.

  In the image processing apparatus according to the aspect of the invention, in addition to the configuration described above, it is preferable that the pixel value gradient feature amount is a feature amount indicating at least a gradient direction of the pixel value.

  As described above, since the gradient direction of the pixel value is a feature quantity indicating the direction, it can be directly linked to the designated direction, and the designated quantity of the target object can be easily specified.

  On the other hand, a scalar quantity such as the magnitude of the gradient is a feature quantity that is difficult to be directly linked to the designated direction because it is not a feature quantity indicating the direction.

  Therefore, by using the gradient direction of the pixel value as the feature quantity, it is possible to more easily specify the target direction of the object.

  Further, in the image processing apparatus according to the present invention, in addition to the above configuration, the degree-of-match calculation means includes, for each gradient direction, the gradient direction included in the matching region and the gradient direction included in the model pattern. The number of matching pixels indicating the number of matching pixels may be calculated as the degree of matching.

  Here, in the pattern matching using the gradient direction, it can be determined whether or not the gradient directions match for each pixel included in the matching region.

  Therefore, the number of matching pixels calculated over all the pixels in the collation area can be calculated as the degree of matching, and the number of matching pixels calculated for each gradient direction can also be calculated as the degree of matching. Furthermore, the degree of match (match / mismatch) can also be calculated for each pixel included in the collation area.

  Here, when the number of matched pixels is calculated for each gradient direction, if the number of matched pixels for each gradient direction is biased toward a specific gradient direction, the specified direction can be easily identified using this bias. .

  Therefore, it can be said that the gradient direction is a pixel value gradient feature amount suitable for specifying the designated direction.

  In addition to the above configuration, the image processing apparatus according to the present invention further includes association information indicating a correspondence relationship between the distribution of the number of matched pixels for each gradient direction and the indication direction, and the direction identification The means may identify the indicated direction from the distribution of the number of matching pixels for each gradient direction calculated by the matching degree calculation means with reference to the association information.

  Here, as an example of the association information, when the deviation in the gradient direction matched with the model pattern is large, only the presence / absence of the matched pixel is indicated for each gradient direction (the number of matching pixels is discarded). The pattern and the instruction direction may be associated with each other.

  In addition, when the deviation in the gradient direction that matches the model pattern is small, the distribution of the number of matched pixels for each gradient direction may be associated with the designated direction. At this time, the association information is set on the assumption that the gradient direction having a large number of matching pixels has a strong relationship with the instruction direction.

  In the image processing apparatus of the present invention, in addition to the above configuration, the direction specifying unit may specify the indicated direction from a distribution of first direction vectors having a length corresponding to the number of coincident pixels for each gradient direction. good.

  Here, since the gradient direction with a large number of matching pixels has a strong relationship with the indication direction of the object, if the number of matching pixels for each gradient direction is used, the edge pixels can be selected according to the strength of the relationship with the indication direction. As a result, the pointing direction can be specified with high accuracy.

  In the image processing apparatus according to the present invention, in addition to the above-described configuration, the direction specifying unit may include a second direction vector having a length as a product of the number of coincident pixels for each gradient direction and the magnitude of the pixel value gradient feature amount. The indicated direction may be specified from the distribution of

  In the first direction vector described above, the designated direction is specified by regarding the number of matching pixels as the magnitude of the gradient. However, strictly speaking, the pixel value gradient feature quantity has a degree of freedom of the magnitude of the gradient, so that the indicated direction is specified from the distribution of the second direction vector including the degree of freedom of the magnitude of the gradient. By doing so, it is possible to specify the pointing direction with higher accuracy.

  In addition, as described above, the magnitude of the gradient tends to increase from the base of the finger toward the vicinity of the tip of the finger. Therefore, in the second direction vector, the resultant vector of the second direction vector finally calculated On the other hand, since such a tendency can be included, the pointing direction can be specified with higher accuracy.

  Further, in the image processing apparatus of the present invention, in addition to the above-described configuration, the direction specifying unit may further include the third direction vector distribution whose length is the size of the pixel value gradient feature amount for each gradient direction. The indicated direction may be specified.

  As described above, the gradient tends to increase from the base of the finger toward the vicinity of the tip of the finger. Therefore, in the third direction vector, the finally calculated third direction vector is compared with the resultant vector. Since such a tendency can be included, the pointing direction can be specified with high accuracy.

  By the way, when the center position such as the belly of the finger is set as the target position of the target object, the entire contact surface of the target object may not be specified as the target position of the target object by mistake. However, it is desirable to match the shape of the object relatively precisely by pattern matching so that it is specified as an edge pixel. Therefore, the predetermined threshold value for specifying the edge pixel cannot be set too large.

  On the other hand, for example, when detecting a human hand or finger, the luminance of the finger base side of the contacted part varies depending on the angle of the hand or finger with respect to the display surface, and accordingly, the finger base side changes accordingly. The area and shape of the image changes. Therefore, if the predetermined threshold value is too small, the area on the base side of the finger that is not the tip among the parts that are in contact with each other will increase, and the degree of match for the edge pixel specified using the predetermined threshold value will be determined. This makes it difficult to stably specify the indicated position. Therefore, there is a problem that the designated position of the object cannot be stably acquired unless the predetermined threshold value is adjusted very carefully.

  In other words, it can be said that the indicated position of the object described above is easily affected by noise or the like, or a change in the shape of the entire contact surface of the object.

  Therefore, in order to solve such a problem, in addition to the above configuration, in the image processing apparatus of the present invention, the position specifying unit includes the distribution of the gradient direction for each of the edge pixels included in the matching region, The tip position of the object on the captured image may be specified from the indicated direction specified by the direction specifying means.

  Here, for example, an area where there are many pixels having a gradient direction that is the same or opposite to the indicated direction specified by the direction specifying means is considered to be an area close to the tip position of the object on the matching area.

  In addition, even if the shape of the entire contact surface of the object, especially the area and shape of the image on the base side of the finger changes, the position of the region close to the tip position on the matching region or the pixel included in the region close to the tip position It is considered that the gradient direction does not change much.

  Therefore, for example, if the tip position of the object is specified from an area where pixels having gradient directions that are the same as or opposite to the indicated direction specified by the direction specifying means exist, the shape of the entire contact surface of the object In particular, the tip position can be accurately identified regardless of whether the area or shape of the image on the base side of the finger has changed.

  As described above, the tip position of the target can be specified with high accuracy while preventing the position of noise or the like in the captured image from being erroneously specified as the target position of the target.

  Further, in the image processing apparatus of the present invention, in addition to the above configuration, the position specifying unit includes a distribution of the gradient direction for each edge pixel included in the matching region, the indication direction specified by the direction specifying unit, The tip position of the object on the captured image may be specified from the magnitude of the pixel value gradient feature amount.

  By the way, as described above, the gradient tends to increase as it goes from the base of the finger toward the tip of the finger. Therefore, in the region where there are many pixels having a gradient direction that is the same or opposite to the indicated direction specified by the direction specifying means, the position of the pixel having a large gradient is close to the tip position of the object. It is believed that there is.

  Therefore, according to the above-described configuration, for example, the region near the tip of the finger is specified from the distribution of the gradient direction for each edge pixel included in the matching region, the indication direction specified by the direction specifying unit, and the region near the tip of the finger. Since the position of a pixel having a large gradient from the region can be set as the tip position, the tip position of the object can be specified with higher accuracy.

  In the image processing apparatus according to the aspect of the invention, in addition to the configuration, the position specifying unit may be the same as the indication direction specified by the direction specifying unit out of the gradient directions for each of the edge pixels included in the matching region. The tip position of the object on the captured image may be specified from the distribution of the edge pixels from which the gradient direction of the direction is calculated.

  By the way, in the case of a liquid crystal display with a built-in image sensor, for example, the image sensor detects the superposed light of the reflected light reflected by the object and the light emitted from the backlight and the outside light.

  In a captured image obtained by imaging such superposed light with an imaging sensor, the gradient directions of the edge portions tend to be opposite to each other when the surrounding is bright and when the surrounding is dark.

  Therefore, according to the above configuration, the tip position is specified from the distribution of edge pixels in which the gradient direction in the same direction as the indicated direction is calculated among the gradient directions for each edge pixel, assuming that the surrounding is bright. Yes.

  Thereby, even when the surroundings are bright, the tip position can be specified with high accuracy.

  In the image processing apparatus according to the present invention, in addition to the configuration described above, the position specifying unit may reverse the indication direction specified by the direction specifying unit out of the gradient directions for each of the edge pixels included in the matching region. The tip position of the object on the captured image may be specified from the distribution of the edge pixels from which the gradient direction of the direction is calculated.

  According to the above configuration, the tip position is specified from the distribution of edge pixels for which the gradient direction opposite to the indicated direction is calculated among the gradient directions for each edge pixel, assuming that the surrounding is dark.

  Thereby, even when the surroundings are dark, the tip position can be specified with high accuracy.

  Further, in the image processing apparatus of the present invention, in addition to the above configuration, the position specifying unit may be configured such that the indicated position is specified in the previous operation and the tip position is specified in the previous and current operations. The position where the indicated position specified in the previous operation is displaced by the amount of displacement from the tip position specified in the previous operation to the tip position specified in the current operation is The designated position may be used.

  As described above, in order to specify the center position of the finger belly as the designated position, strict pattern matching is required. Therefore, the shape of the entire contact surface of the object, particularly the image area on the base side of the finger When the shape changes, it is difficult to specify the indicated position.

  On the other hand, as described above, if the region near the tip of the finger is specified from the distribution of the gradient direction for each edge pixel included in the matching region and the indication direction specified by the direction specifying means, the specification of the tip position is performed. Thus, the shape of the entire contact surface of the object, particularly the area or shape of the image at the base of the finger, can be identified stably.

  Therefore, according to the above configuration, the position specifying means specifies the pointing position in the previous operation, and when the tip position is specified in the previous and current operations, from the tip position specified in the previous operation. The position where the indicated position specified in the previous operation is displaced by the amount of displacement up to the tip position specified in the current operation is set as the indicated position of the target object.

  Therefore, the specified position is specified using the tip position specified in the previous and current actions even if the specified position is specified in the previous action, even if the specified position cannot be specified in the current action for some reason. Therefore, the target position of the target can be specified with high accuracy while preventing the position of noise or the like in the captured image from being erroneously specified as the target position of the target.

  In addition to the above configuration, in the image processing apparatus of the present invention, the matching degree calculation unit sets the matching degree to 0 when the calculated matching degree is less than a predetermined threshold. Also good.

  By the way, when an unreasonable error pixel caused by noise or the like in image data occurs, the degree of matching calculated by the degree-of-matching calculating means tends to be relatively low.

  Therefore, according to the above configuration, by setting the upper limit value (threshold value) of the matching degree, it is possible to delete from the processing data the matching degree that is affected by an invalid error pixel caused by noise or the like in the image data. It is possible to eliminate the influence of an unreasonable error pixel and specify the target position of the object stably.

  Each means, each function, each process, and each step and each process in the image processing apparatus and its control method may be realized by a computer. In this case, the computer is used as each means. A control program that causes a computer to realize the image processing apparatus and the control method thereof by causing the computer to realize the functions or causing the computer to execute the processes, the steps, or the processes. A computer-readable recording medium on which is recorded also falls within the scope of the present invention.

  As described above, the image processing apparatus of the present invention includes a gradient feature amount calculating unit that calculates a pixel value gradient feature amount from image data of a captured image, and the pixel value gradient calculated by the gradient feature amount calculating unit. Edge pixel specifying means for specifying an edge pixel whose feature amount is equal to or larger than a predetermined threshold, a matching area of a predetermined size set on the image data, and a predetermined model pattern are collated. A matching degree calculating means for calculating a matching degree indicating a degree of matching between the matching area and the model pattern; and specifying the matching area having the maximum matching degree calculated by the matching degree calculating means as a matching area. From the distribution of the pixel value gradient feature quantity for each edge pixel included in the matching area specified by the matching area specifying means and the matching area specifying means, the finger of the object on the captured image It is configured to and a direction specifying means for specifying a direction.

  In addition, as described above, the control method of the image processing apparatus according to the present invention is an image processing apparatus having a function of specifying an instruction direction of an object on the captured image using image data of the captured image. A gradient feature amount calculating step for calculating a pixel value gradient feature amount from the image data, and the magnitude of the pixel value gradient feature amount calculated in the gradient feature amount calculation step is a predetermined threshold value or more. An edge pixel specifying step for specifying an edge pixel, a matching area of a predetermined size set on the image data, and a predetermined model pattern are matched, and matching between the matching area and the model pattern A matching degree calculating step for calculating a matching degree indicating the degree of matching, and a matching area that specifies the matching area that maximizes the matching degree calculated in the matching degree calculating step as a matching area. A region specifying step, and a direction specifying step for specifying the indicated direction of the object on the captured image from the distribution of the pixel value gradient feature amount for each of the edge pixels included in the matching region specified in the matching region specifying step It is a method that includes.

  Therefore, it is possible to stably detect the indication direction of the imaging object.

It is a block diagram which shows one Embodiment of the image processing apparatus (electronic device) in this invention. (A) is the schematic diagram which shows the mode of imaging of the belly of the finger (finger belly) when the surrounding is dark, and (b) is the captured image of the finger pad when the surrounding is dark. It is a schematic diagram showing a feature, (c) is a schematic diagram showing a state of imaging of the finger pad when the surrounding is bright, and (d) shows a feature of the captured image of the finger pad when the surrounding is bright. FIG. (A) is an example of the half of the reference table referred in order to quantize a gradient direction into 8 directions, (b) is an example of the remaining half of the said reference table. It is a flowchart which shows the outline | summary of the whole operation | movement of the said image processing apparatus. (A) is a schematic diagram showing an example of the distribution in the gradient direction when the surrounding is dark, and (b) is a schematic diagram showing an example of the distribution in the gradient direction when the surrounding is bright. (A) is a schematic diagram showing another example of the distribution in the gradient direction when the surrounding is dark, and (b) is a schematic diagram showing another example of the distribution in the gradient direction when the surrounding is bright. It is a flowchart which shows an example of the operation | movement of a pattern matching process among operation | movement of the said image processing apparatus. (A) is an example of a correspondence table indicating a correspondence relationship between a matching pixel number pattern (direction matching pattern) for each gradient direction and a target object indication direction with respect to the image processing apparatus, and (b) is the correspondence relationship. It is another example of a table. (A) is the schematic diagram which shows an example of distribution of the gradient direction for every edge pixel regarding the said image processing apparatus, (b) is a schematic diagram which shows an example of the number of matched pixels for every gradient direction, ( (c) is a schematic diagram showing an example of the specified direction of the specified object, (d) is a schematic diagram showing another example of the distribution of the gradient direction for each edge pixel, and (e) is It is a schematic diagram which shows another example of the number of coincidence pixels for every gradient direction, (c) is a schematic diagram which shows another example of the instruction | indication direction of the specified target object. (A) is a schematic diagram showing still another example of the distribution in the gradient direction for each edge pixel in the image processing apparatus, and (b) shows still another example of the number of coincident pixels for each gradient direction. It is a schematic diagram, and (c) to (e) are schematic diagrams showing an example of an indication direction of an object specified by a synthesized vector synthesized from a vector whose length is the number of matching pixels for each gradient direction. It is a flowchart which shows another example of the operation | movement of a pattern matching process among operation | movement of the said image processing apparatus. (A) is an outline figure showing an example of distribution of the gradient direction for every 1st edge pixel and every 2nd edge pixel about the above-mentioned image processing device, and (b) is every gradient direction in the 1st edge pixel. It is a schematic diagram which shows an example of the number of coincidence pixels, (c) is a schematic diagram which shows an example of the coincidence pixel number for every gradient direction in a 2nd edge pixel. 6 is a flowchart illustrating an example of an operation of an object (instruction) position specifying process with respect to the image processing apparatus. (A) is a schematic diagram showing a state in which a direction vector opposite to an instruction direction of an object is obtained with respect to the image processing apparatus, and (b) is an edge pixel having a gradient direction that matches the direction vector. It is a schematic diagram which shows an example of a presence area | region. It is a flowchart which shows another example of operation | movement of an object (instruction | indication) position specific process regarding the said image processing apparatus.

  An embodiment of the present invention will be described with reference to FIGS. Configurations other than those described in the following specific embodiments may be omitted as necessary, but are the same as the configurations described in the other embodiments. For convenience of explanation, members having the same functions as those shown in each embodiment are given the same reference numerals, and the explanation thereof is omitted as appropriate.

  In each of the following embodiments, a case where a liquid crystal display is employed as an example of the image display unit will be described. However, a case where an image display unit other than the liquid crystal display is employed is also included in the scope of application of the present invention.

[First Embodiment]
The image processing apparatus 1 (electronic device 20) according to the first embodiment of the present invention will be described below with reference to FIGS.

  First, main functions of the image processing apparatus 1 will be described, and then the configuration of the image processing apparatus 1 of the present embodiment will be described based on FIG.

  The main functions of the image processing apparatus 1 include the following two functions.

  The first function is an image display function, and the image processing apparatus 1 includes a liquid crystal display composed of a plurality of pixels (not shown) and a backlight that emits light to the liquid crystal display, and can display an image. It has become.

  The second function is an image capturing function. Each pixel in the liquid crystal display of the image processing apparatus 1 includes an optical sensor (or an imaging sensor), and the optical sensor approaches the display screen of the liquid crystal display. A captured image obtained by capturing an external object or the like (target object) can be captured as image data.

  Note that the liquid crystal display may have a built-in photosensor for every predetermined number of pixels among the plurality of pixels. However, from the viewpoint of the resolution of an image captured by the photosensor, It is preferable to be built in every.

  In addition to the photodiode, any optical sensor may be used as long as it can be built in each pixel of a liquid crystal display or the like based on the photoelectric effect.

  Next, as shown in FIG. 1, the image processing apparatus 1 includes a gradient feature amount calculation unit (gradient feature amount calculation unit) 2, an edge extraction unit (edge pixel identification unit) 3, a calculation / extraction data holding unit 4, a match Pixel number calculation unit (matching degree calculation unit) 5, model pattern storage unit 6, matching region specifying unit (matching region specifying unit) 7, matching region information storage unit 8, object direction specifying unit (direction specifying unit) 9, target An object direction specifying information storage unit 10 and an object coordinate specifying unit (position specifying means) 11 are provided.

  The gradient feature amount calculation unit 2 calculates a gradient (pixel value gradient feature amount) S of the pixel value of the target pixel from the pixel value of the target pixel and the pixel values of a plurality of adjacent pixels for each pixel on the image data. .

  Here, “image data” in the present embodiment is assumed to be digital data, but may be analog data.

  The “pixel value” is a luminance value or a density value. In the case of analog data, each divided area obtained by dividing image data into a plurality of areas is called a pixel, and an average density value in each divided area is called a pixel value. When the “pixel value” is a density value, the “pixel value gradient feature value” may be referred to as a “density gradient feature value”.

  In the present embodiment, the gradient S is defined by (lateral gradient Sx, vertical gradient Sy), and the gradient S has a magnitude [gradient magnitude (pixel value gradient feature amount) POW (S)] and direction [ Gradient vector (pixel value gradient feature value) ANG (S)].

  Mathematically, the gradient S is a vector that takes a difference in pixel value between a specific pixel and an adjacent pixel, sets the difference as a magnitude, and sets the direction from the specific pixel to the adjacent pixel as positive. It is calculated by adding all the adjacent pixels.

  More specifically, the gradient S may be calculated using an edge extraction operator such as a Sobel operator or a Prewit operator.

  For example, the local vertical gradient Sy and the horizontal gradient Sx at the pixel position x (i, j) of each pixel are obtained using the Sobel operator, and then the pixel is calculated using the following equations (1) and (2). The gradient direction ANG (S) and the gradient magnitude POW (S) at the position x (i, j) can be obtained. Here, i represents the position of the pixel along the horizontal direction, and j represents the position of the pixel along the vertical direction. i and j are positive integers.

Gradient direction ANG (S) = tan ⁻¹ (Sy / Sx) (1)
The magnitude of the gradient POW (S) = √ (Sx ^ 2 + Sy ^ 2) (2)
Next, the edge extraction unit 3 extracts (identifies) edge pixels whose gradient magnitude POW (S) calculated by the gradient feature value calculation unit 2 is equal to or greater than a predetermined threshold.

  Next, the calculation / extraction data holding unit 4 calculates data regarding the gradient S for each pixel on the image data calculated by the gradient feature amount calculation unit 2 and the position of the edge pixel extracted by the edge extraction unit 3 on the image data. Data related to etc. is retained.

  As a result, out of all the pixels on the image data, edge pixels whose gradient magnitude POW (S) is equal to or greater than a predetermined threshold are other than edge pixels whose gradient magnitude POW (S) is less than the predetermined threshold. Can be selected from the pixels.

  Next, the coincidence pixel number calculation unit 5 reads data relating to the gradient S for each pixel on the image data from the calculation / extraction data holding unit 4 described above, reads a model pattern from a model pattern storage unit 6 to be described later, Are compared with the model pattern (hereinafter referred to as “pattern matching”), and the number of pixels in which the gradient direction included in the verification region matches the gradient direction included in the model pattern (degree of matching: This is referred to as “the number of matching pixels”).

  More specifically, the matching pixel number calculating unit 5 scans the matching area pixel by pixel while matching the target pixel included in the matching area with all the pixels included in the image data. In addition to setting, the number of matching pixels calculated for each matching region is calculated for each target pixel corresponding to each matching region, so that the number of matching pixels can be calculated for all pixels included in the image data. It has become.

  The matching pixel number calculation unit 5 determines whether the gradient direction ANG (S) matches or does not match for each edge pixel included in the matching region, and determines the matching pixel number for each gradient direction ANG (S) included in the matching region. It can be calculated.

  Here, the gradient direction ANG (S) is a continuous amount that changes in the range of 0 rad or more and less than 2π rad. In the coincidence pixel number calculation unit 5 of the present embodiment, the quantized direction is quantized into 8 directions. Used as

  Next, based on FIGS. 3A and 3B, a method of quantizing the gradient direction ANG (S) in eight directions will be described.

  FIG. 3A is an example of a half of the reference table referred to quantize the gradient direction into 8 directions, and FIG. 3B is an example of the remaining half of the reference table.

  First, for each pixel, the lateral gradient Sx obtained by Equation (2) is quantized according to the gradient direction ANG (S) according to the reference table shown in FIG. 3A if Sx ≧ 0. Set the gradient direction.

  On the other hand, if Sx <0, the gradient direction quantized according to the gradient direction ANG (S) is set according to the reference table shown in FIG.

  Here, the case where the gradient direction is quantized into eight directions will be described, but the number of quantized gradient directions is not limited to eight, and accuracy conditions required for pattern matching and hardware In view of the performance and the like, it can be increased or decreased.

  As described above, the image processing apparatus 1 uses at least the gradient direction ANG (S) indicating the gradient direction of the pixel value as the feature amount.

  Since the gradient direction ANG (S) of the pixel value is a feature quantity indicating the direction, it can be directly linked to the designated direction, and is a feature quantity that makes it easy to specify the designated direction of the object.

  On the other hand, a scalar quantity such as the gradient magnitude POW (S) is a feature quantity that is difficult to be directly linked to the designated direction because it is not a feature quantity indicating the direction.

  Therefore, by using the gradient direction ANG (S) of the pixel value as the feature quantity, it is possible to more easily specify the target direction of the object.

  In the pattern matching using the gradient direction ANG (S) described above, it can be determined whether or not the gradient direction ANG (S) matches for each pixel included in the collation area.

  Therefore, the number of matching pixels calculated over all the pixels in the matching region can be calculated as the matching degree, and the number of matching pixels calculated for each gradient direction ANG (S) can also be calculated as the matching degree. Furthermore, the degree of match (match / mismatch) can also be calculated for each pixel included in the collation area.

  Here, when the number of matching pixels is calculated for each gradient direction ANG (S), if the number of matching pixels for each gradient direction ANG (S) is biased to a specific gradient direction ANG (S), this bias is used. The pointing direction can be easily specified. Therefore, it can be said that the gradient direction ANG (S) is a feature amount suitable for specifying the designated direction.

  Next, another mode for quantizing the gradient S will be described. First, the matching pixel number calculation unit 5 matches the number of pixels in which the difference between the gradient direction ANG (S) included in the matching region and the corresponding gradient direction ANG (S) included in the model pattern is equal to or smaller than a predetermined angle. It may be calculated as the number of pixels.

  Thereby, for example, a pixel whose difference between the gradient direction ANG (S) included in the matching region and the corresponding gradient direction ANG (S) included in the model pattern is equal to or smaller than a predetermined angle is determined as a pixel that matches the model pattern. Then, by determining a pixel whose difference between the gradient direction ANG (S) included in the matching region and the corresponding gradient direction ANG (S) included in the model pattern exceeds a predetermined angle as a pixel that does not match the model pattern, It is possible to calculate the number of matching pixels of all the pixels included in the matching area.

  As a method of quantizing the gradient direction, in addition to the above-described example, for example, by dividing an angle range of 0 rad or more and less than 2π rad by an equal upper limit angle, a gradient direction having infinite freedom can be finite. It is possible to quantize in the direction of the number.

  In the present embodiment, for the sake of simplicity, it is assumed that each of the captured image, the collation region, and the model pattern is a rectangle, but each shape is not limited to a rectangle. In addition, a single pixel to be noticed is determined in advance from the pixels included in the collation area, and the single pixel is referred to as a noticed pixel.

  Here, for example, pattern matching starts from a state in which the pixel in the upper left corner of the image data is set to the position of the target pixel in the matching area, and the matching area is set from the upper left corner to the lower right corner of the image data with reference to the target pixel. Matching with the model pattern is performed while scanning pixel by pixel.

  Note that when the pixel at the end (outermost circumference) of the image data is set to the position of the target pixel in the collation area, there may be a case where no pixel exists in a part of the collation area. However, in the case where such a boundary problem occurs, pattern matching may be performed by excluding the pixel in the portion where the pixel in the matching area does not exist. Further, pattern matching may be performed by setting a periodic boundary condition or the like and replacing a pixel in a region where a pixel in the matching region on the matching region does not exist with a model having an appropriate pixel value gradient feature amount.

  Further, the size and shape of the collation region may be set as appropriate according to the size and shape of the model pattern.

  By the pattern matching as described above, there are a plurality of collation areas scanned for each pixel, and the number of matching pixels is calculated for each collation area scanned for each pixel. That is, the number of matching pixels is calculated for each of all the pixels on the image data.

  Next, the model pattern storage unit 6 stores the model pattern as described above.

  From the above, it is possible to recognize the degree of matching between the matching region and the model pattern from the number of matching pixels calculated by the matching pixel number calculation unit 5.

  Here, as an amount used for pattern matching, a scalar amount such as a pixel value (density value or luminance value) is also conceivable. However, even if such a scalar quantity is quantized, it can always change according to the condition of the object, and so it is difficult to set a model pattern in advance. Note that quantization means that a quantity within a predetermined range is treated as a constant quantity.

  On the other hand, the gradient S described above is a feature amount having a degree of freedom in direction. The gradient S is a vector quantity obtained by taking a difference in pixel values between a specific pixel and an adjacent pixel and adding these differences for all adjacent pixels, and the gradient magnitude POW (S ) And gradient direction ANG (S).

  In addition, the gradient direction ANG (S) is quantized in eight directions as described above, so that the gradient direction ANG (S) calculated for one pixel can take 8 states (9 including a non-direction). The states can be discretized into a very small number of states, and each state can have a feature that allows easy identification of different directions. Therefore, the gradient S is an amount suitable for the pattern matching described above.

  Next, in the present embodiment, the case where the number of matching pixels is calculated as an example of the “matching degree” calculated by pattern matching has been described.

  However, the calculation method of the “matching degree” can be roughly divided into two calculation methods.

  The first calculation method is a method for calculating the number of coincident pixels in which the gradient S for each pixel included in the collation area matches the gradient S for each corresponding pixel included in the model pattern as in the present embodiment. At this time, the greater the number of matching pixels, the greater the degree of matching, and the smaller the number of matching pixels, the smaller the degree of matching.

  However, since the gradient magnitude POW (S) and the gradient direction ANG (S) have an infinite degree of freedom, it is necessary to devise in order to obtain the number of matching pixels. For example, as described above, the gradient S may be equally divided into an angle range of 0 rad or more and less than 2π rad by an appropriate upper limit angle by discarding the gradient magnitude POW (S). Alternatively, the gradient magnitude POW (S) may be discarded from the gradient S and the gradient direction ANG (S) may be quantized in 8 directions or 16 directions.

  Next, the second calculation method is a method of calculating the difference between the gradient S for each pixel included in the collation region and the model pattern and calculating the sum of these differences. At this time, the greater the sum of differences, the smaller the degree of match, and the smaller the sum of differences, the greater the degree of match.

  In the first calculation method, matching / non-matching can be determined for each pixel, and in the second calculation method, a difference can be calculated for each pixel. Therefore, in these methods, it is possible to set the degree of match for each pixel based on the match / mismatch for each pixel or the difference for each pixel.

  In the second calculation method, if the difference in the gradient direction ANG (S) for each pixel is equal to or smaller than the predetermined threshold angle, it is determined that the gradient direction ANG (S) of the model pattern matches, If the gradient direction ANG (S) is the gradient direction ANG (S) of the pixel, the number of matching pixels for each gradient direction ANG (S) included in the matching region can be calculated.

  Next, the matching area specifying unit 7 specifies the matching area in which the number of matching pixels calculated by the matching pixel number calculation unit 5 is the maximum as the matching area.

  As a result, it is possible to identify a matching area indicating the area where the object exists in the image data.

  The matching area information storage unit 8 includes data relating to the position of the matching area specified by the matching area specifying unit 7 on the image data, data relating to the gradient S for each pixel included in the matching area, and edge pixels extracted by the edge extracting unit 3. The data regarding the position on the matching area and the data regarding the number of matching pixels for each pixel included in the matching area are stored.

  Next, the object direction specifying unit 9 receives from the matching region information storage unit 8 data on the position of the matching region on the image data, data on the gradient S for each pixel included in the matching region, and on the matching region of the edge pixel. Is read out from the object direction specifying information storage unit 10 to be described later, and a direction matching pattern for each gradient direction ANG (S) and an instruction direction predetermined as indicating the instruction direction of the object. Correspondence table (correspondence information: see FIG. 8A, etc.) indicating the correspondence relationship, or correspondence table (correlation) indicating the correspondence relationship between the distribution of the number of matching pixels for each gradient direction ANG (S) and the designated direction Information: see FIG. 9B, etc.) and referring to the correspondence table, the specified direction is specified from the distribution of the gradient direction ANG (S) for each pixel included in the matching area.

  The designation direction may be specified based on the above-described correspondence table, or only the gradient direction ANG (S) and the number of matching pixels for each pixel included in the matching region may be used without using the correspondence table. May be used.

  In the object direction specifying information storage unit 10, as described above, the correspondence table indicating the correspondence between the direction matching pattern for each gradient direction ANG (S) and the designated direction, or for each gradient direction ANG (S). A correspondence table indicating the correspondence between the distribution of the number of matched pixels and the designated direction is stored.

  By the way, there is a possibility that error pixels due to noise or pixel defects generated at the time of imaging affect the gradient S, and the accuracy of specifying the pointing direction is lowered.

  Therefore, in order to solve such a problem, the above-described coincidence pixel number calculation unit 5 performs collation between the region including the edge pixel specified by the edge extraction unit 3 on the collation region and the model pattern. May be.

  As a result, pixels other than the less important edge pixels are excluded, and pattern matching is performed only on the edge pixels, so that most of the error pixels described above can be excluded. As a result, it is possible to improve the accuracy of specifying the pointing direction, so that the pointing direction can be detected stably.

  Next, the object coordinate specifying unit 11 receives data regarding the position of the matching region on the image data from the matching region information storage unit 8, data regarding the position of the edge pixel on the matching region, and matching for each pixel included in the matching region. Data related to the number of pixels is read, and the designated position of the object on the captured image is specified from the distribution of the number of matching pixels calculated for each pixel included in the matching region.

  Therefore, the designated position of the object on the captured image can be specified from the distribution of the number of matching pixels for each pixel included in the matching region.

  For example, the position of the pixel having the maximum number of matching pixels among the number of matching pixels for each pixel included in the matching area may be set as the target position of the object, and the number of matching pixels for each pixel included in the matching area is weighted. The position obtained by the weighted average may be used as the target position of the object.

  The calculation / extraction data holding unit 4, model pattern storage unit 6, matching area information storage unit 8, and object direction specifying information storage unit 10 described above are, for example, a tape system such as a magnetic tape or a cassette tape, a floppy tape, etc. (Registered trademark) disks / hard disks and other magnetic disks and compact disks-ROM / MO (magneto-optic) / MD (magnetic disk) / digital video disks / compact disks-R, etc. A card system such as an optical card or a semiconductor memory system such as a mask ROM / EPROM (Erasable Programmable ROM) / EEPROM (Electrically Erasable Programmable ROM) / flash ROM or the like can be used.

  Next, as shown in FIGS. 2A to 2D, a finger pad (hereinafter referred to as “finger pad”) is used as an object to be imaged by a photodiode incorporated in each pixel of the liquid crystal display of the image processing apparatus 1. An outline of the characteristics of each captured image (image data) will be described.

  FIG. 2A is a schematic diagram showing how the finger pad is imaged when the surroundings are dark, and FIG. 2B is a schematic diagram showing the characteristics of the captured image of the finger pad when the surroundings are dark. .

  First, in a situation where the surroundings are dark, as shown in FIG. 2A, when the user touches the index finger's belly with the display screen of the liquid crystal display, the captured image (image data) 31 shown in FIG. Since the backlight is an image obtained by reflection on the imaging target (finger's belly), the white circular shape is blurred.

  Further, the gradient direction ANG (S) in each pixel shows a tendency from the edge portion on the captured image 31 to the vicinity of the center of the region surrounded by the edge portion. The gradient direction ANG (S) is a direction in which a density difference occurs, and here, the direction from the dark part to the bright part is positive. In addition, the gradient POW (S) (density difference) in each pixel is generally small at the base of the finger and tends to increase toward the tip of the finger.

  Next, FIG. 2C is a schematic diagram showing how the finger pad is imaged when the periphery is bright, and FIG. 2D is an overview showing the characteristics of the captured image of the finger pad when the periphery is bright. FIG.

  In a dark environment, as shown in FIG. 2C, when the user touches the index finger's belly with the display screen of the liquid crystal display, the captured image (image data) 32 shown in FIG. Becomes the image obtained by entering the display screen of the liquid crystal display (if touched, the reflected light from the backlight is also mixed), so the shadow of the finger caused by the external light being blocked by the index finger and the liquid crystal display The components reflected by the backlight are mixed with the finger pad in contact with the display screen to form a gray circle close to black.

  In the case of the captured image 32, since the surroundings are bright with external light, the gradient direction ANG (S) in each pixel is opposite to the gradient direction ANG (S) in the white circular portion of the captured image 31 shown in FIG. It shows the tendency of the direction.

  On the other hand, the magnitude of the gradient POW (S) in each pixel in the captured image 32 tends to be smaller toward the tip of the finger and larger toward the tip of the finger. The same as the trend.

  Therefore, if the gradient S is used as the feature amount, the above-described tendency indicated by the distribution of the gradient S for each edge pixel among the pixels included in the matching region can be used. Therefore, the gradient S for each edge pixel included in the matching region can be used. From the distribution, the pointing direction of the object on the captured image can be specified.

  Therefore, since the pointing direction can be specified using only a single matching area, the object does not span the two areas of the first area and the second area as in the operation input device disclosed in Patent Document 1. In addition, there is no problem that the operation direction cannot be detected. Therefore, it is possible to specify a stable pointing direction.

  Further, since the gradient S is a difference between pixel values as described above, for example, even if the pixel value on the base side of the finger changes in conjunction with a change in the angle of the finger, the magnitude of the gradient POW ( S) has a tendency that the base side of the finger is small and becomes larger toward the tip of the finger, and the gradient direction ANG (S) does not vary so much, like the operation input device disclosed in Patent Document 1, There is no problem that the operation direction cannot be stably calculated due to the change in the pixel value on the base side of the finger. Therefore, it is possible to specify a stable pointing direction.

  As described above, according to the image processing apparatus 1 of the present embodiment, it is possible to stably detect the indication direction of the imaging target.

(Overview of overall operation)
Next, based on FIG.1 and FIG.4, the outline | summary of operation | movement of the image processing apparatus 1 (electronic device 20) is demonstrated. FIG. 4 is a flowchart showing an outline of the overall operation of the image processing apparatus 1.

  First, in step S101 (hereinafter described as “S101”), the gradient feature value calculation unit 2 calculates a gradient S for each pixel on the image data, and proceeds to S102 (gradient feature value calculation process: Gradient feature calculation step).

  In S102, the edge extraction unit 3 identifies a pixel having a gradient magnitude POW (S) equal to or greater than a predetermined threshold as an edge pixel, and the gradient direction is “no direction” for pixels that are not identified as edge pixels. The magnitude of the gradient is reset to “0” (edge pixel specifying process: edge pixel specifying step).

  In this way, by specifying the gradient direction ANG (S) in non-important pixels, error pixels at the time of imaging, etc. as a uniform non-direction and / or the gradient magnitude POW (S) as 0, it is possible to specify the indicated position described later. Further, the accuracy of specifying the pointing direction can be further improved.

  In S103, after the matching pixel number calculation unit 5 performs matching between the matching region and the model pattern and calculates the matching pixel number, the matching region specifying unit 7 calculates the matching pixel calculated by the matching pixel number calculation unit 5. The matching area having the maximum number is specified and the process proceeds to S104 (pattern matching process: matching degree calculating step / matching area specifying step).

  In S104, the object direction specifying unit 9 specifies the indication direction of the object from the distribution of the gradient S for each edge pixel included in the matching region, and the object coordinate specifying unit 11 is calculated by the coincident pixel number calculating unit 5. The designated position of the object is identified from the coincident pixel number distribution for each pixel, and becomes “end” (object position / direction identification process: position identification / direction identification step).

  In specifying the target position of the target object, the center of gravity position of the distribution of the number of matching pixels (the position obtained by weighted average using the number of matching pixels for each pixel included in the matching area as a weight) is used as the target position of the target object. Although the method to identify can be considered, it is not limited to this method.

  For example, the position where the number of matching pixels is the maximum may be specified as the target position of the object. Further, the matching area may be specified as it is as the existence range (position and size) of the object. Further, a range corresponding to the size of the model pattern centering on the center of gravity of the distribution of the number of matched pixels or the position where the number of matched pixels is the maximum may be set as the existence range of the object.

  Further, in specifying the target direction of the object, a method of specifying the gradient direction ANG (S) specified at the position (pixel) where the number of matching pixels is the maximum in S103 as the target direction of the target can be considered. However, it is not limited to this method. For example, the gradient direction ANG (S) specified in each pixel is set as the vector direction, and the direction vector (the first number) is the length of either the number of matching pixels or the product of the gradient size POW (S) and the number of matching pixels. 1 direction vector or second direction vector) may be defined, and the direction indicated by the combined vector obtained by combining the direction vectors for each edge pixel may be specified as the target direction of the object.

  For each gradient direction ANG (S), an average value of gradient magnitudes POW (S) may be calculated, and a vector (third direction vector) having the average value as a length may be defined.

  Further, for each gradient direction ANG (S), a median (median value) of the gradient magnitude POW (S) may be calculated, and a vector (third direction vector) having the median as a length may be defined.

  Here, since the gradient direction ANG (S) having a large number of matching pixels has a strong relationship with the indication direction of the object, the distribution of the first direction vector having the length of the number of matching pixels in each gradient direction ANG (S) is represented. If used, the edge pixels can be selected according to the strength of the relationship with the pointing direction, so that the pointing direction can be specified with high accuracy.

  In the first direction vector described above, the specified direction is specified without the degree of freedom of the gradient magnitude POW (S). However, strictly speaking, since the gradient S has a degree of freedom of the gradient magnitude POW (S), it is indicated from the distribution of the second direction vector including the degree of freedom of the gradient magnitude POW (S). If the direction is specified, it is possible to specify the pointing direction with higher accuracy.

  Further, as described above, the gradient magnitude POW (S) tends to increase from the base of the finger toward the vicinity of the tip of the finger, so the second direction vector described above is finally calculated. Since such a tendency can be included in the combined vector of the two-way vectors, the pointing direction can be specified with higher accuracy.

  Further, as described above, since the gradient tends to increase from the base of the finger toward the vicinity of the tip of the finger, the third direction vector described above is the final calculated third direction vector. Since such a tendency can be included in the composite vector, the pointing direction can be specified with high accuracy.

  The above is the outline of the overall operation of the image processing apparatus 1. Next, details of each operation of the image processing apparatus 1 in the pattern matching process will be described with reference to FIGS. 1 and 5A to 9D.

  First, a specific example of a model pattern to be collated with a collation area will be described based on FIGS. 5A and 5B and FIGS. 6A and 6B.

  Fig.5 (a) is a schematic diagram which shows an example of the model pattern of a finger pad. FIG. 5B is a schematic diagram showing an example of a shadow model pattern surrounding the periphery of the finger pad when the surrounding is bright. Compared to the model pattern of FIG. The direction is opposite (180 degrees different).

  FIG. 5 (a) assumes image data picked up based on a state in which the reflected light of the backlight hits the finger pad, and shows a state where the image data becomes brighter toward the center. Yes. On the other hand, FIG. 5B assumes image data in which the shadow of a finger is captured, and shows a state where the image is brighter toward the edge portion of the image.

  FIG. 6A is a schematic diagram illustrating another example of a model pattern that is collated with image data when the surrounding is dark. The difference from the model pattern in FIG. 5A is that two degrees of freedom in the gradient direction are given for each pixel. By devising such a model pattern, it is possible to improve collation accuracy while improving collation efficiency. On the other hand, FIG. 6B assumes image data in which a shadow of a finger is imaged, and two degrees of freedom in gradient directions are given for each pixel, as in FIG. 6A. ing.

  Next, FIG. 7 is a flowchart showing the operation of the pattern matching process among the operations of the image processing apparatus 1 shown in FIG.

  First, in S201, the coincidence pixel number calculation unit 5 initializes all coincidence pixel numbers in each gradient direction ANG (S) to “0”, and proceeds to S202.

  In S202, the coincidence pixel number calculation unit 5 performs a matching process using a model pattern for each pixel, and proceeds to S203.

  In S203, the matching pixel number calculation unit 5 determines whether or not the model pattern and the gradient direction ANG (S) match by the matching process in S202. If the gradient directions ANG (S) match (Yes), the process proceeds to S204, and if they do not match (No), the process proceeds to S205.

  In S204, the coincidence pixel number calculation unit 5 updates the coincidence pixel number in each gradient direction ANG (S) by adding the number of pixels in which the model pattern coincides with the gradient direction ANG (S) in S202, and proceeds to S205. .

  In S205, the coincidence pixel number calculation unit 5 determines whether or not collation has been completed for all the elements (pixels) of the model pattern. If the collation is completed (Yes), the process proceeds to S206, and if not completed (No), the process proceeds to S202, and the collation process is performed on the next element (pixel) of the model pattern.

  In S206, the coincidence pixel number calculation unit 5 outputs a value obtained by adding the coincidence pixel numbers for each gradient direction ANG (S), and the process proceeds to S207. The total value output in S206 is used in the object position / direction specifying process in S104 of FIG.

  In S207, the target direction specifying unit 9 specifies and outputs the target direction of the target based on the number of matching pixels for each gradient direction ANG (S), and ends the pattern matching process. The method for specifying the target direction of the object in this embodiment will be described in detail below.

  Next, a specific example of the object indication direction specifying process by the object direction specifying unit 9 will be described with reference to FIGS. 8 (a) to 9 (f).

  FIGS. 8A and 8B show the pattern of the number of matching pixels (or direction matching pattern) calculated by the matching pixel number calculation unit 5 and indicating the presence / absence of the matching direction for each gradient direction ANG (S), and the target object. It is a schematic diagram which shows an example of the correspondence table which shows a correspondence with the instruction | indication direction. Here, the direction matching pattern is data having a value of “Yes” or “No” for each gradient direction ANG (S), and “Yes” indicates the number of matching pixels in the gradient direction ANG (S). 1 or more and “None” indicates that the number of matching pixels in the gradient direction ANG (S) is zero.

  Note that the correspondence relationship table shown in FIG. 8A is suitable when the deviation in the gradient direction matching the model pattern is large.

  The object direction specifying unit 9 includes a correspondence table in which the number of matching pixels for each gradient direction ANG (S) calculated by the matching pixel number calculation unit 5 is as illustrated in FIGS. 8A and 8B. It is determined which pattern corresponds to the number of matching pixels, and the indication direction of the object is specified.

  Next, the specification process of the specified direction using the correspondence table will be described in more detail.

  FIG. 8A is a schematic diagram showing an example of the correspondence table when the surroundings are dark, and FIG. 8B is a schematic diagram showing an example of the correspondence table when the surroundings are bright. Compared with the correspondence table in (a), the indication direction of the object is reverse (a direction different by 180 degrees).

  FIG. 8A assumes image data captured based on a state in which the reflected light of the backlight hits the belly of the finger, and the captured image tends to become brighter toward the center. Therefore, the indication direction of the object is associated with the gradient direction ANG (S) in the image in the opposite direction (direction different by 180 degrees).

  On the other hand, FIG. 8B assumes image data in which a shadow of a finger is captured, and the captured image shows a tendency to become brighter toward the edge portion of the image. Are associated with the same direction as the gradient direction ANG (S) in the image.

  8A and 8B show only the case where three types of gradient directions ANG (S) are matched by the matching process, but four or more types of gradient directions ANG (S) are matched. In this case, the same correspondence table can be defined.

  The correspondence table may be defined for all possible matching combinations, or only some of the matching combinations may be defined as shown in FIGS. 8 (a) and 8 (b). Also good. When only some combinations are defined, the object direction specifying unit 9 specifies the same direction as other matching states when a matching state of the undefined gradient direction ANG (S) appears. Alternatively, the direction may be undefined (no direction).

  Next, FIG. 9A is a schematic diagram illustrating an example of the distribution of the gradient direction ANG (S) for each pixel calculated by the gradient feature amount calculation unit 2, and FIG. 9B illustrates the number of matching pixels. It is a schematic diagram which shows an example (correspondence information) of distribution of the number of coincidence pixels for every gradient direction ANG (S) which the calculation part 5 calculated. In addition, the pixel (pixel which is not an edge pixel) identified as “non-directional” by the gradient feature amount calculation unit 2 is shown as an empty box in FIG. 9A. FIG. 9A is an example of image data obtained by imaging a state in which the reflected light of the backlight hits the belly of the finger when the surrounding is dark.

  On the other hand, FIG. 9D is a schematic diagram illustrating another example of the distribution of the gradient direction ANG (S) for each pixel calculated by the gradient feature amount calculation unit 2, and FIG. It is a schematic diagram which shows another example (association information) of the distribution of the number of coincident pixels for each gradient direction ANG (S) calculated by the number calculation unit 5. In addition, the pixel (pixel which is not an edge pixel) specified as “non-directional” by the gradient feature amount calculation unit 2 is shown as an empty box in FIG. 9D. FIG. 9D is another example of image data obtained by imaging a state in which the reflected light of the backlight hits the belly of the finger when the surrounding is dark.

  Note that the distribution of the number of coincident pixels shown in FIGS. 9A and 9B is suitable when the deviation in the gradient direction matched with the model pattern is small. Note that the distribution of the number of matching pixels is set such that the gradient direction ANG (S) having a large number of matching pixels has a strong relationship with the instruction direction.

  First, in FIG. 9A, edge pixels are concentrated on the upper part of the image, and the edge pixels matched by the matching process are indicated by shaded portions. Referring to the number of matched pixels for each gradient direction ANG (S) in FIG. 9B, three gradient directions ANG (S) in the lower right direction, the lower direction, and the lower left direction match.

  Since FIG. 9A shows image data when the surrounding is dark, the object direction specifying unit 9 uses the correspondence table shown in FIG. 8A as the correspondence table. Referring to FIG. 8A, the pointing direction of the object associated with the pattern in which the three gradient directions ANG (S) of the lower right direction, the lower direction, and the lower left direction are matched is defined as the upward direction. ing. Accordingly, in the captured image shown in FIG. 9A, the object direction specifying unit 9 specifies that the designated direction of the object is the upward direction (FIG. 9C).

  On the other hand, in FIG. 9D, edge pixels are concentrated in the upper left part of the image, and the edge pixels matched by the matching process are indicated by shaded portions. Referring to the number of matched pixels for each gradient direction ANG (S) in FIG. 9E, three gradient directions ANG (S) in the right direction, the lower right direction, and the lower direction are matched. As in the example shown in FIG. 9A, referring to the correspondence table shown in FIG. 8A, a pattern in which three gradient directions ANG (S) in the right direction, the lower right direction, and the lower direction are matched is obtained. The instruction direction of the associated object is defined as the upper left direction. Therefore, in the captured image shown in FIG. 9D, the object direction specifying unit 9 specifies the instruction direction of the object as the upper left direction (FIG. 9F).

  For the image data when the surroundings are bright, the object direction specifying unit 9 specifies the instruction direction of the object by referring to the correspondence relationship table of FIG.

  As described above, according to the image processing apparatus 1 of the present embodiment, pixels having a gradient magnitude POW (S) equal to or greater than a predetermined threshold are identified as edge pixels from the image data of the captured image. Since the pointing direction and pointing position of the object to be imaged are specified using the pattern matching technique above, it is possible to prevent the position of noise or the like in the picked-up image from being erroneously specified as the pointing position of the object. The pointing direction and pointing position of an object can be specified with high accuracy.

  In the above description, the object direction specifying unit 9 has the pattern of the number of coincident pixels for each gradient direction ANG (S) and the instruction direction of the object as shown in FIGS. 8 (a) and 8 (b). However, the present invention is not limited to such a configuration. That is, it is good also as a structure which specifies the instruction | indication direction of a target object without using a correspondence table.

  Next, a specific example will be described based on FIGS. 10A to 10E in the case where the correspondence relationship table is not used.

  FIG. 10A is a schematic diagram illustrating still another example of the distribution of the gradient direction ANG (S) for each pixel calculated by the gradient feature value calculation unit 2, and FIG. 10B illustrates the number of matching pixels. It is a schematic diagram which shows another example of distribution of the number of coincidence pixels for every gradient direction ANG (S) which the calculation part 5 calculated. 9A and 9D, pixels that are identified as “non-directional” by the gradient feature value calculation unit 2 (pixels that are not edge pixels) are shown as empty boxes.

  Further, FIG. 10 (a) is a view similar to FIGS. 9 (a) and 9 (b), showing still another image data obtained by imaging a state in which the reflected light of the backlight hits the belly of the finger when the surroundings are dark. It is an example.

  In FIG. 10A, edge pixels matched by the matching process are shown as shaded portions, and three gradient directions ANG (S) in the lower right direction, the lower direction, and the lower left direction match.

  Here, for each gradient direction ANG (S), defining a vector (first direction vector) whose length is the number of matching pixels, three first direction vectors such as the vector 51 shown in FIG. 10C are obtained. It is done.

  Next, when the three first direction vectors shown in the vector 51 shown in FIG. 10C are combined, a combined vector like the vector 52 shown in FIG. 10D is obtained.

  Here, FIG. 10A is an example of image data captured based on a state in which the reflected light of the backlight hits the belly of the finger, and the captured image tends to become brighter toward the center. Therefore, it is considered that the target direction of the object is opposite to the gradient direction ANG (S) in the image (direction different by 180 degrees).

  Therefore, the object direction specifying unit 9 specifies the direction obtained by converting the vector 52 shown in FIG. 10D in the reverse direction (direction different by 180 degrees), that is, the upward direction as the instruction direction of the object. (FIG. 10 (e)).

  For image data when the surroundings are bright, the object direction specifying unit 9 defines the first direction vector described above for each gradient direction ANG (S), and obtains a result of combining all the first direction vectors. The direction indicated by the combined vector thus specified is directly specified as the target direction of the object.

  In the above description, the first direction vector whose length is the number of matching pixels is defined for each gradient direction ANG (S). However, the matching pixel is not the number of matching pixels itself but the gradient direction ANG (S). A vector (second direction vector) whose length is the product of the number and the gradient magnitude POW (S) may be defined.

  For each gradient direction ANG (S), an average value of gradient magnitudes POW (S) may be calculated, and a vector (third direction vector) having the average value as a length may be defined.

  Further, for each gradient direction ANG (S), a median (median value) of the gradient magnitude POW (S) may be calculated, and a vector (third direction vector) having the median as a length may be defined.

  Further, in the above description, the object coordinate specifying unit 11 specifies the target position of the object using all the edge pixels for which the number of matching pixels can be obtained by the pattern matching process. It is not limited to a specific method.

  For example, if a threshold value for the lower limit of the number of matching pixels is prepared in advance, and the total value of the number of matching pixels obtained by the pattern matching process is lower than the threshold value, for the edge pixel from which the number of matching pixels is obtained, It may not be used for specifying the designated position of the object (the number of matching pixels is 0). By adopting such a configuration, it is possible to eliminate the influence of an unreasonable error pixel caused by noise or the like in the image data and to specify the target position of the target object stably.

[Second Embodiment]
The image processing apparatus 1 (electronic device 20) according to the second embodiment of the present invention will be described below with reference to FIGS. 1, 8, 11, and 12. FIG.

  In the first embodiment, there is only one predetermined threshold when the edge extraction unit 3 specifies an edge pixel.

  However, when it is desired to obtain the target direction with higher accuracy, only the vicinity of the front end of the target from which a relatively stable gradient magnitude POW (S) can be obtained with respect to variations in the angle at which the target is placed is obtained. Therefore, it is necessary to set the predetermined threshold value so as to be specified as an edge pixel.

  On the other hand, in order not to mistakenly specify the position where noise or the like appears as the target position of the target object, not only the vicinity of the tip of the target object but also the entire contact surface of the target object is specified as an edge pixel. Thus, there arises a problem that the predetermined threshold cannot be set too large.

  In the image processing apparatus 1 (electronic device 20) of the second embodiment, in order to solve the above-described problems, the edge extraction unit 3 can set two thresholds for specifying edge pixels. The point which did is different from 1st Embodiment, and the structure of other than that is the same as that of 1st Embodiment. Note that the configuration of the image processing apparatus 1 in the second embodiment is the same as the configuration of the image processing apparatus 1 in the first embodiment and only part of the operation is different, so description of each configuration is omitted. Here, an outline of the operation of the image processing apparatus 1 of the present embodiment will be described.

  Note that the outline of the overall operation is substantially the same as the outline of the overall operation in the image processing apparatus 1 of the first embodiment shown in FIG. However, in the edge pixel specifying process, the first point is that the edge extraction unit 3 shown in FIG. 1 specifies the first edge pixel using the first threshold and specifies the second edge pixel using the second threshold. Different from the embodiment.

  Next, a pattern matching process of the image processing apparatus 1 according to the present embodiment that performs an operation different from that of the first embodiment will be described with reference to FIGS. 8, 11, and 12.

  FIG. 11 is a flowchart showing the operation of the pattern matching process in the image processing apparatus 1 of the present embodiment.

  First, in S301, the coincidence pixel number calculation unit 5 initializes all the coincidence pixel numbers in each gradient direction ANG (S) for the first edge pixel to “0”, and the process proceeds to S302.

  In S302, the coincidence pixel number calculation unit 5 initializes all the coincidence pixel numbers in each gradient direction ANG (S) for the second edge pixel to “0”, and the process proceeds to S303.

  In S303, the coincidence pixel number calculation unit 5 performs a matching process using the model pattern for the first edge pixel for each pixel, and the process proceeds to S304.

  In S304, the matching pixel number calculation unit 5 determines whether or not the model pattern and the gradient direction ANG (S) are matched by the matching process in S303. If the gradient directions ANG (S) match (Yes), the process proceeds to S305, and if they do not match (No), the process proceeds to S307.

  In S305, the coincidence pixel number calculation unit 5 performs a matching process using the model pattern on the second edge pixel, and the process proceeds to S306.

  In the present embodiment, the second threshold value for specifying the second edge pixel is set larger than the first threshold value for specifying the first edge pixel. Therefore, the second edge pixel has a larger gradient magnitude POW (S) than the first edge pixel. In addition, the first edge pixel always exists at the position of the second edge pixel due to the magnitude relationship between the two threshold values of the first threshold value and the second threshold value.

  Therefore, the matching result at the second edge pixel is always obtained by the collation processing in S305. In other words, when the second edge pixel and the model pattern are collated, a result in which the gradient directions ANG (S) coincide is always obtained.

  In S306, for the number of matching pixels for each gradient direction ANG (S) of the first edge pixel and the number of matching pixels for each gradient direction ANG (S) of the second edge pixel, the model pattern and the gradient direction ANG ( The number of pixels that match S) is newly added and updated, and the process proceeds to S307.

  In S307, the coincidence pixel number calculation unit 5 determines whether or not collation has been completed for all the elements (pixels) of the model pattern. If the collation is completed (Yes), the process proceeds to S308, and if not completed (No), the process proceeds to S303 and the collation process is performed on the next element of the model pattern.

  In S308, the coincidence pixel number calculation unit 5 adds the coincidence pixel number for each gradient direction ANG (S) of the first edge pixel and the coincidence pixel number for each gradient direction ANG (S) of the second edge pixel. Is added, and the process proceeds to S309. The combined value output in S308 is used in the object position / direction specifying process in S104 of FIG.

  In S309, the object direction specifying unit 9 specifies and outputs the instruction direction of the object based on the distribution of the number of coincident pixels for each gradient direction ANG (S) of the second edge pixels, and the pattern matching process ends. .

  Next, based on FIGS. 8A and 8B and FIGS. 12A to 12C, a specific example of the target direction specifying process of the target by the target direction specifying unit 9 will be described. The method for specifying the target direction of the object in this embodiment will be described in detail below.

  FIG. 12A is a schematic diagram showing the distribution of the gradient direction ANG (S) for each edge pixel calculated by the gradient feature value calculation unit 2.

  FIG. 12B is a schematic diagram illustrating an example of the distribution of the number of matching pixels for each gradient direction ANG (S) of the first edge pixel calculated by the matching pixel number calculation unit 5, and FIG. These are schematic diagrams showing an example of the distribution of the number of coincident pixels for each gradient direction ANG (S) of the second edge pixels.

  In FIG. 12A, pixels (non-edge pixels) identified as “non-directional” by the gradient feature value calculation unit 2 are shown as empty boxes. FIG. 12A shows still another example of image data obtained by imaging a state in which the reflected light of the backlight hits the belly of the finger when the surroundings are dark.

  In FIG. 12A, the edge pixels matched by the collation process are indicated by a dark shaded portion and a light shaded portion. The thin shaded portion is a pixel specified as the first edge pixel and the second edge pixel, and the dark shaded portion is a pixel specified only as the first edge pixel.

  Here, as described above, the vicinity of the tip of the object is not significantly affected by the contact angle of the object to the display surface, and has a relatively stable gradient magnitude POW (S). The second threshold is set so that only the edge near the tip of the object is specified as the two-edge pixel.

  Next, referring to the distribution of the number of matched pixels for each gradient direction ANG (S) of the first edge pixel shown in FIG. 12B, all the gradient directions ANG (S) in the eight directions are matched. Therefore, it is considered that the shape of the contact portion of the object is close to a circle, and it is considered that the object is correctly detected by the pattern matching process instead of noise. However, since all the gradient directions ANG (S) are matched, there is a possibility that the target direction of the target cannot be correctly specified as it is.

  On the other hand, referring to the distribution of the number of coincident pixels for each gradient direction ANG (S) of the second edge pixel shown in FIG. 12C, there are three gradient directions ANG (S) in the right direction, the lower right direction, and the lower direction. Matching.

  Here, since only the edge near the tip of the object is specified as the second edge pixel, the object is determined based on the number of matching pixels for each gradient direction ANG (S) of the second edge pixel. By specifying the pointing direction, it is possible to specify the pointing direction with high accuracy.

  FIG. 12A is an example of image data obtained by imaging the state in which the reflected light of the backlight hits the belly of the finger. Therefore, the distribution of the number of coincident pixels for each gradient direction ANG (S) and the object Referring to the correspondence table shown in FIG. 8A as the association information of the designated direction, the three gradient directions ANG (S) of the right direction, the lower right direction, and the lower direction are associated with the matched pattern. The pointing direction of the object is defined as the upper left direction.

  Therefore, in the example of FIG. 12A, the object direction specifying unit 9 can specify that the designated direction of the object is the upper left direction (FIG. 12C).

  For the image data when the surroundings are bright, the object direction specifying unit 9 specifies the instruction direction of the object by referring to the correspondence relationship table of FIG.

  As described above, according to the image processing apparatus 1 in the second embodiment, the target position of the target object can be determined while preventing the position where noise or the like appears from being erroneously specified as the target position of the target object. In addition to being able to identify, the pointing direction of the object can be identified more accurately.

  In the above description, the object direction specifying unit 9 uses a correspondence table that indicates the correspondence between the direction matching pattern and the designated direction of the object, as shown in FIGS. Although configured, it is not limited to such a configuration. For example, as described in the first embodiment, a configuration may be adopted in which the target direction of an object is specified without using a correspondence table. That is, the object direction specifying unit 9 defines, for each gradient direction ANG (S), a first direction vector whose length is the number of coincident pixels, and a combined vector obtained as a result of combining all the first direction vectors is It is good also as a structure which specifies the instruction | indication direction of a target object from the direction shown.

  Further, in the above description, the object coordinate specifying unit 11 is configured to specify the target position of the object using all the edge pixels for which the number of matching pixels can be obtained by the pattern matching process. It is not limited to such a configuration. For example, as described in the first embodiment, a lower limit threshold value for the number of matching pixels is prepared in advance, and the total value of the matching pixel number obtained by the pattern matching process is lower than the lower limit threshold value. The edge pixels for which the number of matching pixels is obtained may be configured not to be used for specifying the target position of the object (the number of matching pixels is set to 0). By adopting such a configuration, it is possible to eliminate the influence of an unreasonable error pixel caused by noise or the like in the image data and to specify the target position of the target object stably.

  As described above, the predetermined threshold value may include at least a predetermined first threshold value and a second threshold value that is larger than the first threshold value.

  In addition, the coincidence pixel number calculation unit 5 of the image processing apparatus 1 according to the present embodiment determines, for each gradient direction ANG (S) related to the first edge pixel, from the result of matching the matching region with the model pattern for the first edge pixel. The matching pixel count (second matching score) for each gradient direction ANG (S) related to the second edge pixel is calculated from the result of collation for the matching pixel count (first matching score) and the second edge pixel.

  Here, as described above, for example, the gradient magnitude POW (S) near the tip of the finger is relatively large, and the gradient magnitude POW (S) on the base side of the finger tends to be small. .

  Therefore, for example, if the threshold value for detecting the gradient S near the tip of the finger is the second threshold value and at least the threshold value for detecting the gradient S on the base side of the finger is the first threshold value, it is equal to or greater than the first threshold value. An edge of the entire finger (first edge pixel) can be detected by the pixel, and an edge (second edge pixel) in the vicinity of the tip of the finger can be detected by a pixel having a second threshold value or more.

  The first matching degree is the number of matching pixels related to the edge of the entire finger, and the second matching degree is the number of matching pixels related to the edge near the tip of the finger.

  On the other hand, the object coordinate specifying unit 11 of the image processing apparatus 1 of the present embodiment calculates the number of coincident pixels for each gradient direction ANG (S) related to the first edge pixel and the second edge calculated for each first edge pixel. The indicated position is specified from the distribution on the matching area of the number of matching pixels for each gradient direction ANG (S) related to the second edge pixel calculated for each pixel.

  Here, as described above, for example, if the threshold value for detecting the gradient S near the tip of the finger is the second threshold value, and the threshold value for detecting at least the gradient S on the base side of the finger is the first threshold value, A first edge pixel having a large number of matching pixels tends to be close to the center of the finger belly, and a second edge pixel having a large number of matching pixels tends to be close to a position near the tip of the finger (hereinafter referred to as a tip position).

  Therefore, the indication position specified from the distribution of the number of coincident pixels for each gradient direction ANG (S) related to the first edge pixel is close to the center of the belly of the finger, and the coincidence for each gradient direction ANG (S) related to the second edge pixel. The indicated position specified from the distribution of the number of pixels is close to the tip position.

  In addition, the indicated position specified from the distribution of the number of matching pixels for each gradient direction ANG (S) regarding the first edge pixel and the number of matching pixels for each gradient direction ANG (S) regarding the second edge pixel is the center of the belly of the finger. , It is close to the middle position of the tip position.

  Therefore, if the designated position is specified from the distribution on the matched region of at least one of the first match degree and the second match degree, a position close to the center of the finger, the tip position, or the middle position is set as the designated position. It becomes possible to specify.

  In addition, the number of matching pixels for each gradient direction ANG (S) regarding the first edge pixel, the number of matching pixels for each gradient direction ANG (S) regarding the second edge pixel, or a combination of these matching pixel numbers is selected. Thus, any one of the center of the finger, the tip position, or a position close to the middle position can be specified as the designated position of the object.

[Embodiment 3]
In the first embodiment and the second embodiment, as a method for specifying the target position of the object, a method for specifying the center of gravity of the distribution of the number of matching pixels for each pixel, and a position where the number of matching pixels is the maximum. A method of specifying, a method of specifying a matching area as it is as an existence range (position and size) of an object, a model pattern centered on a centroid of a matching pixel number distribution or a position where the matching pixel number distribution is maximum A method for specifying a range corresponding to the size of the target object as an existing range (position and size) of the object has been described.

  By the way, in order not to mistakenly specify the position where noise or the like appears as the target position of the object, the entire contact surface of the object is specified as an edge pixel, and pattern matching is performed. It is desirable to check the shape of an object relatively closely. Therefore, the threshold value for specifying the edge pixel in the first embodiment and the first threshold value for specifying the first edge pixel in the second embodiment cannot be set too large.

  However, for example, when detecting a human hand or finger, the luminance of the side that is not the tip (the base side in the case of a finger) out of the parts in contact with each other due to fluctuations in the angle of the hand or finger with respect to the display surface. It fluctuates, and the area and shape of the base side edge change accordingly.

  Therefore, if the threshold value (first threshold value) is too small, the area on the non-tip side of the contacted part becomes large, and the number of matching pixels of the edge pixels specified using the threshold value (first threshold value) If the position is specified using, stability is lacking. Therefore, unless the threshold value (first threshold value) is adjusted very carefully, there arises a problem that the indicated position of the object cannot be obtained stably.

  In other words, it can be said that the indicated position of the object described above is easily affected by noise or the like, or a change in the shape of the entire contact surface of the object.

  In the image processing apparatus 1 (electronic device 20) according to the third embodiment, in order to solve the above-described problems, an instruction direction of an object is first specified, and then an instruction direction of the specified object is specified. The point which employ | adopts the method of specifying the designated position (or front-end | tip position) of a target object using is different from embodiment mentioned above.

  As described in the description of the second embodiment, the data near the front end of the object that provides a relatively stable gradient magnitude POW (S) with respect to fluctuations in the angle of the object with respect to the contact surface is used. Thus, by specifying the indication direction of the object, the indication direction can be specified with high accuracy.

  Moreover, if the pointing direction with high accuracy can be specified, the tip position of the object can be specified with high accuracy. Therefore, if the tip position of the target is specified as the target position of the target, a stable position can be specified.

  The image processing apparatus 1 according to the third embodiment of the present invention will be described below with reference to FIGS. Note that the configuration of the image processing apparatus 1 in the third embodiment is the same as the configuration of the image processing apparatus 1 in the first and second embodiments, and a description thereof will be omitted.

  First, an outline of the operation of the image processing apparatus 1 according to the third embodiment of the present invention will be described.

  The outline of the overall operation is the same as the outline of the overall operation in the image processing apparatus 1 of the first embodiment shown in FIG.

  Next, as an operation of the image processing apparatus 1 of the third embodiment, an object position specifying process that performs an operation different from that of the first embodiment and the second embodiment will be described with reference to FIGS. 13 and 14. To do.

  FIG. 13 is a flowchart showing the operation of the object position specifying process in the image processing apparatus 1 of the present embodiment. Here, it is assumed that when the surroundings are dark, the operation is performed on image data obtained by imaging the state in which the reflected light of the backlight hits the belly of the finger.

  First, in S401, the object coordinate specifying unit 11 specifies the gradient direction ANG (S) obtained by rotating the indication direction of the object specified by the object direction specifying unit 9 by 180 degrees, and then the gradient direction ANG. The edge pixel in the matching area having (S) is specified, and the process proceeds to S402.

  In S401, the direction in which the target object is rotated 180 degrees assumes image data when the surrounding is dark, and the captured image tends to become brighter toward the center. This is because the designated direction and the gradient direction of the pixel at the tip position of the object are opposite to each other (the two directions form an angle of 180 degrees).

  On the other hand, for image data when the surrounding is bright, in S401, an edge pixel having the same gradient direction ANG (S) as the target direction specified by the target object specifying unit 9 is specified. do it.

  In S402, the object coordinate specifying unit 11 checks the gradient magnitude POW (S) of each edge pixel specified in S401, and selects the pixel having the largest gradient magnitude POW (S) as the target position of the object. And the object position specifying process is terminated.

  Next, a specific example of the object position specifying process of the object by the object coordinate specifying unit 11 will be described with reference to FIGS.

  FIGS. 14A and 14B show an example of the gradient direction ANG (S) for each edge pixel calculated by the gradient feature value calculation unit 2 and the indication direction of the target specified by the target direction specifying unit 9. FIG. Note that a pixel (a pixel that is not an edge pixel) identified as “non-directional” by the gradient feature amount calculation unit 2 is shown as an empty box in FIG.

  FIG. 14B is an example of image data obtained by imaging a state in which the reflected light of the backlight hits the belly of the finger when the surrounding is dark.

  As shown in FIG. 14A, the target direction of the object is specified as the upper left direction, and therefore the gradient direction ANG (S) obtained by rotating the target direction of the object 180 degrees is the lower right direction. Further, when an edge pixel whose gradient direction ANG (S) is in the lower right direction is specified, a thin shaded portion and a dark shaded portion in FIG. 14B are obtained. The pixel having the largest gradient magnitude POW (S) is identified as a dark shaded pixel by referring to the gradient magnitude POW (S) (not shown). Therefore, the target object coordinate specifying unit 11 specifies the position of the pixel in the dark shaded portion as the target position of the target object.

  Here, for example, an area where there are many pixels having a gradient direction that is the same as or opposite to the indicated direction specified by the object direction specifying unit 9 is considered to be an area close to the tip position of the object among the matching areas. It is done.

  In addition, even if the shape of the entire contact surface of the object, especially the area and shape of the image on the base side of the finger changes, the position of the region close to the tip position on the matching region or the pixel included in the region close to the tip position It is considered that the gradient direction does not change much.

  Therefore, as described above, the object coordinate specifying unit 11 determines the tip position of the object from the region where pixels having gradient directions that are the same as or opposite to the indicated direction specified by the object direction specifying unit 9 exist. If specified, the tip position can be specified with high accuracy regardless of whether the shape of the entire contact surface of the object, particularly the area and shape of the image at the base of the finger, has changed.

  As described above, according to the image processing device 1 in the third embodiment, the target position of the target object is prevented from being erroneously specified as the target position of the target object, where noise or the like appears. It becomes possible to acquire (tip position) more stably.

  Further, the object coordinate specifying unit 11 determines the distribution of the gradient direction ANG (S) for each edge pixel included in the matching area, the indication direction specified by the object direction specifying unit 9, and the gradient magnitude POW (S). The tip position of the object on the captured image may be specified.

  By the way, as described above, the gradient magnitude POW (S) tends to increase as it goes from the base of the finger toward the vicinity of the tip of the finger. Therefore, among the regions where there are many pixels having the gradient direction ANG (S) close to the same or opposite direction as the indication direction specified by the object direction specifying unit 9, the pixel having the large image gradient size POW (S) is selected. The position is considered to be a position close to the tip position of the object.

  Therefore, the object coordinate specifying unit 11 determines, for example, an area near the tip of the finger from the distribution of the gradient direction ANG (S) for each edge pixel included in the matching area and the indication direction specified by the object direction specifying unit 9. Further, since the position of the pixel having a large gradient POW (S) from the region near the tip of the finger can be set as the tip position, the tip position of the target can be specified with higher accuracy.

  The object coordinate specifying unit 11 calculates a gradient direction in the same direction as the designated direction specified by the object direction specifying unit 9 among the gradient directions ANG (S) for each edge pixel included in the matching region. The tip position of the object on the captured image may be specified from the distribution of edge pixels.

  By the way, in the case of a liquid crystal display with a built-in image sensor, for example, the image sensor detects the superposed light of the reflected light reflected by the object and the light emitted from the backlight and the outside light.

  In a captured image obtained by imaging such superposed light with an imaging sensor, the gradient direction ANG (S) of the edge portion tends to be opposite to each other when the surrounding is bright and when the surrounding is dark.

  Therefore, the object coordinate specifying unit 11 assumes that the surrounding is bright, and the edge in which the gradient direction ANG (S) in the same direction as the indicated direction is calculated among the gradient directions ANG (S) for each edge pixel. The tip position is specified from the pixel distribution.

  Thereby, even when the surroundings are bright, the tip position can be specified with high accuracy.

  Further, the object coordinate specifying unit 11 has a gradient direction ANG (S) opposite to the indicated direction specified by the object direction specifying unit 9 among the gradient directions ANG (S) for each edge pixel included in the matching region. The tip position of the object on the captured image may be specified from the calculated distribution of edge pixels.

  The object coordinate specifying unit 11 assumes that the surrounding is dark, and among the gradient directions ANG (S) for each edge pixel, the object coordinate specifying unit 11 calculates the gradient direction ANG (S) opposite to the indicated direction. The tip position is specified from the distribution.

  Thereby, even when the surroundings are dark, the tip position can be specified with high accuracy.

  In the above description, the object coordinate specifying unit 11 determines the position of the pixel having the largest gradient magnitude POW (S) among the edge pixels in the gradient direction ANG (S) corresponding to the instruction direction of the object. Although the configuration is such that it is specified as the indicated position of the object, it is not limited to this configuration. For example, it is good also as a structure which specifies the gravity center of each edge pixel which has the gradient direction ANG (S) corresponding to the instruction | indication direction of a target object as a target position of a target object.

  In addition, among the edge pixels having the gradient direction ANG (S) corresponding to the indication direction of the object, the indication direction or the direction different from the indication direction by 180 degrees from the position of the pixel having the largest gradient magnitude POW (S). The position moved by a predetermined amount may be specified as the designated position of the object.

[Fourth Embodiment]
By the way, in the specified position specifying method described in the first embodiment or the second embodiment, the specified position of the target object is specified near the center of the contact portion of the target object.

  However, as described in the third embodiment, in the specifying method of the first embodiment or the second embodiment, the indication position of the object is not adjusted unless the threshold (first threshold) is adjusted very carefully. The problem that can not be obtained stably.

  On the other hand, in the third embodiment, the tip position of the object is specified at a position near the tip of the contact portion of the object. However, if it is desired to specify the vicinity of the center of the contact portion of the object as the target position of the object, it is difficult to stably acquire which position is near the center with the third embodiment alone. The problem remains.

  In the image processing apparatus 1 (electronic device 20) of the fourth embodiment, in order to solve the above-described problems, the object coordinate specifying unit 11 performs the first embodiment or the second operation in the previous operation. When the indicated position is specified by the method of the embodiment and the tip position is specified by the method of the third embodiment in the previous and current operations, the current operation is performed from the tip position specified in the previous operation. It differs from the above-described embodiment in that the position where the designated position identified in the previous operation is displaced by the amount of displacement up to the identified tip position is set as the designated position of the current object. .

  As a result, the specified position is specified in the previous operation, but even if the specified position cannot be specified in the current operation for some reason, the tip position in which the tip position was specified in the previous and current operations is used. Since the designated position is specified, it is possible to specify the designated position of the target object with high accuracy while preventing the position of noise or the like in the captured image from being erroneously specified as the designated position of the target object.

  Next, an object position specifying process of the image processing apparatus 1 according to the present embodiment that performs operations different from those of the first to third embodiments will be described with reference to FIGS. 1 and 15.

  Note that the configuration of the image processing apparatus 1 in the fourth embodiment is the same as the configuration of the image processing apparatus 1 in the first to third embodiments, and a description thereof will be omitted. The outline of the overall operation of the image processing apparatus 1 is also the same as the outline of the overall operation of the image processing apparatus 1 according to the first embodiment shown in FIG.

  FIG. 15 is a flowchart showing the operation of the object position specifying process in the image processing apparatus 1 of the present embodiment.

  In the following, an object position specifying process for performing an operation different from the first to third embodiments among the overall operations in the image processing apparatus 1 of the present embodiment will be described.

  First, for each predetermined unit time, a first position (indicated position) of the object and a second position (tip position) different from the first position are stored.

  Here, the object coordinate specifying unit 11 uses the specified method of specifying the target position of the object as in the first embodiment or the second embodiment as the first position from the captured image data. The position near the center of the contact portion of the object is specified.

  Next, the target object coordinate specifying unit 11 uses the method for specifying the front end position of the target object as in the third embodiment as the second position from the captured image data, to the front end of the target object. Identify close locations.

  Next, the object coordinate specifying unit 11 tries to read out the stored first position. When the reading of the first position is successful, that is, when the first position has already been stored, the object coordinate specifying unit 11 reads the stored second position, and the captured image data The coordinate difference between the second position identified from the above and the second position already stored is calculated, and the calculated first difference is added to the stored first position to obtain a new first position. The position of is calculated.

  On the other hand, when the reading of the first position fails, that is, when the first position is not yet stored, the first position specified from the captured image data is used as it is as the new first position. And

  Then, the object coordinate specifying unit 11 specifies the new first position as the designated position of the object, and sets the new first position and the second position newly specified from the captured image data. Store and store the object position identification process. If the first position is not specified, the stored first position is deleted.

  Next, FIG. 15 is a flowchart showing the operation of the object position specifying process in the fourth embodiment described above. In the fourth embodiment, it is assumed that the image processing apparatus 1 repeatedly executes the operation shown in this flowchart every predetermined unit time.

  First, in S501, the target object coordinate specifying unit 11 checks whether or not the first position of the target object is stored. If stored (Yes), the process proceeds to S504. If not stored (No), the process proceeds to S502.

  In S502, the object coordinate specifying unit 11 uses the method for specifying the target position of the object in the first or second embodiment as the first position from the captured image data. After trying to specify the position near the center of the contact part, the process proceeds to S503.

  In S503, the object coordinate specifying unit 11 checks whether the position near the center has been specified in S502. If it can be identified (Yes), the process proceeds to S505, and if it cannot be identified (No), the process proceeds to S512.

  In S504, the object coordinate specifying unit 11 reads the stored first position, and proceeds to S505.

  In S505, the object coordinate specifying unit 11 uses the method for specifying the tip position of the target object according to the third embodiment as a second position from the captured image data. After the identification is attempted, the process proceeds to S506.

  In S506, the object coordinate specifying unit 11 checks whether or not the position near the tip has been specified in S505. If it can be identified (Yes), the process proceeds to S507, and if it cannot be identified (No), the process proceeds to S512.

  In S507, the object coordinate specifying unit 11 checks whether or not the second position of the object is stored. If stored (Yes), the process proceeds to S508, and if not stored (No), the process proceeds to S509.

  In S508, the object coordinate specifying unit 11 calculates a coordinate difference between the new second position specified in S505 and the already stored second position, and the calculated difference is already stored. The new first position is calculated by adding the first position, and the process proceeds to S509.

  In S509, the object coordinate specifying unit 11 stores the new second position specified in S505, and proceeds to S510.

  In S510, the object coordinate specifying unit 11 specifies the first position as the designated position of the object, and the process proceeds to S511. Here, the position specified as the target position of the target object is the newly calculated first position if the process of S508 is performed, and is stored if the process of S508 is not performed. The first position or the first position specified in S502 (not yet stored).

  In S511, the object coordinate specifying unit 11 stores the (new) first position, and ends the object position specifying process.

  In S512, the object coordinate specifying unit 11 deletes the stored first position and second position, and ends the object position specifying process without specifying the indicated position of the object. . In this case, the designated position of the object cannot be specified.

  As described above, according to the image processing apparatus 1 in the fourth embodiment, it is possible to continue to acquire the position near the center of the contact portion of the target object with high accuracy and stability.

  Finally, each block of the image processing apparatus 1 (electronic device 20), in particular, the gradient feature amount calculation unit 2, the edge extraction unit 3, the matching pixel number calculation unit 5, the matching region specification unit 7, the object direction specification unit (direction specification) The means) 9 and the object coordinate specifying unit (position specifying means) 11 may be configured by hardware logic, or may be realized by software using a CPU as follows.

  That is, the image processing apparatus 1 (electronic device 20) includes a CPU (central processing unit) that executes instructions of a control program that realizes each function, a ROM (read only memory) that stores the program, and a RAM that expands the program. (Random access memory), a storage device (recording medium) such as a memory for storing the program and various data. An object of the present invention is to enable a computer to read the program code (execution format program, intermediate code program, source program) of the control program of the image processing apparatus 1 (electronic device 20), which is software that implements the functions described above. It is also achieved by supplying the recorded recording medium to the image processing apparatus 1 (electronic device 20), and the computer (or CPU or MPU: micro processing unit) reads and executes the program code recorded on the recording medium. Is possible.

  Examples of the recording medium include a tape system such as a magnetic tape and a cassette tape, a magnetic disk such as a floppy (registered trademark) disk / hard disk, and a compact disk-ROM / MO / MD / digital video disk / compact disk-R. A disk system including an optical disk, a card system such as an IC card (including a memory card) / optical card, or a semiconductor memory system such as a mask ROM / EPROM / EEPROM / flash ROM can be used.

  In addition, if the system configuration is capable of connecting to a communication network including the Internet, the recording medium is preferably a recording medium that fluidly carries the program so as to download the program from the communication network.

  Further, when the program is downloaded from the communication network as described above, it is preferable that the download program is stored in the main device in advance or installed from another recording medium.

  Further, the image processing apparatus 1 (electronic device 20) may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Further, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention is a computer program encoded in a computer-readable medium, and the image processing apparatus includes the readable medium, and when executed by a computer, each of the image processing apparatuses It can also be realized as a computer program for realizing the function of the means.

  The present invention can also be expressed as follows.

  That is, the image processing apparatus according to the present invention includes a feature amount calculation unit that calculates a feature amount according to a pixel value difference with an adjacent pixel for each pixel on image data of a captured image, and the feature amount calculation. An edge specifying means for specifying an edge pixel whose feature amount calculated by the means is equal to or greater than a threshold; a collation area included in a partial area on the image data based on a result of specifying the edge pixel; A matching degree calculation means for calculating a matching degree indicating a matching degree between the matching area and the model pattern, and a distribution for each pixel of the matching degree calculated by the matching degree calculation means. From the position specifying means for specifying the position of the object on the captured image, the distribution of the edge pixels specified by the edge specifying means for each pixel, and the matching degree calculating means All matching degree, the distribution of each pixel may comprise a direction specifying means for specifying a direction in which the object on the captured image is indicated.

  In the image processing apparatus of the present invention, the threshold value includes at least two threshold values: a first threshold value and a second threshold value that is larger than the first threshold value, and the edge specifying means includes the feature value calculating means. At least two types of edge pixels, a first edge pixel whose calculated feature value is equal to or greater than the first threshold value and a second edge pixel whose feature value calculated by the feature value calculation means is equal to or greater than the second threshold value. And the degree-of-match calculation means includes at least two types of matches: a first degree of matching based on the result of specifying the first edge pixel and a second degree of matching based on the result of specifying the second edge pixel. The position specifying means specifies the position of the object using one or both of the first match degree and the second match degree calculated by the match degree calculation means, and the direction specifying means The degree of match calculation means A second matching degree may be specified in the direction of the object.

  In the image processing apparatus of the present invention, the feature amount may include at least a gradient direction indicating a direction of luminance gradient of the pixel.

  In the image processing apparatus according to the aspect of the invention, the degree-of-match calculation unit calculates the number of coincident pixels in which the gradient direction included in the matching region matches the gradient direction included in the model pattern for each gradient direction. You may do it.

  In the image processing apparatus of the present invention, association information in which the number of matching pixels for each gradient direction is associated with the direction indicated by the object is determined in advance, and the direction specifying unit The direction indicated by the object on the captured image may be specified from the information and the distribution of the matching degree for each pixel.

  Further, in the image processing device of the present invention, the direction specifying unit is configured to detect a target on the captured image from a vector amount having the length of the number of coincident pixels for each gradient direction and a distribution of the degree of coincidence for each pixel. The direction indicated by may be specified.

  In the image processing apparatus of the present invention, the feature amount includes at least a vector amount having a gradient direction indicating a magnitude of a pixel luminance gradient and a direction of the pixel luminance gradient, and the direction specifying unit includes: The direction indicated by the object on the captured image is specified from the vector amount obtained by multiplying the number of matching pixels for each gradient direction by the magnitude of the previous gradient and the distribution of the matching degree for each pixel. May be.

  Further, in the image processing apparatus of the present invention, the position specifying unit specifies the distribution of the edge pixels for each pixel specified by the edge specifying unit, the direction specified by the target specified by the direction specifying unit, The position of the object on the captured image may be specified from the distribution of the degree of coincidence calculated by the degree of coincidence calculating means for each pixel.

  In the image processing apparatus according to the aspect of the invention, the feature amount includes at least a magnitude of a luminance gradient of a pixel, and the position specifying unit specifies the luminance gradient of the edge pixel specified by the edge specifying unit. The position of the object on the captured image based on the size and distribution for each pixel, the direction indicated by the object specified by the direction specifying unit, and the distribution for each pixel of the matching degree calculated by the matching degree calculating unit. May be specified.

  Further, in the image processing apparatus of the present invention, the position specifying unit has a pixel-by-pixel distribution of the edge pixels having the same gradient direction as the direction indicated by the object specified by the direction specifying unit, and the matching degree calculating unit. The position of the object may be determined from the distribution of the degree of match calculated for each pixel.

  In the image processing apparatus of the present invention, the position specifying unit has a gradient direction obtained by rotating the direction indicated by the target specified by the direction specifying unit by 180 degrees and the edge pixel distribution for each pixel, and the match The fingertip position may be determined from the distribution of the degree of match calculated by the degree calculation means for each pixel.

  In addition, the image processing apparatus of the present invention is characterized in that an image is captured every predetermined unit time, and the position of the object on the captured image and the pointing direction are identified from the captured image. An image processing apparatus comprising: a position storage unit that stores at least a first position of an object and a second position different from the first position for each predetermined unit time; The position specifying means specifies, as a first position, a position of an object on the captured image specified from a distribution for each pixel of the degree of match calculated by the match degree calculating means, and the edge specifying means Specified by the distribution of the edge pixels for each pixel, the direction specified by the direction specifying unit, the direction indicated by the object, and the distribution for each pixel of the matching degree calculated by the matching degree calculating unit, Object on the captured image Is determined as the second position, and when the first position is stored in the position storage means, the specified first position is stored in the position storage means. Recalculating from the first position, the second position stored in the position storage means, and the specified second position, and specifying the first position as the position of the object Also good.

  In the image processing apparatus of the present invention, the matching degree calculation unit may set the matching degree to 0 when the calculated matching degree is less than a predetermined threshold.

  In each of the above configurations, a pixel having a feature amount equal to or greater than a predetermined threshold is specified as an edge pixel from image data of a captured image that has been mainly captured, and then the pointing direction of the imaging target is determined using a pattern matching technique. And the specified position. Therefore, it is possible to specify the target position and the pointing direction with high accuracy while preventing the position of noise or the like in the captured image from being erroneously specified as the target position of the target object.

  Note that the present invention is not limited to the above-described example of the image processing apparatus (electronic device), and various modifications are possible within the scope of the claims, and [Embodiment 1] to [Embodiment 1] Embodiments obtained by appropriately combining the technical means disclosed in the respective items of [Embodiment 4] are also included in the technical scope of the present invention.

  The image processing apparatus of the present invention can be applied to an apparatus that performs operations and instructions by touching a display of a display device such as a liquid crystal with an object such as a finger, such as a mobile phone or a PDA. it can. Specifically, as a display device, for example, it can be used for an active matrix liquid crystal display device, and an electrophoretic display, a twist ball display, a reflective display using a fine prism film, a digital mirror device, etc. In addition to displays using light modulation elements, organic EL light-emitting elements, inorganic EL light-emitting elements, displays using variable-light-emitting elements such as LEDs (Light Emitting Diodes), and field emission displays (FED) It can also be used for plasma displays.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 Gradient feature-value calculation part (Gradient feature-value calculation means)
3 Edge extraction unit (edge pixel identification means)
5 Matching pixel number calculation unit (matching degree calculation means)
7 Matching area specifying part (matching area specifying means)
9 Object direction identification part (direction identification means)
11 Object coordinate specifying part (position specifying means)
31, 32 Captured image S gradient (pixel value gradient feature)
Sx Horizontal gradient amount (pixel value gradient feature amount)
Sy Vertical gradient (pixel value gradient feature)
POW (S) Gradient magnitude (pixel value gradient feature)
ANG (S) Gradient direction (pixel value gradient feature)

Claims (19)

A gradient feature amount calculating means for calculating a pixel value gradient feature amount from image data of a captured image;
Edge pixel specifying means for specifying an edge pixel whose magnitude of the pixel value gradient feature value calculated by the gradient feature value calculating means is equal to or greater than a predetermined threshold;
A matching degree calculation for matching a matching area of a predetermined size set on the image data with a predetermined model pattern and calculating a matching degree indicating a degree of matching between the matching area and the model pattern Means,
A match area specifying means for specifying, as a match area, the matching area where the match degree calculated by the match degree calculating means is maximum;
Direction specifying means for specifying the indicated direction of the object on the captured image from the distribution of the pixel value gradient feature value for each of the edge pixels included in the matching area specified by the matching area specifying means. An image processing apparatus.   The image processing apparatus according to claim 1, wherein the matching degree calculating unit performs matching between the model pattern and a region including the edge pixel specified by the edge pixel specifying unit on the matching region. . The matching degree calculation unit sets a plurality of matching areas by scanning the matching area pixel by pixel while matching the target pixel included in the matching area with all the pixels included in the image data. By calculating the degree of matching calculated for each matching area for each pixel of interest corresponding to each matching area, the degree of matching is calculated for all the pixels included in the image data,
3. The image according to claim 1, further comprising: a position specifying unit that specifies an indicated position of an object on the captured image from the distribution of the matching degree for each pixel included in the matching area. Processing equipment. The predetermined threshold includes at least a predetermined first threshold and a second threshold larger than the first threshold,
The edge pixel specifying means includes a first edge pixel whose magnitude of the pixel value gradient feature value calculated by the gradient feature quantity calculation means is greater than or equal to the first threshold value, and a second edge pixel that is greater than or equal to the second threshold value. Are identified as the edge pixels,
The degree-of-match calculation means calculates the first match from the result of matching the matching area and the model pattern for the first edge pixel, and the second match from the result of matching for the second edge pixel. Degree is calculated as the degree of match,
The direction specifying means specifies the indication direction from the distribution of the pixel value gradient feature value for each of the second edge pixels included in the matching region,
The said position specification means specifies the said instruction | indication position from distribution of at least one among the said 1st coincidence degree and the said 2nd coincidence degree for every pixel contained in the said coincidence area. Image processing apparatus.   The image processing apparatus according to claim 3, wherein the pixel value gradient feature amount is a feature amount indicating at least a gradient direction of the pixel value.   For each gradient direction, the matching degree calculation means calculates the matching pixel number indicating the number of pixels in which the gradient direction included in the matching region and the gradient direction included in the model pattern match. The image processing apparatus according to claim 5, wherein Correspondence information indicating a correspondence relationship between the distribution of the number of matched pixels for each gradient direction and the indication direction is predetermined,
The direction specifying unit specifies the indication direction from the distribution of the number of matched pixels for each gradient direction calculated by the matching degree calculation unit with reference to the association information. Image processing apparatus.   The image processing apparatus according to claim 6, wherein the direction specifying unit specifies the indication direction from a distribution of first direction vectors having a length corresponding to the number of coincident pixels for each gradient direction.   The direction specifying means specifies the indication direction from a distribution of second direction vectors having a length as a product of the number of matching pixels for each gradient direction and the magnitude of the pixel value gradient feature amount. Item 8. The image processing apparatus according to Item 6 or 7.   The said direction specification means specifies the said instruction | indication direction from distribution of the 3rd direction vector which sets the magnitude | size of the said pixel value gradient feature-value for every said gradient direction as length. The image processing apparatus described.   The position specifying means specifies the tip position of the object on the captured image from the distribution of the gradient direction for each edge pixel included in the matching region and the indicated direction specified by the direction specifying means. The image processing apparatus according to claim 6.   The position specifying unit is configured to calculate the position on the captured image from the distribution of the gradient direction for each edge pixel included in the matching region, the indication direction specified by the direction specifying unit, and the size of the pixel value gradient feature amount. The image processing apparatus according to claim 6, wherein a tip position of the target object is specified.   The position specifying means is based on a distribution of the edge pixels in which the gradient direction in the same direction as the indicated direction specified by the direction specifying means is calculated among the gradient directions for each edge pixel included in the matching region. The image processing apparatus according to claim 6, wherein a tip position of an object on the captured image is specified.   The position specifying means is based on a distribution of the edge pixels in which the gradient direction opposite to the indicated direction specified by the direction specifying means is calculated among the gradient directions for each edge pixel included in the matching region. The image processing apparatus according to claim 6, wherein a tip position of an object on the captured image is specified.   The position specifying means specifies the current position from the tip position specified in the previous operation when the indicated position is specified in the previous operation and the tip position is specified in the previous and current operations. The position of the target position specified by the previous operation by the amount of displacement up to the specified tip position is set as the current target position of the object. The image processing apparatus according to any one of the above.   The match level calculation unit sets the match level to 0 when the calculated match level is less than a predetermined threshold value, according to any one of claims 1 to 15. The image processing apparatus described.   17. A control program for an image processing apparatus for causing a computer to operate as each means in the image processing apparatus according to claim 1.   The computer-readable recording medium which recorded the control program of Claim 17. A control method of an image processing apparatus having a function of specifying an instruction direction of an object on the captured image using image data of a captured image,
A gradient feature amount calculating step for calculating a pixel value gradient feature amount from the image data;
An edge pixel specifying step for specifying an edge pixel in which the magnitude of the pixel value gradient feature calculated in the gradient feature calculating step is equal to or greater than a predetermined threshold;
A matching degree calculation for matching a matching area of a predetermined size set on the image data with a predetermined model pattern and calculating a matching degree indicating a degree of matching between the matching area and the model pattern Steps,
A matching area specifying step for specifying, as a matching area, the matching area where the matching degree calculated in the matching level calculation step is maximum;
And a direction specifying step for specifying the indicated direction of the object on the captured image from the distribution of the pixel value gradient feature quantity for each of the edge pixels included in the match area specified in the match area specifying step. And a control method for the image processing apparatus.

Images