JP3499608B2 - Input coordinate judgment device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an input coordinate judging device for detecting the degree of overlap between input image information and automatically or manually input coordinate information.

[0002]

2. Description of the Related Art Conventionally, such a process has been inspected by, for example, inspecting whether or not a correct input is made by using a positional relationship of input coordinate information, or by checking a neighboring pixel at an input coordinate. That is, in the former case, the ratio of the distances between the input coordinates is calculated and compared with the previously registered reference data. In the latter case, the gradient vector of the image at the input coordinates is obtained and the inclination is used as the reference. I was comparing it with the data.

[0003]

By the way, the input image information is transformed and matched by using the input coordinate information,
In the image processing for outputting the obtained image, if the input coordinate information is incorrect, there is a problem that the output image will also be incorrect. Therefore, in order to solve such a problem, it is necessary to check the consistency between the input coordinate information and the image information.

However, in the conventional technique, when the inspection is performed only by the coordinate information, the deviation from the image cannot be determined. In addition, when examining the neighboring pixels, the inspection may be erroneous when the coordinate information is largely deviated.

In consideration of the above-mentioned problems of the conventional technique, the present invention can stably detect the error between the input image and the input coordinates, and can perform image transformation using the input image. An object is to provide a coordinate determination device.

[0006]

According to the present invention, there is provided an image input section for inputting an image, an image storage section for storing the input image, and a coordinate input section for inputting coordinates of a specific part in the image. A reference structure model part in which a reference structure of an input image is described in advance by a set of patches; a structure model matching part for deforming and matching the reference structure model based on the coordinates input by the coordinate input part; For each patch of the matched structural model, a brightness calculation unit that calculates brightness from the image in the image storage unit, a reference data unit that records a standard brightness distribution of the image to be input, and the brightness calculation unit. The brightness of each patch obtained by the above is compared with the brightness of each patch of the reference data section to determine whether the coordinates input from the coordinate input section are correctly input. Part and An input coordinate determining device was e.

[0007]

According to the present invention, the position information obtained by the coordinate input unit is used to deform and match the structural model of the reference structural model unit by the structural model matching unit. Further, the brightness of each patch of the obtained structural model is calculated by the brightness calculator. If the input position information is correct, the deformation-matched structural model is correctly matched to the image in the image storage unit.Therefore, the lightness distribution in each patch shows that the input image is of the same type recorded in the reference data unit. If it ’s an image,
The brightness distribution should be similar.

Therefore, the lightness distribution of the same kind of image serving as a reference is stored as the data of the reference data part, and the difference from the lightness obtained by the lightness calculation part is calculated in the judgment part,
The difference between the position information and the input image is detected from the calculation result.

[0009]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing the configuration of an embodiment of the invention described in claim 1. In the figure, 1 is an image input section for inputting an image, 2 is an image storage section for storing an image, 3 is a coordinate input section for inputting coordinates of a specific part of the image, and 4 is a patch of the structure of the input image. A reference structural model unit described as a set, 5 is a structural model matching unit that deforms and matches the structural model according to the input coordinates, and 6 is the brightness calculated from the image in the image storage unit based on the matched structural model. A lightness calculation unit, 7 is a reference data unit recording an average lightness distribution of an image to be input, 8 is a difference between the lightness obtained by the lightness calculation unit 6 and the lightness in the reference data unit 7, This is a determination unit that determines whether or not the coordinates input from the coordinate input unit 3 are correctly present on the image obtained from the image input unit 1. Hereinafter, in the present embodiment, a face will be taken as an example of an input image, and the image will be described more specifically.

The image input unit 1 uses a television camera, specifically, to capture a color face image and input it to the image storage unit 2. For the face image in the image storage unit 2, the coordinate input unit 3
Position information of each feature point such as eyebrows, eyes, nose, mouth, etc. can be manually input by using a mouse, or, for example, a patent application has already been made (Japanese Patent Laid-Open No. 5-197793). It is automatically entered using the device described in ".

FIG. 2 shows an example of coordinates to be input to a face image of a person. In this embodiment, the coordinate P to be input is 49
I am individual. For example, the apex of the head, lateral points of the left and right eyes, lateral points of the left and right mouth, and the like. At that time, it may be input incorrectly or improperly, regardless of whether it is manual or automatic.

FIG. 3 shows an example of the structural model described in the reference structural model section 4. In this way, the structural model is composed of a set of triangular patches. More specifically, the structural model is the vertex data v0 (x0, y0), v1 (x1, y1), v2 (x2, y2)
… And the concatenated data p0 (v0, v1, v
2), p1 (v0, v2, v3) ... And. In the case of the structural model of FIG. 3, it is composed of 205 vertex data and 380 connected data. Among these data, there are head vertices, left and right eye lateral points, left and right oral lateral points, and the like.

Next, the structural model matching unit 5 deforms and matches the structural model so as to match the coordinates P obtained by the coordinate input unit 3. Specifically, the vertices on the structural model are classified into vertices corresponding to the coordinates P obtained from the coordinate input unit 3 (head vertices, left and right eye lateral points, etc.) and vertices not. For the corresponding vertices, the coordinate value of the obtained coordinate P is used as it is. For non-corresponding vertices, linear interpolation is performed as follows while maintaining the positional relationship with the corresponding vertices to update the coordinates of all vertices and generate a structural model that matches the input image. . When this linear interpolation is expressed by an equation, q (x, y) ... Vertex to be interpolated in the structural model p0 (x0, y0) p1 (x1, y1) p2 (x2, y2) ... In the structural model, Feature points m0: m1: m2 that satisfy the following conditions: Ratio of distances between vertex q and feature points p0, p1, p2 p0 '(x0', y0 ') p1' (x1 ', y1') p2 '(x2 ', y2') ・ ・ ・ The vertex q '(x', y ') obtained by interpolation is the feature point obtained from the input image,
It is calculated by (Equation 1).

[0015]

[Equation 1] The conditions mentioned here are: -On the structural model, the vertex to be interpolated is the feature point 3
It must be inside a triangle obtained by connecting points. -The phase of the triangle obtained by connecting the feature points is the same in the structural model and the extracted feature points. -The feature points are close to the vertex to be interpolated. Is. An example of the matched structural model is shown in FIG.

Next, the lightness calculation unit 6 calculates the lightness of each patch of the matched structural model from the image data of the image storage unit 2. Specifically, the average brightness of the pixels surrounded by the three vertices may be obtained. After that, the determination unit 8 calculates a difference between the reference lightness in the reference data unit 7 and the lightness obtained by the lightness calculation unit 6 as described below, and when the difference is smaller than a certain threshold value T, the coordinate input value Determines that it is correctly on the input image.

FIG. 5 shows the input image (left figure) and the lightness obtained by the lightness calculating section 6 (right figure), and FIG. 6 shows the reference lightness in the reference data section 7. Further, FIG. 7 shows coordinate vertices (white dots) obtained by erroneous automatic extraction, the input image (left figure), and the lightness obtained by the lightness calculation unit 6 (right figure). In the case of the present embodiment, for example, when the determination unit 8 normalizes the sum of the brightness of each patch to a certain value for each shape model and obtains the sum of the absolute values of the differences from the reference data, the case of FIG. 0.80, 1.2 in the case of Fig. 7
It becomes 0. Therefore, by setting a certain threshold value T to T = 1.00, it can be seen that the coordinate input value (white dot) in FIG. 7 was incorrect coordinate automatic extraction.

In the case of the invention according to claim 2 of the present invention, for example, front, right, left,
Data when the light sources are present in the upper and lower five directions is prepared, and the difference with each reference data is compared in the determination unit 8, and the one having the smallest sum of absolute values is set as the distance from the input image. By doing so, it is possible to make a strong judgment against the fluctuation of the light source.

In the case of the third aspect of the present invention, what is calculated in the judging section 8 is not the lightness of each patch of the structural model but the difference of lightness between each patch. By using the difference in brightness, it is possible to make a judgment that is more robust to the illumination condition.

FIG. 8 is a block diagram showing the configuration of an embodiment of the invention according to claim 5 of the present invention. In the figure, 9 is a lightness correction unit for correcting the lightness of the face image, which is an example added to the latter stage of the lightness calculation unit 6. In addition, the same functional blocks as those in FIG. 1 are designated by the same reference numerals. To perform the brightness correction processing in the brightness correction unit 9, the reference structure model unit 4 has three-dimensional vertex data so that the normal vector of each patch can be obtained. The lightness correction unit 9 estimates the position of the light source from the lightness of each patch obtained from the lightness calculation unit 6 and the normal vector. The light source can be estimated by using the patch of the skin region of the face for estimation and using the average of the normal vectors of a plurality of patches having high brightness. That is, it is estimated that the light source exists on the side where the bright region exists. When the estimated light source and the reference data in the reference data unit 7 have different light source positions, the lightness is corrected so as to match one of the light sources. That is, the brightness of the patch close to the normal of the old light source direction is lowered, and the brightness of the patch close to the normal of the new light source direction is increased. By performing such a correction, it is possible to make a determination under a wide range of illumination conditions.

Each means of the present invention can be realized by software using a computer, or can be realized by using a dedicated hardware circuit having these respective functions.

Further, the reference structure model is not limited to the triangular patch, and may be a patch having another shape.

The deformation matching performed by the structural model matching unit is not limited to linear interpolation, but can be performed by other methods.

[0024]

As is apparent from the above description,
The input coordinate determination device of the present invention generates a structural model matching the image from the input image and coordinates, obtains the brightness from the input image, and adds the brightness information to the structural model. By comparing the brightness on the obtained structural model with the brightness of the reference data, the consistency of the input image and the coordinates is judged, so even if the lighting conditions change slightly, the change in brightness is relatively small. It is possible to stably detect the error between the input image and the input coordinates, and as a result, it becomes possible to perform image transformation using the input image.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of an embodiment of the invention according to claim 1 of the present invention.

FIG. 2 is a diagram showing an example of feature points P to be input.

FIG. 3 is a diagram showing an example of a reference structure model.

FIG. 4 is a diagram showing an example of a structural model matched to an input image.

FIG. 5 is a diagram showing an input image and the brightness obtained by a brightness calculation unit.

FIG. 6 is a diagram showing an example of brightness of reference data stored in a reference data section.

FIG. 7 is a diagram showing an input image and feature points automatically extracted by mistake.
It is a figure showing the brightness obtained from it.

FIG. 8 is a block diagram showing the configuration of an embodiment of the invention according to claim 5 of the present invention.

[Explanation of symbols]

1 ... Image power department 2 ... Image storage unit 3 ... Coordinate input section 4 ... Standard structure model section 5 ... Structural model matching section 6 ... Brightness calculator 7 ... Standard data section 8 ... Judgment unit 9 ... Brightness correction unit

Continuation of the front page (56) Reference JP-A-3-158997 (JP, A) JP-A-4-306782 (JP, A) JP-A-6-259532 (JP, A) Takaaki Akimoto 1 person, "front" Generation of 3D Head Model Based on Image and Side View ", NTT R & D, The Telecommunications Society of Japan, April 10, 1993, Vol. 42, No. 4, p. 455-464 Hiroshi Hasegawa, 1 person, "Study on face image matching using pattern matching", Technical Report of The Television Society of Japan, Television Society of Japan, July 23, 1992, Volume 16, No. 47. , P. 49-56 (58) Fields investigated (Int.Cl. ⁷ , DB name) G06T 1/00 G06B 11/00 G06T 11/80 CSDB (Japan Patent Office)

Claims (5)

(57) [Claims] 1. An image input unit for inputting an image, an image storage unit for storing the input image, a coordinate input unit for inputting coordinates of a specific part in the image, and a reference structure of the input image. A reference structure model part described in advance as a set of patches, a structure model matching part that deforms and matches the reference structure model based on the coordinates input by the coordinate input part, and each patch of the matched structure model On the other hand, a brightness calculation unit that calculates brightness from the image in the image storage unit, a reference data unit that records a standard brightness distribution of an image to be input, and each patch obtained by the brightness calculation unit. The lightness and the lightness of each patch of the reference data section are compared and calculated, and a determination section for determining whether or not the coordinates input from the coordinate input section are correctly input is provided. Input to Coordinate determination device. 2. The input coordinate determination device according to claim 1, wherein the reference data section records a plurality of standard brightness distribution data for each light source direction for the image to be input. 3. The judgment unit calculates the difference between the brightness difference value between the patches obtained by the brightness calculation unit and the brightness difference value between the patches of the reference data unit. The input coordinate determination device according to claim 1, wherein: 4. The judgment unit calculates a brightness difference value between patches obtained by the brightness calculation unit and a brightness difference value between patches of a plurality of data in the reference data unit. The input coordinate determination device according to claim 2, wherein 5. The reference structure model section has the information in a three-dimensional manner, and the lightness correction section uses the lightness of each patch obtained from the lightness calculation section and the three-dimensional information for each patch. The input coordinate determination device according to claim 1 or 3, wherein the average brightness is corrected.