JP3272584B2 - Region extraction device and direction detection device using the same - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an area extracting apparatus for detecting a face position, a face outline, a face part, and the like from an input image, and a head direction and a line of sight using the apparatus. The present invention relates to a direction detecting device that detects a direction.

[0002]

2. Description of the Related Art A great deal of research has been carried out on image processing relating to faces for a very long time, which has been around 20 years. Image processing related to a face covers a very wide range, such as personal identification by extracting feature points of a face, extraction of face parts by template matching, data compression using color information, application of animation such as morphing, and application of phase information. Among them, the problem of extracting a face part such as eyes, nose, and mouth from a face image in the input image, and the problem of extracting facial parts such as eyes, nose, and mouth are the most basic and very wide-ranging useful technologies. It is. This is the first key point of the face image processing, and various methods have been conventionally proposed.

[0003] As a method of detecting a face position from an input image, there are a method of performing template matching with a grayscale image by lowering the resolution of the image, and a method of using color information and segmentation. Any of these methods is susceptible to rotation in the optical axis direction, noise, and the like, and is likely to cause erroneous extraction.

[0004] For extraction of facial parts, the method using projection is the simplest and fastest algorithm in terms of the algorithm. However, this method involves variations in the intensity and direction of illumination, the position, size and rotation of the face in the image. It has been found that the method is very effective for images captured in an environment where the distance to the subject of illumination and the image quality are considerably restricted. In addition, a method for extracting face parts using template matching has also been proposed. However, similar to the method using projection, shadows may be generated due to variations in illumination, etc.
It can be said that it is extremely vulnerable to changes in the face direction and rotation and variations due to individual differences. Hereinafter, the projection and the template matching will be described in detail.

[0005] Projection is the conversion of an image into an image viewed from a certain direction. The direction in this case is different from the three-dimensional direction. For example, assume that there is an input face image shown in FIG. 8 used for describing the present embodiment. When this is differentiated in the horizontal direction and binarized, the result is as shown in FIG.
Differentiating in the horizontal direction means that, as shown in FIG. 48, the brightness of a pixel is viewed in order in the horizontal direction, and the difference between the brightness of a certain pixel and the brightness of the previous pixel is defined as the brightness of a new pixel. It is. Therefore, when differentiating an image in which the brightness is rapidly changing, the differential value is large where the brightness is changing, so that the edge can be extracted.

When this binarized horizontal differential image is projected in the horizontal direction, it becomes as shown in FIG. 47 (b), and when it is projected in the vertical direction, it becomes as shown in FIG. 47 (c). At this time, projecting in the horizontal direction means that pixels of the binarized differential image are viewed in the horizontal direction, and pixels that are ON (FIG. 47)
(A), the number of black pixels) is counted and graphed. In the vertical projection, the pixels of the binarized differential image are viewed vertically, the number of ON pixels is counted, and the result is graphed. By examining the projection image, a face region and a face part region can be extracted. For example, by examining the peak of the projected image in the horizontal direction in FIG. 47B, the vertical position of the eye can be determined. Also, by examining the left and right peaks of the projected image in the vertical direction in FIG. 47C, the left and right edges of the face area can be examined.

[0007] However, this method has a problem that when the illumination condition is changed, the way the shadow is formed changes, so that the differential image changes greatly and the projected image changes due to noise, background, face rotation, and the like. Therefore, it can be understood that the face technique cannot extract a face region or a face part unless the image is captured under very limited conditions. That is, the projection method is not effective unless the image is captured in an environment in which the lighting, the size of the image, the face position and size in the image, the face direction, and the like are strictly controlled.

Next, template matching will be described. The template is one in which an image of a region to be extracted is stored in advance, such as an image of an eye shown in FIG. The template matching is to compare a template with a corresponding region of an input image, and extract a region having the highest score as a region matching the template.

[0009] The template is an image of an appropriate size smaller than the input image. As shown in FIG. 50, assuming that the size of the template is 5 × 5, first, an upper left 5 × 5 region of the input image is compared with the template to give an evaluation value. The comparison method uses, for example, the difference between the sum of squares of the brightness of the corresponding pixels between the template and the area of the input image corresponding to the template as a score. This results in the following equation.

[0010]

(Equation 1) Here, V (p) is the brightness of a certain pixel P, T _ij is the pixel at the coordinates (i, j) of the template, and I _ij is the pixel at the coordinates (i, j) of the area corresponding to the template.

[0011] As shown in FIG. 51, 5 × in the upper left of the input image
After calculating the evaluation values for the five regions, the evaluation values are calculated in the same manner for the 5 × 5 region shifted by one pixel in the horizontal direction. Similarly, the evaluation value is calculated between all the 5 × 5 regions of the input image and the template up to the lower right 5 × 5 region. Then, the one with the highest score among all the evaluation values is extracted as the region most similar to the template.

However, this method is used when the region to be extracted is shifted from the template, for example, when the region is larger or smaller than the template, or when a region similar to the template is formed due to noise or background. Prone to incorrect extraction. As with the method using projection, this method is not effective unless the illumination and the size of the image, the position, size, and orientation of the face in the image are strictly controlled.

The disadvantages of the above-described projection and template matching are illumination and image size, face position and size in the image,
As a prerequisite to alleviate the prerequisite of images taken in an environment where the orientation is strictly controlled, De
Extraction of facial parts using a method called formable template has been proposed.

As shown in FIG. 52, a template (dotted line in the drawing) expressing the feature of the region to be extracted by a combination of simple mathematical graphic functions and an image corresponding to the template The energy is calculated between them, and the area is extracted by deforming and moving the template so that the energy is minimized.

The template is set, for example, as follows. As shown in FIG. 53, a circle having a center x _c and a radius r corresponds to a pupil. The energy is defined so that the circumference is drawn to the edges of the pupil and the white eye, while the inside of the circle is drawn to the area where the brightness is reduced.

The boundary of the eye is formed by combining two vertically convex parabolas having a focal point of x _e . Focus x _e
The width from b to one end of the eye is b ', that is, the width of the eye is 2
b ′, the height of the upwardly convex parabola is a ′, the height of the downwardly convex parabola is c ′, the horizontal line passing through the focal point x _e and the center line of the eye (the dotted line passing through the focal point x _{e in the} figure) Is the rotation angle θ. Energy defines the eye boundaries to be drawn to the edges.

The centers of the white-eye regions divided into two by the pupil are x _e + p ₁ (cos θ, sin θ) and x _e + p
₂ (cos θ, sin θ) (p ₁ ≧ 0, p ₂ ≦
0). Energy is defined so that the centers of the two white-eye regions are drawn to the center of the brightest region among the neighboring pixels.

The energy between the boundary of the eye and the pupil corresponds to the white eye, and the energy is defined so as to be drawn to a region having a high brightness.

These elements include a center x _c and a focus x _e
Attracts each other and minimizes energy when they match each other, the eye width 2b 'is almost four times the radius r, and the two center points of the white-eye region act on the center line of the eye. I do.

By changing the above parameters little by little, the template is reduced so that the energy function E _c (x _c , x _e , p ₁ , p ₂ , r, a ′, b ′, c ′, θ) is minimized. Matches eyes by deforming. Energy function E _c is expressed by the following equation.

[0021]

(Equation 2) Where E _v is the valley energy, E _e is the edge energy, E _i is the image energy, _Ep is the peak energy,
E _int is an internal energy and is defined as follows.

[0022]

(Equation 3)

[0023]

(Equation 4)

[0024]

(Equation 5)

[0025]

(Equation 6)

[0026]

(Equation 7)

[0027]

(Equation 8) The above equations are defined such that each energy is minimized when the geometric configuration is optimized. This method is characterized in that it is hardly affected by the size of the image, rotation parallel to the image plane, rotation of the head, and illumination.

In addition to the detection of the face area and the face parts as described above, a technique for detecting the head direction and the gaze direction has also been disclosed.

As a method of detecting the head direction, there is a method using a magnetic sensor or the like. This is because, as shown in FIG. 54, a user wears an electromagnetic wave radiating device that emits three or more electromagnetic waves on his / her head and mounts an electromagnetic wave receiver that receives electromagnetic waves from the electromagnetic wave radiating device on a floor or the like. It is configured to be installed in more than one place. According to this, the three-dimensional position of each electromagnetic wave radiation device is detected by using the phase difference of signals received by three or more electromagnetic wave receivers for one electromagnetic wave radiation device, and the electromagnetic wave radiation device is attached. The direction of the head can be detected.

There is also a method of detecting the position of the head without using dedicated hardware. This is because, as shown in FIG. 55, the user puts three or more conspicuous color stickers or the like at appropriate positions on the head or wears headgear or the like on which the stickers are stuck. It is a structure provided with. According to this, the three-dimensional position of the seal can be measured from two or more images of the seal, and the direction of the head can be detected.

In order to detect the gaze, a wearable gaze tracker is usually used. In the eye-gaze tracker, an imaging device such as a CCD is arranged immediately near the eyeball in order to constantly capture the eyeball and capture an image of the eyeball.

[0032]

However, the above De
The formable template method has a problem that the initial position of the template representing the part to be extracted must be set using another method. This is because if the template is not located near the region to be extracted as the initial position, erroneous extraction increases and the amount of calculation becomes enormous. Other methods include the above-mentioned projection and template matching.However, there is a major inconsistency that a method for overcoming the drawbacks of the conventional method such as template matching must use such a conventional technique as a premise. Has become.

As a result, the same problem as that of the projection or template matching method occurs in the extraction of the face outline / face part by the Deformable template method. That is, De
The formable template method uses the extraction accuracy, lighting conditions, image size, face position and size in the image, rotation,
It is vulnerable to changes in the background and the like, and is effective only for images taken in a strictly controlled environment.
In other words, when photographing a person in a normal environment, it is unlikely that the same lighting will always be used and the background, direction, and orientation will always be the same. Applying this technique may cause the extraction of an incorrect region. Another problem is that energy must be defined for each component.

The method of detecting the head direction using an electromagnetic wave radiating device requires that the electromagnetic wave radiating device be firmly fixed so as not to move to the head, and that a mechanism for supplying power thereto is also required. Become. Further, the electromagnetic wave receiver must be fixedly placed on a floor or the like which is at least a certain distance from the electromagnetic wave radiation device. As a result, there are drawbacks in that it is troublesome to attach hardware to the head and the cost is high. On the other hand, a method of attaching a marker such as a seal to the head can be realized at low cost, but has a problem that it is not possible to eliminate the troublesomeness of attaching a seal or wearing a headgear-shaped object.

Furthermore, the method of detecting the direction of the line of sight has a problem that the user has to wear a line of sight tracker having a rigid structure that is fixed to the face and does not move. Have.

[0036]

In order to achieve the above object, an area extracting apparatus according to the first aspect of the present invention is an image pickup means (for example, an image pickup apparatus) which picks up an object and converts it into a color digital image. A probability density function storage unit (for example, a probability density function storage unit) that stores in advance a probability density function indicating a desired region having a similar color uniformly in the color digital image; A region probability calculation means (for example, a region probability calculation device) for calculating the probability of the color of each pixel by calculating the probability density function based on the color of the pixel; Three or more points (eg, grid points)
And a network information storage unit (for example, a face area extraction network information storage unit) configured to store virtual network information provided by covering the entire image and connected to the point. The internal energy of the net defined based on
Energy calculating means (for example, an energy calculating device) for calculating the image energy of the net defined based on the likelihood of the color of the pixel where the point is located, and
The total value of the internal energy of the net and the image energy when moving one point is compared with the total value before moving the point, and when the total value changes, the point moves in the direction in which the net contracts. And the movement of the point is stopped when the total value no longer changes.

According to the above arrangement, first, the object is photographed by the imaging means, and a color digital image of the object is obtained. Then, the region probability calculation unit refers to the probability density function storage unit, and substitutes the color value of each pixel constituting the region to be extracted in the color digital image into the probability density function for calculation, thereby calculating the color of each pixel. Probability is calculated. At this time, the desired area is uniformly formed with the same color, and the probability density function is also a function indicating that the area is the desired area. Therefore, the likelihood of the color of the desired area increases.

Next, the points constituting the virtual net provided to cover the entire input color digital image are moved one by one, and the energy of the net changes before and after the point is moved. Check if there is. Here, the energy of the net is the total value of the internal energy that reflects the shape of the net and the image energy that reflects the color of the pixel at which the points forming the net are located, which is obtained by the energy calculating means.

When there is a change in the total value of the energy, the point is moved in the direction in which the net contracts, and when there is no change in the total value, the movement of the point is stopped. That is, the net that initially covers the entire image contracts and deforms toward the desired area where the likelihood of color is high, and eventually stops around the predetermined area. Therefore, a desired area can be extracted.

Thus, the present region extraction apparatus can extract a desired region based on the likelihood of a color that is resistant to changes in lighting conditions, and is thus affected by the location, time, presence or absence of auxiliary lighting, direction, etc. No more. Further, since the net can have any shape, the size of the input image and the position and size of the face in the image are not limited. As a result, the degree of freedom of photographing can be increased.

According to a second aspect of the present invention, there is provided an area extracting apparatus.
In addition to the configuration described above, the object is a person, and the desired area is an area of a face of the person.

According to the above arrangement, first, a person image is obtained by photographing a person by the image pickup means. If a virtual mesh is used for this image, the mesh shrinks to the face region and the face region can be extracted.

Thus, it can be used for various appliances such as home appliances. That is, by extracting the face region by the region extraction device provided in the device, it is possible to determine what positional relationship the device and the user have. And whether there is a user in front of the device,
Also, if it exists, information about where it is can be obtained. Using such information, it is possible to control the device based on the position of the user.

According to the third aspect of the present invention, there is provided an area extracting apparatus.
Or, in addition to the configuration described in 2, the probability density function indicating a partial area having a color different from the desired area is stored in advance in the probability density function storage means, and the desired area is surrounded by the network information storage means. Information of the stopped network is stored as an initial value of the partial area extraction network for extracting the partial area,
When there is one partial area in the desired area, the area probability calculation means calculates the probability density function based on the color of each pixel constituting the partial area, thereby calculating the likelihood of the color of each pixel. Extracting the partial region by calculating the internal energy and image energy of the partial region extraction network and the moving energy defined so as to converge on the partial region. I have.

According to the above construction, the information of the network stopped surrounding the predetermined area (face area) in claim 1 or 2 is stored in the network information storage means as an initial value of the partial area extracting network for extracting the partial area. Is done. Further, a probability density function indicating a partial region having a color different from the desired region is stored in the probability density function storage means in advance. Then, when there is one partial region in the desired region, the likelihood of the color of each pixel in the partial region is determined, and then a net having a moving energy defined to be converged on the partial region is moved. Extract a partial region. That is, the net initially covering the predetermined area is
Shrinking deformation toward partial area where color certainty is large
It will move and eventually stop around the partial area.

As a result, the net used for the first extraction area can be used for extracting another area in the area, so that it is not necessary to change the setting of the net for each area. Also,
According to the second aspect, when the face region is extracted, a lip region or an eye region having a different color from the face region can be extracted.

According to the fourth aspect of the present invention, there is provided an area extracting apparatus.
In addition to the configuration described above, a barycenter calculating means (for example, a center of gravity of a region surrounded by the partial region extraction network with the brightness of the image as a weight, as well as a frame centroid from a point forming the outer periphery of the partial region extraction network) , A center of gravity calculation device), and the moving energy is defined so that the frame center of gravity is drawn in the direction of the center of gravity of the region.

According to the above configuration, the brightness of the image is weighted by the center of gravity of the area surrounded by the partial area extraction, so that the position of the area center of gravity is closer to the area to be extracted than the position of the frame center of gravity. Become. At this time, since the moving energy is set so that the frame center of gravity is drawn in the direction of the area center of gravity, the net moves toward the partial area. Thereby, even if there is an area having a color similar to the partial area in the predetermined area, the partial area can be extracted without being drawn to the area. As a result,
The processing speed is high, and it is possible to extract a partial region with high accuracy.

According to the fifth aspect of the present invention, there is provided an area extracting apparatus.
In addition to the configuration described above, a center of gravity calculating means for obtaining a frame center of gravity from a point forming the outer periphery of the network, and obtaining a center of gravity of an area surrounded by the network using the brightness of the image as a weight, the probability density function storage means The probability density function indicating the lip region is stored in advance, and the information of the net stopped surrounding the face region is stored in the net information storage means as initial values of the net for extracting the lip region and the left and right eye regions. The region probability calculation means calculates the probability of each pixel by calculating the probability density function based on the color of each pixel constituting the lip region, and configures the left and right eye regions. The likelihood of the color of each pixel is calculated by calculating the probability density function of the face area or the lip area based on the color of each pixel, and the energy calculation means calculates the internal energy of each area and the image energy. And the center of gravity of each frame is drawn in the direction of the center of gravity of each region, and the positional relationship of the centers of gravity of the three regions is defined to be equal to the previously measured and stored positional relationship of the lips and left and right eyes. The lip and the left and right eye regions are extracted by calculating the balance energy.

According to the above configuration, the information of the net stopped surrounding the face region in claim 2 is stored in the net information storage means as the initial value of the net for extracting the lip region and the left and right eye regions. Further, the probability density function indicating the likelihood of the lip region is stored in the probability density function storage means in advance. Then, after obtaining the certainty of the color of each pixel in the lip area and the certainty of the color of each pixel in the left and right eye areas, by moving three nets having internal energy, image energy, and balance energy, The lips and the left and right eye regions are extracted. That is, the net that initially covers the face area contracts and deforms and moves toward three areas having high color certainty, and finally stops around each area.

Thus, the face area extracting net can be the lip area extracting net and the left and right eye area extracting nets, so that it is not necessary to change the setting of the net for each area.

Further, since the position of the face part with respect to the face area is known, the face rotation angle (face inclination) can be recognized. If the angle of rotation of the face is known, for example, when creating a person database, when a photograph is read by a scanner or the like, it is possible to input without worrying about the direction of the photograph and the like, and labor is greatly reduced. As a result, the degree of freedom during photographing and processing increases.

Further, the approximate direction of the face can be estimated based on the balance between the face area and the lip area. That is, if the center of the lip region is located at a position close to the main axis of the face region, the user is facing the direction of the imaging unit, and if the center is shifted to the left or right, the user is facing each direction. .

According to a sixth aspect of the present invention, there is provided a direction detecting device.
A plurality of the above-mentioned image pickup means are provided using the region extraction device described above, and a lip area and a left and right eye area are respectively extracted from a plurality of images obtained simultaneously from the plurality of image pickup means, and the lip area and the eye area are compared with each other. Is derived.

According to the above arrangement, when the same person is photographed simultaneously by a plurality of image pickup means, a plurality of images having different positions of the person on the screen are obtained. Then, the lip region and the eye region in the plurality of images are respectively extracted by the region extraction device having the configuration described in claim 5. When the corresponding face parts are compared with each other in the plurality of images, three points in the three-dimensional space of the lips, the left eye, and the right eye are determined by the shift between the images. This makes it possible to detect in which direction the head is facing.

In this way, the head direction can be easily detected without using dedicated hardware as in the related art. As a result, it is possible to eliminate the trouble of mounting hardware and the like, and to reduce the cost of the apparatus.

According to a seventh aspect of the present invention, there is provided a direction detecting device.
In addition to the configuration described above, the estimated eye direction is calculated by comparing the extracted eye image with an eye image that has been measured and stored with the eye direction information added thereto in advance, and the estimated eye direction is further calculated. Then, the true gaze direction is detected by comparing the detected head direction.

According to the above configuration, since the relationship between the position of the iris and the line of sight is known to some extent by the pre-measurement, the estimated line of sight is obtained by using the eye image extracted by the structure of claim 6. Can be calculated. When the estimated gaze direction is obtained, the estimated gaze direction is further compared with the direction of the head to perform correction, and the true gaze direction can be detected.

Thus, the gaze direction can be easily detected without using dedicated hardware as in the related art. As a result, it is possible to eliminate the trouble of mounting hardware and the like, and to reduce the cost of the apparatus.

[0060]

BEST MODE FOR CARRYING OUT THE INVENTION

[Embodiment 1] An embodiment of the present invention will be described with reference to FIGS. 1 to 9 and FIGS. 11 to 19.
It is as follows.

The area extracting apparatus according to the present embodiment is an apparatus for extracting a face area by using a virtual net model formed by connecting arbitrary points (pixels) on a color digital image. is there.

As shown in FIG.
An input device 1 which is an imaging device (imaging means) such as a CCD which images a target object such as a person and converts it into a color digital image;
An arithmetic unit 2 performs various calculation processes, a storage device 3 that stores data in advance or temporarily stores data, and an output device 4 that outputs a result when the network converges on a target area.

The arithmetic unit 2 has a region probability calculation device (region probability calculation means) 5 and an energy calculation device (energy calculation means) 6. The region probability calculating device 5 derives a probability density function of a region as color information by referring to a probability density function storage unit 7 described below as a pre-stage for extracting a target region in the input image, and This is an apparatus for generating a region probability image. The energy calculation device 6 calculates the internal energy (E _int ) of the network by referring to a face area extraction network information storage unit 9 described later, and also stores the face area extraction network information storage unit 9 and the area probability image storage unit. 8 to calculate the image energy (E _image ).

The storage device 3 has a probability density function storage unit (probability density function storage unit) 7, an area probability image storage unit 8, and a face area extraction network information storage unit (network information storage unit) 9.

The probability density function storage unit 7 stores the probability density function of the skin color of the face area obtained by sampling in advance. The region probability image storage unit 8 stores the region probability image generated by the region probability calculation device 5.

The face area extraction network information storage 9 stores information on a virtual network formed by connecting a plurality of points on the input image and provided so as to cover the entire image. That is, the initial coordinates of the grid points that make up the virtual network are stored in advance. The initial coordinates are automatically calculated and set so as to be evenly positioned with respect to the input image. Further, theoretically, an area can be formed if there are three grid points, but the number of grid points is set according to the accuracy of extraction, the shape to be extracted, and the size of the image. The face area extracting mesh information storage unit 9 also stores the coordinates of the lattice points that have been changed by the movement of the mesh.
Further, the internal energy (E) calculated and obtained by the energy calculator 6 is stored in the face area extracting net information storage unit 9.
_int ) and the image energy (E _image ) are stored.

Hereinafter, the operation of the region extracting apparatus having the above configuration, the method of defining a network, and the like will be described in detail.

First, in order to calculate a probability density function representing a face region, the color distribution of the skin of a human face is examined. The steps of this operation will be described based on the flowchart of FIG. More images of the human face are taken to obtain statistics, and an image of only the skin of the face is manually created (S1). For example,
From the input person image shown in FIG. 3A, only the skin color portion of the face is extracted as shown in A of FIG. 3B. An appropriate number of pixels (pixels) are randomly selected from a digital image obtained by cutting out only the face region. The color of the pixel is HSV
The number of appearances of H (hue) and S (saturation) in a color system is counted, and their frequency distribution is obtained (S2).

Hue is an attribute for distinguishing between colors such as red, blue, and yellow, and is characterized by being hardly affected by reflection or shading due to illumination or the like. Saturation is an index indicating the vividness of a color. Human faces are often relatively saturated, while indoors and the like are often constructed with relatively low saturation.

FIG. 4 shows the image data of 24 persons.
The chromaticity distribution of the hue of the face area when sampling the points of 1000 pixels per image is shown. The vertical axis in the figure is the probability density (shown in the range of 0 to 1) indicating the probability of being a face area. The horizontal axis is hue, and red is 0 °.
The direction of yellow and green is taken to be up to + 180 ° as positive, and the direction of magenta and blue is taken up to -180 ° as being negative. FIG. 5 shows the chromaticity distribution of the saturation of the face area when similarly sampling. The vertical axis of the figure is the probability density (represented in the range of 0 to 1) indicating the probability of being a face area, and the horizontal axis is the saturation when the achromatic color is 0 and the single color is 100.

When the chromaticity distribution of hue and saturation in the face area is obtained, a two-dimensional normal probability density function representing the face area is derived assuming that the variation between hue and saturation follows a normal distribution ( S3). That is, μ ₁ is the average value of hue, μ
Assuming that ₂ is an average value of saturation and σ _ij is a variance-covariance matrix of a distribution of hue and saturation, first, these values are calculated from a distribution of colors cut out and counted manually.

The two-dimensional normal probability density function is

[0073]

(Equation 9) Therefore, the obtained μ ₁ , μ ₂ , and σ _ij are represented by (1)
By substituting into the expression, a probability density function representing the face region can be derived. This probability density function is stored in the probability density function storage unit 7 in advance. FIG. 6 shows a plot of the probability density function obtained based on the chromaticity distributions of FIG. 4 and FIG. In the figure, the height axis represents the probability density, the left axis represents the hue, and the right axis represents the saturation.

[0074] a region probability calculation unit 5, a probability density function obtained, hue values x _1, by calculating by substituting saturation to x _2, for all pixels of the input digital image, the face color Is determined in the range of 0 to 1. Then, an image is generated in which the likelihood value (0 to 1) obtained from the hue and saturation of the pixel is set as a new value of the pixel. The image obtained in this manner is referred to as a region probability image.

The process of generating a region probability image will be described with reference to the flowchart of FIG. When an image is input (S
11) Pixels are extracted one by one sequentially from the upper left of the input image (S12). The hue and saturation of the extracted pixel are obtained (S13), and the certainty of the area is calculated using the hue and saturation as arguments of the probability density function obtained earlier. The value of the certainty is set as a new value of the pixel (S14). Then, it is determined whether new values have been obtained for all the pixels (S15). If the calculation has not been performed for all the pixels yet, the above S12 to S12 are executed.
Step 14 is repeated, and when the calculation has been performed for all the pixels, the process ends. Thereby, a region probability image is generated. FIG. 9 shows a face area probability image generated from the input image shown in FIG. This indicates that a brighter pixel is more likely to be a face area.

Next, a network model will be described. The net is
As shown in FIG. 11, it is formed by connecting black dots corresponding to pixels on a digital image. The black dots that form this net are called grid points. The energy of the net is defined by the positional relationship between the grid points. The energy of the net has two energies: internal energy generated from the shape of the net itself, and image energy determined by the net and the image it covers. The target area is extracted by moving the grid points and deforming the net so that the total value of these two energies becomes smaller.

A certain grid point has a connection relation with another adjacent grid point, and the internal energy of the network is defined by the positional relation between these points. The internal energy of the net is
The smaller the distance between the grid points, the smaller the energy, and the smoother the lines that make up the net connecting the grid points, the smaller the energy,
Opposite sides of a grid formed by connecting grid points are defined to be parallel. The image energy is defined by the brightness of the pixel at the grid point. That is, the region is extracted by reflecting the state of the pixel on the grid point in the energy.

The region extraction is performed by applying the above-described network model to the region probability image generated by the region probability calculation device 5. FIG. 12 (a)
(D) shows how to extract a certain area using a network model. An area probability image is calculated from the input image shown in FIG. 12A, and a net covering the entire image is covered as shown in FIG. 12B. Then, the net shrinks toward the target area (see FIG. 3C), and finally stops surrounding the area (see FIG. 4D). When a face region is extracted, the result is as shown in FIGS. That is, the face area probability image is first covered with a net covering the entire image. Then, the grid points are moved so that the energy of the net becomes small, and the movement of the grid points is stopped when there is no change in the energy. Then, the net is deformed to surround the face area.

(1) Shape of Net The shape of the net is defined as shown in FIG. The radius is gradually increased around a certain point to create a plurality of concentric circles. Then, a line segment is extended radially from the center point to the outermost concentric circle. The intersection of the circumference of the concentric circle and the radially extending line segment is defined as a lattice point. Therefore, the number of lattice points is T × S + 1, where T is the number of concentric circles and S is the number of radial segments. Here, layer the concentric circles,
The radial segments will be called spokes. The outermost layer is referred to as layer 0, and the layers are referred to as layer 1, layer 2,. The intersection between the outermost layer 0 and each spoke is called an outermost grid point, and the other grid points are called inner grid points. Note that, although the shape of the net is circular, as shown in FIG. 15, even if the net is changed to a rectangle, the connection relation between the grid points does not change and the grid can be handled in the same manner.

(2) Internal Energy of Network A grid point formed by a certain layer t and a spoke s is represented by p (t, s). A layer inside one layer t is defined as a layer (t + 1), and a layer outside one layer is defined as a layer (t-1). Of course, the layer (t-1) does not exist at the outermost lattice point. Similarly, the next spoke in the clockwise direction from a certain spoke s is referred to as a spoke (s + 1), and the next spoke in the counterclockwise direction is referred to as a spoke (s-1).

The internal energy E of the lattice point p (t, s)
_int (t, s) is expressed as follows. Note that p in the text and FIG. 16 represents a vector.

[0082]

(Equation 10)

[0083]

[Equation 11]

[0084]

(Equation 12)

[0085]

(Equation 13)

[0086]

[Equation 14]

[0087]

(Equation 15) The above _Et , as shown in FIG. 16 (a), represents the distance between adjacent grid points p (t, s) and p (t + 1, s) on the same spoke s. As the distance is small E _t becomes smaller, p (t,
If s) and p (t + 1, s) match, then E _t = 0.

E _s represents the distance between adjacent grid points p (t, s) and p (t, s + 1) on the same layer t. As the distance is smaller E _s becomes small, and p (t, s) and p (t, s + 1) when the match E _s = 0.

E _tt is, as shown in FIG.
(t, s) and whether or not grid points p (t−1, s) and p (t + 1, s) before and after p (t, s) on the same spoke s are connected smoothly. If three grid points are straight lines, E _tt
= 0.

E _ss is such that p (t, s) and p (t, s−1) and p (t, s + 1) on the left and right of p (t, s) on the same layer t are smooth. Indicates whether it is connected. If the three grid points are straight lines, E _ss = 0.

_Ets is p (t, s−1) −p (t, s) and p (t−
1, s-1) -p and (t-1, s) are the same vector, the E _ts = 0 when ie parallel magnitude to each other are the same.

Attention is paid to a certain grid point p (t, s), and E _int (t, s) obtained above is calculated from the positions of the surrounding grid points. This is the internal energy of the lattice point.
When this is calculated for all grid points and the sum is obtained, the internal energy E _{int of the} net is calculated. In addition,
For each term constituting E _int (t, s) in equation (2), the weight may be appropriately determined depending on the ratio between the size of the image and the number of grid points and the range of certainty in color. .

(3) Network image energy Image energy E _image (t, s) of a certain grid point p (t, s)
Is defined as the state of a pixel on the grid point as follows.

[0094]

(Equation 16) Here, FRPI (t, s) is the lattice point p in the region probability image.
(t, s) indicates the brightness of the pixel on (the certainty of the area). That is, at the internal grid point (t> 1), the brightness of the pixel on the grid point p (t, s) in the region probability image is given a minus sign, and at the outermost grid point (t = 0), the sign is given. The image energy of the grid point p (t, s) is used as it is.

Attention is paid to a certain grid point p (t, s), and E _image (t, s) obtained above is calculated. This is the image energy of the grid point. When this is calculated for all grid points and the sum is obtained, the image energy E
_image is calculated. Note that E _{image in} equation (3)
The weights of the terms constituting (t, s) may be appropriately determined as in the case of the equation (2).

The energy E _{net of an} arbitrary lattice point p (t, s)
(t, s) is represented by the sum of the internal energy E _int (t, s) of the grid point and the image energy E _image (t, s).

[0097]

[Equation 17] The energy E _net of the entire network is obtained by calculating Expression (4) for all grid points and summing them, and is expressed as follows.

[0098]

(Equation 18) In this way, the energy E of the entire network defined is_netTo
Calculate the case so that the magnitude of this energy is small.
Move the child point. When there is no change in energy
When the movement of the grid points is stopped with, the mesh converges around the face area
(See FIG. 13).

(4) Moving Grid Points Next, how to move grid points will be described with reference to the flowchart of FIG.

First, as described above, the region probability calculation device 5
To generate a face area probability image (S21). Next, an initial network is set. This setting is performed so as to cover the entire input image as shown in FIG. 13 (S22). First, one grid point is selected from the outermost grid points, and the energy E _net (t, s) of the grid point is calculated (S23).
Then, as shown in FIG. 18, eight pixels (hereinafter, referred to as pixels) arranged around the pixel where the grid point is located
(Referred to as 8 neighborhoods). At this time, the energy when the grid point is moved to each of the vicinity of 8 is calculated (S24), and the grid point is moved to the pixel having the lowest energy among them (S2).
5). When the energy at the position of the first grid point is the smallest, there is no need to move the grid point.

The grid points are moved one by one from the outermost grid point to the inner grid point, and it is checked where the energy is minimized for all the grid points in the vicinity of 8, and the grid points are moved there. . It is checked whether or not the movement processing for moving all pixels by one pixel (hereinafter, referred to as one-unit deformation) has been completed (S26). If the processing has not been completed, the process returns to the step S23 and has been completed. In this case, the process proceeds to step S27.

In S27, it is checked whether or not the energy E _{net of the} entire network obtained by summing the energies at the respective grid points is changed in comparison with the energy E _net before moving the grid points. If there is a change in the energy E _net of the entire network in S27, the process returns to the step S23 and the processes of S23 to S26 are performed again. If there is no change in the energy E _net of the whole network in S27, the movement of the grid point is performed. To end. When the movement of the grid points is completed, the net surrounds the entire face area (see FIG. 13).

As described above, the region extracting apparatus according to the present embodiment changes the total value of the internal energy of the net based on the positional relationship between the grid points and the image energy based on the likelihood of the color of the pixel where the grid point is located. In this configuration, the mesh is shrunk and deformed until it stops, and the movement of the grid points is stopped when the total value no longer changes.

Therefore, the net covering the entire input image is
It contracts toward the face area where the likelihood of color is large, and finally stops around the face area. As a result, a face region can be extracted.

At this time, since the likelihood of the color of the face area is determined based on the hue which is not influenced by the lighting conditions, the present area extracting apparatus does not always need to shoot under the same lighting conditions. That is, since the present region extraction apparatus is resistant to changes in lighting conditions, it is not necessary to worry about the location, time, presence / absence of auxiliary lighting, direction, etc. Conventionally, the same lighting conditions had to be always set by photographing a person from the same direction using the same lighting equipment at the same position in a room without a lighting window.

Further, since the area extracting apparatus of the present invention can take any shape of the net, it can handle any size of input image. Generally, in order to perform a region extraction process, it is necessary to handle an image digitally. The primary input image may be a digital image or a normal film photograph, but may be in any form. However, the image must be a digital image at the stage of performing the extraction process. Therefore, regardless of the form of the input image, conversion to a digital image is necessary as a pre-stage for performing the region extraction processing.

At this time, the fact that the image size does not depend on the size of the image means that, for example, whether a single digital image is 100 × 100 dots or 512 × 512 dots has nothing to do with it. If a technique such as the conventional template matching is used, the template must be set according to the size of the component to be extracted, so that the size of the image is naturally limited. Also, Defo
Even with the rmable template method, only parts corresponding to the set template can be extracted.

On the other hand, since the present region extracting apparatus does not depend on the size of the image, it can deal with an image of any size. Accordingly, an imaging device having any resolution may be used, and any rate may be used when sampling an analog image, so that the degree of freedom of photographing is increased.

Similarly, since the present region extraction apparatus does not depend on the face position and size in the image, if the face is projected, even if a person is projected on the entire screen, or a small image is projected on a corner of the screen. You can extract the face even if you are. As a result, a person can be photographed from a desired position with a desired composition, so that the degree of freedom in photographing is improved. In the conventional template matching method, as described above,
Since the template is set according to the region to be extracted, not only the size of the input image but also the size of the face in the image is limited. Also, in the method using projection, if the size of the face in the image is not known to some extent, the accuracy of component extraction is extremely reduced.

Further, since the face area is extracted, it can be used for information compression. For example, it is possible to compress information by reducing the resolution of an area other than the face area or reducing the number of colors, so that many human images can be stored with a small storage capacity. This is a great merit when creating a person database or the like. Also, when transferring images centered on a person over a line with a small transmission capacity per unit time, the amount of information is reduced by roughening information other than people, and a smooth image with many frames is sent over a narrow line. Will be able to send a lot of information. This is applicable to applications such as videoconferencing, videophone, electronic secretary, and navigation system that include explanations by persons.

The present region extracting apparatus can be used for controlling various devices such as home appliances. FIG. 19 is a basic block diagram in the case where the present region extraction device 11 is applied to control of the device 13. The imaging device 12 attached to the device 13 captures the user's figure and sends the digital image to the region extraction device 11. Region extraction device 11
Extracts the face area of the user from the image and sends the information to the control device 14. The control device 14 is connected to the device 13 and controls the device 13 based on information sent from the region extraction device 11. By repeating this series of operations in a short time, a system capable of controlling the device 13 in real time can be constructed.

As described above, the region extracting device 11 provided in the device 13 extracts the face region, and thereby the device 1
3 and the user can be determined in a positional relationship. Then, it is possible to obtain information as to whether or not the user exists in front of the device 13 and, if so, where the user is.

As a result, it becomes possible to control home electric appliances and the like in the direction in which the person is facing. For example, by attaching an area extraction device to an air conditioner,
Since it is possible to detect at which position in the room a person is and where it is not, it is possible to control such as sending a wind toward a person or sending a wind while avoiding a person. Further, it is also possible to detect the absence of a person and stop blowing.

Further, when the present invention is applied to a television or an audio product, the direction of the screen and the direction of the sound can be controlled in the direction of the user, and the volume and the balance between the left and right volumes in the case of stereo can be adjusted. It is also possible to switch off when no one is present and switch on when a person appears.

As described above, turning on / off the device when the user comes close to the device and controlling the device according to the position of the user can be applied to most electric appliances. is there. This can reduce the trouble of turning on and off the switch, forgetting to turn off the switch, and trouble of adjusting the device.

In an application such as an electric appliance that controls according to the presence or absence of a person,
Conventionally, humans turn on / off the switch one by one,
Infrared images were used to determine the presence / absence of a person. In the former case, however, the operation was cumbersome. In the latter case, an expensive infrared imaging device was used in order to increase the recognition accuracy. There is a problem that it has to be used and the cost is high. The use of the present region extraction device enables low-cost control of electrical products related to the presence / absence of people.

[Second Embodiment] Another embodiment of the present invention will be described with reference to FIGS. 3, 8 to 10, 20 to 31.
This will be described below. For convenience of explanation, the same members as those shown in the drawings of the above-described embodiment are denoted by the same reference numerals, and description thereof will be omitted.

The region extracting apparatus according to the present embodiment extracts a face region and a lip region from an input human image. Here, after extracting the face area, a lip area extraction network is set with the extracted face area as an initial state, and moved to extract the lip area.

As shown in FIG. 20, the above-described region extracting device has a configuration in which the arithmetic unit 2 has a centroid calculating device (centroid calculating means) 21 and the storage device 3 has a lip region extracting network in addition to the configuration of the first embodiment. An information storage unit (network information storage means) 22 is provided.

The center-of-gravity calculator 21 calculates the center of gravity (g) of the outermost frame obtained by averaging the coordinates of the outermost lattice points and the coordinates of the pixels inside the net using the brightness of the image in the lip region as a weight. This is a device for calculating the center of gravity (G) of the area surrounded by the net, which is obtained by the calculation.

The face area extraction net information storage unit 9 stores the two centroids of the face area extraction net obtained by the centroid calculator 21 and the internal energy (E) of the face area extraction net as in the first embodiment. _int ) and the image energy (E _image ) are stored. The lips area extraction net information storage unit 22 stores the coordinates of the net that has shrunk to the face area as an initial state. Further, the two centroids of the lip region extraction net, the internal energy (E _int ) and the image energy (E _image ) of the lip region extraction net, the shape energy (E _form ) described later, and the two centroids are obtained. And the moving energy (E _move ) to be stored. Further, the probability density function representing the lip region is stored in the probability density function storage unit 7 in addition to the probability density function representing the face region.

According to the above configuration, the region probability calculation device 5 refers to the region probability image storage unit 8 and calculates a probability density function based on the color of each pixel constituting the face region and the lip region, thereby obtaining each function. The likelihood of the color of the pixel is calculated, and a face region probability image and a lip region probability image are generated.
Next, as in the first embodiment, the energy calculation device 6 calculates the internal energy (E _int ) and the image energy (E _image ) of the face area extraction network, and stores them in the face area extraction network information storage unit 9. To be stored. The mesh is shrunk, deformed and moved so that the sum of these energies is reduced, and a face region is first extracted. This result is output to the output device 4.

The center-of-gravity calculating device 21 refers to the coordinates of the mesh stored in the face area extracting mesh information storage 9 to determine the center of gravity (g) of the outermost frame and the coordinates of the mesh and the area probability image storage. The center of gravity (G) of the area surrounding the face area extraction network is calculated with reference to the face area probability image stored in 8 and stored in the face area extraction network information storage unit 9. Similarly, the center-of-gravity calculation device 21 refers to the coordinates of the lip region extraction network stored in the lip region extraction network information storage unit 22, and calculates the center of gravity (g) of the outermost frame as the coordinates of the network. Referring to the lip area probability image stored in the area probability image storage unit 8, the center of gravity (G) of the area surrounding the lip area extraction network is calculated, and stored in the lip area extraction network information storage unit 22.

The energy calculator 6 calculates the internal energy (E _int ), the image energy (E _image ), the shape energy (E _form ), and the moving energy (E _move ) of the lip region extraction net, and calculates them. The information is stored in the lip area extraction net information storage unit 22. The lip region extraction net is deformed and contracted / moved so that the total value of the energy of these lip region extraction nets becomes small, and the lip region is extracted. The result is output to the output device 4.

Hereinafter, the operation of the region extracting apparatus having the above configuration, the method of defining a network, and the like will be described in detail.

First, in order to calculate a probability density function representing a lip region, for example, as shown in FIG.
As shown in FIG. 3B, only the lips are extracted from the input human image shown in FIG. 3A, and the color distribution of the lips is examined. FIG. 21 shows a chromaticity distribution of hues in the lip region when 100 pixel points are sampled per image from 24 person image data. FIG. 22 shows the chromaticity distribution of the saturation of the lip region when similarly sampled. Note that the ordinate and abscissa of each figure correspond to FIG. 4 of the first embodiment.
5 and FIG.

FIG. 23 shows a plot of the two-dimensional normal probability density function representing the lip region obtained in the same manner as in the first embodiment. In the figure, the height axis represents the probability density, the left axis represents the hue, and the right axis represents the saturation.

The region probability calculating device 5 calculates the probability density function by substituting the hue value and the saturation into the obtained probability density function, so that for all the pixels of the input digital image,
The likelihood of the color of the lips is determined in the range of 0 to 1. Then, a lip region probability image is generated in which the likelihood value (0 to 1) obtained from the hue and saturation of the pixel is set as a new value of the pixel. FIG. 10 is a lip region probability image generated from the input image shown in FIG. 8, and shows that brighter pixels are more likely to be lip regions.

Next, (1) the outermost frame, (2) the center of gravity of the outermost frame, and (3) the center of gravity of the area surrounded by the net are defined.

As shown in FIG. 24, the outermost frame is an outline formed by connecting outermost lattice points in order, and is layer 0. That is, it represents the outer shape of the net.

The center of gravity of the outermost frame is the average of the coordinates of the outermost lattice points. That is, assuming that the number of outermost lattice points is n and the vector representing the i-th outermost lattice point is pi, the vector g representing the center of gravity is expressed as follows.

[0132]

[Equation 19] As shown in FIG. 25, the center of gravity of the area surrounded by the mesh is obtained by averaging the coordinates of the pixels inside the mesh as the weight of the brightness. That is, assuming that the number of pixels in the network is N, the vector representing the pixel j in the network is pj, and the brightness at the pixel j is I (pj), the center of gravity G of the area surrounded by the network
Is defined as follows: Note that Σ is the sum of all pixels inside the net.

[0133]

(Equation 20) (1) Internal Energy of Net The definition of the internal energy of the net for extracting the lip region is as follows. The internal energy of the present embodiment is:
To the internal energy according to the first embodiment, energy is applied so that the net shrinks to the average shape and size of the lips estimated from the face width. This energy is used as the shape energy E
_We will call it _form (t, s).

Specifically, first, as shown in FIG.
And the center of gravity g _f outermost frame F _f of the face region extraction network, perpendicular to a line connecting the center of gravity G _m area A _m surrounding the lip region extraction network, draw a straight line through the center of gravity g _f. This is called a face width straight line. Two intersections F _a between the face width straight line and the outermost frame F _f
· The estimate F _ab face width a length of a line connecting the F _b. By multiplying the estimated value F _ab of the face width by a fixed ratio, the average size of the lips with respect to the face width, that is, the width and height of the lips is estimated.

As shown in FIG. 27, assuming that the shape of the lips is close to an ellipse, the center of gravity _Gm is the center, the estimated width of the lips is the major axis, the estimated height of the lips is the minor axis, and the major axis is consider an ellipse e _m having the same inclination as the face width linearly.

The shape energy E _form (t,
s) of when a straight line is drawn toward the center of the ellipse e _m (the center of gravity G _m) to one of the outermost grid point m ₁ of the lip region extraction network, the intersection of the straight line and the elliptic e _m , Is calculated as the distance from the outermost lattice point m ₁ . Also, the internal grid point m
Similarly the outermost grid point m ₁ for _2, the shape energy E _form (t, s) but is computed, E _form (t, s) when the inner grid point m ₂ are inside the ellipse e _m = Set to 0. The value obtained by adding the shape energy E _form (t, s) thus obtained to the equation (1) of the first embodiment is defined as the internal energy of the lip region extracting net.

(2) Image energy of net The image energy uses the brightness of the pixels of the lip region probability difference image. The lip region probability difference image is obtained by calculating the difference in brightness of the corresponding pixels between the lip region probability image generated by the region probability calculation device 5 and the face region probability image. If the pixel of the lip region probability image is brighter than the pixel of the face region probability image, the difference in the brightness is calculated, and if not, it is set to 0. That is, as shown in FIG. 28, the brightness of the pixel of the lip region probability image at the coordinates (x, y) is I _m (x, y), and the brightness of the pixel of the face region probability image is I _f (x, y). ), Assuming that the brightness of the pixel of the lip region probability difference image is I _mf (x, y).

[0138]

(Equation 21) In the actual image, when the difference between the face region probability image shown in FIG. 9 and the lip region probability image shown in FIG. 10 is obtained, a lip region probability difference image shown in FIG. 29 is obtained. The value obtained by multiplying the pixel brightness of the lip region probability difference image generated in this manner by −1 is defined as the energy at the internal grid point. The energy of the outermost lattice point is obtained by multiplying the pixel brightness of the face area probability image by -1. That is, the image energy at the lattice point p (t, s) of the lip region extraction net is as follows.

[0139]

(Equation 22) Here, MFDRPI (t, s) is the brightness at the lattice point p (t, s) in the lip region probability difference image, and FRPI (t, s) is the lattice point p (t, s) in the face region probability image. ).

(3) Moving Energy In order to move the lip region extracting net, the moving energy of the net is defined in addition to the internal energy of the net and the image energy. This is energy for the net for extracting the lip region to move the entire net toward the lip region in order to extract the lip region with high accuracy. This transfer energy E
_move is defined as follows.

[0141]

(Equation 23) Here, G _net is a vector representing the center of gravity of the area surrounded by the lip region extraction network, and g _net is a vector representing the center of gravity of the outermost frame of the lip region extraction network. In other words, E _{mo ve} is, 2
The force causes the net to move so that the two centers of gravity overlap.
Note that, in the initial state, the center of gravity of the outermost frame of the lip region extraction network matches the center of gravity of the outermost frame of the face region extraction network.

Next, a process of extracting a lip region by the lip region extraction net defined as above will be described with reference to a flowchart of FIG.

First, a face area is extracted by the face area extracting net in the same manner as in the first embodiment (S31). Therefore, the virtual net has a shape and size surrounding the face area as shown in the upper part of FIG. Since the lip area is inside the face area, the lip area extraction net is started to be moved using the net at the time of extraction of the face area as the initial value of the lip area extraction net (S32). In the same manner as in the face area extraction net, 8 neighborhoods are moved for all grid points,
The net is deformed by one unit (S33). Next, the center of gravity G _{net of the} region surrounded by the lip region extraction network and the center of gravity g _net of the outermost frame of the lip region extraction network are calculated, and the net is moved in a direction in which the distance between them is reduced. (S34).
It is checked whether there is a change in the energy of the entire network (S3
5) If there is a change in energy in S35, the process returns to step S33, and if there is no change in energy in S35, the deformation of the net is stopped. When the network stops, the network
As shown in the lower part of FIG.

As described above, the region extracting apparatus according to the present embodiment sets the lip region extracting net with the initial value of the net surrounding the face region, and sets the internal energy of the net based on the positional relationship between the grid points and the grid point. The total value of the image energy based on the likelihood of the color of the pixel where is located and the moving energy defined so that the center of gravity of the outermost frame of the lip region extraction network matches the center of gravity of the region surrounded by the network changes In this configuration, the mesh is shrunk, deformed and moved until it stops, and the movement of the grid points is stopped when the total value no longer changes.

Therefore, the net covering the face region contracts toward the lip region having a high degree of certainty in color, and finally stops around the lip region. As a result, a lip region can be extracted.

Further, in addition to the effect of the first embodiment, the angle between the vertical axis and the straight line connecting the center of gravity of the outermost frame of the face area and the center of gravity of the area surrounded by the lip area extraction net is obtained. , The face rotation angle can be recognized. Here, the rotation of the face refers to the inclination of the face, that is, rotation around an axis extending in the shooting direction from the imaging device.

When the rotation angle of the face is known, for example, when creating a person database, when a photograph is read by a scanner or the like, it is possible to input the photograph without ignoring the direction of the photograph, thereby greatly reducing labor. Similarly, the present invention can be applied to an apparatus that aligns the direction of a photograph that has fallen apart. Furthermore, it is not necessary to be concerned about a direction error such as blurring during shooting or digitizing a film image. As a result, the degree of freedom during photographing and processing increases.

Further, the approximate direction of the face can be estimated from the balance between the face area and the lip area. That is, if the center of the lip region is located at a position close to the main axis of the face region, the user is facing the direction of the imaging device, and if the center is shifted to the left or right, the user is facing each direction. .

Further, the movement of the lips can be detected by continuously extracting the lip regions from the moving image, and the recognition accuracy of the voice recognition can be improved by combining with a microphone or the like.
It is possible to determine whether the sound picked up by the microphone is talking by the user or heard from another place.

Further, the present region extracting apparatus is similar to that of the first embodiment.
It can be used to control various devices as well as.

For example, by attaching the present area extraction device to a television and detecting the direction of the user's face, the screen is darkened only when the user is not facing the television to protect the display surface. In combination with a sound sensor, the switch can be turned ON / OFF by simple generation, or simple device control can be performed. In addition, it is possible to determine whether the voice picked up by the microphone belongs to the user,
Control by erroneous detection can be prevented.

[Embodiment 3] The following will describe another embodiment of the present invention with reference to FIGS. For convenience of explanation, the same members as those shown in the drawings of the above-described embodiment are denoted by the same reference numerals, and description thereof will be omitted.

The region extracting apparatus according to the present embodiment extracts a face region, a lip region, and left and right eye regions from an input human image. Here, after extracting the face area, a lip area extraction net, a left eye area extraction net, and a right eye area extraction net with the extracted face area as an initial state are set, and the positional relationship between the respective nets is set. The lips region and the left and right eye regions are simultaneously extracted by simultaneously moving those nets while taking into account.

As shown in FIG. 32, in addition to the configuration of the second embodiment, the above-mentioned area extracting device stores a left-eye area extracting network information storage section 23, a right-eye area extracting network information storing section 24 in E _balance storage unit 2
5 and a face part positional relationship storage unit 26.

Left / right eye area extraction network information storage 23.2
4 stores the coordinates of the screen contracted in the face area as an initial state. In addition, the two centroids of the left and right eye region extracting nets, the internal energy (E _int ), the image energy (E _image ), and the shape energy (E _form ) of the net are stored. The balance energy storage unit 25 stores balance energy (E _balance ) for controlling the positional relationship between the three nets. Face part positional relation storage unit 2
6 stores in advance the positional relationship of the face parts obtained in advance by sampling.

According to the above configuration, as in the second embodiment, the region probability calculation device 5 refers to the region probability image storage unit 8 and calculates the probability based on the color of each pixel constituting the face region and the lip region. The likelihood of the color of each pixel is calculated by calculating a density function, and a face region probability image and a lip region probability image are generated. Next, the energy calculator 6 calculates the internal energy (E _int ) and the image energy (E _image ) of the face area extraction network, and stores them in the face area extraction network information storage unit 9. The mesh is shrunk, deformed and moved so that the sum of these energies is reduced, and a face region is first extracted. This result is output to the output device 4.

The center-of-gravity calculation device 21 stores the face area extraction net information storage unit 9, the lip area extraction net information storage unit 22, the left eye area extraction net information storage unit 23, and the right eye area extraction net information storage unit 24, respectively. The center of gravity (g) of the outermost frame is calculated with reference to the stored coordinates of the net and stored.
Further, referring to the coordinates of each net and the face area probability image stored in the area probability image storage unit 8, the area surrounding the face area extraction net, the lip area extraction net, and the left and right eye area extraction nets is determined. The center of gravity (G) is calculated, and the face area extraction net information storage unit 9 is calculated.
・ They are stored in the lip area extraction net information storage unit 22, the left eye area extraction net information storage unit 23, and the right eye area extraction net information storage unit 24, respectively.

The energy calculation device 6 calculates the internal energy (E (E) of the lip region extraction net and the left and right eye region extraction nets.
_int ), image energy (E _image ), and shape energy (E _form ) are calculated, and they are stored in the lip area extraction net information storage unit 22, the left eye area extraction net information storage unit 23, and the right eye area extraction net information storage. Stored in the unit 24. Further, the energy calculator 6 calculates and stores the balance energy (E _balance ) by comparing the face part positional relationship storage unit 26 with the centroids (g · G) stored in the four net information storage units. .

The three meshes are deformed, contracted and moved so that the total value of the energy of the lip region extracting mesh and the energy of the left and right eye region extracting meshes is reduced, and the lip region and the left and right eye regions are extracted. The result is output to the output device 4.

Next, the energy of the lip region extracting net, the left eye region extracting net, and the right eye region extracting net will be described. The energy of the lip region extracting net is the same as in the second embodiment.

(1) Internal energy of the eye area extraction net The internal energy of the left and right eye area extraction nets is the average energy of the eyes estimated from the face width, similar to the lip area extraction net. Apply energy so that the net shrinks in size and size. That is, the shape energy E _form obtained by assuming an ellipse having a major axis and a minor axis obtained by multiplying the face width by a certain ratio is added to the expression (1) of the first embodiment to obtain the eye energy. The internal energy of the area extraction network is defined.

(2) Image Energy of Eye Area Extraction Net The brightness of the pixels of the eye area extraction image is used as the image energy of the left and right eye area extraction net energies. The eye region extraction image is obtained by taking the larger of the brightness of the pixels corresponding to the lip region probability image and the face region probability image generated by the region probability calculation device 5. That is, FIG.
As shown in the coordinates (x, y) the brightness of the lip region probability image at pixel I _m (x, y), the brightness of the face area probability image pixel I _f (x, y), the eye area Assuming that the brightness of the pixel of the extracted image is I _e (x, y), the following is obtained.

[0163]

(Equation 24) The brightness of the pixel of the eye region extracted image generated in this way is defined as the image energy at the internal grid point. The image energy of the outermost lattice point is obtained by multiplying the pixel brightness of the face area probability image by -1. That is, the image energy at a certain grid point p (t, s) of the eye area extracting net is as follows.

[0164]

(Equation 25) Here, ERPI (t, s) is the lattice point p in the eye region extracted image.
The brightness at (t, s) and FRPI (t, s) are the brightness at lattice point p (t, s) in the face area probability image.

As described above, the internal energy and the image energy of the eye region extracted image are defined.

(3) Net Balance Energy These three types of nets for extracting the left and right eye areas and the lip area are used.
A description will be given of the balance energy of a network that is defined so that two networks move in cooperation.

Normally, the positional relationship between the left and right eyes and the lips of a human is the same, and although there are individual differences, for example, the distance between the eyes and the height of the eyes and lips are constant with respect to the face width. Have a ratio. Therefore, the positional relationship of the net from which each region is extracted is controlled using the a priori knowledge regarding the positional relationship between these face parts.

Table 1 shows data relating to the size and positional relationship of face parts measured from sample data of 24 persons.
This corresponds to the face width h, as shown in FIGS.
The average, the variance, and the standard deviation of the values of the respective measurement widths a to l when is 1 are shown. Each data is measured at the location shown in FIG. The positional relationship of the face parts calculated based on these data is stored in the face part positional relationship storage unit 26.
Is stored in

[0169]

[Table 1] The balance energy _Ebalance is defined so as to constrain the positional relationship of each network to specific constraints in addition to the internal energy and the image energy. The balance energy _Ebalance is expressed as follows.

[0170]

(Equation 26) As shown in FIG. 35, E _mg is energy such that the center of gravity g _m of the outermost lattice point of the lip region extraction net is drawn to the center of gravity G _m of the region surrounded by the net.

E _leg is energy such that the center of gravity g _le of the outermost lattice point of the left eye area extraction net is drawn to the center of gravity G _le of the area surrounded by the net.

[0172] E_regIs the outermost case of the right eye region extraction network
Center of gravity g of child point_reIs the center of gravity G of the area surrounded by the net_rePull to
It is energy that can be sent. The center of gravity g
_m, Center of gravity g_leAnd the center of gravity g_reIs the face area extraction in the initial state.
The center of gravity g of the outermost grid point of the destination net_fMatches.

[0173] E_iodIs the area surrounded by the left and right eye area extraction nets.
Area center of gravity G_le・ G_reDistance is a specific ratio to face width
It is the energy that is about to become.

E _emh is the distance between the midpoint of the line connecting the centroids G _le and G _{re of the} area surrounded by the left and right eye area extraction nets and the center of gravity G _{m of the} area encircled by the lip area extraction net. The energy that is trying to reach a certain ratio to width.

E _fa is the axis connecting the midpoint of the line connecting the centroids G _le and G _{re of the} area surrounded by the left and right eye area extraction nets and the center of gravity G _{m of the} area encircled by the lip area extraction net. , The energy that the angle formed with the axis representing the inclination of the face tends to become zero.

E _ema is a line segment connecting the centroids G _le and G _{re of the} region surrounded by the left and right eye region extraction nets, the center of the line segment and the centroid G _{m of the} region surrounded by the lip region extraction net. Is the energy at which the angle between the axis and the axis connecting the two is going to be a right angle.

As described above, the internal energy and image energy of the lip / left eye / right eye area extraction network are shrunk to each area in accordance with the image characteristics of each area while keeping the net shrunk and smooth. Live and balance energy E _balance keeps the balance of each net constant and parallel. The three nets are deformed and shrunk toward the respective areas, and the positional relations of the nets are moved so as to maintain a constant equilibrium state, so that the respective nets finally converge so as to surround the individual areas. FIG. 36 shows the result of applying the present region extraction device to an input image. (A) to (c) of FIG.
In this order, the lip, left eye, and right eye region extraction nets extract each region.

Next, the process of extracting the respective regions by the lip / left eye / right eye region extracting net will be described with reference to the flowchart of FIG.

First, a face area is extracted by the face area extraction net in the same manner as in the first embodiment (S41). Therefore, as shown in the upper part of FIG. 38, the virtual net has a shape and size surrounding the face area. Since the lip area and the left and right eye areas are inside the face area, each of the nets is started to be moved using the net at the time of extracting the face area as an initial value of the lip / left eye / right eye area extraction net (S42). . In the same manner as in the face area extraction net, 8 neighborhoods are moved for all grid points of the lip area extraction net, and it is checked whether or not the net is converged (S43). If not, the net is deformed by one unit (S44), and if it converges in S43, then the same is examined for the left and right eye region extraction nets (S45 to S48).

After each net is deformed by one unit, the balance energy is calculated and each net is moved (S49). Then, it is checked whether or not there is a change in the energy of the lip region extraction net (S50). If there is no change, the convergence flag of the lip region extraction net is turned ON (S51). Similarly, the left and right eye region extraction nets are checked (S52 to S52).
S55). Thereafter, it is checked whether or not there is a change in the energy of all nets (S56). If there is a change in energy, the process returns to step S43, and if there is no change in the energy of all nets, the deformation of the net is stopped. When all the networks are stopped, each network surrounds each area as shown in the lower three tiers of FIG.

As described above, the region extracting apparatus according to the present embodiment sets the lip / left and right eye region extracting nets with the net surrounding the face region as the initial value, and sets the internal energy of the net based on the positional relationship between the grid points. And the image energy based on the likelihood of the color of the pixel at which the grid point is located, and the positional relationship between the centers of gravity of the three regions are defined to be equal to the previously measured and stored positional relationship between the lips and the left and right eyes. In this configuration, the mesh is contracted and deformed and moved until the total value of the balance energy does not change, and the movement of the lattice point is stopped when the total value does not change.

Therefore, the net covering the face area contracts toward the lips and the left and right eye areas having high color certainty, and finally stops around each area. As a result, the lips and the left and right eye regions can be extracted.

As a result, in addition to the effects of the first and second embodiments, the shape of the lips and eyes can be recognized. Since these have unique values depending on the individual, individual identification is performed using this. be able to. That is, images of a plurality of persons are input in advance, each area is extracted by an area extracting device, and the relative positional relationship, size, and shape of each part are digitized and stored in a database for each person. . By attaching an imaging device and a region extraction device to the database search device, the face, lip, and eye regions of the person reflected on the imaging device are extracted, and the positional relationship and size of each part are similarly quantified. I do. Matching is performed between the obtained numerical value and the numerical value in the database to determine which person in the database the input image is. As a result, an application that creates a database in advance at the reception of a company with many visitors or a hotel, etc., photographs a visitor, automatically searches the profile of that person from the database, etc., and thereby assumes an application that responds smoothly Can be.

Further, if it becomes possible to extract not only a face outline but also a face part, the information compression rate will be further increased. As described in the first embodiment, the information can be compressed by extracting the face area and reducing the resolution and the number of colors other than the face, but the face part area is the highest among the face areas. By reducing the resolution and the number of colors in the order of the resolution and the number of colors and then the face area and the area other than the face area, it is possible to compress the information while keeping the deterioration of the face area of the image small. Furthermore, the number of colors used in the face part and the face area is limited, and by having their own color codes, it is possible to further compress information. Thus, more images can be stored with a smaller capacity than in the first and second embodiments, or a smooth image with a larger number of frames can be transmitted with a narrow bandwidth.

In an application such as a person database that requires a large number of images to be filed, the size of each image greatly depends on the number of images that can be stored in the device. As described above, if one person image can be stored with a smaller capacity, a database that can store many images at low cost can be created. In addition, a person image can be transmitted without increasing the bandwidth. Therefore, cost reduction of a video conference, a video phone, an electronic secretary, and the like can be realized.

Also, when creating a person database,
Until now, humans manually registered data representing the individual, which is a key for search along with the image, as a word, and manually entered the keyword when searching, or
Although the database had to be viewed from the beginning until it was found in order from the beginning, this area extraction device can automatically extract the face area and face parts from the image. Thus, it is only necessary to project the device itself on the imaging device connected to the database search device. As a result, an application that instantly finds out who is a visitor or a person who has met after a long time becomes possible.

Furthermore, when the eye and lip regions are extracted,
It is possible to recognize facial expressions. Since human facial expressions have features in facial parts such as eyes and mouth, by examining the shape of eyes and mouth by extracting areas, it is possible to determine what kind of facial expression that person is Become.
This makes it possible to perform automation such as robots and psychoanalysis that change correspondence according to human emotions.

Similarly, the ability to recognize a facial expression means that a facial expression can also be generated. When an eye or mouth region is extracted, an expression can be created by transforming the region into an emotion-specific shape. For example, a typical eye and mouth shape when a person is laughing is measured in advance and compiled into a database, and if the eye or mouth region extracted from the input image is deformed according to the database, the image is obtained. You can make the expression inside laugh. If various expressions are stored in a database in advance, it is possible to make the face of the captured image freely laugh or anger.

This makes it possible to develop a device having a human-friendly interface. Therefore,
It is possible to freely change the facial expression appearing in the operation explanation of the device or the like, and to provide a device which is easy for the user to access. Further, special effects in video works, such as animation that continuously changes one person into another person or another animal, can be fully automatically performed by using this technology. Its value as entertainment content in the coming multimedia era is great.

In addition, the automatic generation of facial expressions and portraits can reduce the cost of supplying multimedia contents in the entertainment field. What used to require manual work and unique senses and techniques can now be automatically performed, so that it can respond to demands that will increase in the future. .

Further, similarly to the first and second embodiments, by extracting the face area and the face parts in real time, it becomes possible to perform more detailed control of the device. For example,
The inclination of the face can be accurately detected based on the positional relationship between the face parts. Depending on the magnitude of the inclination of the face, it is possible to control the volume of the audio device and control the brightness of the lighting. Alternatively, it is possible to detect blinking or the like and change the tuning of the radio while driving both hands, for example, while driving a car.

Alternatively, by extracting facial parts and identifying individuals, automatic control of the strength of the air conditioner, the volume of televisions and audios, etc. according to personal preference, and security measures such as personal computer login and electronic lock can be performed. Application is also possible. This makes it necessary to adjust the device each time, to enter the password,
Input mistakes are reduced.

In the first to third embodiments, a region probability image is once generated by the region probability calculation device 5 and the image energy is calculated using this region probability image. And the processing can be sped up. However, when calculating the image energy, the image energy can also be calculated by calculating the likelihood of each region from the color of the pixel on the grid point in each case. In this case, there is an advantage that it is not necessary to generate a region probability image, so that the required memory can be reduced, but on the other hand, it is necessary to calculate the certainty of the region of the pixel on the grid point each time, so the processing speed is increased. Decrease.

Further, the energy of the net in the above embodiment is defined so that the energy becomes minimum when the shape of the net surrounds the face area and the face parts. The same effect can be obtained even if the energy is defined so as to be maximized when the face part is surrounded.

Although the shape of the net in this embodiment is radial, other shapes such as a grid may be used depending on the setting of energy. That is, any shape can be used as long as there is information on the connection relationship between points. However, in the case of extracting a face, a lip, an eye region or the like having a shape close to an ellipse as in the present embodiment, a radial shape is preferable because energy can be easily set.

In the present embodiment, the lattice points are moved from the outermost lattice point when moving the lattice points. However, the present invention is not limited to this, and the lattice points may be moved from any lattice point.
However, it is necessary to prevent the same grid point from moving twice until all the grid points have moved by one unit.

In this embodiment, the movement of the lattice point is 8
Although the vicinity is set, it is not limited to this, and may be, for example, four. Furthermore, you may move to a distant pixel instead of an adjacent pixel, in which case,
It is necessary to set the moving distance so as not to jump over the boundary of the region to be extracted.

Although the internal energy in Embodiments 2 and 3 is set to the shape energy obtained by assuming an ellipse, it is not necessary to set the internal energy if the shape of the object to be extracted is not known. Good. However, when the shape of the extraction target is known as in the present embodiment, it is desirable to set the shape energy because the accuracy is improved.

Further, the transfer energy in the second embodiment does not need to be set in the same manner, but it is preferable to set the transfer energy because the accuracy is improved.

[Fourth Embodiment] The following will describe another embodiment of the present invention with reference to FIGS. For convenience of explanation, the same members as those shown in the drawings of the above-described embodiment are denoted by the same reference numerals, and description thereof will be omitted.

The direction detecting apparatus according to the present embodiment uses the area extracting apparatus according to the third embodiment and extracts face parts from a human image reflected on each image capturing apparatus using two image capturing apparatuses. The three-dimensional direction of the head is detected by examining the displacement of the position of the face part shown in each image.

As shown in FIG. 39, the imaging devices (imaging means) 42L and 42R attached to the device 43 capture the user's figure and send the digital image to the region extraction device 41. The region extraction device 41 extracts a user's face region from each image, and sends the information to the head direction detection device 45. The head direction detecting device 45 detects the face part 3 from the two images.
The direction of the head is detected by detecting the dimensional information, and the information is sent to the control device 44. The control device 44 is connected to the device 43, and controls the device 43 based on information sent from the head direction detection device 45.

The method of detecting the head direction using the two image pickup devices 42L and 42R is as follows. First,
The arrangement of the imaging devices 42L and 42R complies with the epipolar constraint. As shown in FIG. 40, the epipolar constraint refers to a condition in the digital image L of one of the imaging devices 42L,
This is a restriction such that the digital image R of the other imaging device 42R appears parallel to the scanning line. That is, when two imaging devices are installed, the imaging devices having similar characteristics must be installed such that the optical axes are on the same plane.

First, as shown in FIG. 41, a lip region and left and right eye regions are respectively extracted from images L and R as two person images. Next, it is checked where the three face parts in the image L are located in the image R. The horizontal displacement of the lip position in the image L from the lip position in the image R is dm, the horizontal displacement of the left eye position in the image L is dle, and the horizontal displacement of the left eye position in the image R is dle, and the right eye position in the image L is the image R. Let dre be the horizontal displacement of the right eye position. The reciprocal of the shift of the face part between these two images L and R corresponds to the relative distance of the face part from the image plane.

Therefore, as shown in FIG. 42, each of the face parts on the image plane is vertically shifted 1 /
Assuming that points at distances dm, 1 / dle, and 1 / dre are pm, ple, and pre, respectively, the three-dimensional direction of the head passes through the midpoint of the line segment connecting ple and pre and pm. It is a straight line from pm to the middle point.

The process of the direction detecting device will be described with reference to the flowchart of FIG. First, an image of the user is captured by the two imaging devices 42L and 42R (S6).
1). Then, the lip / left and right eye regions are extracted from each image by the region extracting device 41 (S62). The horizontal position shift of the face parts of the lips, left eye, and right eye between the two images is calculated (S63). The distance of each face part from the image plane is calculated (S64). Then, the three-dimensional direction of the head is calculated (S65). By repeating this process, it is possible to detect the head direction of the user in front of the imaging device in real time.

That is, the face images are simultaneously photographed by the two image pickup devices 42L and 42R, and the face region and the face parts of each image are extracted, so that the lips, the left eye, the right eye, and the three Since three points in the dimensional space are determined, it is possible to detect which direction the head is facing.

As a result, the direction detecting device does not need to use dedicated hardware, as compared with the conventional device in which the direction of the head cannot be detected unless the dedicated device is mounted on the head. In addition, it is possible to realize a product at low cost, and it is possible to eliminate troublesomeness at the time of mounting. Also,
This eliminates the need for dedicated hardware, thus expanding the range of applications.

[0209] Further, a mechanism that detects the direction of the face and operates only when the user is facing the user is provided. Audio products become possible.

[0210] It is also possible to apply to virtual reality. That is, the position and direction of the virtual user in the virtual world represented on the computer can be determined according to the position and direction of the head of the person in front of the imaging device, and the actual user's head can be determined. And the movement of the user's head in the virtual world can be synchronized without using special headgear or the like. Further, in the three-dimensional CAD, it is possible to specify a point at a specific three-dimensional position, and to simultaneously adjust one having a plurality of attributes, for example, to simultaneously adjust the color, contrast, brightness and the like of a television.

Note that the number of imaging devices is not limited to two, but may be three or more. In this case, the arrangement of the imaging device does not need to follow the epipolar constraint.

[Fifth Embodiment] The following will describe another embodiment of the present invention with reference to FIGS. 44 to 46. For convenience of explanation, the same members as those shown in the drawings of the above-described embodiment are denoted by the same reference numerals, and description thereof will be omitted.

As shown in FIG. 44, in addition to the configuration of the fourth embodiment, information on the head direction detected by head direction detecting device 45 is input to the direction detecting device according to the present embodiment, as shown in FIG. A gaze direction detecting device 46 is provided.

The method of detecting the line-of-sight direction is as follows. First, as an experiment, the line-of-sight detection accuracy is set. The number of directions to be detected is determined from the accuracy, and the subject is gazed at that direction by the number as shown in FIGS. 45 (a) to (f), for example. Then, an image of the eye during the gaze is taken. The image of the eyes serves as a gaze database indicating the position of the pupil when a human looks in a certain direction. The direction of the user's line of sight can be estimated by comparing the image of the eye detected by the region extraction device with the line of sight database. Then, by combining the information on the three-dimensional direction of the head detected by the head direction detection device 45 of the fourth embodiment with the information on the gaze direction, it is possible to detect the true gaze direction. .

The process of the direction detecting device will be described with reference to the flowchart of FIG. First, the head detecting device detects the three-dimensional direction of the head (S71). At this point, since the eye region has already been detected, the size of the extracted eye region is changed according to the image in the eye-gaze database. Then, matching is performed between the image of the changed eye region and the image of the eye gaze database (S72). For this matching, a technique such as denplate matching is used. Then, the line-of-sight direction is estimated by taking the difference between the two images and selecting the one with the smallest difference (S73). The true gaze direction is detected by matching the three-dimensional direction of the head already detected with the estimated gaze direction (S74).

Thus, the present direction detecting device can detect the direction of the line of sight without using any special device. As a result, it is possible to eliminate the trouble of mounting hardware and the like, and to reduce the cost of the apparatus. In addition, instead of a pointing device such as information equipment, for example, any point of the computer display can be specified without using a mouse or pen, or a large device such as an existing eye tracker,
Various electronic devices can be controlled. In addition, it is possible to visually track a point of interest while operating the keyboard, thereby improving the efficiency of device operation. Further, other operations can be performed even when both hands are closed. It can also be applied to people with physical disabilities.

[0219]

As described above, the region extracting apparatus according to the first aspect of the present invention has an image pickup means for picking up an object to form a color digital image, and the color digital image has the same color uniformly. Probability density function storage means for storing in advance a probability density function indicating a desired area, and calculating the probability density function based on the color of each pixel constituting the desired area, thereby ascertaining the color accuracy of each pixel. And network information for storing virtual network information formed by connecting any three or more points on the color digital image and provided to cover the entire image. The storage means calculates the internal energy of the net defined based on the positional relationship between the points and the image energy of the net defined based on the likelihood of the color of the pixel where the point is located. Energy calculation means, the total value of the internal energy of the net and the image energy when one point is moved is compared with the total value before moving the point, and when the total value changes, In this configuration, the point is moved in the direction in which the net contracts, and the movement of the point is stopped when the total value no longer changes.

Thus, the present region extracting apparatus can extract a desired region based on the likelihood of a color that is resistant to a change in lighting conditions, and is thus affected by the location, time, presence / absence of auxiliary lighting, direction, etc. No more. Further, since the net can have any shape, the size of the input image and the position and size of the face in the image are not limited. As a result, there is an effect that the degree of freedom of photographing can be increased.

[0219] The region extracting apparatus according to the second aspect is the first aspect.
In addition to the configuration described above, the object is a person, and the desired area is an area of a face of the person.

As a result, the face region can be extracted, and can be used for controlling various appliances such as home appliances. That is, there is an effect that it is possible to control the device based on the position of the user.

The region extracting apparatus according to the third aspect is the first aspect.
Or, in addition to the configuration described in 2, the probability density function indicating a partial area having a color different from the desired area is stored in advance in the probability density function storage means, and the desired area is surrounded by the network information storage means. Information of the stopped network is stored as an initial value of the partial area extraction network for extracting the partial area,
When there is one partial area in the desired area, the area probability calculation means calculates the probability density function based on the color of each pixel constituting the partial area, thereby calculating the likelihood of the color of each pixel. Then, the energy calculation means calculates the internal energy and the image energy of the partial area extraction network and the moving energy defined so as to converge on the partial area, thereby extracting the partial area.

Thus, since the net used for the first area extraction can be used for extracting another area in the area, it is not necessary to change the setting of the net for each area.

The region extracting apparatus according to the fourth aspect is the third aspect.
In addition to the configuration described above, there is provided a center of gravity calculating means for obtaining a frame center of gravity from a point forming the outer periphery of the partial area extraction network, and obtaining a center of gravity of an area surrounded by the partial area extraction network with the brightness of the image as a weight. The moving energy is defined such that the frame center of gravity is drawn in the direction of the center of gravity of the region.

Since the moving energy is set so that the center of gravity of the frame is drawn in the direction of the center of gravity of the region, even if there is a region having a color similar to the partial region, the region is not drawn to that region. Partial regions can be extracted. As a result, there is an effect that the processing speed is high, and a high-precision partial region can be extracted.

According to the fifth aspect of the present invention, there is provided an area extracting apparatus.
In addition to the configuration described above, a center of gravity calculating means for obtaining a frame center of gravity from a point forming the outer periphery of the network, and obtaining a center of gravity of an area surrounded by the network using the brightness of the image as a weight, the probability density function storage means The probability density function indicating the lip region is stored in advance, and the information of the net stopped surrounding the face region is stored in the net information storage means as initial values of the net for extracting the lip region and the left and right eye regions. The region probability calculation means calculates the probability of each pixel by calculating the probability density function based on the color of each pixel constituting the lip region, and configures the left and right eye regions. The likelihood of the color of each pixel is calculated by calculating the probability density function of the face area or the lip area based on the color of each pixel, and the energy calculation means calculates the internal energy of each area and the image energy. And the center of gravity of each frame is drawn in the direction of the center of gravity of each region, and the positional relationship of the centers of gravity of the three regions is defined to be equal to the previously measured and stored positional relationship of the lips and left and right eyes. By calculating the balance energy, the lips and the left and right eye regions are extracted.

Thus, the face area extracting net can be the lip area extracting net and the left and right eye area extracting nets, so that it is not necessary to change the setting of the net for each area. In addition, since the position of the face part with respect to the face area is known, the rotation angle of the face (the inclination of the face) can be recognized, and the degree of freedom in photographing and processing increases. Further, there is an effect that the approximate direction of the face can be estimated.

The direction detecting device according to claim 6 is the same as in claim 5.
A plurality of the above-mentioned image pickup means are provided using the region extraction device described above, and a lip area and a left and right eye area are respectively extracted from a plurality of images obtained simultaneously from the plurality of image pickup means, and the lip area and the eye area are compared with each other. Is derived.

Thus, the head direction can be easily detected without using dedicated hardware as in the related art. As a result, it is possible to eliminate the trouble of mounting the hardware and the like, and it is possible to reduce the cost of the apparatus.

The direction detecting device according to claim 7 is the same as in claim 6.
In addition to the configuration described above, the estimated eye direction is calculated by comparing the extracted eye image with an eye image that has been measured and stored with the eye direction information added thereto in advance, and the estimated eye direction is further calculated. And the direction of the detected head is compared to detect the true gaze direction.

As a result, the gaze direction can be easily detected without using dedicated hardware as in the related art. As a result, it is possible to eliminate the trouble of mounting the hardware and the like, and it is possible to reduce the cost of the apparatus.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an area extraction device according to an embodiment of the present invention.

FIG. 2 is a flowchart showing an operation of a region probability calculation device in the region extraction device.

3A is an explanatory diagram illustrating an input image, and FIG. 3B is an explanatory diagram illustrating a state in which a skin color portion and a lip portion of a face are extracted from the input image.

FIG. 4 is a graph showing a chromaticity distribution of hues in a face area.

FIG. 5 is a graph showing a chromaticity distribution of saturation of a face area.

FIG. 6 is a graph showing a two-dimensional normal probability density function representing a face region.

FIG. 7 is a flowchart illustrating a process of generating a region probability image.

FIG. 8 is a drawing substitute photograph showing an image input to the input device and showing a halftone image displayed on a display.

FIG. 9 is a drawing substitute photograph showing a face region probability image generated from the input image shown in FIG. 8, and showing a halftone image displayed on a display;

FIG. 10 is a drawing substitute photograph showing a lip region probability image generated from the input image shown in FIG. 8, and showing a halftone image displayed on a display.

FIG. 11 is an explanatory diagram showing grid points forming a net;

FIGS. 12A and 12B are explanatory diagrams showing how a region is extracted using a network model. FIG. 12A is an input image, FIG. 12B is an initial state of the network, FIG. 12C is a state of a converging network, and FIG. d) shows the state of the network after convergence, respectively.

13A and 13B are explanatory diagrams showing how a face region is extracted using a network model, where FIG. 13A is an input image, FIG. 13B is an initial state of the network, FIG. 13C is a state of a converging network, (D) shows the state of the network after convergence.

FIG. 14 is an explanatory diagram showing the shape of a net.

FIG. 15 is an explanatory diagram showing another net shape.

FIGS. 16A and 16B are explanatory diagrams showing definitions of energies constituting an internal energy of a network.

FIG. 17 is a flowchart showing how to move a grid point.

FIG. 18 is an explanatory diagram showing eight neighborhoods of a certain pixel.

FIG. 19 is a block diagram showing a configuration when the area extraction device is used in a device.

FIG. 20 is a block diagram showing a configuration of an area extracting device according to another embodiment of the present invention.

FIG. 21 is a graph showing a chromaticity distribution of hues in a lip region.

FIG. 22 is a graph showing the chromaticity distribution of the saturation of the lip region.

FIG. 23 is a graph showing a two-dimensional normal probability density function representing a lip region.

FIG. 24 is an explanatory diagram showing the outermost frame and the center of gravity of the outermost frame.

FIG. 25 is an explanatory diagram showing the center of gravity of a region surrounded by a net.

FIG. 26 is an explanatory diagram showing parameters required for defining shape energy.

FIG. 27 is an explanatory diagram showing the definition of shape energy.

FIG. 28 is an explanatory diagram showing how to obtain a lip region probability difference image.

FIG. 29 is a drawing substitute photograph showing a lip region probability difference image calculated from the face region probability image shown in FIG. 9 and the lip region probability image shown in FIG. 10, and showing a halftone image displayed on the display. .

FIG. 30 is a flowchart showing a process in which the lip region extraction net extracts a lip region.

FIG. 31 is a schematic diagram illustrating a process of extracting a lip region by a lip region extraction net;

FIG. 32 is a block diagram showing a configuration of an area extracting device according to another embodiment of the present invention.

FIG. 33 is an explanatory diagram showing a method of obtaining an eye area extracted image.

FIG. 34 is an explanatory diagram showing data measurement points when data relating to the size and the positional relationship of the face part is measured.

FIG. 35 is an explanatory diagram showing parameters required for defining balance energy.

FIG. 36 (a) is an explanatory diagram showing a state where a lip region extracting network, FIG. 36 (b) is a left eye region extracting network, and FIG.

FIG. 37 is a flowchart showing a process in which the lip and left and right eye region extraction nets extract each region.

FIG. 38 is a schematic diagram showing a process in which the lip and the left and right eye region extraction nets extract each region.

FIG. 39 is a block diagram showing a configuration of a direction detection device using an area extraction device according to another embodiment of the present invention.

FIG. 40 is an explanatory diagram showing a state of epipolar constraint.

FIG. 41 is an explanatory diagram showing a process of calculating a shift of a face part between two images.

FIG. 42 is an explanatory diagram showing three-dimensional position information of a face part calculated from the value of the shift.

FIG. 43 is a flowchart showing a head detecting operation of the direction detecting device.

FIG. 44 is a block diagram showing a configuration of a direction detecting device using an area extracting device according to another embodiment of the present invention.

FIGS. 45A to 45F are explanatory diagrams illustrating an example of a gaze database.

FIG. 46 is a flowchart showing a line-of-sight detection operation of the direction detection device.

47 (a) and 47 (b) show a conventional extraction method using projection. FIG. 47 (a) is an explanatory diagram showing a state in which the input image shown in FIG. 8 is differentiated in the horizontal direction. Graph that counts the number of pixels when projected to
(C) shows a graph in which the number of pixels when the image of (a) is projected in the vertical direction is counted.

FIG. 48 is an explanatory diagram showing the horizontal differentiation process.

FIG. 49 is an explanatory diagram showing an input image and a template used in a conventional template-based extraction method.

FIG. 50 is an explanatory diagram showing a state of comparison between a template having a size of 5 × 5 and an input image.

FIG. 51 is an explanatory diagram showing the order of comparison between a template and an input image.

FIG. 52 is an explanatory diagram showing a state in which eyes are extracted by using a conventional extraction method using a deformable template.

FIG. 53 is an explanatory diagram showing parameters required for defining a template in the extraction method.

FIG. 54 is an explanatory diagram showing a conventional head direction detection method.

FIG. 55 is an explanatory diagram showing another conventional head direction detection method.

[Explanation of symbols]

Reference Signs List 1 input device (imaging means) 5 region probability calculation device (region probability calculation device) 6 energy calculation device (energy calculation device) 7 probability density function storage unit (probability density function storage unit) 9 face region extraction network information storage unit ( 21) Center of gravity calculation device (centroid calculation unit) 22 Network information storage unit for lip region extraction (net information storage unit) 23 Network information storage unit for left eye region extraction (net information storage unit) 24 Net for right eye region extraction Information storage (network information storage means)

──────────────────────────────────────────────────続 き Continuing the front page (56) References Katsuhiko Sakagami, one other, Dynamic Network Model Active Net and Its Application to Region Extraction, Journal of the Institute of Television Engineers of Japan, The Institute of Television Engineers of Japan, 1991 October 20, Vol. 45, No. 10, pp. 146-64 1155-1163 (58) Field surveyed (Int. Cl. ⁷ , DB name) G06T ⁷ /00-7/60 G06T 1/00 JICST file (JOIS)

Claims (7)

(57) [Claims] 1. An image pickup means for picking up an image of an object to form a color digital image, and a probability density function storage means for storing in advance a probability density function indicating a desired area having a uniform color in the color digital image. And area probability calculation means for calculating the likelihood of the color of each pixel by calculating the probability density function based on the color of each pixel constituting the desired area; Network information storage means for storing information of a virtual network formed by connecting two or more points and provided so as to cover the entire image; and inside the network defined based on the positional relationship of the points. Energy calculating means for calculating the energy and the image energy of the net defined based on the likelihood of the color of the pixel at which the point is located, and moving one point The total value of the internal energy of the net and the image energy is compared with the total value before moving the point, and when the total value changes, the point is moved in a direction in which the net contracts, and the total value changes. An area extracting apparatus, wherein the movement of a point is stopped when the point no longer exists. 2. The image processing apparatus according to claim 1, wherein the object is a person, and the desired area is an area of a face of the person. 3. A probability density function indicating a partial area having a color different from that of the desired area is stored in advance in the probability density function storage means, and the network information storage means stores information of a network stopped surrounding the desired area. The information is stored as an initial value of a partial area extraction network for extracting a partial area, and when there is one partial area in the desired area, the area probability calculating means sets the color of each pixel constituting the partial area. Calculate the probability of each pixel by calculating the probability density function based on the above, and define the energy calculation means to converge on the internal energy and image energy of the partial area extraction network and the partial area. The region extracting apparatus according to claim 1, wherein a partial region is extracted by calculating each of the determined transfer energies. 4. A method for calculating a center of gravity of a frame from a point forming an outer periphery of a partial area extracting network, and a center of gravity calculating means for determining a center of gravity of an area surrounded by the partial area extracting network with weight of image brightness. 4. The region extracting apparatus according to claim 3, wherein the moving energy is defined such that the center of gravity of the frame is drawn toward the center of gravity of the region. 5. A method for calculating a center of gravity of a frame from a point forming an outer periphery of a network, and calculating a center of gravity of an area surrounded by the network using the brightness of an image as a weight. Is stored in advance, and the network information storage means stores information on the network stopped surrounding the face region as initial values of the network for extracting the lip region and the left and right eye regions, The area probability calculation means calculates the probability of each pixel by calculating the probability density function based on the color of each pixel forming the lip area, and calculates each pixel forming the left and right eye areas. Calculating the probability density function of the face area or the lip area based on the color of each pixel to calculate the likelihood of the color of each pixel; A balance defined such that the center of gravity of each frame is drawn in the direction of the center of gravity of each region, and the positional relationship of the centers of gravity of the three regions is equal to the previously measured and stored positional relationship of the lips and left and right eyes. 3. The region extracting apparatus according to claim 2, wherein the lip and the left and right eye regions are extracted by calculating energy. 6. A three-dimensional position of a lip and an eye by extracting a plurality of lip regions and left and right eye regions from a plurality of images simultaneously obtained from the plurality of imaging units. 6. A direction detecting device using the region extracting device according to claim 5, wherein information is derived. 7. An estimated line-of-sight direction is calculated by comparing the extracted eye image with an eye image which has been measured and added with information on the line-of-sight direction and is stored in advance. 7. The direction detecting device using the region extracting device according to claim 6, wherein a true line-of-sight direction is detected by comparing the detected direction of the head.