JP2008059108A - Image processing apparatus, image processing method, its program, and flow of people monitoring system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of producing a face image of an arbitrary angle, which is more precise than conventional one and is considered to be closer to a person himself when the person is recognized by using the face image. <P>SOLUTION: The image processing apparatus comprises an image storing means for storing an image to be inputted from a camera, a facial parts extracting means for detecting a position etc. of a facial organ from the image, a 3-dimensional shape model storing means for storing 3-dimensional shape information of an average personal head portion configured by a wire frame, a simple model producing means for producing the 3-dimensional shape model adapted to facial parts arrangement of the person included in the image by deforming the 3-dimensional shape model based on a facial parts position acquired from the facial parts extracting means, an attribute calculating means for calculating a personal attribute from the image, and an attribute reflecting means for producing a detailed model after deforming the 3-dimensional shape model acquired from the simple model producing means based on the personal attribute acquired from the attribute calculating means. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, a program thereof, and a human flow monitoring system for recognizing a person using a face image.

  Currently, surveillance cameras are installed in the entrances and parking lots of buildings, entrances to sites such as factories and schools, and stores such as large-scale stores and convenience stores, and monitoring images are stored on storage media such as magnetic tape and magnetic disks. In addition, it is monitored in real time by facility managers and security guards. The main purpose of such human flow monitoring is to detect criminals and suspicious persons such as terrorists from the hustle and bustle who are requested to investigate. However, it is extremely expensive to discover the target person by looking at the monitor screen at all times, and it can be overlooked due to fatigue and reduced concentration, so it is surely discovered. It is difficult.

  For this reason, it is important to perform image processing and image recognition processing on the monitoring image obtained from the monitoring camera to automatically find the target person.

  In a system for identifying a person using an image obtained by photographing a face, due to a change in the posture of the person with respect to the camera between a pre-registered face image obtained by identification or the like and a face image photographed by the camera used for identification Apparent distortion and deformation occur, which is a factor that decreases the authentication accuracy. In particular, many products that perform only face recognition processing (hereinafter referred to as face recognition engines) that have become commercially available in recent years target only the front face, and such front face face recognition engines have apparent distortion. Susceptible to deformation and deformation.

  Therefore, in order to solve such a problem, for example, in Patent Document 1, the front face image registered in advance is directed in the same direction as the input face image at an arbitrary angle in the following procedure. There is disclosed a method of performing conversion and collation as required.

  First of all, facial organs such as eyes, nose and mouth are extracted automatically or manually from pre-registered frontal face images, and based on the contour position of the organs, three-dimensionally formed with a wire frame Assign a head model. After the assignment, the color information of each point in the input image is made to correspond to the 3D head model to color the 3D head model. The three-dimensional head model thus formed is rotated to face an arbitrary angle, and is returned to the image again to generate a face image in the same direction as the input image.

  The wire frame is a model in which the contour of a target object (here, a facial part such as an eye or nose) is connected by a plurality of line segments.

JP 2003-263639 A Paul Viola and Michael Jones, “Rapid Object Detection using a Boosted Cascade of Simple Features”, IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2001 (CVPR01)

  In the face image recognition method disclosed in Patent Document 1, a three-dimensional head model based on a wire frame is assigned by calculating an edge of an image from an input image with respect to the boundary position of face parts such as eyes and mouth. Although calculation was possible, wireframe allocation was not performed well in areas where accurate face parts positions such as forehead, gills, cheeks, and mandibular jaws were difficult to obtain regardless of the head orientation, resulting in standard models, etc. The information is complemented using a shape model that does not reflect the individual characteristics of the person. For this reason, appropriate shape restoration and deformation correction cannot be performed for these parts which are considered to be different for each individual.

  For parts that cannot be captured directly from the camera due to hiding, etc., these three-dimensional shape restoration uses these standard three-dimensional head model shapes stored in the system in advance. Complementing the information deficient site. For this reason, the image generated by the above method includes a portion that does not reflect the characteristics of the person, and there is a problem that an inaccurate face image is generated.

  In order to achieve the above object, the image processing apparatus and the image processing method of the present invention have a viewpoint (camera direction (vertical, horizontal, etc.), human posture or facial expression, lighting, camera parameters (zoom magnification, lens characteristics). ) Or the like, and a point-of-view face image taken according to the user) is input, and face parts (eye, nose, mouth, eyebrows, mole, outer ear, hairline, etc.) are input to this input image. (Including at least one of the above) by image processing, and based on the facial part extraction result, the 3D head shape model formed by the wire frame is assigned to the input image while deforming according to the facial part arrangement Model (hereinafter referred to as a simple model). Then, a person attribute (including at least one of head size, face head outline shape, age, gender, physique, etc.) is calculated from the input image or face part information, and the simplified model is corrected based on the person attribute Thus, by creating a detailed model, more appropriate three-dimensional personal shape data for the input image is created.

  According to the present invention, the arrangement and shape of face parts are extracted, and the 3D head shape model is transformed and adjusted to a shape suitable for each individual, and then color and pattern information (hereinafter referred to as texture) obtained from the input image is obtained. The shape near the extracted facial parts is considered to reflect the characteristics of each individual. This is equivalent to the prior art, but when the entire human head is viewed, it cannot be said that this alone sufficiently reflects the characteristics of each individual.

  Therefore, in the image processing apparatus and the image processing method of the present invention, the person attribute is calculated, and based on the calculation result, based on the information that is lost only by the arrangement and shape of only the face parts, the three-dimensional head shape By correcting the model, more accurate three-dimensional personal shape data can be generated.

  In addition, once the three-dimensional personal shape data is generated, it is easy to generate an image of an arbitrary viewpoint using the shape data. For this reason, three-dimensional personal shape data is generated from the head image of unknown position and orientation input from a monitoring camera or the like using the means of the present invention, for example, by generating a front face image, It becomes possible to perform person verification using the face recognition engine.

  In the image processing apparatus and the image processing method of the present invention, a three-dimensional head detailed shape model (hereinafter referred to as a detailed model) having different characteristics is selected based on the calculated person attributes, and a simple model is selected. By deforming this detailed model using the deformation parameter, it becomes possible to reflect the person attribute in the three-dimensional personal shape data.

  Furthermore, in the image processing apparatus and the image processing method of the present invention described above, based on the calculated person attributes, three-dimensional detailed head shape models (hereinafter referred to as detailed models) having different characteristics are selected, and the details are selected. It is also possible to adopt a configuration in which the person attribute is reflected in the three-dimensional personal shape data by deforming the basic model using the deformation parameter in the model.

  The present invention includes a program for executing the above image processing apparatus and image processing method.

  According to the present invention, in the face recognition system, there is an effect that three-dimensional personal shape data can be generated with respect to each of various viewpoints such as up / down, left / right, oblique direction of shooting or various facial expressions.

  Further, it is possible to generate an image from a viewpoint different from the input image from the three-dimensional personal shape data, and there is an effect that the recognition accuracy can be kept high in the face recognition system.

  By using such a highly accurate face image recognition device, there is an effect of realizing a crime prevention system, a monitoring system, a security system, etc. with high reliability and accuracy.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described in detail with reference to FIGS.

  FIG. 1 is a diagram showing an example of the configuration of an image processing apparatus according to the first embodiment of the present invention.

  As shown in FIG. 1, the image processing apparatus 101 is connected to a camera 102 and a display apparatus 103.

FIG. 2 is an example showing the inside of the image processing apparatus 101. Image input unit 104, image storage unit 105, face part extraction unit 106, three-dimensional shape model storage unit 107, simple model generation unit 108, detailed model generation unit (attribute calculation unit) 109, attribute reflection unit 110, image generation unit 111 It is comprised including. The image input unit receives an image signal sent from the camera 102. When the image signal is an analog signal, the image input unit converts the converted image data into a digital signal using an A / D (Ana1ogtoDigital) conversion device. Store in the storage unit 105. When the image signal is a digital signal, the image data is stored in the image storage unit 105 after performing communication processing with the camera 102, error correction, and the like. The image storage unit 105 stores the image data input from the camera 102 by the image input unit 104, reads and writes the stored image data in response to requests from the face part extraction unit 106 and the attribute calculation unit 109, and appropriately , Change or add all or part of the image data.

  In this specification, “image data” may be simply referred to as “image”. The face parts extraction unit 106 extracts parts such as eyes, eyebrows, and mouth from the image stored in the image storage unit 105. This part extraction can be realized by using a known active contour search method, template matching method, graph matching method, or the like.

  The three-dimensional shape model storage unit 107 stores model information in which a three-dimensional shape of a face is expressed in advance using a wire frame. As shown in FIG. 6, the wire frame model is composed of a vertex 133 representing three-dimensional coordinates and its connection information 134. This model information includes not only the coordinates of each vertex of the wire frame in the three-dimensional space but also the correspondence information with the face organ position such as which vertex of the wire frame corresponds to the vertex of the right eye. It is.

  The simple model generation unit 108 deforms the three-dimensional shape model based on the face part wrinkle position obtained by the face part extraction unit 106, and the three-dimensional shape model conforms to the human face part arrangement included in the input image. (Hereinafter referred to as a simple model).

  The simple model generation unit 108 deforms the three-dimensional shape model based on the face part position obtained by the face part extraction unit 106, and the three-dimensional shape model conforms to the human face part arrangement included in the input image. (Hereinafter referred to as a simple model). The shape of this simple model can be adjusted based on the face part coordinates extracted by the face part extraction unit 106.

  For example, with regard to the width of the simple model, the right and left eye corners / eyes can be found using the dynamic contour method, and the width of the head can be estimated from the interval between the right eye corner and the left eye corner. This is, for example, the detection of eye contours by face detection and the dynamic contour method, and specifically the following flow.

  First, by using the technique of Viola et al. Shown in Non-Patent Document 1, the approximate position of the eye can be determined from the position and size of the face simultaneously with the face detection from the image. The initial contour line of the dynamic contour method is arranged largely. After that, by contracting the contour line so as not to exceed the edge in the image, the position of the contour line converges around the eye. For this reason, the position of face parts without clear edges on the face, such as the forehead, gills, cheeks, and lower jaw, cannot be obtained, and the simple model is only for parts such as the eyebrows, eyes, nose, and mouth. It becomes a model that reflects individual characteristics.

  The attribute calculation unit 109 calculates personal attributes such as face width, head size, head outline shape, age, gender, physique, and height. At this time, not only the face area of the input image but also the image of the entire person shown in the input image is used. For example, the face width can be calculated by using a ratio of a person silhouette extracted using a known background difference method to an input image and a face region extracted using a known skin color extraction method. Further, the head size can be calculated by obtaining the head position based on the person silhouette and the face part position and obtaining the width of the silhouette corresponding to the head position. Furthermore, the age can be calculated from the amount of wrinkles around the face parts, and the sex can be calculated from the clothes.

  The detailed model generation unit 110 generates a detailed model by reflecting the shape feature inferred from the person attribute obtained from the attribute calculation unit 109 in the simple model. For example, the horizontal scale of the simple model is changed using the face width information, the amount of flesh of the simple model is changed using the physique information, or the contour of the error is determined using gender information. To change. Alternatively, not only information that can be calculated mechanically but also other human impressions and sighting information may be used as attribute information. For example, it may be possible to increase the height of the nose based on information that seems to be Westerners. Note that the detailed model generation unit 110 may store a model reflecting the characteristics of each attribute, and reflect the characteristic shape information in the simple model when the attribute information is input.

  In the image generation unit 111, the detailed model is moved to a designated position and orientation and projected onto a two-dimensional plane, and an image is generated in consideration of camera parameters, illumination conditions, and the like as necessary.

The image processing apparatus 101 described above is a CPU including an arithmetic device (not shown).
(Central Processing Unit) and a computer including a storage device such as a semiconductor memory or a hard disk storage device. In that case, the entities such as the image storage unit 105 and the three-dimensional shape model storage unit 107 are configured in a storage area allocated to a part of the storage device, and the face parts extraction unit 106, the simple model generation unit 108, The attribute calculation unit 109, the detailed model generation unit (attribute reflection unit) 110, and the image generation unit 111 are realized by the CPU executing a predetermined program stored in the storage device. Further, part of the image input unit 104 and the image generation unit 111 is realized by the CPU executing an input or output interface circuit and a program for driving and controlling the circuit.

  FIG. 3 shows how the internal data is sequentially created by the image processing apparatus according to the first embodiment of the present invention.

  The face image captured by the image input unit is stored as an input image 112 in the image storage unit. For this input image 112, a simple model 113 is created by the face part extraction and simple model generation unit, human attributes such as age and physique are estimated, and detailed models 114 are created by reflecting shape information for each attribute. This represents a process of rotating and translating the detailed model 114 to a designated position and orientation to create a personal shape 115 as a two-dimensional image.

  FIG. 4 shows a human flow monitoring system using the image processing apparatus according to the first embodiment of the present invention.

  An image photographed by the camera 102 is input to the image processing apparatus 101 and a front face is generated, and then passed to the face matching apparatus 116. The face matching device 116 calculates a numerical vector (hereinafter referred to as a feature vector) peculiar to each individual face calculated from the front face, and a face image database 117 (hereinafter referred to as a “matching face image database”) registered in advance. The feature vectors of the face images stored in the matching face DB) are calculated, and the degree of similarity is calculated. The face matching device 116 determines which person in the matching face DB the input image is based on the similarity. The determination result is sent to the storage / retrieval device 119 through, for example, a wired / wireless network 118 represented by the Internet, and an alarm is sent to the portable terminal 120 as necessary. As described above, according to the present invention, an automatic human flow monitoring system using a monitoring camera can be constructed, and there is an effect of improving security and convenience of searching for a specific person.

  FIG. 5 shows another example of the human flow monitoring system using the image processing apparatus according to the first embodiment of the present invention.

  As shown in FIG. 5, the image processing apparatus 101 is configured to receive images from one or a plurality of cameras 102 connectable to a network typified by a Web camera via the network 118, and a number of cameras Image data from the image processing apparatus 101 is processed by one image processing apparatus 101. In this case, except for the part particularly mentioned, it is the same as the embodiment of FIG. 4, but is excellent in view of the installation volume and cost of the apparatus.

  FIG. 7 shows an example of a face collation system when a person attribute is not obtained.

  When the person attribute of the input image 112 is not obtained, the image processing apparatus 101 assumes a plurality of person attributes, outputs a plurality of different generated images 121, and sends all of them to the face matching apparatus 116. It has become.

  By adopting such a configuration, it is possible to apply face matching in consideration of a plurality of variations of information related to individual faces that are lost during face matching, and therefore, there is an effect that it is possible to prevent overlooking of matching.

  FIG. 9 is a diagram illustrating an example of a configuration when the result of attribute estimation is also used for matching.

  It is conceivable that a plurality of person attributes such as the person's age, sex, and physique may be input to the matching face DB 117 in addition to the face image. In this case, comparing the person attribute 130 calculated from the input image 112 with the person attribute of the person determined by the face matching engine 129 has an effect of improving the matching accuracy of the face matching.

  FIG. 10 shows an embodiment of a face image generation system when manually assigning person attributes.

  For example, in a criminal investigation, when an image is obtained from the camera 102 and qualitative person attribute information 130 is obtained from the sighting information 132, there is a demand for creating a more accurate arrangement photograph using both. In such a case, instead of calculating the person attribute directly from the input image, the person attribute 130 based on the sighting information 132 is manually input, thereby selecting the attribute-specific detailed shape information 131 and creating the detailed model 114. It has become.

  FIG. 8 shows an example of an interface screen for inputting the person attribute at this time.

The GUI interface 122 includes a field 123 for inputting gender, a field 124 for inputting age, a field 125 for inputting physique, and the like, and attribute value ranges 126 automatically recognized by the system are shown. With reference to this attribute value range 126, the knob 127 for designating the input value is manually moved to input the person attribute. By providing the column 128 for displaying the model reflecting the attribute value for visualizing the input person attribute, it becomes possible to input the data of the person attribute in more detail.

  FIG. 11 shows an embodiment of a system that performs collation after converting the collation face image DB of the face collation system using the person attribute calculation result obtained from the input image.

  It is considered that the viewpoint of capturing the collation face image DB and the viewpoint of capturing the input image may be different. For example, aging is a typical example, but the face image registered in the matching face image DB 140 is aging using the person attribute 130 (aging information, etc.) obtained by calculating the person attribute. The configuration is such that the input image is collated with the object.

  In this case, for example, even if a secular change occurs, there is an effect that the matching accuracy of the face matching engine 129 can be improved.

  According to the present invention, by using it in a face recognition system, it is possible to generate three-dimensional personal shape data for each of various viewpoints such as up / down, left / right, oblique direction of shooting or various facial expressions.

  Further, an image having a different viewpoint from the input image can be generated from the three-dimensional personal shape data, and the recognition accuracy can be increased in the face recognition system.

  Furthermore, by using such a highly accurate face image recognition system, a crime prevention system, a monitoring system, a security system, etc. with high reliability and accuracy can be realized.

The figure explaining the input part and output part of the image processing apparatus of this patent. The figure explaining the internal processing flow of the image processing apparatus of this patent. The figure showing a mode that data are sequentially converted inside the image processing apparatus of this patent. The figure showing the people flow monitoring system using the image processing apparatus of this patent. The figure showing the example of a different structure of the people flow monitoring system using the image processing apparatus of this patent. The figure showing the wire frame model contained in the three-dimensional shape storage part of the image processing apparatus of this patent. The figure showing the system which does not calculate a person attribute, obtains many generated images on the assumption of a plurality of person attributes, and inputs all of them to the face matching device. The figure showing the interface in the case of inputting manually, without calculating a person attribute. The figure which shows the structure which utilizes the person attribute calculated with the image processing apparatus of this patent for collation. The figure showing the structure which combines a camera image | video and the person attribute by sighting information, and produces | generates an image. The figure showing the structure which converts and compares the face DB for collation using the person attribute calculated with the image processing apparatus of this patent.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 ... Image processing apparatus, 102 ... Camera, 103 ... Display apparatus, 104 ... Image input part,
DESCRIPTION OF SYMBOLS 105 ... Image memory | storage part 106 ... Face part extraction part 107 ... Three-dimensional shape model storage part 108 ... Simple model production | generation part 109 ... Attribute calculation part 110 ... Detailed model production | generation part 111 ... Image production | generation part 112 ... Input image 113 ... Simple model 114 ... Detailed model 115 ... Personal three-dimensional shape 116 ... Face collation device 117 ... Collation face DB 118 118 Network 119 Accumulation / retrieval device 120 Mobile terminal 121 ... generated image, 122 ... GUI interface, 123 ... field for inputting gender, 124 ... field for inputting age, 125 ... field for inputting physique, 126 ... range of attribute value automatically recognized by system, 127 ... attribute input value , A field for displaying a model reflecting attribute values, 129, a face matching engine, 130, person attribute information, 131, by attribute Fine shape information 133 ... vertices of the wire frame model,
134: Connection information of the wire frame model.

Claims (8)

In an image processing apparatus that processes an image input from a camera,
Image storage means for storing an input image input from the camera;
Facial part extraction means for detecting the position, shape and characteristics of a facial organ from the input image;
3D shape model storage means for storing 3D shape information of an average human head composed of wire frames;
Simple model generating means for deforming the three-dimensional shape model based on the face part position obtained from the face part extracting means, and generating a three-dimensional shape model suitable for a human face part arrangement included in the input image; ,
Attribute calculation means for calculating a person attribute from the input image;
Based on the person attribute obtained from the attribute calculation means, attribute reflection means for deforming the three-dimensional shape model obtained from the simple model generation means and generating a detailed model;
An image processing apparatus comprising: an image generation unit configured to generate a two-dimensional image from a detailed model output from the attribute reflection unit. The image processing apparatus according to claim 1.
The facial organ includes at least one of eyes, nose, mouth, eyebrows, mole, outer ear, hairline, etc.
The image processing apparatus according to claim 1, wherein the person attribute includes at least one of a face width, a head size, a head outline shape, an age, a gender, a physique, and a height. The image processing apparatus according to claim 1.
With respect to the attribute reflecting means for preparing the detailed model, means for manually inputting an arbitrary person attribute as a parameter;
An image processing apparatus characterized by using a value input by the input means instead of or selectively using an attribute calculation means. In claim 1,
A human flow monitoring system in which an image generated by the image generation means is input to a face recognition engine and a corresponding person is detected by calculating a similarity with a face image registered in advance. In an image processing method for processing an image input from a camera,
Storing information of a three-dimensional shape model of an average human head composed of wireframes;
Storing an input image input from the camera;
A facial part extraction step for detecting the position, shape and characteristics of a facial organ from the input image;
A simple model generation step of deforming the three-dimensional shape model based on the face part position obtained from the face part extracting means and generating a three-dimensional shape model suitable for a human face part arrangement included in the input image; ,
An attribute calculation step of calculating a person attribute from the input image;
Based on the person attribute obtained from the attribute calculating means, an attribute reflecting step for generating a detailed model by deforming the three-dimensional shape model obtained from the simple model generating means,
An image processing method comprising: an image generation step of generating a two-dimensional image from the output detailed model.   A program for causing a computer to execute the image processing method according to claim 5. The image processing method according to claim 5,
The facial organ includes at least one of eyes, nose, mouth, eyebrows, mole, outer ear, hairline, etc.
The person attribute includes at least one of a face width, a head size, a head outline shape, an age, a gender, a physique, and a height. The image processing method according to claim 5,
An image processing method, wherein the person attribute is manually input as an arbitrary parameter.

Images