KR101875966B1 - A missing point restoration method in face recognition for vehicle occupant - Google Patents

Abstract

The present invention relates to a method for restoring missing feature points in the case where some feature points are lost in the process of recognizing a face of a vehicle occupant, and more particularly, to a feature point extraction method for extracting one or more feature points for an occupant's face from an image signal captured by a camera ; A distance ratio information calculation step of calculating a distance ratio between the extracted minutiae points; A minutia information storing and managing step of continuously storing and managing valid information among the information calculated in the distance ratio information calculating step in a memory; Comparing the information calculated in the distance ratio information calculation step with the information stored in the memory and checking whether there is a missing error in the minutia information extracted in the minutiae point extraction step based on the comparison result, Minutiae identification step; As a result of checking, if the lost feature point information is confirmed, the distance ratio information between the lost feature points and the associated linked points is read from the memory, and based on the read out information, A lost feature point estimation step of calculating position information of feature points; And a feature point reconstruction step of reconstructing a set of feature point information including feature points lost based on the calculated feature point position information.
According to the present invention configured as described above, it is possible to shorten the time required for reconstructing lost minutia points, reduce the amount of computation, and reduce the load of the processor.

Description

TECHNICAL FIELD [0001] The present invention relates to a method for restoring missing feature points in a vehicle occupant's face tracking process,

BACKGROUND OF THE INVENTION 1. Field of the Invention [0002] The present invention relates to face recognition of an occupant in a vehicle, and in particular, when a part of a minutiae is not extracted or an error occurs due to an obstacle such as a passenger's face being obstructed And extracting the extracted minutia information based on a ratio of distance between related minutiae and linked minutiae points.

Among the recently developed vehicle technologies, the vehicle system automatically recognizes the driver's face, guides the destination of the navigation to the route frequently operated by the driver, automatically adjusts the seat of the driver's seat to the driver's physical condition, There are various forms of assistant functions or convenience functions according to recognition. In addition, for various other safety assist functions or convenience functions, the vehicle also needs to constantly monitor where the driver's gaze is watching during driving, The user extracts a plurality of feature points such as eyes, nose, and mouth from the face of the user and performs a process of continuously tracking the driver's face through the extracted feature points.

On the other hand, in the feature point extracting process for identifying the driver or the feature point extracting process for facial tracking, the system of the vehicle detects a feature point from the driver's face for reasons such as a part of the driver's face is blocked by an obstacle, If the extraction fails, the driver should not be provided with the functions to be provided smoothly, which may cause the driver to feel uncomfortable. In addition, if the obstacle occurs in the facial recognition information provided to the driving assistance device, do.

Meanwhile, the basic facial tracking technique is performed in three stages of (1) facial detection, (2) feature point detection, and (3) facial tracking. In the conventional technique, when an error occurs in the driver's face tracking process, So that it takes a relatively long time to recover it.

In the past, the optical flow / Lucas-Kanade theory was mainly used in the facial tracking process because it is designed to use a designed algorithm based on the speed and direction of the tracked object during facial tracking, The conventional method of estimating lost feature points based on the distance between feature points is more effective in reducing the distance between the face of the driver and the camera So that there is an inexact problem in the accuracy thereof.

The present invention has been made in view of the above circumstances, and it is an object of the present invention to provide a method and system for extracting a feature point from a feature point extraction process for recognizing a face of a vehicle occupant, And a method of reconstructing lost feature points in a face tracking process of a vehicle occupant in which face recognition of a passenger can be accurately and efficiently performed even when a feature point is lost.

In order to achieve the above object, a lost feature point restoration method according to the present invention is a restraining face recognition method for recognizing a face of a vehicle occupant using a camera and at least one processor, A feature point extracting step of extracting at least one feature point with respect to a face; A distance ratio information calculation step of calculating a distance ratio between the extracted minutiae points; A minutia information storing and managing step of continuously storing and managing valid information among the information calculated in the distance ratio information calculating step in a memory; Comparing the information calculated in the distance ratio information calculation step with the information stored in the memory and checking whether there is a missing error in the minutia information extracted in the minutiae point extraction step based on the comparison result, Minutiae identification step; As a result of checking, if the lost feature point information is confirmed, the distance ratio information between the lost feature points and the associated linked points is read from the memory, and based on the read out information, A lost feature point estimation step of calculating position information of feature points; And a feature point reconstruction step of reconstructing a set of feature point information including feature points lost based on the calculated feature point position information.

According to another aspect of the present invention, there is provided a lost feature point restoration method, wherein the lost feature point estimation step comprises: calculating a ratio of the distance from the associated feature points by a ratio of the distance from the associated feature points, And the position information of the intersection close to the position confirmed before the corresponding feature point is lost in the intersection of the formed circle is estimated as the position information of the lost feature point .

The lost feature point restoration method in the vehicle occupant's face tracking process according to the present invention is characterized in that the information stored in the memory updates and stores the values of the one or more users that are changed according to the face angle, And the lost feature point estimating step calculates the current face angle of the driver based on the information stored in the memory and calculates the distance ratio information between the linked points according to the calculation result Is read out.

Also, the feature point restoration method according to the present invention is characterized in that the linked feature points are one or more feature points in which the distances to the lost feature points are calculated and managed.

In the method of reconstructing a lost feature point in a vehicle occupant's face tracking process according to the present invention, creation of a database managed in the memory is performed based on information generated at the time of recognizing the front face of the driver at the beginning and a 3D model ).

That is, according to the present invention having the above-described configuration, it is possible to restore lost feature points based on the ratio between feature points, instead of estimating feature points lost based on the direction, speed, and distance between feature points It is possible to always restore consistent and precise feature points irrespective of the distance between the occupant's face and the camera, and to restore the feature points immediately in the face tracking process by avoiding the conventional method of restarting facial detection in the case of facial tracking failure The restoration time can be shortened and the amount of computation can be reduced, thereby reducing the load of the processor.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing a configuration of a system having a lost feature point restoration function according to an embodiment of the present invention; FIG.
FIG. 2 is a flowchart for explaining the operation of the system having the configuration of FIG. 1; FIG.
3 is a diagram for explaining a process of extracting missing feature points from an occupant's face image;
4 is a diagram for explaining a process of calculating the distance between minutia points and the distance ratio;
5 is a view for explaining a change in the distance between minutiae according to the passenger's face angle.
FIG. 6 is a diagram for explaining a lost point estimation process; FIG.

Hereinafter, embodiments according to the present invention will be described with reference to the accompanying drawings.

FIG. 1 illustrates the structure of a system equipped with a lost feature point restoration function in the vehicle occupant's face tracking process according to an embodiment of the present invention. The function according to the present invention is applied to a vehicle navigation system 71 and an IMS Integrate Memory Seat) 72, a driving safety support system 73, and the like. In FIG. 1, reference numeral 10 denotes a camera installed on the driver's cab of the vehicle for photographing the driver's face 100, and may be implemented to be operated when the driver boarded the vehicle. However, Or may be implemented so as to photograph the face image of the driver at a time required by the system as required.

In the present embodiment, it is exemplified that the camera 10 photographs the driver's face, but it may be used that the driver 10 is photographed on the face of a passenger other than the driver, A plurality of cameras for passengers other than the camera may be additionally provided.

Reference numeral 20 denotes a video signal processing unit for processing the video signal picked up by the camera 10 and converting it into a digital signal and outputting it. Reference numeral 60 denotes a control unit for controlling the overall operation of the system. The MCU (Main Control Unit) May be used, but a processor associated with image processing of the camera may be used or may be implemented by other local processors.

Next, the operation of the system configured as described above will be described with reference to the flowchart of Fig.

When the camera 10 is driven, a video signal of the driver's face is processed by the video signal processing unit 20 and output to the facial feature point extracting unit 30 as a digital signal. The facial feature point extraction unit 30 checks the face contour of the driver from the digital image signal output from the image signal processing unit 20 and extracts the feature point A from the driver's eyes, nose, mouth, , B, C, D, E, F) are extracted (ST 1)

At this time, the detection of each feature point can be detected using a search algorithm and can be obtained by generating a 3D model from the initial image recognized by the driver. On the other hand, the six feature points as in the present embodiment are merely examples, and more or less number of feature points may be extracted and used.

The feature point information extracted by the facial feature point extraction unit 30 is applied to the feature point distance ratio calculation unit 40. As a result, the feature point distance ratio calculation unit 40 calculates the distance between each feature point The distance b between A and B (between two pupils), the distance b between A and C (between the left pupil and the nose of the nose), the distance between B and D (between the right pupil and the right end of the nose) b2 , Distance c between C and D (between the nose of the nose), distance d1 between A and E (between the left pupil and the mouth's left end), distance between B and F (between the right pupil and the right end of the mouth) d2 Distance e1 between C and E (between the left end of the nose and the left end of the mouth), distance e2 between D and F (between the right end of the nose and the right end of the mouth), between E and F ) Are calculated. Then, based on the distance information between the respective minutiae points calculated by the above-mentioned procedure, the ratio of the distance between each minutiae points Is calculated. (ST 2)

B2, c, d1, d2, e1, e2, and f are obtained when the ratio a of each distance is set as a reference (a = 1) in the present embodiment In other embodiments, the distance between the feature points other than a may be used as a reference, and the distance between the remaining feature points may be converted as a relative ratio.

The control unit 50 stores the distance information between the minutiae points calculated by the above process in the minutia information storage unit 60 in the form of a database and manages the information. When the distance ratio information between the other minutiae points is inputted, the minutiae information is stored in the minutia information storage 60 as information about the new user and stored.

That is, when the feature point distance ratio information for the driver is input from the feature point distance ratio calculating unit 40, the control unit 50 compares the feature point distance ratio information with the existing information stored in the feature point information storage unit 60 (Step ST3). If the comparison result matches any of the registered ones, the corresponding user is recognized as the matching user, and the previously registered information is updated (ST4) . If there is no match in the previously registered information, the new user is recognized as a new user and the corresponding minutia information is newly registered on the minutia information storage 60 (ST 6).

Through the above process, the control unit 50 continuously updates the information stored in the minutia information storage unit 60 and manages and manages the information stored in the minutia information storing unit 60, thereby maintaining more accurate reference values through rich data and various experience values.

The driver's face is not a plane but a three-dimensional structure. Therefore, when the driver's face is out of the front-facing state, The distance between each feature point is different from the measured information when the driver looks at the front side and the distance between each feature point and the ratio of the distance is changed according to the angle of inclination of the driver's face.

In consideration of this feature, the controller 50 stores the information stored in the minutia information storing unit 60 so that the values of the same user, which vary according to the face angle of the driver, are updated and stored at any time, Information is managed in the form of a database. The creation of the first database may be based on the information generated at the initial recognition of the driver's frontal face and the projection of the 3D model based on the generated information. In addition, the information stored in the minutia information storage unit 60 may be stored individually for each of a plurality of drivers using the vehicle.

Meanwhile, the controller 50 can continuously perform facial recognition and facial tracking of the driver until the operation of the vehicle is finished, if necessary, and the results can be transmitted to the navigation system 71 or the IMS 72, The safety support system 73 or the like.

In this case, the control unit 50 checks whether the function termination is selected (ST (ST)) or not (step ST). In this case, 7), the facial recognition or facial tracking operation according to the present invention ends.

On the other hand, during execution of facial recognition or face tracking, the driver's face is partially obscured by the obstacle or fails to extract the feature points from the driver's face under the influence of strong light rays. As a result, (ST 8), the control unit 50 determines whether or not an error has occurred in the information inputted from the feature point distance ratio calculating unit 40, that is, Point information storage unit 60 to calculate the current face angle of the driver and reads the stored distance ratio information of the feature points corresponding to the calculated angle, The position information of the feature points linked to the missing points is estimated (ST 9). Generally, you can calculate the face angle (Roll, Pitch, Yaw) if you know the information about the three points. On the other hand, the linked feature points refer to feature points related to the calculation of the mutual distance and distance ratio, and refer to one or more feature points in which the distances to the missing feature points are calculated and managed.

For example, referring to the example of FIG. 4, the associated minutiae for minutiae A are minutiae B, C, E related to the distance a, b1, d1 and the ratio value calculation, a, b2, d2 and characteristic points A, D, F related to the calculation of the ratio value.

Assuming a case where the minutiae point A is lost and a process of estimating the position information of the missing point A lost by referring to Fig. 6 will be described in detail. First, the controller 50 calculates a value Information on the ratio of the distance (a, b1, d1) between the associated minutiae at the corresponding angle is read out from the minutia information storage 60. 6, the control unit 50 calculates a set of points a spaced apart by a distance a from which the missing point A should be located, from the extracted associated point B, And a set of points spaced apart by a distance b1 where the lost feature point A should be located from the extracted related feature point C (a circle having a radius b1) (Indicated by a circle with d1 as a radius, denoted by 'd') separated by a distance d1 from which the minutiae point A lost from the associated minutiae point E should exist, Will be selected by the third party.

As a result, three circles ('a', 'b', 'd') are represented by the above process, and these three circles have two intersection points. The controller 50 can accurately estimate the positional information of the missing feature point A by selecting a point close to the position that was confirmed immediately before the feature point A was lost among the two intersection points.

As a result, the control unit 50 restores the lost feature point A based on the positional information of the feature point A calculated by the above process, and outputs the restored information to the navigation system 71, the IMS 72, System 73 or the like (ST 10)

In other words, according to the present invention, it is possible to quickly and accurately restore lost feature points based on the ratio of distances between feature points, thereby making it possible to always perform constant and accurate feature point restoration regardless of the distance between the occupant's face and the camera By restoring the feature points directly from the face tracking process, it is possible to shorten the time required for restoration and reduce the load of the processor by reducing the time required for restoration It is possible to provide a feature point restoration function.

The present invention is not limited to the above-described embodiments, and various modifications may be made without departing from the technical scope of the present invention. For example, in the above-described embodiment, the function according to the present invention is implemented when a feature point is lost during facial tracking. However, regardless of whether the face tracking process is performed or not, It may be applied to the process of executing facial recognition for confirmation.

In addition, the respective steps (ST 1 to ST 9) in the above embodiment are disclosed as only one example, and the order of execution of each step may be changed as necessary, The steps may be omitted depending on the situation.

If an error occurs in the feature point extraction process, the process of correcting the error is performed by the control unit 50 continuously updating the result output from the feature point distance ratio calculation unit 40 with the information stored in the feature point information storage unit 60 A function of correcting the feature point extraction error by correcting the error based on the information of the feature point information storage unit 60 can be implemented.

10: camera 20: video signal processor
30: facial feature point extracting unit 40: feature point distance ratio calculating unit
50: control unit 60: minutia point information storage unit
71: navigation system 72: IMS (Integrate Memory Seat)
73: Driving safety support system 100: Driver's face

Claims (5)

A passenger facial recognition method for recognizing a face of a vehicle occupant using a camera and one or more processors, the method comprising:
A feature point extracting step of extracting at least one feature point of a passenger's face from an image signal photographed by a camera;
A distance ratio information calculation step of calculating a distance ratio between the extracted minutiae points;
A minutia information storing and managing step of continuously storing and managing valid information among the information calculated in the distance ratio information calculating step in a memory;
Comparing the information calculated in the distance ratio information calculation step with the information stored in the memory and checking whether there is a missing error in the minutia information extracted in the minutiae point extraction step based on the comparison result, Minutiae identification step;
As a result of checking, if the lost feature point information is confirmed, the distance ratio information between the lost feature points and the associated linked points is read from the memory, and based on the read out information, A lost feature point estimation step of calculating position information of feature points; And
And a feature point reconstruction step of reconstructing a set of feature point information including feature points lost based on the calculated feature point position information,
Wherein the associated feature points are at least one feature point in a relationship in which a distance to the missing feature points is calculated and managed. A passenger facial recognition method for recognizing a face of a vehicle occupant using a camera and one or more processors, the method comprising:
A feature point extracting step of extracting at least one feature point of a passenger's face from an image signal photographed by a camera;
A distance ratio information calculation step of calculating a distance ratio between the extracted minutiae points;
A minutia information storing and managing step of continuously storing and managing valid information among the information calculated in the distance ratio information calculating step in a memory;
Comparing the information calculated in the distance ratio information calculation step with the information stored in the memory and checking whether there is a missing error in the minutia information extracted in the minutiae point extraction step based on the comparison result, Minutiae identification step;
As a result of checking, if the lost feature point information is confirmed, the distance ratio information between the lost feature points and the associated linked points is read from the memory, and based on the read out information, A lost feature point estimation step of calculating position information of feature points; And
And a feature point reconstruction step of reconstructing a set of feature point information including feature points lost based on the calculated feature point position information,
The lost feature point estimation step may include a step of constructing a circle having a distance of a distance from the associated feature points by a ratio of the distance from the associated feature points according to the distance ratio information between feature points read out from the memory And the location information of the intersection close to the position confirmed before the corresponding feature point is lost is estimated as the position information of the missing feature point in the intersection of the drawn circle. How to restore feature points. A passenger facial recognition method for recognizing a face of a vehicle occupant using a camera and one or more processors, the method comprising:
A feature point extracting step of extracting at least one feature point of a passenger's face from an image signal photographed by a camera;
A distance ratio information calculation step of calculating a distance ratio between the extracted minutiae points;
A minutia information storing and managing step of continuously storing and managing valid information among the information calculated in the distance ratio information calculating step in a memory;
Comparing the information calculated in the distance ratio information calculation step with the information stored in the memory and checking whether there is a missing error in the minutia information extracted in the minutiae point extraction step based on the comparison result, Minutiae identification step;
As a result of checking, if the lost feature point information is confirmed, the distance ratio information between the lost feature points and the associated linked points is read from the memory, and based on the read out information, A lost feature point estimation step of calculating position information of feature points; And
And a feature point reconstruction step of reconstructing a set of feature point information including feature points lost based on the calculated feature point position information,
Wherein the information stored in the memory is configured to manage the different minutia information for each angle in the form of a database by updating the values of the one or more users,
Wherein the lost feature point estimation step calculates the current face angle of the driver based on the information stored in the memory and reads the distance ratio information between the linked points according to the calculation result. A method for reconstructing missing feature points in a facial tracking process. delete The method of claim 3,
Wherein the generation of the database managed in the memory is generated based on information generated at the time of recognizing the front face of the driver at the initial stage and a projection of the 3D model based on the information generated at the time of recognizing the front face of the driver Of the lost feature points.

Images