WO2017114285A1

WO2017114285A1 - Eye recognition method and system

Info

Publication number: WO2017114285A1
Application number: PCT/CN2016/111515
Authority: WO
Inventors: 冯亮; 尹亚伟; 蔡子豪
Original assignee: 中国银联股份有限公司
Priority date: 2015-12-30
Filing date: 2016-12-22
Publication date: 2017-07-06
Also published as: CN105590103B; CN105590103A; TWI641999B; TW201727540A

Abstract

The present invention provides an eye recognition method, comprising: a) acquiring a facial image of a user; b) in the acquired facial image, marking out a rectangle containing a facial contour, the rectangle being a rectangular image containing the facial contour; c) recording the coordinates of the marked out rectangular image in a display system; d) with respect to the marked out rectangular image, making a correction based on the symmetry and projection amplitude of the facial image, to obtain a corrected facial image; and e) based on the corrected facial image and a recorded position, recognizing the position of the eye.

Description

Eyeball recognition method and system

Technical field

The present invention relates to face recognition technology and, more particularly, to eyeball recognition technology.

Background technique

Eye tracking is mainly to study the acquisition, modeling and simulation of eye movement information. With the widespread use of cameras in mobile phones, notebook computers, PCs, etc., eye tracking has been widely used in live detection, vehicle driver fatigue detection, command control and other scenarios.

Face plane rotation correction is an important part of eye tracking. Many eye movement recognition effects depend on whether the image is rotated or not.

Summary of the invention

In view of this, the present invention provides an eyeball recognition method, including:

a) obtaining a user's face image;

b) dividing, in the acquired facial image, a rectangle containing a contour of a face, the rectangle being a rectangular image containing a contour of the face;

c) recording the coordinates of the divided rectangular image in the display system;

d) performing correction on the divided rectangular image based on the symmetry of the face image and the projection amplitude to obtain a corrected face image;

e) Identifying the eyeball position based on the corrected face image and the recorded position.

An eyeball recognition method according to an example of the present invention, wherein the step d includes:

D1) calculating a center point position o of the rectangular image;

D2) converting the rectangular image into a grayscale image P;

D3) in the grayscale image, the plurality of sub-rectangular images q _i are divided by at least different proportions, wherein each sub-rectangular image q _i is centered on the center point, and the ratios are all less than 1, and i is greater than An integer of 1;

D4) rotating each sub-rectangular image q _i around a center point in a plane of the rectangular image by a certain angle α;

D5) For each sub-rectangle image, projecting in the longitudinal direction to obtain a longitudinal direction projection curve f(x), and calculating a peak gray value g max(q _i ) of the projection curve f(x), a trough gray value g max (q _i );

D6) calculating the symmetry Sym(q _i ) for each sub-rectangle image q _i ;

D7) for each sub-rectangle image q _i , respectively calculate h(q _i )=gmax(q _i )−β·gmin(q _i )+η·Sym(q _i ), where β and η are preset parameters, Both are positive numbers; β and η can be set according to the characteristics of the picture, and the larger their values, the greater the weight of the items multiplied by them;

D8) accumulating the h(q _i ) values of the respective sub-rectangle images q _i to obtain an accumulated h value at the rotation α angle;

D9) transforming the magnitude of the rotation angle α within the angular range of (α1, α2), and sequentially performing steps d4 to d8 to obtain h values at a plurality of rotation angles;

D10) The largest h value is selected from a plurality of h values at a plurality of rotation angles, and the image corresponding to the rotation angle corresponding to the h value is a corrected image.

An eyeball recognition method according to an example of the present invention, wherein the step d6 comprises:

For each rectangular image q _i , projecting in the longitudinal direction to obtain a projection curve g(y) in the direction;

When the center of symmetry is in the range of [1/4w, 1/2w], the symmetry intervals are (0, c) and (c, 2c), respectively, where w is the width of the rectangular image p, c is the center of symmetry, then Sym ( q _i , c)=Σ|g(y)-g(2c-y)|, where y is in the range of (0, c);

When the symmetry center c is in the range of [1/2w, 3/4w], the symmetry intervals are (2c-w, c) and (c, w), respectively.

Sym(q _i )=Σ|g(y)-g(2c-y)|, where y is in the range of (c, w).

According to an exemplary eyeball recognition method of the present invention, in the step d3, three sub-rectangle images p ₁ , P ₂ and P ₃ are divided in three different ratios.

According to still another aspect of the present invention, an eyeball recognition system is further provided, the system comprising:

a first unit, configured to acquire a facial image of the user;

a second unit, configured to divide, in the acquired facial image, a rectangle including a contour of a human face, the rectangle being a rectangular image including a contour of the human face;

a third unit for recording coordinates of the divided rectangular image in the display system;

a fourth unit, configured to perform correction on the divided rectangular image based on the symmetry of the face image and the projection amplitude to obtain the corrected face image;

The fifth unit is configured to recognize the position of the eyeball based on the corrected face image and the recorded position.

An eyeball recognition system according to an example of the present invention, wherein the fourth unit comprises:

a first subunit for calculating a center point position of the rectangular image;

a second subunit, configured to convert the rectangular image into a grayscale image P;

a third subunit, configured to divide a plurality of sub-rectangular images q _i in at least different proportions in the rectangular image, wherein each sub-rectangular image q _i is centered on the center point, and the ratios are all less than 1 , i is an integer greater than one;

a fourth subunit, configured to rotate each sub-rectangular image q _i by a certain angle α around a center point in a plane of the rectangular image;

The fifth sub-unit is configured to project a length of the sub-rectangle image into a longitudinal direction projection curve f(x), and calculate a peak gray value g max(q _i ) of the projection curve f(x), a trough Gray value g max(q _i );

a sixth subunit, configured to calculate a symmetry Sym(q _i ) for each sub-rectangular image q _i ;

a seventh subunit, configured to calculate h(q _i )=gmax(q _i )−β·gmin(q _i )+η·Sym(q _i ) for each sub-rectangular image q _i ;

An eighth subunit for accumulating h(q _i ) values of each sub-rectangular image q _i to obtain an accumulated h value at a rotation α angle;

a ninth subunit for transforming the magnitude of the rotation angle α within an angular range of (α1, α2), and transmitting the converted angle to the fourth subunit, and sequentially operating from the fourth subunit to the eighth subunit h value at multiple rotation angles;

The tenth subunit is configured to select a maximum h value from a plurality of h values at a plurality of rotation angles, and the image corresponding to the rotation angle corresponding to the h value is a corrected image.

DRAWINGS

1 is a flow chart of an eyeball recognition method according to an example of the present invention.

Figure 2 shows a flow chart of step 14 in Figure 1.

Figure 3 illustrates a third sub-image is a schematic illustration of the q o [alpha] ₃ angle of rotation about the center point.

4 is a schematic structural view of the eyeball recognition system.

detailed description

An illustrative example of the present invention will now be described with reference to the drawings. The same reference numerals denote the same elements. The embodiments described below are intended to provide a thorough understanding of the invention, and are intended to be illustrative and not limiting. Unless otherwise defined, terms (including scientific, technical, and industrial terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. In addition, the order of the steps in the flowchart is not limited to the order illustrated.

In this paper, both the image and the image represent the user obtained by the image acquisition component such as a camera. Images and images obtained after processing based on the images, images and images are used interchangeably herein.

1 is a flow chart of an eyeball recognition method according to an example of the present invention. Briefly, according to the method shown in FIG. 1, the user's face image is first acquired, and then processed to obtain a corrected image, the position of the eyeball is confirmed in the corrected image, and finally the original is determined based on the confirmed eyeball position. The position of the eyeball in the user's face image.

At step 10, a user's face image is acquired. The user's face image can be acquired by an image acquisition component such as a camera.

In step 12, in the acquired facial image, a rectangle containing a contour of the face is divided, which is a rectangular image containing the contour of the face. The divided rectangular image includes at least the facial features of the person. The division can be done by dividing the existing pattern recognition method.

At step 14, the coordinates of the divided rectangular image in the display system are recorded. The displayed image has a coordinate position in the real device, and by way of example, the coordinate position can be recorded.

At step 16, for the divided rectangular image, based on the symmetry of the face image and the projection amplitude, correction is performed to obtain a corrected face image.

At step 18, the eyeball position is identified based on the corrected face image and the recorded position. After the step of identifying the position of the eyeball, the position of the eyeball in the original image can be determined correspondingly in conjunction with the coordinate position recorded in step 14.

As an example, Figure 2 shows a flow chart of step 14 in Figure 1.

As shown, at step 140, the center point o position of the rectangular image is calculated.

At step 142, the rectangular image is converted to a grayscale image P.

In step 144, in the grayscale image, a plurality of sub-rectangular images q _i are divided in at least different proportions, wherein each sub-rectangular image q _i is centered on the center point, and the ratios are all less than 1, i Is an integer greater than 1. As an example, three sub-rectangle images are respectively divided in proportions of 0.5, 0.6, and 0.7, which are referred to as a first sub-image q ₁ , a second sub-image q _{2 ,} and a third sub-image q ₃ , respectively, in the following examples. .

At step 146, each sub-rectangular image q _{i is} rotated by a certain angle α around the center point o in the plane of the rectangular image. For example, the first sub-image q _{1 is} rotated by an angle α around the center point o, the first sub-image q _{2 is} rotated by an angle α around the center point o, and the first sub-image q _{3 is} rotated by an angle α around the center point o.

In step 148, the sub-rectangle image is projected to the length direction thereof to obtain a longitudinal direction projection curve f(x), and the peak gray value g max(q _i ) and the trough gray value of the projection curve f(x) are calculated. g max(q _i ). Figure 3 illustrates a third sub-image is a schematic illustration of the q o [alpha] ₃ angle of rotation about the center point. As shown in the figure, the rectangular image q has a length w and a width h. Here, in particular, in the example of the present invention, the length of the side of the rectangular image q along the x-axis direction of the display screen is taken as the length side. The length of the side along the y-axis direction of the display screen is defined as the width side. However, this is only an illustration, and the length in the x-axis direction may be used as the width side, and the length of the side along the y-axis direction of the display screen may be used as the height side. The third sub-image q _{3 has} a length w' and a width h'. Projecting the third sub-image q ₃ in the direction of its length side to obtain a projection curve f(x), calculating a peak gray value g max(q _s ) of the projection curve f(x), and a trough gray value g max ( q _s ).

At step 150, the symmetry Sym(q _i ) is calculated for each sub-rectangle image q _i . For each sub-image q _i rotated around the center o, the left and right have symmetry in accordance with the vertical line of the face center. Naturally, we calculate the symmetry value Sym(q _i ) of each candidate image q _i to measure the symmetry of the face. At the same time, in the image, it is impossible to accurately know the position of the center line of the face. Therefore, the system sets the symmetric center c one by one to a range of 1/4w to 3/4w, and calculates the symmetry value of the picture of the symmetric center c. Sym (q _i , c), pick the largest value, as the symmetry value of the picture Sym (q _i , c). Here, it should be understood that Sym(q _i , c) represents Sym(q _i ) obtained with the center of symmetry c as the center of symmetry. Sym(q _i ,c) is calculated as follows:

For each rectangle q, project in the y-axis direction (parallel to the length side) to obtain a y-axis gray value projection curve x=g(y);

When the symmetry center c is in the range of [1/4w, 1/2w], the symmetry intervals are (0, c) and (o, 2c) Sym (q _i , c) = Σ | g (y) - g ( 2c-y)|, where y falls within the range of (0, c);

When the symmetry center c is in the range of [1/2w, 3/4w], the symmetry intervals are (2c-w, c) and (c, w), respectively, and Sym(q _i , c) = Σ | g (y) ) -g(2c-y)|, where y falls within the range of (c, w).

Subsequently, in step 152, h(q _i )=gmax(q _i )−β·gmin(q _i )+η·Sym(q _i ) is calculated for each sub-rectangle image q _i . For example, for the first sub-image q ₁ , h(q ₁ )=gmax(q ₁ )−β·gmin(q ₁ )+η·Sym(q ₁ ,c) is calculated; for the second sub-image q ₂ , Calculate h(q ₂ )=gmax(q ₂ )−β·gmin(q ₂ )+η·Sym(q ₂ ,c); q ₁ and calculate h(q ₃ )=gmax for the third sub-image q ₃ ( q ₃ )-β·gmin(q ₃ )+η·Sym(q ₃ ,c).

At step 154, the h(q _i ) values of the respective sub-rectangle images q _i are accumulated to obtain an accumulated h value at the rotation α angle. Illustratively, the accumulated h is the sum of h(q ₁ ), h(q ₂ ), and h(q ₃ ).

In step 156, the magnitude of the rotation angle α is changed within the angular range of (α1, α2), and Steps 146 through 154 are performed a second to obtain h values at a plurality of rotation angles.

At step 158, the largest h value is selected from the h value obtained in step 154 and the plurality of h values obtained in step 156. The sub-image having the largest h value is the selected corrected image.

After obtaining the corrected image according to the procedure shown in Fig. 2, for example, the position of the eyeball in the corrected image can be known. Further, based on the position and the recorded coordinates of the divided rectangular image in the display system, the eyeball in the user's face image can be identified.

The eyeball recognition method according to various examples of the present invention can be implemented as a software module incorporated into an existing face recognition module or device. Alternatively, it can also be implemented as a combination of software and hardware, or only by hardware.

According to the present invention, an eyeball recognition system is also provided. 4 is a schematic structural view of the eyeball recognition system. As shown, the eyeball recognition system includes a first unit 50, a second unit 52, a third unit 54, a fourth unit 56, and a fifth unit 58.

The first unit 50 is configured to acquire a user facial image, which may be, for example, an image capturing component such as a camera.

The second unit 52 divides a rectangle including a face contour in the acquired face image, and the rectangle is a rectangular image including a face contour. The divided rectangular image includes at least the facial features of the person. The division can be done by dividing the existing pattern recognition method.

The third unit 54 records the coordinates of the divided rectangular image in the display system. The displayed image has a coordinate position in the real device, and by way of example, the coordinate position can be recorded.

The fourth unit 56 performs correction based on the symmetry of the face image and the projection amplitude for the divided rectangular image to obtain a corrected face image.

The fifth unit 58 identifies the eyeball position based on the corrected face image and the recorded position. After the position of the eyeball is recognized, the position of the eyeball in the original image can be determined correspondingly in combination with the recorded coordinate position.

The fourth unit 56 can further include a plurality of subunits. The first sub-unit calculates the center point o position of the rectangular image. The second sub-unit converts the rectangular image into a grayscale image P. The third subunit divides a plurality of sub-rectangular images q _i in at least different proportions in the gray scale image, wherein each sub-rectangular image q _i is centered on the center point, and the ratios are all less than 1, i Is an integer greater than 1. As an example, three sub-rectangle images are respectively divided in proportions of 0.5, 0.6, and 0.7, and in the following examples, they are referred to as a first sub-image q ₁ , a second sub-image q _{2 ,} and a third sub-image q _{3 , respectively.} .

The fourth sub-unit rotates each sub-rectangle image q _i by a certain angle α around the center point o in the plane of the rectangular image. For example, the first sub-image q _{1 is} rotated by an angle α around the center point o, the first sub-image q _{2 is} rotated by an angle α around the center point o, and the first sub-image q _{3 is} rotated by an angle α around the center point o.

The fifth sub-unit performs projection on the length direction of each sub-rectangle image to obtain a longitudinal direction projection curve f(x), and calculates a peak gray value g max(q _i ) and a trough gray value of the projection curve f(x). g max(q _i ). Figure 3 illustrates a third sub-image is a schematic illustration of the q o [alpha] ₃ angle of rotation about the center point. As shown in the figure, the rectangular image q has a length w and a width h. Here, in particular, in the example of the present invention, the length of the side of the rectangular image q along the x-axis direction of the display screen is taken as the length side. The length of the side along the y-axis direction of the display screen is defined as the width side. However, this is only an illustration, and the length in the x-axis direction may be used as the width side, and the length of the side along the y-axis direction of the display screen may be used as the height side. The third sub-image q _{3 has} a length w' and a width h'. Projecting the third sub-image q ₃ in the direction of its length side to obtain a projection curve f(x), calculating a peak gray value g max(q _s ) of the projection curve f(x), and a trough gray value g max ( q _s ).

The sixth sub-unit calculates the symmetry Sym(q _i ) for each sub-rectangle image q _i . For each sub-image q _i rotated around the center o, the left and right have symmetry in accordance with the vertical line of the face center. Naturally, we calculate the symmetry value Sym(q _i ) of each candidate image q _i to measure the symmetry of the face. At the same time, in the image, it is impossible to accurately know the position of the center line of the face. Therefore, the system sets the symmetric center c one by one to a range of 1/4w to 3/4w, and calculates the symmetry value of the picture of the symmetric center c. Sym (q _i , c), pick the largest value, as the symmetry value of the picture Sym (q _i , c). Here, it should be understood that Sym(q _i , c) represents Sym(q _i ) obtained with the center of symmetry c as the center of symmetry. Sym(q _i ,c) is calculated as follows:

The seventh sub-unit calculates h(q _i )=gmax(q _i )−β·gmin(q _i )+η·Sym(q _i ) for each sub-rectangle image q _i . For example, for the first sub-image q ₁ , h(q ₁ )=gmax(q ₁ )−β·gmin(q ₁ )+η·Sym(q ₁ ,c) is calculated; for the second sub-image q ₂ , Calculate h(q ₂ )=gmax(q ₂ )−β·gmin(q ₂ )+η·Sym(q ₂ ,c); q ₁ and calculate h(q ₃ )=gmax for the third sub-image q ₃ ( q ₃ )-β·gmin(q ₃ )+η·Sym(q ₃ ,c).

The eighth subunit accumulates the h(q _i ) values of the respective sub-rectangle images q _i to obtain an accumulated h value at the rotation α angle. Illustratively, the accumulated h is the sum of h(q ₁ ), h(q ₂ ), and h(q ₃ ).

The ninth subunit changes the magnitude of the rotation angle α within the angular range of (α1, α2), and sequentially performs steps 146 to 154 to obtain h values at a plurality of rotation angles.

The tenth subunit selects the largest h value from the h value obtained in step 154 and the plurality of h values obtained in step 156. The sub-image having the largest h value is the selected corrected image.

After the corrected image is obtained, the position of the eyeball in the corrected image can be known. Further, based on the position and the recorded coordinates of the divided rectangular image in the display system, the eyeball in the user's face image can be identified.

An eyeball recognition system such as the example of the present invention can be implemented by software, incorporated into an existing face recognition module or device. Alternatively, it can also be implemented as a combination of software and hardware, or only by hardware.

Although the specific embodiments of the present invention have been disclosed in the foregoing description, the embodiments of the present invention may be modified or modified without departing from the spirit of the invention. modify. The embodiments of the present invention are intended to be illustrative only and not to limit the invention.

Claims

An eyeball recognition method, characterized in that the method comprises:

a) obtaining a user's face image;

b) dividing, in the acquired facial image, a rectangle containing a contour of a face, the rectangle being a rectangular image containing a contour of the face;

c) recording the coordinates of the divided rectangular image in the display system;

d) performing correction on the divided rectangular image based on the symmetry of the face image and the projection amplitude to obtain a corrected face image;

e) Identifying the eyeball position based on the corrected face image and the recorded position.
The eyeball recognition method according to claim 1, wherein the step d comprises:

D1) calculating a center point position o of the rectangular image;

D2) converting the rectangular image into a grayscale image P;

D3) in the grayscale image, the plurality of sub-rectangular images q i are divided by at least different proportions, wherein each sub-rectangular image q i is centered on the center point, and the ratios are all less than 1, and i is greater than An integer of 1;

D4) rotating each sub-rectangular image q i around a center point in a plane of the rectangular image by a certain angle α;

D5) For each sub-rectangle image, projecting in the longitudinal direction to obtain a projection projection curve f(x) in the longitudinal direction, and calculating a peak gray value g max(q i ) of the projection curve f(x), a trough gray value g min (q i );

D6) calculating the symmetry Sym(q i ) for each sub-rectangle image q i ;

D7) for each sub-rectangle image q i , respectively calculate h(q i )=gmax(q i )−β·gmin(q i )+η·Sym(q i );

D8) accumulating the h(q i ) values of the respective sub-rectangle images q i to obtain an accumulated h value at the rotation α angle;

D9) transforming the magnitude of the rotation angle α within the angular range of (α1, α2), and sequentially performing steps d4 to d8 to obtain h values at a plurality of rotation angles;

D10) The largest h value is selected from a plurality of h values at a plurality of rotation angles, and the image corresponding to the rotation angle corresponding to the h value is a corrected image.
The eyeball recognition method according to claim 2, wherein the step d6 comprises:

For each rectangular image q i , projecting in the longitudinal direction to obtain a projection curve g(y) in the direction;

When the center of symmetry is in the range of [1/4w, 1/2w], the symmetry intervals are (0, c) and (c, 2c), respectively, where w is the width of the rectangular image p, c is the center of symmetry, then Sym ( q i , c)=Σ|g(y)-g(2c-y)|, where y is in the range of (0, c);

When the symmetry center c is in the range of [1/2w, 3/4w], the symmetry intervals are (2c-w, c) and (c, w), respectively, then Sym(q i ) = Σ | g (y) - g(2c-y)|, where y is in the range of (c, w).
The eyeball recognition method according to claim 2, wherein in step d3, three sub-rectangle images p 1 , P 2 and P 3 are divided in three different ratios.
An eyeball recognition system, characterized in that the system comprises:

a first unit, configured to acquire a facial image of the user;

a second unit, configured to divide, in the acquired facial image, a rectangle including a contour of a human face, the rectangle being a rectangular image including a contour of the human face;

a third unit for recording coordinates of the divided rectangular image in the display system;

a fourth unit, configured to perform correction on the divided rectangular image based on the symmetry of the face image and the projection amplitude to obtain the corrected face image;

The fifth unit is configured to recognize the position of the eyeball based on the corrected face image and the recorded position.
The eyeball recognition system of claim 5 wherein said fourth unit comprises:

a first subunit for calculating a center point position of the rectangular image;

a second subunit, configured to convert the rectangular image into a grayscale image P;

a third subunit, configured to divide a plurality of sub-rectangular images q i in at least different proportions in the rectangular image, wherein each sub-rectangular image q i is centered on the center point, and the ratios are all less than 1 , i is an integer greater than one;

a fourth subunit, configured to rotate each sub-rectangular image q i by a certain angle α around a center point in a plane of the rectangular image;

The fifth sub-unit is configured to project a length of the sub-rectangle image into a longitudinal direction projection curve f(x), and calculate a peak gray value g max(q i ) of the projection curve f(x), a trough Gray value g min(q i );

a sixth subunit, configured to calculate a symmetry Sym(q i ) for each sub-rectangular image q i ;

a seventh subunit, configured to calculate h(q i )=gmax(q i )−β·gmin(q i )+η·Sym(q i ) for each sub-rectangular image q i ;

An eighth subunit for accumulating h(q i ) values of each sub-rectangular image q i to obtain an accumulated h value at a rotation α angle;

a ninth subunit for transforming the magnitude of the rotation angle α within an angular range of (α1, α2), and transmitting the converted angle to the fourth subunit, and sequentially operating from the fourth subunit to the eighth subunit h value at multiple rotation angles;

The tenth subunit is configured to select a maximum h value from a plurality of h values at a plurality of rotation angles, and the image corresponding to the rotation angle corresponding to the h value is a corrected image.