CN106933339A

CN106933339A - A kind of eyeball driving method and electronic equipment

Info

Publication number: CN106933339A
Application number: CN201511026866.0A
Authority: CN
Inventors: 周世威
Original assignee: Zhang Ying Information Technology (shanghai) Co Ltd
Current assignee: Zhang Ying Information Technology (shanghai) Co Ltd
Priority date: 2015-12-31
Filing date: 2015-12-31
Publication date: 2017-07-07

Abstract

The invention discloses a kind of eyeball driving method and electronic equipment, belong to image processing field.Method includes：Multiple 2D key point coordinates of face in current video frame is obtained, human face region is determined according to multiple 2D key point coordinates；Human face region is filtered by right and left eyes wave filter, obtains the 2D coordinates of the eyeball position of face in current video frame；It is the 3D coordinates of eyeball position by the 2D Coordinate Conversions of eyeball position according to the transformational relation between multiple 2D key point coordinates and corresponding multiple 3D keys point coordinates；3D coordinates and the 3D coordinates of the initial position of eyeball according to eyeball position, drive the eyeball of destination object.So as to the 3D coordinates according to eyeball position and the 3D coordinates of the initial position of the eyeball, drive the eyeball position of destination object, improve the accuracy and stability of Ins location, simultaneously can be in mobile terminal Effec-tive Function, Consumer's Experience is improve, demand of the user by Rotation of eyeball control targe object is met.

Description

Eyeball driving method and electronic equipment

Technical Field

The present invention relates to the field of image processing, and in particular, to an eyeball driving method and an electronic device.

Background

With the popularization of face recognition in computer technology, the technology of recognizing faces through computers has been applied to various fields, the recognition of eye states in face recognition is of great importance, the eyes are positioned through eye state recognition, and the requirements of users on controlling targets in games through eye recognition and the like can be met.

The prior art human eyeball positioning method mainly comprises the following steps: the method based on geometric features, such as a gray level projection method, Hoffman transformation and the like, firstly calculates the edge contour of a human eye by using a Canny operator in an eye region of interest, and then positions the eyeball by utilizing the Hoffman transformation; statistical-based methods, such as those using neural networks or support vector machines, knowledge-based methods.

However, in the prior art, the method based on the geometric characteristics is very susceptible to the influence of illumination, and is not enough in stability and precision, while the method based on statistics has certain adaptability to illumination, a large number of samples are needed to train a classifier, so that the method applied to the mobile terminal is poor in instantaneity, and meanwhile, the method based on knowledge is used for positioning human eyes by adopting a large number of priori knowledge, and is not enough in universality.

Disclosure of Invention

In order to improve the accuracy and stability of eyeball positioning, enable the method to operate efficiently at a mobile end, improve user experience, and meet the requirements of a user for controlling a target object in a game and the like through eyeball rotation, the embodiment of the invention provides an eyeball driving method and electronic equipment. The technical scheme is as follows:

in a first aspect, an eye ball driving method is provided, the method comprising:

acquiring a plurality of 2D key point coordinates of a face in a current video frame, and determining a face area according to the plurality of 2D key point coordinates;

filtering the face region through a left eye filter and a right eye filter to obtain 2D coordinates of eyeball positions of the face in the current video frame;

converting the plurality of 2D keypoint coordinates into a corresponding plurality of 3D keypoint coordinates;

converting the 2D coordinates of the eyeball position into 3D coordinates of the eyeball position according to the conversion relation between the 2D key point coordinates and the corresponding 3D key point coordinates;

driving the eyeball position of the target object according to the 3D coordinates of the eyeball position and the 3D coordinates of the initial position of the eyeball;

and displaying the driven eyeballs of the target object.

With reference to the first aspect, in a first possible implementation manner, before the filtering the face region by using the left and right eye filters to obtain the 2D coordinates of the eyeball position of the face in the current video frame, the method further includes:

normalizing the size and/or illumination of the face area to obtain a normalized face area;

and converting the normalized face area into frequency domain data.

With reference to the first possible implementation manner of the first aspect, in a second possible implementation manner, the filtering the face region through left and right eye filters, and acquiring 2D coordinates of an eyeball position of a face in the current video frame includes:

filtering the frequency domain data through a left eye filter and a right eye filter, and respectively constructing interested areas of the left eye and the right eye according to the filtered output images;

respectively determining the position coordinates of the maximum value in the filtering output result in the interested areas of the left eye and the right eye, and acquiring the position coordinates of the left eye and the right eye corresponding to the interested areas of the left eye and the right eye respectively;

judging whether the position coordinate of the left eye is in a left eye area, if so, determining that the position coordinate of the left eye is a 2D coordinate of the eyeball position of the left eye, otherwise, calculating the position coordinate of the eyeball position according to key point information of the left eye, and taking the calculated position coordinate as the 2D coordinate of the eyeball position of the left eye; and

judging whether the position coordinate of the right eye is in a right eye area, if so, determining that the position coordinate of the right eye is a 2D coordinate of the eyeball position of the right eye, otherwise, calculating the position coordinate of the eyeball position according to key point information of the right eye, and taking the calculated position coordinate as the 2D coordinate of the eyeball position of the right eye.

With reference to the second possible implementation manner of the first aspect, in a third possible implementation manner, after the step of determining the position coordinates or the step of calculating the position coordinates of the eyeball position is performed, the method further includes:

and performing integral projection in a peripheral preset area of the position coordinate, and adjusting the 2D coordinate of the eyeball position of the left eye and/or the 2D coordinate of the eyeball position of the right eye according to the coordinate of the position of the integral minimum value.

With reference to the first aspect to any one of the third possible implementation manners of the first aspect, in a fourth possible implementation manner, the method further includes:

obtaining a plurality of human eye training samples;

generating a plurality of accurate filters respectively corresponding to the plurality of human eye training samples;

and generating the left and right eye filters according to the plurality of precise filters.

With reference to the first aspect, in a fifth possible implementation manner, before converting the plurality of 2D keypoint coordinates into a corresponding plurality of 3D keypoint coordinates, the method further includes:

and calculating conversion relations between the 2D key point coordinates and the corresponding 3D key point coordinates according to the 3D model data.

With reference to the first aspect or the fifth possible implementation manner of the first aspect, in a sixth possible implementation manner, the method further includes:

acquiring a plurality of 2D key point coordinates of the target object and 2D coordinates of eyeball positions;

and calculating a plurality of 3D key point coordinates and 3D coordinates of eyeball positions of the target object.

With reference to the first aspect to the sixth possible implementation manner of the first aspect, in a seventh possible implementation manner, the driving, according to the 3D coordinates of the eyeball position and the 3D coordinates of the initial position of the eyeball, the eyeball position of the target object includes:

acquiring relative offsets of the 3D coordinates of the eyeball positions of the left and right eyes and the 3D coordinates of the initial positions of the eyeballs of the left and right eyes respectively;

mapping the relative offsets of the left and right eyes to 3D coordinates of eyeball positions of the left and right eyes of the target object respectively; and

converting the 3D coordinates of the eyeball positions of the left and right eyes of the target object into 2D coordinates, and driving the eyeball positions of the target object;

wherein the initial position of the eyeball corresponds to the initial position of the eyeball of the target object.

With reference to the seventh possible implementation manner of the first aspect, in an eighth possible implementation manner, the obtaining the relative offset between the 3D coordinates of the eyeball positions of the left and right eyes and the 3D coordinates of the initial positions of the eyeballs of the left and right eyes respectively includes:

respectively comparing the 3D coordinates of the eyeball positions of the left eye and the right eye with the 3D coordinates of the initial positions of the eyeballs of the left eye and the right eye, and respectively acquiring the offset of the left eye and the right eye according to the comparison result;

and respectively calculating the relative offset according to the offset and the width and the height of the interested areas of the corresponding left eye and the right eye.

With reference to the seventh possible implementation manner of the first aspect, in a ninth possible implementation manner, the converting the 3D coordinates of the eyeball positions of the left and right eyes of the target object into 2D coordinates further includes:

and performing preset proportion weighting adjustment on the currently converted 2D coordinate according to the converted 2D coordinate acquired from the previous frame to acquire the adjusted 2D coordinate.

With reference to the ninth possible implementation manner of the first aspect, in a tenth possible implementation manner, the displaying the driven eyeball of the target object includes:

and displaying the driven eyeballs of the target object according to the adjusted 2D coordinates.

In a second aspect, an electronic device is provided, the electronic device comprising:

the key point acquisition module is used for acquiring a plurality of 2D key point coordinates of the face in the current video frame;

the face area determining module is used for determining a face area according to the coordinates of the 2D key points;

the filtering module is used for filtering the face region through a left eye filter and a right eye filter to obtain a 2D coordinate of an eyeball position of a face in the current video frame;

the key point conversion module is used for converting the 2D key point coordinates into a plurality of corresponding 3D key point coordinates;

the key point conversion module is further used for converting the 2D coordinates of the eyeball position into the 3D coordinates of the eyeball position according to the conversion relation between the 2D key point coordinates and the corresponding 3D key point coordinates;

the driving module is used for driving the eyeball position of the target object according to the 3D coordinate of the eyeball position and the 3D coordinate of the initial position of the eyeball;

and the display module is used for displaying the driven eyeballs of the target object.

With reference to the second aspect, in a first possible implementation manner, the apparatus further includes:

the normalization processing module is used for performing normalization processing on the size and/or illumination of the face area to obtain a normalized face area;

and the frequency domain conversion module is used for converting the normalized human face area into frequency domain data.

With reference to the first possible implementation manner of the second aspect, in a second possible implementation manner,

the filtering module comprises a construction module, a position determination module, a judgment module and a calculation module;

the construction module is used for filtering the frequency domain data through a left eye filter and a right eye filter, and respectively constructing interested areas of the left eye and the right eye according to the filtered output images;

the position determining module is used for determining the position coordinates of the maximum value in the filtering output result in the interested areas of the left eye and the right eye respectively, and acquiring the position coordinates of the left eye and the right eye corresponding to the interested areas of the left eye and the right eye respectively;

the judging module is used for judging whether the position coordinate of the left eye is in a left eye area, if so, the position determining module is triggered to determine that the position coordinate of the left eye is a 2D coordinate of the eyeball position of the left eye, if not, the calculating module is triggered to calculate the position coordinate of the eyeball position according to key point information of the left eye, and the calculated position coordinate is used as the 2D coordinate of the eyeball position of the left eye; and

the judging module is further configured to judge whether the position coordinate of the right eye is in a right eye region, if so, trigger the position determining module to determine that the position coordinate of the right eye is a 2D coordinate of the eyeball position of the right eye, and if not, trigger the calculating module to calculate the position coordinate of the eyeball position according to the key point information of the right eye, and use the calculated position coordinate as the 2D coordinate of the eyeball position of the right eye.

With reference to the second possible implementation manner of the second aspect, in a third possible implementation manner, the apparatus further includes:

and the integral determining module is used for performing integral projection in a peripheral preset area of the position coordinate and adjusting the 2D coordinate of the eyeball position of the left eye and/or the 2D coordinate of the eyeball position of the right eye according to the coordinate of the position of the integral minimum value.

With reference to the second aspect to any one of the third possible implementation manners of the second aspect, in a fourth possible implementation manner, the apparatus further includes:

the system comprises a sample acquisition module, a human eye training module and a human eye training module, wherein the sample acquisition module is used for acquiring a plurality of human eye training samples;

the accurate filter generation module is used for generating a plurality of accurate filters respectively corresponding to the plurality of human eye training samples;

and the left and right eye filter generating module is used for generating the left and right eye filters according to the plurality of accurate filters.

With reference to the second aspect, in a fifth possible implementation manner, the apparatus further includes:

and the conversion relation calculation module is used for calculating the conversion relation between the plurality of 2D key point coordinates and the corresponding plurality of 3D key point coordinates according to the 3D model data.

With reference to the second aspect or the fifth possible implementation manner of the second aspect, in a sixth possible implementation manner, the apparatus further includes:

the target object key point acquisition module is used for acquiring a plurality of 2D key point coordinates of the target object and 2D coordinates of eyeball positions;

and the target object 3D coordinate calculation module is used for calculating a plurality of 3D key point coordinates of the target object and 3D coordinates of eyeball positions.

With reference to the second aspect to any one of the sixth possible implementation manners of the second aspect, in a seventh possible implementation manner, the driving module further includes:

an offset acquisition module, configured to acquire relative offsets of the 3D coordinates of the eyeball positions of the left and right eyes and the 3D coordinates of the initial positions of the eyeballs of the left and right eyes, respectively;

a mapping module for mapping the relative offsets of the left and right eyes to 3D coordinates of eyeball positions of the left and right eyes of the target object, respectively; and

the target object coordinate conversion module is used for converting the 3D coordinates of the eyeball positions of the left eye and the right eye of the target object into 2D coordinates and driving the eyeball positions of the target object;

With reference to the seventh possible implementation manner of the second aspect, in an eighth possible implementation manner, the offset obtaining module further includes:

the comparison module is used for comparing the 3D coordinates of the eyeball positions of the left eye and the right eye with the 3D coordinates of the initial positions of the eyeballs of the left eye and the right eye respectively, and acquiring the offset of the left eye and the right eye respectively according to the comparison result;

and the relative offset calculating module is used for respectively calculating the relative offset according to the offset and the width and the height of the corresponding interested areas of the left eye and the right eye.

With reference to the seventh possible implementation manner of the second aspect, in a ninth possible implementation manner, the target object coordinate conversion module further includes:

and the adjusting module is used for performing preset proportion weighting adjustment on the currently converted 2D coordinate according to the converted 2D coordinate acquired from the previous frame to acquire the adjusted 2D coordinate.

With reference to the ninth possible implementation manner of the second aspect, in a tenth possible implementation manner, the display module is specifically configured to:

In a third aspect, an electronic device is provided, where the electronic device includes a display screen, a camera, a memory, and a processor connected to the display screen, the camera, and the memory, where the memory is configured to store a set of program codes, and the processor calls the program codes stored in the memory to perform the following operations:

and displaying the driven eyeballs of the target object.

With reference to the third aspect, in a first possible implementation manner, the processor calls the program code stored in the memory to perform the following operations:

and converting the normalized face area into frequency domain data.

With reference to the first possible implementation manner of the third aspect, in a second possible implementation manner, the processor calls the program code stored in the memory to perform the following operations:

With reference to the second possible implementation manner of the third aspect, in a third possible implementation manner, the processor calls the program code stored in the memory to perform the following operations:

With reference to any one of the third to fourth possible implementation manners of the third aspect, in a fourth possible implementation manner, the processor calls the program code stored in the memory to perform the following operations:

obtaining a plurality of human eye training samples;

With reference to the third aspect, in a fifth possible implementation manner, the processor calls the program code stored in the memory to perform the following operations:

With reference to the third aspect or the fifth possible implementation manner of the third aspect, in a sixth possible implementation manner, the processor calls the program code stored in the memory to perform the following operations:

With reference to any one of the sixth possible implementation manner of the third aspect to the third aspect, in a seventh possible implementation manner, the processor calls the program code stored in the memory to perform the following operations:

With reference to the seventh possible implementation manner of the third aspect, in an eighth possible implementation manner, the processor calls the program code stored in the memory to perform the following operations:

With reference to the seventh possible implementation manner of the third aspect, in a ninth possible implementation manner, the processor calls the program code stored in the memory to perform the following operations:

With reference to the ninth possible implementation manner of the third aspect, in a tenth possible implementation manner, the processor calls the program code stored in the memory to perform the following operations:

The invention discloses an eyeball driving method and electronic equipment, wherein the method comprises the following steps: acquiring a plurality of 2D key point coordinates of a face in a current video frame, and determining a face area according to the plurality of 2D key point coordinates; filtering the face region through a left eye filter and a right eye filter to obtain 2D coordinates of eyeball positions of the face in the current video frame; converting the plurality of 2D key point coordinates into a plurality of corresponding 3D key point coordinates; converting the 2D coordinates of the eyeball position into the 3D coordinates of the eyeball position according to the conversion relation between the 2D key point coordinates and the corresponding 3D key point coordinates; driving the eyeball position of the target object according to the 3D coordinates of the eyeball position and the 3D coordinates of the initial position of the eyeball; and displaying the eyeballs of the driven target object. According to the embodiment of the invention, the eyeball position of the target object is driven according to the 3D coordinate of the eyeball position and the 3D coordinate of the initial position of the eyeball, so that the accuracy and the stability of eyeball positioning and driving are improved, meanwhile, the high-efficiency operation at a moving end is realized, the user experience is improved, and the requirement of a user for controlling the target object through eyeball rotation is met; in addition, the 2D coordinates of the eyeball positions of the human faces in the current video frame are obtained by filtering the human face regions through the left eye filter and the right eye filter, so that compared with the traditional method of simply relying on picture texture information or only positioning the eyeballs through the filters, the method provided by the embodiment of the invention combines the filter positioning eyeballs and the human face key point positioning technology, so that the accuracy and the stability of eyeball positioning are further improved, and meanwhile, because the filters can be obtained by offline training, the method does not need to occupy processing resources when a mobile terminal runs, the running speed at the mobile terminal is further improved, the user experience is improved, and the user requirements are met; in addition, the 2D key point coordinates are converted into the corresponding 3D key point coordinates, so that the scales of key points of the face in the acquired instant video frame are kept consistent, and meanwhile, the front face is performed, so that the influence on eyeball positioning caused by the fact that the face is positioned on the side face or the scales are not consistent is avoided, meanwhile, the 2D coordinates of the eyeball position are converted into the 3D coordinates of the eyeball position, so that the 3D coordinates of the eyeball position are acquired in the 3D space with the consistent scales and the front face, the eyeball positioning accuracy is further improved, the eyeball position of a target object is driven through the 3D coordinates, and the eyeball driving accuracy and stability are further improved; in addition, the eyeball driving method provided by the embodiment of the invention can be applied to eyeball control of target objects in games and the like, enriches the modes of controlling the target objects in the games, and meets the requirements of users on controlling the target objects through eyeball rotation, thereby further improving the user experience.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a flowchart of an eyeball driving method provided by an embodiment of the invention;

fig. 2 is a flowchart of an eyeball driving method provided by an embodiment of the invention;

fig. 3 is a schematic diagram of a determined face region according to an embodiment of the present invention;

fig. 4 is a flowchart of an eyeball driving method provided by an embodiment of the invention;

FIG. 5 is a schematic diagram of a region of interest for the left and right eyes according to an embodiment of the present invention;

fig. 6 is a flowchart of an eyeball driving method provided by an embodiment of the invention;

fig. 7 is a flowchart of an eyeball driving method provided by an embodiment of the invention;

fig. 8 is a flowchart of an eyeball driving method provided by an embodiment of the invention;

FIG. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The embodiment of the invention provides an eyeball driving method, which can be applied to an interactive system, wherein the system can comprise at least one piece of electronic equipment and a user, wherein the electronic equipment can be a smart phone, a tablet personal computer or wearable equipment, or other mobile terminals; the electronic device comprises at least a video input module and a video display module, wherein the video input module can comprise a camera, and the video display module can comprise a display screen.

The method can be applied to a scene for controlling the eyeball rotation of the target object in the game, at least one game program can be run in the electronic equipment in the scene, and the electronic equipment obtains the eyeball position of the user by running the game program and starting the camera function so as to control the eyeball position of the target object in the game.

In addition, the method may also be applied to other interactive systems and application scenarios, and the specific interactive system and application scenario are not limited in the embodiment of the present invention.

Example one

An embodiment of the present invention provides an eyeball driving method, and with reference to fig. 1, the method includes:

101. obtaining a plurality of 2D key point coordinates of the face in the current video frame, and determining the face area according to the plurality of 2D key point coordinates.

Optionally, before step 102, the method may further include:

and converting the normalized face area into frequency domain data.

102. And filtering the face region through the left and right eye filters to obtain the 2D coordinates of the eyeball position of the face in the current video frame.

Specifically, the process may include:

filtering the frequency domain data through a left eye filter and a right eye filter, and respectively constructing interested areas of the left eye and the right eye according to the filtered output graphs;

judging whether the position coordinate of the left eye is in a left eye area, if so, determining that the position coordinate of the left eye is a 2D coordinate of the eyeball position of the left eye, otherwise, calculating the position coordinate of the eyeball position according to the key point information of the left eye, and taking the calculated position coordinate as the 2D coordinate of the eyeball position of the left eye; and

and judging whether the position coordinate of the right eye is in the right eye area, if so, determining that the position coordinate of the right eye is the 2D coordinate of the eyeball position of the right eye, otherwise, calculating the position coordinate of the eyeball position according to the key point information of the right eye, and taking the calculated position coordinate as the 2D coordinate of the eyeball position of the right eye.

Preferably, after the step of determining the position coordinates or the step of calculating the position coordinates of the eyeball position is performed, the method further includes:

Optionally, the left-eye and right-eye filters according to the embodiment of the present invention may be generated in advance, where the process of generating the left-eye and right-eye filters may include:

obtaining a plurality of human eye training samples;

generating a plurality of accurate filters respectively corresponding to a plurality of human eye training samples;

left and right eye filters are generated from the plurality of precision filters.

103. The plurality of 2D keypoint coordinates are converted into a corresponding plurality of 3D keypoint coordinates.

Optionally, before performing step 104, the method further includes:

and calculating conversion relations between the plurality of 2D key point coordinates and the corresponding plurality of 3D key point coordinates according to the 3D model data.

104. And converting the 2D coordinates of the eyeball position into the 3D coordinates of the eyeball position according to the conversion relation between the 2D key point coordinates and the corresponding 3D key point coordinates.

105. And driving the eyeball position of the target object according to the 3D coordinates of the eyeball position and the 3D coordinates of the initial position of the eyeball.

Specifically, the process comprises the following steps:

respectively acquiring the relative offset of the 3D coordinates of the eyeball positions of the left eye and the right eye and the 3D coordinates of the initial positions of the eyeballs of the left eye and the right eye;

mapping the relative offset of the left eye and the right eye to the 3D coordinates of the eyeball positions of the left eye and the right eye of the target object respectively; and

the 3D coordinates of the eyeball positions of the left and right eyes of the target object are converted into 2D coordinates, and the eyeball position of the target object is driven.

Optionally, the obtaining of the relative offset between the 3D coordinates of the eyeball positions of the left and right eyes and the 3D coordinates of the initial positions of the eyeballs of the left and right eyes respectively includes:

the relative offsets are calculated based on the offsets and the corresponding widths and heights of the regions of interest for the left and right eyes, respectively.

Optionally, converting the 3D coordinates of the eyeball positions of the left and right eyes of the target object into 2D coordinates further includes:

106. And displaying the eyeballs of the driven target object.

Optionally, the method further comprises:

acquiring a plurality of 2D key point coordinates of a target object and 2D coordinates of eyeball positions;

a plurality of 3D keypoint coordinates of the target object and 3D coordinates of the eyeball position are calculated.

Optionally, the eyeball position of the driven target object is displayed according to the adjusted 2D coordinate.

Optionally, displaying the driven eyeball of the target object may include:

and displaying the eyeballs of the driven target object according to the adjusted 2D coordinates.

The embodiment of the invention provides an eyeball driving method, which drives the eyeball position of a target object according to the 3D coordinate of the eyeball position and the 3D coordinate of the initial position of the eyeball, improves the accuracy and stability of eyeball positioning and driving, can efficiently operate at a moving end, improves user experience, and meets the requirement of a user for controlling the target object through eyeball rotation; in addition, the 2D coordinates of the eyeball positions of the human faces in the current video frame are obtained by filtering the human face regions through the left eye filter and the right eye filter, so that compared with the traditional method of simply relying on picture texture information or only positioning the eyeballs through the filters, the method provided by the embodiment of the invention combines the filter positioning eyeballs and the human face key point positioning technology, so that the accuracy and the stability of eyeball positioning are further improved, and meanwhile, because the filters can be obtained by offline training, the method does not need to occupy processing resources when a mobile terminal runs, the running speed at the mobile terminal is further improved, the user experience is improved, and the user requirements are met; in addition, the 2D key point coordinates are converted into the corresponding 3D key point coordinates, so that the scales of key points of the face in the acquired instant video frame are kept consistent, and meanwhile, the front face is performed, so that the influence on eyeball positioning caused by the fact that the face is positioned on the side face or the scales are not consistent is avoided, meanwhile, the 2D coordinates of the eyeball position are converted into the 3D coordinates of the eyeball position, so that the 3D coordinates of the eyeball position are acquired in the 3D space with the consistent scales and the front face, the eyeball positioning accuracy is further improved, the eyeball position of a target object is driven through the 3D coordinates, and the eyeball driving accuracy and stability are further improved; in addition, the eyeball driving method provided by the embodiment of the invention can be applied to eyeball control of target objects in games and the like, enriches the modes of controlling the target objects in the games, and meets the requirements of users on controlling the target objects through eyeball rotation, thereby further improving the user experience.

Example two

An embodiment of the present invention provides an eyeball driving method, which is shown in fig. 2 and includes:

201. obtaining a plurality of 2D key point coordinates of the face in the current video frame, and determining the face area according to the plurality of 2D key point coordinates.

Specifically, the obtaining of the coordinates of the plurality of 2D key points of the face in the current video frame may be performed by positioning the key points of the face in the current video frame by a Method based on shape regression, obtaining the coordinates of the plurality of 2D key points of the face by an SDM (supervisory gradient Descent) algorithm, or obtaining the coordinates of the plurality of 2D key points of the face by other methods.

The 2D key point coordinates of the face may include key point coordinates for describing a peripheral contour of the face and/or internal key point coordinates, the internal key point coordinates include key point coordinates for describing contours of eyebrows, eyes, a nose, and a mouth of the face, respectively, and the 2D key point coordinates of the face may also include others.

For example, the 2D key point coordinates of the face may include 66 key point coordinates for describing the peripheral outline and the interior of the face.

Determining a face region according to the coordinates of the plurality of 2D key points, which may be a rectangular region composed of two points located on the left and right of the face and representing the widest horizontal distance and two points located above and below the face and representing the longest vertical distance, for example, referring to the face region shown in a in fig. 3, the face region is a rectangular region composed of two points on the left and right sides indicated by arrows and two points above and below indicated by arrows; or a preset face region including a plurality of 2D key point coordinates for describing a face, for example, the face region may be shown in fig. 3 b; in addition, the face region determined according to the coordinates of the plurality of 2D key points may also be other face regions, and the specific face region is not limited in the embodiment of the present invention.

It should be noted that the face area determined according to the coordinates of the plurality of 2D key points may be a rectangle, an ellipse, or another shape, and the specific shape of the face area is not limited in the embodiment of the present invention.

202. And carrying out normalization processing on the size and/or illumination of the face area to obtain a normalized face area.

Specifically, the process of normalizing the size of the face area may include:

scaling the size of the face region to a preset size, preferably, the preset size may be 128 × 128;

the process of normalizing the illumination of the face region may be:

and carrying out log transformation on the illumination of the face area.

By carrying out normalization processing on the size of the face area, the influence on an eyeball positioning result caused by size difference can be avoided, and the illumination of the face area is subjected to normalization processing, so that the influence of shadow and illumination on eyeball positioning is reduced, and the accuracy and the stability of eyeball positioning are further improved.

It should be noted that step 202 is an optional step, and in practical applications, after step 201, step 203 may be directly performed without performing step 202.

203. And converting the normalized face area into frequency domain data.

Specifically, a two-dimensional gaussian distribution image G of the face region is obtained, and if step 202 is executed before step 203, a two-dimensional gaussian distribution image G of the normalized face region is obtained;

performing fast fourier transform on the two-dimensional gaussian distribution image G to generate F corresponding to the face region, and if step 202 is executed before step 203, acquiring F corresponding to the normalized face region.

Based on G and F, H corresponding to the face region is generated, and if step 202 is executed before step 203, H corresponding to the normalized face region is generated.

The H corresponding to the face region may be generated according to a preset formula, where the preset formula may be:

multiplying the H by the face area to obtain frequency domain data, and if step 202 is executed before step 203, multiplying the H by the normalized face area to obtain frequency domain data.

204. And filtering the face region through the left and right eye filters to obtain the 2D coordinates of the eyeball position of the face in the current video frame.

Specifically, referring to fig. 4, the process may include:

2041. and filtering the frequency domain data through a left eye filter and a right eye filter, and respectively constructing interested areas of the left eye and the right eye according to the filtered output graphs.

Specifically, the process comprises the following steps:

inputting the frequency domain data into a left eye filter and a right eye filter, filtering the gray values of pixel points included in the frequency domain data, and outputting a filtering result, wherein the filtering result comprises a left eye interest area and a right eye interest area and the gray values of all the pixel points in the left eye interest area and the right eye interest area;

the central points of the left eye and the right eye and the length and width of the left eye and the right eye can be calculated according to the key point coordinates for describing the left eye and the right eye in the output graph, and the interested areas of the left eye and the right eye can be respectively constructed according to the central points of the left eye and the right eye and the length and width of the left eye and the right eye.

Illustratively, referring to FIG. 5, the center points of the left and right eyes and the lengths and widths of the left and right eyes are calculated respectively according to 6 key points with serial numbers 36-41 for describing the left eye and 6 key points with serial numbers 42-47 for describing the right eye, respectively, to construct the regions of interest of the left and right eyes respectively.

It should be noted that the construction of the regions of interest of the left and right eyes here can also be implemented in other ways, and the specific way is not limited by the embodiment of the present invention, where the regions of interest of the left and right eyes shown in fig. 5 are only exemplary.

2042. And respectively determining the position coordinates of the maximum value in the filtering output result in the interested areas of the left eye and the right eye, and acquiring the position coordinates of the left eye and the right eye corresponding to the interested areas of the left eye and the right eye.

Specifically, the position of the pixel point where the maximum gray value included in the filtering output result is determined is an eyeball position coordinate, and the eyeball position coordinate may be an eyeball position coordinate of a right eye and/or a left eye.

In practical application, the left eye may be filtered first to output the eyeball position coordinate of the left eye, then the right eye is filtered to output the eyeball position coordinate of the right eye, or the right eye may be filtered first to output the eyeball position coordinate of the right eye, then the left eye is filtered to output the eyeball position coordinate of the left eye, or the left eye and the right eye may be filtered simultaneously to output the eyeball position coordinate of the left eye and the eyeball position coordinate of the right eye, and a specific filtering sequence is not limited in the embodiment of the present invention.

2043. Judging whether the position coordinate of the left eye is in the left eye area, if so, executing step 2044; if not, go to step 2045.

Here, the left-eye region refers to a left-eye contour region composed of key points for describing a left-eye contour.

Specifically, it is determined whether the eyeball position coordinate of the left eye output in step 2043 is within the range of the left eye contour area, if yes, step 2044 is executed; if not, go to step 2045.

Illustratively, as described with reference to the example in step 2041, if the left-eye contour region is an eye contour region composed of 6 key points with serial numbers 36 to 41 for describing the left-eye contour, it is determined whether the eye position coordinates of the left eye output in 2043 are within the eye contour region composed of 6 key points with serial numbers 36 to 41, and if so, step 2044 is executed; if not, go to step 2045.

2044. And determining the position coordinates of the left eye as the 2D coordinates of the eyeball position of the left eye.

Specifically, the eye position coordinates of the left eye output in step 2043 are taken as the 2D coordinates of the eye position of the left eye, and the specific determination method is not limited in the embodiment of the present invention.

2045. And calculating the position coordinates of the eyeball position according to the key point information of the left eye, and taking the calculated position coordinates as the 2D coordinates of the eyeball position of the left eye.

Specifically, according to the key point information of the left eye, the position coordinate of the position of the left eye ball is calculated by a preset formula, and the preset formula may be:

wherein, I refers to a gray level image of the face image,denotes gaussian distribution, μ denotes mean and denotes variance.

And taking the position coordinate calculated by the preset formula as the 2D coordinate of the eyeball position of the left eye.

2046. Judging whether the position coordinate of the right eye is in the right eye area, if so, executing step 2047; if not, go to step 2048.

Wherein, the right eye region refers to a right eye contour region composed of key points for describing a right eye contour.

Specifically, it is determined whether the eyeball position coordinate of the right eye output in step 2043 is within the range of the right eye contour area, if yes, step 2047 is executed; if not, go to step 2048.

Exemplarily, referring to the example in step 2041, if the right-eye contour region is an eye contour region composed of 6 key points with serial numbers 42 to 47 for describing the right-eye contour, it is determined whether the eyeball position coordinates of the right eye output in 2043 are within the eye contour region composed of 6 key points with serial numbers 42 to 47, and if so, step 2047 is executed; if not, go to step 2048.

2047. And determining the position coordinate of the right eye as the 2D coordinate of the eyeball position of the right eye.

Specifically, the eyeball position coordinates of the left and right eyes output in step 2043 are taken as the 2D coordinates of the eyeball position of the right eye, and the specific determination method is not limited in the embodiment of the present invention.

2048. And calculating the position coordinates of the eyeball position according to the key point information of the right eye, and taking the calculated position coordinates as the 2D coordinates of the eyeball position of the right eye.

Specifically, the position coordinates of the position of the right eyeball are calculated by a preset formula according to the key point information of the right eye, the preset formula is the same as that in step 2045, and details are not repeated here.

And taking the position coordinate calculated by the preset formula as the 2D coordinate of the eyeball position of the right eye.

It is noted that, the steps 2043-.

Whether the 2D coordinates of the left eyeball position and the right eyeball position of the face obtained after filtering are respectively in the left eye area and the right eye area or not is judged, so that the interference caused by the peak positions of eyeball positioning when a user has black frame glasses or similar eyeball black colors exist around the eyes is avoided, the condition that eyeball positioning errors are caused by wrong peak positions is avoided, the eyeball positioning accuracy is further improved, the accuracy of target object eyeball driving through eyeball positioning is ensured, and the user experience is further improved.

Optionally, referring to fig. 6, in step 2044 and step 2045, the method further includes:

2049. and performing integral projection in a peripheral preset area of the position coordinate of the left eye, and adjusting the 2D coordinate of the eyeball position of the left eye according to the coordinate of the position of the minimum integral value.

Specifically, in order to further obtain a positioning point with higher precision, integral projection is performed on texture information in horizontal and vertical directions in preset areas around the obtained 2D coordinate points of the left and right eyeball positions, respectively, and then the coordinate of the position of the minimum integral value is determined to be the 2D coordinate of the final eyeball position.

Preferably, the predetermined region may be a region 7 × 7 around the 2D coordinate point of the left and right eyeball positions.

After the position coordinate where the maximum value in the filtering output result is located is determined as the eyeball position coordinate, integral projection calculation is further carried out, and the accuracy of the obtained eyeball position coordinate is verified, so that the accuracy of the eyeball position coordinate is ensured, and the accuracy of eyeball positioning is further improved.

After performing steps 2047 and 2048, the method further comprises:

20410. and performing integral projection in a peripheral preset area of the position coordinate of the right eye, and adjusting the 2D coordinate of the eyeball position of the right eye according to the coordinate of the position of the integral minimum value.

Specifically, the process is similar to step 2049 and will not be described herein again.

205. And calculating conversion relations between the plurality of 2D key point coordinates and the corresponding plurality of 3D key point coordinates according to the 3D model data.

The 3D model data may be a 3D linear shape gain model, the process specifically comprising:

firstly training a 3D linear shape gain model, adopting a 3D reconstruction technology of 'recovering structure from motion' in a data set, such as a Multi-PIE data set, calculating corresponding 3D point coordinate points of a plurality of 2D key point coordinate data sets, dividing key points for describing the interior of a human face into three parts, wherein the three parts are respectively used for describing key points of the outlines of eyes, eyebrows and a mouth or a nose, and carrying out singular value decomposition on each part to obtain psi_eyes，Ψ_eyebrows，Ψ_mouthAnd recombining each part to obtain the 3D expression shape gain model psi.

According to the 3D expression shape gain model Ψ obtained by training, the conversion relationship is calculated by a preset formula, where the preset formula may be:

where s denotes the scaling scale, R denotes the rotation matrix, t denotes the offset,3D coordinates of neutral expression.

The optimal solution for minimizing the second product is calculated for the transformation relationship, and the formula is as follows:

wherein x is_i，y_iIs a 2D key point coordinate point, z, of a face in a current video frame_iIs a relative depth value, is a threshold, and p represents a penalty term.

The conversion relation is calculated by training the 3D linear shape gain model, and the optimal solution of minimizing the second-order product is calculated for the conversion relation, so that the accuracy of the 3D coordinate calculated by the conversion relation is ensured, and meanwhile, because the conversion relation between the 2D key point coordinate and the corresponding 3D key point coordinate is calculated on line, the occupation of processing resources when the mobile terminal operates is avoided, and the operation speed of the mobile terminal is further improved.

206. The plurality of 2D keypoint coordinates are converted into a corresponding plurality of 3D keypoint coordinates.

Specifically, a plurality of 2D keypoint coordinates are computationally converted into a corresponding plurality of 3D keypoint coordinates according to the conversion relationship computed in step 205.

The coordinates of the plurality of 3D key points obtained by calculation are normalized and normalized coordinates.

The 2D key point coordinates are converted into the corresponding 3D key point coordinates, so that the scales of the key points of the face in the acquired instant video frame are kept consistent, and meanwhile, the obverse processing is carried out, so that the influence on eyeball positioning caused by the fact that the face is positioned on the side or the scales are inconsistent is avoided, the eyeball positioning accuracy and precision are further improved, and the accuracy of driving the eyeball position of the target object through the 3D coordinates is improved.

207. And converting the 2D coordinates of the eyeball position into the 3D coordinates of the eyeball position according to the conversion relation between the 2D key point coordinates and the corresponding 3D key point coordinates.

Specifically, the 2D coordinates of the eyeball position are converted into the 3D coordinates of the eyeball position according to the conversion relationship calculated in step 205.

The 2D coordinates of the eyeball position are converted into the 3D coordinates of the eyeball position according to the conversion relation, so that the 3D coordinates of the eyeball position are acquired in the 3D space with the consistent dimension and the orthogonalization, the eyeball positioning accuracy is further improved, the eyeball position of the target object is driven through the 3D coordinates, and the eyeball driving accuracy and stability are further improved.

208. And driving the eyeball position of the target object according to the 3D coordinates of the eyeball position and the 3D coordinates of the initial position of the eyeball.

Specifically, the process comprises the following steps:

A. relative offsets of the 3D coordinates of the eyeball positions of the left and right eyes and the 3D coordinates of the initial positions of the eyeballs of the left and right eyes are acquired, respectively.

Specifically, the process may include:

The regions of interest of the left and right eyes may be the same as or different from the regions of interest of the left and right eyes constructed in step 2041, which is not limited in this embodiment of the present invention.

Preferably, the length and width of the interested regions of the left and right eyes are acquired according to the interested regions of the left and right eyes constructed in step 2041, respectively.

Wherein the acquired offset amounts of the left and right eyes have components of x-axis and y-axis.

And dividing the components of the offset of the left eye and the right eye on the x axis and the y axis by the length and the width of the interested region of the corresponding left eye and the right eye respectively, and calculating the relative offset of the x axis and the y axis respectively so as to obtain the relative offset respectively.

B. The relative offsets of the left and right eyes are mapped to the 3D coordinates of the eyeball positions of the left and right eyes of the target object, respectively.

Specifically, the relative offset of the x-axis and the y-axis is multiplied by the length and the width of the interested regions of the left eye and the right eye of the target object respectively, the offset of the length and the width of the interested regions of the left eye and the right eye of the target object on the x-axis and the y-axis is obtained, and then the 3D coordinates of the eyeball positions of the left eye and the right eye of the target object are adjusted according to the offset of the x-axis and the y-axis.

Optionally, after the step B of the step 208 is executed, the method may further include:

Preferably, the preset ratio may be 0.4 and 0.6.

Specifically, the converted 2D coordinate obtained from the previous frame and the currently converted 2D coordinate are weighted according to a preset ratio, and the adjusted 2D coordinate is obtained.

Through weighting the converted 2D coordinate acquired from the previous frame and the currently converted 2D coordinate according to a preset proportion, the acquired driven 2D eyeball coordinate is excessive and more natural, so that the user experience is improved, and the user requirements are met.

C. The 3D coordinates of the eyeball positions of the left and right eyes of the target object are converted into 2D coordinates, and the eyeball position of the target object is driven.

Specifically, the 3D coordinates of the eyeball positions of the left and right eyes of the target object are converted back to 2D coordinates according to the conversion relationship calculated in step 205, and the eyeball position of the target object is driven.

209. And displaying the eyeballs of the driven target object.

Specifically, the eyeballs of the left and right eyes are displayed in the target object according to the 2D coordinates converted back in step C in step 208.

Optionally, if performing, before performing step 209, a preset ratio weighting adjustment on the currently converted 2D coordinate according to the converted 2D coordinate obtained from the previous frame, and obtaining an adjusted 2D coordinate, the process of step 2012 may further be:

The eyeball of the driven target object is displayed according to the adjusted 2D coordinate, so that the eyeball of the driven target object is displayed more naturally, the user experience is improved, and the user requirement is met.

Optionally, as shown in fig. 7, if the left-eye and right-eye filters of the embodiment of the present invention are generated in advance, the method further includes a process of generating the left-eye and right-eye filters, and before executing step 201, the method further includes the following steps:

701. a plurality of human eye training samples are obtained.

Specifically, the plurality of human eye training samples comprise a plurality of right eye training samples and a plurality of left eye training samples, and in practical application, the corresponding right eye training samples and the corresponding left eye training samples can be respectively generated according to the same human eye training sample.

The process of acquiring the plurality of right eye training samples and the plurality of left eye training samples may be:

acquiring a plurality of face images from a local sample library; or

Acquiring a plurality of face images from a database on the network side, wherein the database can be Bio1D or MUCT for example; or

And a plurality of pictures meeting preset conditions can be identified from a plurality of pictures at the network side or in the local area by identification, and the pictures are used as face images.

The method comprises the steps of identifying the right eyes in the face images, generating a plurality of right eye training samples, identifying the left eyes in the face images, and generating a plurality of left eye training samples.

Positioning the human face in the right eye training samples, acquiring a plurality of 2D key point coordinates of the human face, determining a human face area according to the plurality of 2D key point coordinates, and cutting the determined human face area in the right eye training samples into a rectangle; and positioning the face in the left eye training samples, acquiring a plurality of 2D key point coordinates of the face, determining a face area according to the plurality of 2D key point coordinates, and cutting the determined face area in the left eye training samples into a rectangle.

The sizes of the plurality of right eye training samples and the plurality of left eye training samples may also be scaled to a preset size, preferably, the preset size may be 128 × 128;

by carrying out normalization processing on the sizes of the right eye training samples and the left eye training samples, the training result difference caused by the size difference can be avoided, and the accuracy of the left eye filter and the right eye filter for identifying the human eyes is improved, so that the accuracy of eyeball positioning is further improved.

Optionally, in order to obtain more right-eye training samples and left-eye training samples, the following operations may be further performed on the plurality of right-eye training samples and the plurality of left-eye training samples:

and respectively performing at least one of left-right rotation +/-pi/20, up-down left-right translation of each 4 pixels and image scaling by 1.0 +/-0.1 times on the plurality of right eye training samples and the plurality of left eye training samples to obtain more right eye training samples and left eye training samples.

By generating a plurality of samples according to one sample, the accuracy of the left and right eye filters can be checked by comparing training results among a plurality of samples generated by the one sample in addition to acquiring more right eye training samples and left eye training samples.

Optionally, the plurality of right eye training samples and the plurality of left eye training samples are log transformed. By carrying out log transformation on a plurality of right eye training samples and a plurality of left eye training samples, the influence of shadow and illumination can be reduced, the accuracy of identifying human eyes by the left and right eye filters is improved, and the accuracy and the stability of eyeball positioning are further improved.

Furthermore, the plurality of right eye training samples and the plurality of left eye training samples may also be normalized, e.g., to [0,1 ].

702. A plurality of precision filters are generated that correspond to the plurality of human eye training samples, respectively.

Specifically, the two-dimensional gaussian distribution image G corresponding to the plurality of right eye training samples and the plurality of left eye training samples are respectively obtained, the process may be implemented by a first preset formula, and the first preset formula may be:

wherein x_cy_cThe coordinate position of the left or right eye is indicated, and the range of the control peak is indicated.

Wherein, the value directly influence the detection accuracy and the anti-noise performance of whole wave filter, so predetermine the value that the formula was confirmed through the second:

wherein, subscript left represents the left eye, subscript right represents the right eye, Test represents the eyeball coordinate point located by the filter, True represents the eyeball coordinate point really marked manually, and Error represents the measurement criterion.

And respectively taking positive integers, and verifying according to the preset formula, wherein in practical application, positive integers between [2 ] and [ 4] can be taken.

And performing fast Fourier transform on the two-dimensional Gaussian distribution image G of the right eye and/or the left eye to generate F corresponding to the plurality of right eye training samples and the plurality of left eye training samples respectively.

Generating a plurality of accurate filters corresponding to the plurality of right eye training samples and the plurality of left eye training samples respectively by using a relationship between a spatial domain and a frequency domain, where the process may be implemented by a third preset formula, and the third preset formula may be:

wherein H^*(w, v) is a plurality of precise filters corresponding to the plurality of right eye training samples and the plurality of left eye training samples, wherein the precise filters include a precise right eye filter and/or a precise left eye filter.

703. Left and right eye filters are generated from the plurality of precision filters.

Specifically, the right-eye filter and the left-eye filter are respectively generated by averaging the accurate right-eye filter and the accurate left-eye filter through the linear operating characteristic of fourier transform, and the process may be implemented by a fourth preset formula, where the fourth preset formula may be:

h_ava(x，y)＝H_ava ^*(w，v)

wherein,is the process of performing an inverse Fourier transform, h_ava(x, y) is a right-eye or left-eye filter, and the right-eye filter and the left-eye filter are collectively referred to as a left-eye and right-eye filter.

The right-eye filter and the left-eye filter are generated by averaging the accurate right-eye filter and the accurate left-eye filter respectively, so that the influence of the difference between training samples on the accuracy of the left-eye filter and the accuracy of the right-eye filter on human eye identification is avoided, and the accuracy of the left-eye filter and the right-eye filter on human eye identification is improved.

It should be noted that, the parallel operation of the left eye and the right eye to obtain the left-right eye filter in the above-mentioned step 702-703 is only an example, and the left-right eye filter may be obtained by performing the above-mentioned correlation operation on the left eye to obtain the left-eye filter and then performing the above-mentioned correlation operation on the right eye to obtain the right-eye filter, or the left-right eye filter may be obtained by performing the above-mentioned correlation operation on the right eye to obtain the right-eye filter and then performing the above-mentioned correlation operation on the left eye to obtain the left-eye filter, which is not limited in this embodiment of the present invention.

Optionally, as shown in fig. 8, before performing step 208, the method further includes:

801. and acquiring a plurality of 2D key point coordinates of the target object and 2D coordinates of the eyeball position.

Specifically, the obtaining of the coordinates of the plurality of 2D key points of the target object and the 2D coordinates of the eyeball position may be obtaining of an offline manual mark, or may be performing positioning in the manner from step 201 to step 205 on the target object obtained in real time, which is not limited in this embodiment of the present invention.

802. A plurality of 3D keypoint coordinates of the target object and 3D coordinates of the eyeball position are calculated.

Specifically, the coordinates of the plurality of 3D key points and the 3D coordinates of the eyeball position of the target object are calculated through the conversion relationship calculated in step 205, which is not described herein again.

It should be noted that step 201 to step 207 may be performed first, and then step 801 and step 802 may be performed, step 801 and step 802 may be performed first, and then step 201 to step 207 may be performed, step 801 and step 802 may be performed during the process of performing step 201 to step 207, or step 801 and step 802 may be performed offline.

EXAMPLE III

An embodiment of the present invention provides an electronic device 9, and as shown in fig. 9, the electronic device includes:

the key point obtaining module 91 is configured to obtain coordinates of a plurality of 2D key points of a face in a current video frame;

a face region determining module 92, configured to determine a face region according to the coordinates of the plurality of 2D key points;

the filtering module 93 is configured to filter the face region through the left and right eye filters to obtain a 2D coordinate of an eyeball position of a face in the current video frame;

a keypoint conversion module 94 for converting the plurality of 2D keypoint coordinates into a corresponding plurality of 3D keypoint coordinates;

the key point conversion module 94 is further configured to convert the 2D coordinates of the eyeball position into the 3D coordinates of the eyeball position according to a conversion relationship between the plurality of 2D key point coordinates and the corresponding plurality of 3D key point coordinates;

a driving module 95, configured to drive an eyeball position of the target object according to the 3D coordinate of the eyeball position and the 3D coordinate of the initial position of the eyeball;

and a display module 96 for displaying the driven eyeball of the target object.

Optionally, the apparatus further comprises:

the normalization processing module 97 is configured to perform normalization processing on the size and/or illumination of the face region to obtain a normalized face region;

and the frequency domain conversion module 98 is configured to convert the normalized face region into frequency domain data.

Optionally, the filtering module 93 includes a constructing module 931, a position determining module 932, a judging module 933, and a calculating module 934;

the building module 931 is configured to filter the frequency domain data through the left and right eye filters, and respectively build regions of interest of the left and right eyes according to the filtered output maps;

a position determining module 932, configured to determine, in the regions of interest of the left and right eyes, position coordinates where a maximum value in the filtering output result is located, and obtain position coordinates of the left and right eyes corresponding to the regions of interest of the left and right eyes, respectively;

a judging module 933, configured to judge whether the position coordinate of the left eye is in the left eye region, if so, trigger the position determining module 932 to determine that the position coordinate of the left eye is a 2D coordinate of an eyeball position of the left eye, and if not, trigger the calculating module 934 to calculate the position coordinate of the eyeball position according to the key point information of the left eye, and use the calculated position coordinate as the 2D coordinate of the eyeball position of the left eye; and

the determining module 933 is further configured to determine whether the position coordinate of the right eye is in the right eye region, if so, trigger the position determining module 932 to determine that the position coordinate of the right eye is a 2D coordinate of an eyeball position of the right eye, and if not, trigger the calculating module 934 to calculate the position coordinate of the eyeball position according to the key point information of the right eye, and use the calculated position coordinate as the 2D coordinate of the eyeball position of the right eye.

Optionally, the apparatus further comprises:

and an integral determining module 99, configured to perform integral projection in a preset area around the position coordinate, and adjust the 2D coordinate of the eyeball position of the left eye and/or the 2D coordinate of the eyeball position of the right eye according to the coordinate of the position of the integral minimum value.

Optionally, the apparatus further comprises:

a sample obtaining module 910, configured to obtain a plurality of human eye training samples;

a precise filter generating module 911, configured to generate a plurality of precise filters corresponding to the plurality of human eye training samples, respectively;

a left and right eye filter generation module 912 for generating left and right eye filters according to the plurality of precision filters.

Optionally, the apparatus further comprises:

a conversion relation calculating module 913, configured to calculate a conversion relation between the coordinates of the plurality of 2D key points and the corresponding coordinates of the plurality of 3D key points according to the 3D model data.

Optionally, the apparatus further comprises:

a target object key point obtaining module 914, configured to obtain multiple 2D key point coordinates of the target object and 2D coordinates of the eyeball position;

a target object 3D coordinate calculation module 915, configured to calculate a plurality of 3D key point coordinates of the target object and 3D coordinates of an eyeball position.

Optionally, the driving module 95 further includes:

an offset obtaining module 951, configured to obtain relative offsets between the 3D coordinates of the eyeball positions of the left and right eyes and the 3D coordinates of the initial positions of the eyeballs of the left and right eyes respectively;

a mapping module 952, configured to map the relative offsets of the left and right eyes to 3D coordinates of eyeball positions of the left and right eyes of the target object, respectively; and

a target object coordinate conversion module 953 for converting 3D coordinates of eyeball positions of left and right eyes of the target object into 2D coordinates to drive the eyeball position of the target object;

Optionally, the offset obtaining module 951 further includes:

a comparison module 9511, configured to compare the 3D coordinates of the positions of the eyeballs of the left and right eyes with the 3D coordinates of the initial positions of the eyeballs of the left and right eyes, respectively, and obtain offsets of the left and right eyes according to a comparison result;

a relative offset calculation module 9512, configured to calculate relative offsets according to the offsets and the widths and heights of the regions of interest of the corresponding left and right eyes, respectively.

Optionally, the target object coordinate conversion module 953 further includes:

an adjusting module 9531, configured to perform preset proportion weighting adjustment on the currently converted 2D coordinate according to the converted 2D coordinate obtained from the previous frame, so as to obtain an adjusted 2D coordinate.

Optionally, the display module 96 is specifically configured to:

The embodiment of the invention provides electronic equipment, which drives the eyeball position of a target object according to the 3D coordinate of the eyeball position and the 3D coordinate of the initial position of the eyeball, improves the accuracy and stability of eyeball positioning and driving, can efficiently operate at a moving end, improves user experience, and meets the requirement of a user for controlling the target object through eyeball rotation; in addition, the 2D coordinates of the eyeball positions of the human faces in the current video frame are obtained by filtering the human face regions through the left eye filter and the right eye filter, so that compared with the traditional method of simply relying on picture texture information or only positioning the eyeballs through the filters, the method provided by the embodiment of the invention combines the filter positioning eyeballs and the human face key point positioning technology, so that the accuracy and the stability of eyeball positioning are further improved, and meanwhile, because the filters can be obtained by offline training, the method does not need to occupy processing resources when a mobile terminal runs, the running speed at the mobile terminal is further improved, the user experience is improved, and the user requirements are met; in addition, the 2D key point coordinates are converted into the corresponding 3D key point coordinates, so that the scales of key points of the face in the acquired instant video frame are kept consistent, and meanwhile, the front face is performed, so that the influence on eyeball positioning caused by the fact that the face is positioned on the side face or the scales are not consistent is avoided, meanwhile, the 2D coordinates of the eyeball position are converted into the 3D coordinates of the eyeball position, so that the 3D coordinates of the eyeball position are acquired in the 3D space with the consistent scales and the front face, the eyeball positioning accuracy is further improved, the eyeball position of a target object is driven through the 3D coordinates, and the eyeball driving accuracy and stability are further improved; in addition, the eyeball driving method provided by the embodiment of the invention can be applied to eyeball control of target objects in games and the like, enriches the modes of controlling the target objects in the games, and meets the requirements of users on controlling the target objects through eyeball rotation, thereby further improving the user experience.

Example four

An embodiment of the present invention provides an electronic device 10, which, referring to fig. 10, includes a display 1001, a camera 1002, a memory 1003, and a processor 1004 connected to the display 1001, the camera 1002, and the memory 1003, where the memory 1003 is used to store a set of program codes, and the processor 1004 calls the program codes stored in the memory 1003 to perform the following operations:

converting the plurality of 2D key point coordinates into a plurality of corresponding 3D key point coordinates;

converting the 2D coordinates of the eyeball position into the 3D coordinates of the eyeball position according to the conversion relation between the 2D key point coordinates and the corresponding 3D key point coordinates;

and displaying the eyeballs of the driven target object.

Optionally, the processor 1004 calls the program code stored in the memory 1003 for performing the following operations:

and converting the normalized face area into frequency domain data.

and performing integral projection in a peripheral preset area of the position coordinates, and adjusting the 2D coordinates of the eyeball position of the left eye and/or the 2D coordinates of the eyeball position of the right eye according to the coordinates of the position of the integral minimum value.

obtaining a plurality of human eye training samples;

All the above-mentioned optional technical solutions can be combined arbitrarily to form the optional embodiments of the present invention, and are not described herein again.

It is noted that the interfaces mentioned in the embodiments of the present invention are only for further illustrating the methods of the embodiments of the present invention, and are merely exemplary and not specific herein.

It should be noted that: in the electronic device provided in the foregoing embodiment, when implementing the eye driving method, only the division of the functional modules is illustrated, and in practical applications, the functions may be distributed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the electronic device provided by the above embodiment and the eyeball driving method embodiment belong to the same concept, and the specific implementation process thereof is described in the method embodiment, which is not described herein again.

It will be understood by those skilled in the art that all or part of the steps of implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The present invention is not limited to the above preferred embodiments, and any modifications, equivalent replacements, improvements, etc. within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An eye-ball driving method, the method comprising:

and displaying the driven eyeballs of the target object.

2. The method according to claim 1, wherein before the filtering the face region through the left and right eye filters to obtain the 2D coordinates of the eyeball position of the face in the current video frame, the method further comprises:

and converting the normalized face area into frequency domain data.

3. The method according to claim 2, wherein the filtering the face region through left and right eye filters to obtain 2D coordinates of an eyeball position of the face in the current video frame comprises:

4. The method of claim 3, wherein after performing the step of determining the position coordinates or the step of calculating the position coordinates of the eye position, the method further comprises:

5. The method of any one of claims 1 to 4, further comprising:

obtaining a plurality of human eye training samples;

6. The method of claim 1, wherein prior to said converting said plurality of 2D keypoint coordinates to a corresponding plurality of 3D keypoint coordinates, the method further comprises:

7. The method of claim 1 or 6, further comprising:

8. The method according to any one of claims 1 to 7, wherein the driving the eyeball position of the target object according to the 3D coordinates of the eyeball position and the 3D coordinates of the initial position of the eyeball comprises:

9. An electronic device, characterized in that the device comprises:

10. The apparatus of claim 9, further comprising: