CN108090463B

CN108090463B - Object control method, device, storage medium and computer equipment

Info

Publication number: CN108090463B
Application number: CN201711483962.7A
Authority: CN
Inventors: 徐冬成
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2017-12-29
Filing date: 2017-12-29
Publication date: 2021-10-26
Anticipated expiration: 2037-12-29
Also published as: CN108090463A; WO2019128917A1

Abstract

The invention relates to an object control method, an object control device, a storage medium and a computer device, wherein the object control method comprises the following steps: determining an object to be controlled; determining the pupil position in a human face image frame collected according to a time sequence; respectively determining the starting position and the ending position of a pupil moving path in the pupil positions determined in the multi-frame face image frames according to the acquisition time sequence of the corresponding face image frames; determining an object control type according to the starting position and the ending position; and controlling the object to be controlled according to the object control type. The scheme provided by the application improves the accuracy rate of object control.

Description

Object control method, device, storage medium and computer equipment

Technical Field

The present invention relates to the field of computer technologies, and in particular, to an object control method, an object control apparatus, a storage medium, and a computer device.

Background

With the development of internet technology, it is more and more common to control a target object on an internet platform. For example, when an image is processed in an image processing application, the material provided by the image processing application is controlled; or, control the interactive object to interact in the game application, etc.

In the conventional technique, when the object is controlled, the object is usually manually controlled by a human. However, in the manual control process, the hand or finger usually causes occlusion to affect the accuracy of the object control, thereby resulting in low accuracy of the object control.

Disclosure of Invention

In view of the above, it is necessary to provide an object control method, an object control apparatus, a storage medium, and a computer device, which solve the problem that the accuracy of the object control is relatively low.

An object control method, comprising:

determining an object to be controlled;

determining the pupil position in a human face image frame collected according to a time sequence;

respectively determining the starting position and the ending position of a pupil moving path in the pupil positions determined in the multi-frame face image frames according to the acquisition time sequence of the corresponding face image frames;

determining an object control type according to the starting position and the ending position;

and controlling the object to be controlled according to the object control type.

An object control device comprising:

the object determination module is used for determining an object to be controlled;

the position determining module is used for determining the pupil position in the human face image frame acquired according to the time sequence; respectively determining the starting position and the ending position of a pupil moving path in the pupil positions determined in the multi-frame face image frames according to the acquisition time sequence of the corresponding face image frames;

the type determining module is used for determining the object control type according to the starting position and the ending position;

and the object control module is used for controlling the object to be controlled according to the object control type.

A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, causes the processor to perform the steps of:

determining an object to be controlled;

A computer device comprising a memory and a processor, the memory having stored therein a computer program that, when executed by the processor, causes the processor to perform the steps of:

determining an object to be controlled;

According to the object control method, the device, the storage medium and the computer equipment, after the target to be controlled is determined, the pupil position is automatically determined in the face image frames acquired according to the time sequence, and then the starting position and the ending position of the pupil moving path are respectively determined in the pupil positions determined in the multi-frame face image frames according to the acquisition time sequence of the corresponding face image frames, so that the object control type is determined according to the moving path of the pupil to control the target according to different control types. Therefore, the target is controlled through eyes, the problem that the target is shielded by hands or fingers in the manual control process is avoided, and the accuracy of object control is improved.

Drawings

FIG. 1 is a flow diagram illustrating a method for object control in one embodiment;

FIG. 2 is a diagram illustrating an interface of an image processing application in one embodiment;

FIG. 3 is a diagram illustrating the division of the edge regions of the eye in one embodiment;

FIG. 4 is a schematic illustration of determining an angle of rotation in one embodiment;

FIG. 5 is a diagram illustrating binocular pupil location change vectors with their respective corresponding vector directions aligned, according to one embodiment;

FIG. 6 is a diagram illustrating the disparity between the respective vector directions of the binocular pupil location change vectors in one embodiment;

FIG. 7 is a flowchart illustrating an object control method according to another embodiment;

FIG. 8 is a block diagram of an object control device in one embodiment;

FIG. 9 is a block diagram of an object control apparatus according to another embodiment;

FIG. 10 is a diagram showing an internal structure of a computer device according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

Fig. 1 is a flowchart illustrating an object control method according to an embodiment. The embodiment is mainly illustrated by applying the method to computer equipment. Referring to fig. 1, the object control method specifically includes the steps of:

and S102, determining the object to be controlled.

Wherein the object to be controlled is a target to be controlled. The object to be controlled may be an entity object or a virtual object. The entity object is an object having a physical form, such as an intelligent robot or an intelligent home, which actually exists. The virtual object is a digitized object that is operable by the computer device, such as material provided by an image processing application or a virtual character in a game application. The virtual object may be a two-dimensional virtual object or a three-dimensional virtual object.

Specifically, the computer device can display the objects for selection through the computer interface, detect the selection instruction, and determine the object specified by the selection instruction as the object to be controlled. The computer device may specifically detect a predefined trigger operation, upon detection of which a corresponding selection instruction is triggered. The trigger operation may be an operation on a control in the computer interface, such as a touch operation or a cursor click operation on the control. The triggering operation may also be a click on a predefined physical button, or a shaking operation or a voice input operation, etc., which is triggered when an object for selection is presented through a computer interface.

In one embodiment, an image processing application may be run on a computer device, which may provide image processing material. Image processing material such as a two-dimensional sticker or a three-dimensional virtual model, etc. The image processing application can display the image processing materials according to the user instruction, and continuously select the image processing materials selected by the user from the displayed image processing materials according to the user instruction as the object to be controlled. Among them, the image processing application is an application capable of image processing. The image processing application includes a social application, a game application, a live application, or a photo application, etc. The image processing application may be an augmented reality application.

For example, in a photographing application, the photographing application may provide image processing material (such as a sticker), and a user may select the image processing material as an object to be controlled to control the object to be controlled through pupil movement. In a game application, the photographing application may provide image processing materials (e.g., virtual game images) that a user may select as an object to be controlled to control the object to be controlled through pupil movement. Wherein the gaming application may be an augmented reality gaming application.

And S104, determining the pupil position in the human face image frame acquired according to the time sequence.

Wherein, the pupil is a small round hole at the center of the iris in the eye and is a passage for light to enter the eye. The pupil position is the position of the pupil in the image frame of the human face. The pupil position may specifically be a position of a pupil center point in the face image frame. The pupil center is the center of the pupil region.

Specifically, the computer device may acquire the face image frames frame by frame according to an acquisition timing sequence of the face image frames, and when each image frame is acquired, identify a face region in the face image frame through a face detection algorithm, and determine a pupil position in the face region. The face detection algorithm can be customized according to needs, and can be an OpenCV face detection algorithm, a face detection algorithm carried by an IOS or Android system, a best-image face detection algorithm and the like. The face detection algorithm may return whether the face image frame contains a face, a specific face region, and a pupil position in the face region.

The human face image frames acquired according to the time sequence can be image frames acquired by calling a built-in camera by the computer equipment, and can also be image frames acquired by calling an external camera connected with the computer equipment by the computer equipment.

And S106, respectively determining the starting position and the ending position of the pupil moving path in the pupil positions determined in the face image frames of the plurality of frames according to the acquisition time sequence of the corresponding face image frame.

The multi-frame human face image frame can be a human face image frame with any number of frames more than one frame. The pupil movement path is a path formed by the pupil movement. The starting position of the pupil movement path is the position of the pupil at the time of starting control of the object to be controlled by pupil movement. The end position of the pupil movement path is the position of the pupil when the control of the object to be controlled by the pupil movement is ended.

In one embodiment, in any two adjacent human face image frames, a path formed by the positions of the pupils in the human face image frame in the front time sequence and the human face image frame in the back time sequence may be a pupil movement path. The pupil position in the human face image frame at the front of the time sequence is the initial position of the pupil moving path, and the pupil position in the human face image frame at the back of the time sequence is the end position of the pupil moving path.

In one embodiment, the computer device may set in advance a start trigger condition to start controlling the object to be controlled by pupil movement and an end trigger condition to end controlling the object to be controlled by pupil movement. When the computer equipment judges that the human face image frame meets the starting triggering condition, determining the pupil position in the human face image frame as the starting position of the pupil moving path; and when the human face image frame is judged to meet the end triggering condition, determining the pupil position in the human face image frame as the end position of the pupil moving path.

The start triggering condition and the end triggering condition may be triggering conditions set for face features in the face image frame, for example, the face image frame includes face features such as eyes being closed or mouth being opened. The start triggering condition and the end triggering condition may also be triggering conditions set for the pupil position in the human face image frame, for example, the pupil position in the human face image frame is at a specific position. The start trigger condition and the end trigger condition may be the same trigger condition or different trigger conditions. The same triggering condition is that, for example, when the computer device determines that the face image frame meets the triggering condition for the first time, the pupil position in the face image frame is determined as the starting position of the pupil movement path; and when the human face image frame is judged to meet the triggering condition again, determining the pupil position in the human face image frame as the end position of the pupil moving path, and circulating alternately in sequence.

And S108, determining the object control type according to the starting position and the ending position.

Wherein the object control type is a type to which the change made by the control object belongs. An object control type such as an object pan control type, an object rotation control type, or an object zoom control type, etc. It can be understood that different object control types correspond to different position characteristics, and the computer device may set the position characteristics corresponding to the different object control types in advance.

Specifically, after determining the start position and the end position, the computer device may continue to determine the position features of the start position and the end position, compare the determined position features with the position features corresponding to the respective object control types, and use the object control type corresponding to the position feature matched with the determined position feature as the object control type corresponding to the determined start position and end position. Wherein the position feature may be a relative position feature of the pupil position in the eye region.

And S110, controlling the object to be controlled according to the object control type.

Specifically, the computer device controls the object to be controlled according to the object control type after determining the object control type. For example, when the object control type is an object translation type, the object to be controlled is translated; when the object control type is an object rotation type, the object to be controlled is rotated, and the like.

In one embodiment, the control parameter according to which the computer device controls the object to be controlled may be a default parameter corresponding to each object control type that is set in advance. For example, the default parameter corresponding to the object translation control type may be a default translation distance, and the computer device translates the object to be controlled according to the default translation distance each time the computer device translates the object to be controlled.

In one embodiment, the control parameter according to which the computer device controls the object to be controlled may be a control parameter related to the pupil movement path. It will be appreciated that different pupil movement paths may be subject to different control parameters. The computer device can preset a mode of calculating the control parameters according to the pupil movement path, and when the computer device specifically controls the object to be controlled, the computer device calculates the current control parameters according to the current pupil movement path so as to control the object to be controlled according to the control parameters.

According to the object control method, after the target to be controlled is determined, the pupil position is automatically determined in the face image frames acquired according to the time sequence, and then the starting position and the ending position of the pupil moving path are respectively determined according to the acquisition time sequence of the corresponding face image frame in the pupil positions determined in the multi-frame face image frames, so that the object control type is determined according to the moving path of the pupil to control the target according to different control types. Therefore, the target is controlled through eyes, the problem that the target is shielded by hands or fingers in the manual control process is avoided, and the accuracy of object control is improved.

In one embodiment, S104 includes: collecting image frames according to a time sequence; when the collected image frame comprises a human face area, determining and caching the pupil position in the image frame; upon detecting that the next frame includes an image frame of a face region, S106 includes: taking the cached pupil position as the initial position of the pupil movement path; and acquiring the pupil position of the next image frame comprising the human face area as the end position of the pupil moving path.

In particular, the computer device may be run with an image processing application that may be configured with a pupil movement mode. The pupil movement mode is a mode in which an object to be controlled is controlled by movement of a pupil. Specifically, the pupil movement mode may be configured to be started when the image processing application is started, or may be configured to be started according to a specific trigger instruction after the image processing application is executed.

For example, FIG. 2 shows an interface diagram of an image processing application in one embodiment. Referring to fig. 2, it is assumed that the image processing application is a photographing application. The user can set a standing-by pupil opening moving mode in advance; the pupil movement mode can be triggered to enter by a specific action (such as blinking, gesture, etc.) after entering the camera. After the user enters the camera to select the image processing material 210, the user can directly perform finger dragging (compatible with a finger moving mode) on the material by pressing the material, and when the user presses the image processing material 210 for a long time, the user pops up the quick floating layer 220 to quickly click to enter the pupil moving mode. The pupil movement mode is a mode for controlling the material by pupil movement.

Further, after the pupil movement mode should be started in image processing, the computer device may acquire image frames according to a time sequence according to a fixed or dynamic frame rate, so as to obtain image frames acquired according to a time sequence. The computer equipment can also acquire image frames according to the time sequence through a built-in or externally connected camera under the current visual field of the camera to obtain the image frames acquired according to the time sequence. Wherein the field of view of the camera may change due to changes in the pose and position of the computer device.

The computer device may further provide an AR (Augmented Reality) shooting mode through an image processing application, and acquire an image frame from a real scene after the AR shooting mode is selected, and acquire the acquired image frame.

Further, the computer device identifies whether the acquired image frame contains a face region by a face detection algorithm every time the image frame is acquired. Specifically, the computer device may extract image data included in the acquired image frame through a face detection algorithm and detect whether the image data includes face feature data. If the image data is detected to contain the face feature data by the computer equipment, the image frame is judged to contain a face region, and the pupil position is determined in the face region according to the extracted image data. The computer device thus acquires and buffers the pupil position in the currently acquired image frame.

In one embodiment, when the acquired image frame includes a face region, determining and buffering pupil positions in the image frame includes: when the collected image frame comprises a face area, determining a face deflection angle corresponding to the image frame; when the face deflection angle matches a reference deflection angle representing a frontal face image, the pupil position in the image frame is determined and buffered.

The human face deflection angle is an angle deflected when the human face absolutely faces the camera. It can be understood that in a real scene, it is difficult to control the face to be absolutely opposite to the camera, so that in the range of error allowance, the collected face image within a specific deflection angle can be used as a front face image. The specific deflection angle is the reference deflection angle.

In one embodiment, the computer device may calculate a face deflection angle corresponding to a face image in the image frame through a face detection algorithm, match the face deflection angle with a reference deflection angle representing the front face image, and when the face deflection angle does not exceed the reference deflection angle representing the front face image, determine that the face deflection angle corresponding to the image frame matches the reference deflection angle representing the front face image, thereby determining and caching a pupil position in the image frame.

The face deflection angle may be an euler angle of the face region based on a stereo coordinate system of the image frame. The three-dimensional coordinate system of the image frame is a three-dimensional coordinate system including a horizontal axis (X axis), a vertical axis (Y axis), and a vertical axis (Z axis). The axial direction of the horizontal axis of the stereo coordinate system of the image frame is along the horizontal direction of the image frame, the axial direction of the vertical axis is along the vertical direction of the image frame, and the axial direction of the vertical axis is along the direction vertical to the image frame. The computer equipment can calculate the reference deflection angle corresponding to each direction.

For example, the angle of deflection of the face region around the X axis may be Pitch, the angle of deflection around the Y axis may be Yaw, and the angle of deflection around the Z axis may be Roll. The reference deflection angle corresponding to the X-axis direction may be 15 degrees, the reference deflection angle corresponding to the Y-axis direction may be 15 degrees, and the reference deflection angle is not limited in the Z-axis direction. Then, when Pitch is less than 15 degrees and Yaw is less than 15 degrees, the face region included in the acquired image frame may be determined to be a front face region.

In this embodiment, when a face image included in an image frame meets a frontal face condition, a pupil position is determined in the image frame, and then an object to be controlled is controlled according to the pupil position of the image frame, so that the pupil position determined in the frontal face is more accurate, and subsequent object control is more accurate.

In this way, the computer device buffers the pupil position in the currently detected image frame every time one image frame is detected, when it is detected that the image frame includes a face image. And the computer equipment continues to detect the acquired image frames frame by frame according to the acquisition time sequence until the next image frame including the image frame of the face area is detected, and acquires the cached pupil position as the initial position of the pupil moving path and acquires the pupil position of the image frame currently including the face area as the end position of the pupil moving path. In this way, after each two frames of image frames including the face region are continuously determined, the computer device can control the object to be controlled according to the object control type corresponding to the pupil position in the two frames of face image frames in real time.

In the above embodiment, when the face image frame is collected, whether the collected face image frame includes a face region is detected in real time, and when the currently collected image frame includes a face region, the pupil position of the face region is cached, and the face image collected in the next frame is continuously detected. Therefore, when two frames of image frames including the face please-region are continuously detected, the object is controlled in real time according to the change of the pupil positions in the two frames of image frames, and the object control efficiency is improved.

In one embodiment, S106 includes: extracting the human face features of each human face image frame according to the acquisition time sequence of a plurality of human face image frames; when the face features accord with the initial moving features, acquiring the pupil positions of the face image frames from which the face features are extracted as the initial positions of the pupil moving paths; and when the face features accord with the movement ending features, acquiring the pupil position of a face image frame from which the face features are extracted as the ending position of the pupil movement path.

The face features are data reflecting certain features of the face. The start movement feature is a face feature representing an image frame at the time when the control object starts to move through the pupil. The end movement feature is a face feature indicating an image frame at the time of ending the movement of the control object through the pupil. It is to be appreciated that the computer device can implement the set start movement feature and the end movement feature. Ending movement features such as an eye-closing feature or a mouth-opening feature.

Specifically, the computer device may extract the face features of each face image frame according to the acquisition timing sequence of the plurality of face image frames, and sequentially compare each face feature with the initial moving feature. And when the face features accord with the initial moving features, acquiring the pupil positions of the face image frames from which the face features are extracted as the initial positions of the pupil moving paths. The computer equipment can continue to compare the residual face features with the ending movement features one by one, and when the face features accord with the ending movement features, the pupil positions of the face image frames extracted from the face features are obtained to serve as the ending positions of the pupil movement paths.

The starting movement feature and the ending movement feature may be the same face feature or different face features. The same human face features, for example, when the computer device determines that the human face image frame includes the human face features for the first time, the computer device determines the pupil position in the human face image frame as the starting position of the pupil movement path; and when the human face image frame is judged to comprise the human face features again, determining the pupil position in the human face image frame as the end position of the pupil moving path, and sequentially and alternately circulating.

In this embodiment, the image frame indicating that the object starts to be controlled by the pupil and the image frame indicating that the object finishes to be controlled by the pupil are determined by the face features of the image frame, so that it is clear which pupil position changes are effective pupil position changes for object control, and then the object is controlled according to the effective pupil position changes, thereby ensuring the accuracy of object control.

In one embodiment, S108 includes: determining a first eye edge area in a face image frame corresponding to the end position; when the ending position is located in the first eye edge area, determining a second eye edge area in the face image frame corresponding to the starting position; and when the starting position is located in the second eye edge area, judging that the object control type corresponding to the starting position and the ending position is the object rotation control type.

Wherein the eye edge region is an edge region of the eye image. The eye edge region may be a ring-shaped region formed by extending inward a certain distance starting from the outline of the eye image. The eye edge region may be an annular region having a predetermined distance from a boundary line of the eye image, which is formed by four fixed points, i.e., upper, lower, left, and right points of the eye image.

For example, fig. 3 shows a schematic diagram of dividing the eye edge region in one embodiment. Referring to fig. 3, it can be seen that the schematic diagram includes an outline 310 of the eye image, and a ring-shaped region 320 formed by extending inward a distance starting from the outline 310 can be defined as an eye edge region. The outline 310 of the eye image may include an outline upper vertex (top)311, an outline left vertex (left)312, an outline lower vertex (bottom)313, and an outline right vertex (right) 314. The line connecting the top contour vertex 311 and the left contour vertex 312, the left contour vertex 312 and the bottom contour vertex 313, the bottom contour vertex 313 and the right contour vertex 314, and the right contour vertex 314 and the top contour vertex 311 is the boundary line 330 of the eye image. The annular region 340 at a distance from the boundary line of the eye image may also be defined as an eye edge region.

In particular, the computer device may determine a first eye edge region in the image frame of the face corresponding to the end position. When the ending position is judged to be located in the first eye edge area, the second eye edge area is continuously determined in the face image frame corresponding to the starting position. When the starting position is located in the second eye edge area, the pupil positions in the two image frames can be judged to be located in the eye edge area, and therefore the object control types corresponding to the starting position and the ending position are judged to be the object rotation control types. Wherein the object rotation control type indicates that the type to which the control object change belongs is rotation.

In this embodiment, a pupil movement manner corresponding to the object rotation control type is provided, and specifically, the rotation control object is reflected by the pupil movement characteristics that the starting position and the ending position of the pupil movement path are both located in the eye edge region, so that the rotation control object is realized when such pupil movement characteristics are detected.

In one embodiment, controlling an object to be controlled according to an object control type includes: determining the position of the center point of the eyes in the human face image frames corresponding to the starting position and the ending position respectively; determining a rotation angle according to the initial position, the end position and the eye center point position; and rotating the object to be controlled according to the rotation angle.

Wherein the eye center point position is a position of a center point of the eye image. In one embodiment, the computer device may cross the left vertex of the contour of the image of the eye and make a perpendicular line connecting the top vertex of the contour and the bottom vertex of the contour, and use the intersection point of the perpendicular line and the connecting line as the center point of the eye.

Specifically, the computer device may perform image size adjustment on the face image frames corresponding to the start position and the end position, so that the eye images in the two image frames are overlapped, and further, it is ensured that the position of the center point of the eye is determined in the normalized scene, thereby ensuring that the rotation angle is determined based on the normalized scene, so as to rotate the object to be controlled according to the rotation angle.

In one embodiment, the computer device may determine a pupil pixel position and an eye center point position in the face image frames corresponding to the start position and the end position, further establish an eye reference coordinate system with the eye center point position as a reference position, and then map the pupil pixel position to a coordinate in the eye reference coordinate system to obtain the pupil position.

The pupil pixel point position is the position of the pixel point of the pupil center point in the image frame. A pixel coordinate system is typically established for an image frame with the left vertex of the image frame as a reference point. The pupil pixel point position is the coordinate position under the pixel coordinate system. The reference point here may in particular be the origin of a coordinate system. It is understood that the coordinate system herein is a two-dimensional coordinate system, including a horizontal axis (X-axis) and a vertical axis (Y-axis).

Specifically, the computer device may calculate a transformation matrix that changes the origin of the coordinate system from the left vertex of the image frame to the position of the center point of the eye, so that the pupil pixel position may be mapped to the coordinate in the eye reference coordinate system according to the transformation matrix to obtain the pupil position.

For example, the pixel coordinate system established by using the left vertex as the reference point can be changed to the eye reference coordinate system established by using the eye central point as the reference position through rotation and translation. Assuming the angle of rotation required is θ, the distance of translation required along the X-axis is t_xThe distance required to translate along the Y axis is t_y. Then the transformation matrix

Wherein the content of the first and second substances,

in order to be a standard transformation matrix, the transformation matrix,

in order to be a matrix of rotations,

is a translation matrix. At this time, assume that the pupil pixel point position is (x)₁,y₁) Then (x)₁,y₁)*A＝(x₂,y₂) I.e. the pupil position in the eye reference frame.

In one embodiment, the computer device may further perform normalization processing on the pupil position in the eye reference coordinate system, and map the pupil pixel position to the normalized coordinates in the eye reference coordinate system, thereby obtaining a normalized pupil position. For example, the distance from the center point of the eye to the left vertex of the contour of the eye image can be taken as the standard distance EyeW, so that (x) is₂,y₂) Dividing by the standard distance EyeW yields the normalized coordinate (x)₃,y₃)。

In this embodiment, when calculating the pupil positions of multiple frames of image frames, the pupil positions are determined in a unified coordinate system through coordinate system transformation, so as to ensure the accuracy of the subsequent pupil position moving path.

Further, the computer device may use an included angle between a connection line between the start position and the eye center point position and a connection line between the end position and the eye center point position as the rotation angle, calculate a specific angle value of the rotation angle according to coordinates of the start position, the end position and the eye center point position in the eye reference coordinate system and a trigonometric function, and select the object to be controlled according to the angle value. Wherein the direction of the object rotation is the same as the direction of the pupil movement path.

For example, FIG. 4 shows a schematic illustration of determining the angle of rotation in one embodiment. Referring to fig. 4, the diagram includes an eye center point position (O)410, a starting position (a)420 of a pupil movement path, and an ending position (B)430 of the pupil movement path. Then, the rotation angle 440 is the angle θ between OA and OB_ab. Wherein, theta_abCan be calculated by the following formula: theta_ab＝(|OA|²+|OB|²-|AB|²)/(2*|OA|*|OB|)。

In one embodiment, when the pupil position is the dual-purpose pupil position, the computer device may randomly select a rotation angle of one of the pupils, or select a rotation angle of the pupil that rotates the larger pupil, or select an average value of the binocular rotation angles to rotate the object to be controlled.

In the above embodiment, when the object to be controlled is controlled according to the object control type, the object to be controlled is dynamically rotated according to the angle of rotation relative to the center of the eye when the actual pupil moves, so that the object to be controlled is more flexibly rotated.

In one embodiment, the object control method further includes: when the ending position is not in the first eye edge region, or when the ending position is in the first eye edge region and the starting position is not in the second eye edge region, determining a pupil position change vector according to the starting position and the ending position; and acquiring a scalar value of the pupil position change vector, and determining an object control parameter corresponding to the object control type according to the scalar value. Controlling an object to be controlled according to an object control type, comprising: and controlling the object to be controlled according to the object control type and the object control parameter.

The object control parameter is a parameter according to which the control object is changed. The object control parameter may specifically be a translation distance or a scaling, etc.

Specifically, when the end position is not in the first eye edge region, or when the end position is in the first eye edge region and the start position is not in the second eye edge region, the computer apparatus may determine that the object control type may be a panning control type or a zooming control type. At this time, the computer device may further determine whether the current pupil movement corresponds to the translational control type or the zooming control type according to the characteristics of the pupil position.

The computer device may continue to detect whether the determined pupil position is a monocular pupil position or a binocular pupil position. When the determined pupil position is the monocular pupil position, the object control type corresponding to the starting position and the ending position can be determined as the object translation control type. At this time, the computer device may determine the pupil position change vector according to the start position and the end position, and calculate a scalar value of the pupil position change vector, thereby controlling the object to be controlled to translate the distance of the scalar value along the vector direction of the pupil position change vector.

In one embodiment, when the pupil position is a dual purpose pupil position; when the pupil position change vector is a binocular pupil position change vector, the object control method further includes: determining respective corresponding vector directions of binocular pupil position change vectors; and when the vector directions corresponding to the binocular pupil position change vectors are consistent, judging that the object control types corresponding to the starting position and the ending position are the object translation control types. Controlling the object to be controlled according to the object control type and the object control parameter, comprising: and translating the object to be controlled according to the object control parameters.

Specifically, the computer device may determine respective vector directions corresponding to the binocular pupil position change vectors, calculate an included angle between the two vector directions, and when the included angle is smaller than a preset angle, determine that the respective vector directions corresponding to the binocular pupil position change vectors are consistent, thereby determining that the object control types corresponding to the start position and the end position are the object translation control types.

In one embodiment, the vector angle may be calculated by the following equation.

vectorMul＝leftIrisMoveVector2.x*rightIrisMoveVector2.x+leftIrisMoveVector2.y*rightIrisMoveVector2.y；

verctorLeftLen＝sqrt(leftIrisMoveVector2.x*leftIrisMoveVector2.x+leftIrisMoveVector2.y*leftIrisMoveVector2.y)；

verctorRightLen＝sqrt(rightIrisMoveVector2.x*rightIrisMoveVector2.x+rightIrisMoveVector2.y*rightIrisMoveVector2.y)；

leftIrisMoveVector2.x＝currentPosLeft.x-lastPosLeft.x；

leftIrisMoveVector2.y＝currentPosLeft.y-lastPosLeft.y；

rightIrisMoveVector2.x＝currentPosright.x-lastPosright.x；

rightIrisMoveVector2.x＝currentPosright.y-lastPosright.y。

Wherein, vectorMul is the intermediate variable, leftIrisis MoveVector2.X is left eye pupil position along X axle direction change vector, leftIrisis Movector 2.Y is left eye pupil position along Y axle direction change vector, rightIrisis Movector 2.X is right eye pupil position along X axle direction change vector, rightIrisis Movector 2.Y is right eye pupil position along Y axle direction change vector. verctrleftlen is the scalar value of the left eye pupil position change vector, and verctrrightlen is the scalar value of the right eye pupil position change vector. (currentPosLeft. x, currentPosLeft. y) as the left eye end position coordinates,

(lastposleft. x, lastposleft. y) is the left eye starting position coordinate, (currentposright. x, currentposright. y) is the right eye ending position coordinate, (lastposright. x, lastposright. y) is the right eye starting position coordinate.

The angle angleValue ═ acos (anglecmos value) of the binocular pupil position change vector can be obtained from the formula cos (angleValue) · verctorlen ═ vectorMul. Wherein, angleCosValue ═ vectorMul/(vectorleftlen ═ vectorrightlen).

It will be appreciated that the above coordinates are coordinates that map pupil pixel locations into an eye reference coordinate system.

Further, the computer device may calculate a scalar value of the binocular pupil location change vector, and select one scalar value from the scalar values, or a scalar value corresponding to a larger value, or an average value of the two scalar values, thereby controlling the object to be controlled to translate the distance of the selected scalar value along the vector direction of the pupil location change vector.

For example, fig. 5 shows a schematic diagram of the case where the respective vector directions of the binocular pupil position change vectors coincide in one embodiment. The preset angle is assumed to be 90 degrees. Referring to fig. 5, it can be seen that the included angle between the vector directions corresponding to the binocular pupil position change vectors is 0 degree, and 0 degree is smaller than 90 degrees, then the object control types corresponding to the start position and the end position are the object translation control types, so as to control the object translation.

In this embodiment, a pupil movement manner corresponding to the object translation control type is provided, and specifically, the object to be translated is reflected by the pupil movement features corresponding to the binocular pupil position change vectors and having the same vector direction, so that the object to be controlled is translated when the pupil movement features are detected.

In one embodiment, the object control method further includes: and when the vector directions corresponding to the binocular pupil position change vectors are not consistent, judging that the object control types corresponding to the starting position and the ending position are the object scaling control types. Controlling the object to be controlled according to the object control type and the object control parameter, comprising: and adjusting the size of the object to be controlled according to the object control parameters.

Specifically, the computer device may determine respective vector directions corresponding to the binocular pupil position change vectors, calculate an included angle between the two vector directions, and determine that the respective vector directions corresponding to the binocular pupil position change vectors are not consistent when the included angle reaches a preset angle, thereby determining that the object control types corresponding to the start position and the end position are the object rotation control types.

Further, the computer apparatus may calculate a distance twoeye center w1 between center points of the eyes and a distance twiroriss center w1 between pupil positions of the eyes in the image frame corresponding to the start position; the distance twoeyecentrw 2 between the center points of the eyes and the distance tworirisc centrw 2 between the positions of the pupils of the eyes in the image frame corresponding to the end position; further, scale ═ acostvalue @ twoirissecew 1 @ twoeyecew 2/(twoirissecew 2 twoweyecew 1) was calculated. Wherein scale is a scaling scale, and aConstValue is a variable coefficient for adjusting the sensitivity of scaling.

For example, fig. 6 shows a schematic diagram of the case where the respective vector directions of the binocular pupil position change vectors do not coincide in one embodiment. The preset angle is assumed to be 90 degrees. Referring to fig. 6, it can be seen that the included angle between the vector directions corresponding to the binocular pupil position change vectors is 180 degrees, and 180 degrees is greater than 90 degrees, then the object control types corresponding to the start position and the end position are the object zoom control types, so as to control the object zoom. Wherein the distance twoeye center w1 between the center points of the binocular eyes and the distance twcorriscenter w1 between the pupil positions of the binocular eyes.

In this embodiment, a pupil movement manner corresponding to an object zoom control type is provided, and specifically, a pupil movement feature with inconsistent vector directions corresponding to binocular pupil position change vectors is used to reflect the object to be zoomed, so that the object to be controlled is zoomed when the pupil movement feature is detected.

In the above embodiment, when the object to be controlled is controlled according to the object control type, the change amplitude of the object to be controlled is dynamically controlled according to the scalar value of the pupil position change vector formed when the actual pupil moves, so that the object to be controlled is controlled more flexibly.

Fig. 7 is a schematic flowchart of an object control method in a specific embodiment, where the object control method specifically includes the following steps:

s702, determining the object to be controlled.

S704, collecting image frames according to a time sequence, and detecting whether the collected image frames comprise a human face area; if yes, jumping to step S706; if not, the process returns to step S704.

S706, judging whether the face deflection angle corresponding to the image frame is matched with a reference deflection angle representing a front face image; if yes, go to step S708; if not, the process returns to step S704.

S708, determining positions of binocular pupil pixels and positions of center points of eyes in the image frame; establishing an eye reference coordinate system by taking the eye central point position as a reference position; and mapping the positions of the binocular pupil pixels to coordinates in an eye reference coordinate system to obtain the positions of the binocular pupils and cache the positions.

S710, when detecting that the next frame comprises the image frame of the human face area, taking the cached positions of the pupils of the two eyes as the respective initial positions of the movement paths of the pupils of the two eyes; and acquiring positions of the pupils of the two eyes of the image frame including the face area at present as respective end positions of the pupil movement paths of the two eyes.

S712, judging whether the ending position is located in a first eye edge area in the corresponding human face image frame; if yes, go to step S714; if not, the process returns to step S720.

S714, judging whether the starting position is located in a second eye edge area in the corresponding human face image frame; if yes, go to step S716; if not, the process returns to step S720.

S716, the object control type corresponding to the dual-purpose starting position and the dual-purpose ending position is judged to be the object rotation control type.

S718, determining a rotation angle according to the starting position, the ending position and the position of the eye center point; and rotating the object to be controlled according to the rotation angle.

And S720, determining binocular pupil position change vectors according to the binocular starting position and the binocular ending position.

S722, judging whether the vector directions corresponding to the binocular pupil position change vectors are consistent or not; if yes, jumping to step S724; if not, the process returns to step S728.

S724, judging that the object control types corresponding to the dual-purpose starting position and the dual-purpose ending position are the object translation control types.

S726, obtaining a scalar value of the pupil position change vector; determining an object control parameter corresponding to the object control type according to the scalar value; and translating the object to be controlled according to the object control parameters.

And S728, judging the object control type corresponding to the start position and the end position of the dual purpose as the object scaling control type.

S730, acquiring a scalar value of the pupil position change vector; determining an object control parameter corresponding to the object control type according to the scalar value; and adjusting the size of the object to be controlled according to the object control parameters.

In this embodiment, after the target to be controlled is determined, the pupil position is automatically determined in the face image frames acquired according to the time sequence, and then the start position and the end position of the pupil movement path are respectively determined according to the acquisition time sequence of the corresponding face image frame in the pupil positions determined in the plurality of face image frames, so that the object control type is determined according to the movement path of the pupil to control the target according to different control types. Therefore, the target is controlled through eyes, the problem that the target is shielded by hands or fingers in the manual control process is avoided, and the accuracy of object control is improved.

Moreover, in the mode of controlling the object through pupil movement, the hands and feet are inconvenient for users who cannot realize object control through manual operation, and great operation convenience is provided.

It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, the steps are not necessarily executed in sequence as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the above embodiments may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the sub-steps or the stages is not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a part of the sub-steps or the stages of other steps.

As shown in fig. 8, in one embodiment, an object control apparatus 800 is provided. Referring to fig. 8, the object control apparatus 800 includes: an object determination module 801, a location determination module 802, a type determination module 803, and an object control module 804.

An object determining module 801, configured to determine an object to be controlled.

A position determining module 802, configured to determine a pupil position in a face image frame acquired in time series; and respectively determining the starting position and the ending position of a pupil moving path in the pupil positions determined in the multi-frame face image frames according to the acquisition time sequence of the corresponding face image frames.

A type determining module 803, configured to determine an object control type according to the start position and the end position.

And the object control module 804 is configured to control the object to be controlled according to the object control type.

The object control device 800 automatically determines the pupil position in the face image frames acquired according to the time sequence after determining the target to be controlled, and further determines the starting position and the ending position of the pupil movement path in the pupil positions determined in the face image frames of the plurality of frames according to the acquisition time sequence of the corresponding face image frame, so as to determine the object control type according to the movement path of the pupil to control the target according to different control types. Therefore, the target is controlled through eyes, the problem that the target is shielded by hands or fingers in the manual control process is avoided, and the accuracy of object control is improved.

In one embodiment, the position determination module 802 is further configured to acquire image frames in time sequence; when the collected image frame comprises a human face area, determining and caching the pupil position in the image frame; when detecting that the next frame comprises an image frame of a human face area, taking the cached pupil position as the starting position of a pupil moving path; and acquiring the pupil position of the next image frame comprising the human face area as the end position of the pupil moving path.

In one embodiment, the position determining module 802 is further configured to determine a face deflection angle corresponding to the image frame when the acquired image frame includes a face region; when the face deflection angle matches a reference deflection angle representing a frontal face image, the pupil position in the image frame is determined and buffered.

In one embodiment, the position determining module 802 is further configured to extract the facial features of each of the facial image frames according to the acquisition timing sequence of the plurality of facial image frames; when the face features accord with the initial moving features, acquiring the pupil positions of the face image frames from which the face features are extracted as the initial positions of the pupil moving paths; and when the face features accord with the movement ending features, acquiring the pupil position of a face image frame from which the face features are extracted as the ending position of the pupil movement path.

In one embodiment, the type determining module 803 is further configured to determine a first eye edge region in the image frame of the face corresponding to the end position; when the ending position is located in the first eye edge area, determining a second eye edge area in the face image frame corresponding to the starting position; and when the starting position is located in the second eye edge area, judging that the object control type corresponding to the starting position and the ending position is the object rotation control type.

In one embodiment, the object control module 804 is further configured to determine an eye center point position in the face image frame corresponding to each of the start position and the end position; determining a rotation angle according to the initial position, the end position and the eye center point position; and rotating the object to be controlled according to the rotation angle.

As shown in fig. 9, in one embodiment, the object control apparatus 800 further includes: parameter determination module 805.

A parameter determining module 805 configured to determine a pupil position change vector according to the start position and the end position when the end position is not in the first eye edge region or when the end position is in the first eye edge region and the start position is not in the second eye edge region; obtaining a scalar value of the pupil position change vector; and determining the object control parameters corresponding to the object control types according to the scalar values. The object control module 804 is further configured to control the object to be controlled according to the object control type and the object control parameter.

In one embodiment, the pupil location is a dual purpose pupil location; the pupil position change vector is a binocular pupil position change vector. The type determining module 803 is further configured to determine respective corresponding vector directions of the binocular pupil position change vectors; and when the vector directions corresponding to the binocular pupil position change vectors are consistent, judging that the object control types corresponding to the starting position and the ending position are the object translation control types. The object control module 804 is further configured to translate the object to be controlled according to the object control parameters.

In one embodiment, the type determining module 803 is further configured to determine that the object control types corresponding to the starting position and the ending position are the object scaling control types when the respective vector directions of the binocular pupil position change vectors are not consistent. The object control module 804 is further configured to adjust a size of the object to be controlled according to the object control parameter.

In one embodiment, the position determining module 802 is further configured to determine a pupil pixel position and an eye center point position in the face image frames acquired in time series; establishing an eye reference coordinate system by taking the eye central point position as a reference position; and mapping the pupil pixel position to a coordinate in an eye reference coordinate system to obtain the pupil position.

FIG. 10 is a diagram illustrating an internal structure of a computer device in one embodiment. As shown in fig. 10, the computer device includes a processor, a memory, a network interface, a camera, an input device, and a display screen connected through a system bus. Wherein the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program that, when executed by the processor, causes the processor to implement the object control method. The internal memory may also have a computer program stored therein, which, when executed by the processor, causes the processor to execute the object control method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, or an external keyboard, a touch pad or a mouse, and the like. The computer device may be a user terminal, and the user terminal may specifically be at least one of a mobile phone, a tablet computer, a personal digital assistant, or a wearable device. Those skilled in the art will appreciate that the architecture shown in fig. 10 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, the object control apparatus provided in the present application may be implemented in a form of a computer program, the computer program may be executed on a computer device as shown in fig. 10, and a nonvolatile storage medium of the computer device may store various program modules constituting the object control apparatus, such as an object determination module 801, a position determination module 802, a type determination module 803, an object control module 804, and the like shown in fig. 8. The computer program made up of the respective program modules causes the processor to execute the steps in the object control method of the respective embodiments of the present application described in the present specification.

For example, the computer device shown in fig. 10 may be used to determine an object to be controlled by the object determination module 801 in the object control apparatus 800 shown in fig. 8. The position determining module 802 is used for determining the pupil position in the face image frames acquired according to the time sequence; and respectively determining the starting position and the ending position of a pupil moving path in the pupil positions determined in the multi-frame face image frames according to the acquisition time sequence of the corresponding face image frames. The pass-type determination module 803 is used to determine the object control type from the start position and the end position. The object control module 804 is used for controlling the object to be controlled according to the object control type.

In one embodiment, a computer readable storage medium is provided, having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the steps of: determining an object to be controlled; determining the pupil position in a human face image frame collected according to a time sequence; respectively determining the starting position and the ending position of a pupil moving path in the pupil positions determined in the multi-frame face image frames according to the acquisition time sequence of the corresponding face image frames; determining the object control type according to the starting position and the ending position; and controlling the object to be controlled according to the object control type.

In one embodiment, determining pupil positions in time-sequenced frames of images of a human face comprises: collecting image frames according to a time sequence; when the acquired image frame includes a face region, the pupil position in the image frame is determined and buffered. When detecting that the next frame comprises an image frame of a face area, respectively determining the starting position and the ending position of a pupil moving path in the pupil positions determined in the face image frames of multiple frames according to the acquisition time sequence of the corresponding face image frame, wherein the method comprises the following steps: taking the cached pupil position as the initial position of the pupil movement path; and acquiring the pupil position of the next image frame comprising the human face area as the end position of the pupil moving path.

In one embodiment, in the pupil positions determined in the multiple frames of face image frames, the determining the starting position and the ending position of the pupil moving path respectively according to the acquisition timing sequence of the corresponding face image frame includes: extracting the human face features of each human face image frame according to the acquisition time sequence of a plurality of human face image frames; when the face features accord with the initial moving features, acquiring the pupil positions of the face image frames from which the face features are extracted as the initial positions of the pupil moving paths; and when the face features accord with the movement ending features, acquiring the pupil position of a face image frame from which the face features are extracted as the ending position of the pupil movement path.

In one embodiment, determining the object control type from the starting location and the ending location comprises: determining a first eye edge area in a face image frame corresponding to the end position; when the ending position is located in the first eye edge area, determining a second eye edge area in the face image frame corresponding to the starting position; and when the starting position is located in the second eye edge area, judging that the object control type corresponding to the starting position and the ending position is the object rotation control type.

In one embodiment, the computer program, when executed by the processor, further causes the processor to perform the steps of: when the ending position is not in the first eye edge region, or when the ending position is in the first eye edge region and the starting position is not in the second eye edge region, determining a pupil position change vector according to the starting position and the ending position; obtaining a scalar value of the pupil position change vector; and determining the object control parameters corresponding to the object control types according to the scalar values. Controlling an object to be controlled according to an object control type, comprising: and controlling the object to be controlled according to the object control type and the object control parameter.

In one embodiment, the pupil location is a dual purpose pupil location; the pupil position change vector is a binocular pupil position change vector. The computer program, when executed by the processor, further causes the processor to perform the steps of: determining respective corresponding vector directions of binocular pupil position change vectors; and when the vector directions corresponding to the binocular pupil position change vectors are consistent, judging that the object control types corresponding to the starting position and the ending position are the object translation control types. Controlling the object to be controlled according to the object control type and the object control parameter, comprising: and translating the object to be controlled according to the object control parameters.

In one embodiment, the computer program, when executed by the processor, further causes the processor to perform the steps of: and when the vector directions corresponding to the binocular pupil position change vectors are not consistent, judging that the object control types corresponding to the starting position and the ending position are the object scaling control types. Controlling the object to be controlled according to the object control type and the object control parameter, comprising: and adjusting the size of the object to be controlled according to the object control parameters.

In one embodiment, determining pupil positions in time-sequenced frames of images of a human face comprises: determining pupil pixel positions and eye center point positions in a face image frame collected according to a time sequence; establishing an eye reference coordinate system by taking the eye central point position as a reference position; and mapping the pupil pixel position to a coordinate in an eye reference coordinate system to obtain the pupil position.

The storage medium automatically determines the pupil position in the face image frames acquired according to the time sequence after determining the target to be controlled, and further determines the starting position and the ending position of the pupil moving path in the pupil positions determined in the face image frames of multiple frames according to the acquisition time sequence of the corresponding face image frames, so that the object control type is determined according to the moving path of the pupil to control the target according to different control types. Therefore, the target is controlled through eyes, the problem that the target is shielded by hands or fingers in the manual control process is avoided, and the accuracy of object control is improved.

In one embodiment, there is provided a computer device comprising a memory and a processor, the memory having stored therein a computer program that, when executed by the processor, causes the processor to perform the steps of: determining an object to be controlled; determining the pupil position in a human face image frame collected according to a time sequence; respectively determining the starting position and the ending position of a pupil moving path in the pupil positions determined in the multi-frame face image frames according to the acquisition time sequence of the corresponding face image frames; determining the object control type according to the starting position and the ending position; and controlling the object to be controlled according to the object control type.

After the target to be controlled is determined, the pupil position is automatically determined in the face image frames acquired according to the time sequence, and then the starting position and the ending position of the pupil moving path are respectively determined in the pupil positions determined in the face image frames of multiple frames according to the acquisition time sequence of the corresponding face image frame, so that the object control type is determined according to the moving path of the pupil to control the target according to different control types. Therefore, the target is controlled through eyes, the problem that the target is shielded by hands or fingers in the manual control process is avoided, and the accuracy of object control is improved.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a non-volatile computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the program is executed. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above examples only show some embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. An object control method, comprising:

determining an object to be controlled;

determining position characteristics of the starting position and the ending position;

matching the determined position characteristics with the position characteristics corresponding to the object control types, wherein the position characteristics corresponding to the object translation control types are that the vector directions corresponding to the binocular pupil position change vectors are consistent, the position characteristics corresponding to the object rotation control types are that the pupil positions in the multi-frame human face image frames are all located in the eye edge area, and the position characteristics corresponding to the object scaling control types are that the vector directions corresponding to the binocular pupil position change vectors are inconsistent;

taking the object control type corresponding to the position feature matched with the determined position feature as the object control type corresponding to the determined starting position and ending position;

2. The method of claim 1, wherein determining pupil positions in time-sequentially acquired frames of human face images comprises:

collecting image frames according to a time sequence;

when the collected image frame comprises a human face area, determining and caching the pupil position in the image frame;

when detecting that the next frame comprises an image frame of a face area, respectively determining the starting position and the ending position of a pupil moving path in the pupil positions determined in the face image frames of multiple frames according to the acquisition time sequence of the corresponding face image frame, wherein the method comprises the following steps:

taking the cached pupil position as the starting position of a pupil moving path;

and acquiring the pupil position of the image frame of which the next frame comprises the human face area as the end position of the pupil moving path.

3. The method of claim 2, wherein determining and buffering pupil positions in the image frame when the acquired image frame includes a face region comprises:

when the collected image frame comprises a face area, determining a face deflection angle corresponding to the image frame;

and when the human face deflection angle is matched with a reference deflection angle representing a front face image, determining and caching pupil positions in the image frame.

4. The method according to claim 1, wherein the determining the starting position and the ending position of the pupil movement path in the determined pupil positions in the plurality of face image frames according to the acquisition timing sequence of the corresponding face image frame respectively comprises:

extracting the human face features of each human face image frame according to the acquisition time sequence of a plurality of human face image frames;

when the face features accord with the initial moving features, acquiring the pupil positions of the face image frames from which the face features are extracted as the initial positions of pupil moving paths;

and when the face features accord with the movement ending features, acquiring the pupil positions of the face image frames from which the face features are extracted as the ending positions of the pupil movement paths.

5. The method of claim 1, wherein determining the location characteristics of the starting location and the ending location comprises:

determining a first eye edge area in the face image frame corresponding to the end position;

when the end position is located in the first eye edge region, determining a second eye edge region in a face image frame corresponding to the starting position;

when the starting position is located in the second eye edge area, the position characteristics corresponding to the starting position and the ending position are judged to be that the pupil positions in the multi-frame human face image frames are both located in the eye edge area.

6. The method according to claim 5, wherein the controlling the object to be controlled according to the object control type comprises:

determining the position of the center point of the eyes in the human face image frames corresponding to the starting position and the ending position respectively;

determining a rotation angle according to the starting position, the ending position and the eye central point position;

and rotating the object to be controlled according to the rotation angle.

7. The method of claim 5, further comprising:

when the ending position is not in the first eye edge region, or when the ending position is in the first eye edge region and the starting position is not in the second eye edge region,

determining a pupil position change vector according to the starting position and the ending position;

obtaining a scalar value of the pupil position change vector;

determining an object control parameter corresponding to the object control type according to the scalar value;

the controlling the object to be controlled according to the object control type includes:

and controlling the object to be controlled according to the object control type and the object control parameter.

8. The method of claim 7, wherein the pupil location is a dual purpose pupil location; the pupil position change vector is a binocular pupil position change vector; the method further comprises the following steps:

determining respective corresponding vector directions of binocular pupil position change vectors;

when the vector directions corresponding to the binocular pupil position change vectors are consistent, judging that the position characteristics corresponding to the starting position and the ending position are the vector directions corresponding to the binocular pupil position change vectors are consistent;

the controlling the object to be controlled according to the object control type and the object control parameter includes:

and translating the object to be controlled according to the object control parameters.

9. The method of claim 8, further comprising:

when the vector directions corresponding to the binocular pupil position change vectors are not consistent, judging that the position features corresponding to the starting position and the ending position are the vector directions corresponding to the binocular pupil position change vectors are not consistent;

and adjusting the size of the object to be controlled according to the object control parameter.

10. The method according to any one of claims 1 to 9, wherein determining pupil positions in time-sequentially acquired frames of human face images comprises:

determining pupil pixel positions and eye center point positions in a face image frame collected according to a time sequence;

establishing an eye reference coordinate system by taking the eye central point position as a reference position;

and mapping the pupil pixel position to a coordinate in the eye reference coordinate system to obtain the pupil position.

11. An object control apparatus, characterized by comprising:

the type determining module is used for determining the position characteristics of the starting position and the ending position; matching the determined position characteristics with the position characteristics corresponding to the object control types, wherein the position characteristics corresponding to the object translation control types are that the vector directions corresponding to the binocular pupil position change vectors are consistent, the position characteristics corresponding to the object rotation control types are that the pupil positions in the multi-frame human face image frames are all located in the eye edge area, and the position characteristics corresponding to the object scaling control types are that the vector directions corresponding to the binocular pupil position change vectors are inconsistent; taking the object control type corresponding to the position feature matched with the determined position feature as the object control type corresponding to the determined starting position and ending position;

12. The apparatus of claim 11, wherein the position determining module is further configured to acquire image frames in a time sequence; when the collected image frame comprises a human face area, determining and caching the pupil position in the image frame; when detecting that the next frame comprises an image frame of a human face area, taking the cached pupil position as the starting position of a pupil moving path; and acquiring the pupil position of the image frame of which the next frame comprises the human face area as the end position of the pupil moving path.

13. The apparatus of claim 12, wherein the position determining module is further configured to determine a face deflection angle corresponding to an acquired image frame when the image frame includes a face region; and when the human face deflection angle is matched with a reference deflection angle representing a front face image, determining and caching pupil positions in the image frame.

14. The apparatus according to claim 11, wherein the position determining module is further configured to extract the facial features of each of the face image frames according to the acquisition timing sequence of the plurality of face image frames; when the face features accord with the initial moving features, acquiring the pupil positions of the face image frames from which the face features are extracted as the initial positions of pupil moving paths; and when the face features accord with the movement ending features, acquiring the pupil positions of the face image frames from which the face features are extracted as the ending positions of the pupil movement paths.

15. The apparatus of claim 11, wherein the type determining module is further configured to determine a first eye edge region in the image frame corresponding to the end position; when the end position is located in the first eye edge region, determining a second eye edge region in a face image frame corresponding to the starting position; when the starting position is located in the second eye edge area, the position characteristics corresponding to the starting position and the ending position are judged to be that the pupil positions in the multi-frame human face image frames are both located in the eye edge area.

16. The apparatus of claim 15, wherein the object control module is further configured to determine an eye center point location in the face image frame corresponding to each of the start location and the end location; determining a rotation angle according to the starting position, the ending position and the eye central point position; and rotating the object to be controlled according to the rotation angle.

17. The apparatus of claim 15, wherein the type determination module is further configured to determine a pupil position change vector based on the starting position and the ending position when the ending position is not in the first eye edge region or when the ending position is in the first eye edge region and the starting position is not in the second eye edge region; obtaining a scalar value of the pupil position change vector; determining an object control parameter corresponding to the object control type according to the scalar value;

and the object control module is also used for controlling the object to be controlled according to the object control type and the object control parameters.

18. The device of claim 17, wherein the pupil location is a dual purpose pupil location; the pupil position change vector is a binocular pupil position change vector;

the type determining module is also used for determining the vector direction corresponding to the binocular pupil position change vectors; when the vector directions corresponding to the binocular pupil position change vectors are consistent, judging that the position characteristics corresponding to the starting position and the ending position are the vector directions corresponding to the binocular pupil position change vectors are consistent; the object control module is further configured to translate the object to be controlled according to the object control parameter.

19. The apparatus of claim 18, wherein the type determining module is further configured to determine that the position features corresponding to the starting position and the ending position are that the vector directions corresponding to the binocular pupil position change vectors are not consistent when the vector directions corresponding to the binocular pupil position change vectors are not consistent; the object control module is further configured to adjust a size of the object to be controlled according to the object control parameter.

20. The apparatus according to any one of claims 11 to 19, wherein the position determining module is further configured to determine a pupil pixel position and an eye center point position in a time-series acquired human face image frame; establishing an eye reference coordinate system by taking the eye central point position as a reference position; and mapping the pupil pixel position to a coordinate in the eye reference coordinate system to obtain the pupil position.

21. A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, causes the processor to carry out the steps of the method according to any one of claims 1 to 10.

22. A computer device comprising a memory and a processor, the memory having stored thereon a computer program, characterized in that the computer program, when executed by the processor, causes the processor to carry out the steps of the method according to any one of claims 1 to 10.