WO2019128917A1 - 对象控制方法、装置、存储介质和计算机设备 - Google Patents

对象控制方法、装置、存储介质和计算机设备 Download PDF

Info

Publication number
WO2019128917A1
WO2019128917A1 PCT/CN2018/123062 CN2018123062W WO2019128917A1 WO 2019128917 A1 WO2019128917 A1 WO 2019128917A1 CN 2018123062 W CN2018123062 W CN 2018123062W WO 2019128917 A1 WO2019128917 A1 WO 2019128917A1
Authority
WO
WIPO (PCT)
Prior art keywords
pupil
image frame
determining
object control
face image
Prior art date
Application number
PCT/CN2018/123062
Other languages
English (en)
French (fr)
Inventor
徐冬成
Original Assignee
腾讯科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 腾讯科技（深圳）有限公司 filed Critical 腾讯科技（深圳）有限公司
Publication of WO2019128917A1 publication Critical patent/WO2019128917A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/193—Preprocessing; Feature extraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04845—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range for image manipulation, e.g. dragging, rotation, expansion or change of colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/048—Indexing scheme relating to G06F3/048
    • G06F2203/04806—Zoom, i.e. interaction techniques or interactors for controlling the zooming operation

Definitions

  • the present application relates to the field of computer technology, and in particular, to an object control method, apparatus, storage medium, and computer device.
  • an object control method, apparatus, storage medium, and computer apparatus are provided.
  • An object control method is performed by a computer device, the method comprising:
  • the start position and the end position of the pupil movement path are respectively determined according to the acquisition timing of the corresponding face image frame;
  • the object to be controlled is controlled according to the object control type.
  • An object control device includes:
  • An object determining module configured to determine an object to be controlled
  • a position determining module configured to determine a pupil position in the face image frame acquired in time series; in the pupil position determined in the multi-frame face image frame, determine the pupil movement path according to the collection timing of the corresponding face image frame respectively Starting position and ending position;
  • a type determining module configured to determine an object control type according to the starting position and the ending position
  • an object control module configured to control the object to be controlled according to the object control type.
  • a non-volatile storage medium storing computer readable instructions, when executed by one or more processors, causes one or more processors to perform the following steps:
  • the start position and the end position of the pupil movement path are respectively determined according to the acquisition timing of the corresponding face image frame;
  • the object to be controlled is controlled according to the object control type.
  • a computer device comprising a memory and a processor, the memory storing computer readable instructions, the computer readable instructions being executed by the processor such that the processor performs the following steps:
  • the start position and the end position of the pupil movement path are respectively determined according to the acquisition timing of the corresponding face image frame;
  • the object to be controlled is controlled according to the object control type.
  • FIG. 1 is a schematic flow chart of an object control method in an embodiment
  • FIG. 2 is a schematic diagram of an interface of an image processing application in an embodiment
  • Figure 3 is a schematic illustration of dividing an edge region of an eye in one embodiment
  • Figure 4 is a schematic view of determining the angle of rotation in one embodiment
  • FIG. 5 is a schematic diagram showing a case where the vector directions corresponding to the two-position pupil position change vectors are identical in one embodiment
  • FIG. 6 is a schematic diagram showing a case where the vector directions corresponding to the pupil position change vectors of the dual purpose are inconsistent in one embodiment
  • FIG. 7 is a schematic flow chart of an object control method in another embodiment
  • Figure 8 is a block diagram showing the structure of an object control device in an embodiment
  • Figure 9 is a block diagram showing the structure of an object control device in another embodiment.
  • Figure 10 is a diagram showing the internal structure of a computer device in one embodiment.
  • FIG. 1 is a schematic flow chart of an object control method in an embodiment. This embodiment is mainly illustrated by the method applied to a computer device. Referring to FIG. 1, the object control method specifically includes the following steps:
  • the object to be controlled is the target to be controlled.
  • the object to be controlled can be a solid object or a virtual object.
  • An entity object is an object that actually exists in a physical form, such as an intelligent robot or a smart home.
  • a virtual object is a digitized object that can be manipulated by a computer device, such as a material provided by an image processing application or a virtual character in a game application.
  • a virtual object can be a two-dimensional virtual object or a three-dimensional virtual object.
  • the computer device can display the selected object through the computer interface, and detect the selection instruction, and determine the object specified by the selection instruction as the object to be controlled.
  • the computer device can specifically detect a predefined trigger operation, and trigger a corresponding selection instruction when the trigger operation is detected.
  • the triggering operation can be an operation on a control in a computer interface, such as a touch operation on a control or a cursor click operation.
  • the triggering operation may also be a click on a predefined physical button, or a shaking operation or a voice input operation triggered when a selected object is presented through a computer interface.
  • an image processing application can be run on the computer device, the image processing application providing image processing material.
  • Image processing materials such as two-dimensional stickers or three-dimensional virtual models.
  • the image processing application may display the image processing material according to the user instruction, and continue to select the image processing material selected by the user from the displayed image processing material as the object to be controlled according to the user instruction.
  • the image processing application is an application capable of image processing.
  • Image processing applications include social applications, game applications, live applications, or photo applications.
  • the image processing application can be an augmented reality application.
  • the photographing application can provide an image processing material (such as a sticker), and the user can select the image processing material as an object to be controlled to control the object to be controlled through the pupil movement.
  • the photographing application can provide an image processing material (such as a virtual game image), and the user can select the image processing material as an object to be controlled to control the object to be controlled through the pupil movement.
  • the game application may be an augmented reality game application.
  • the pupil is a small round hole in the center of the iris in the eye, which is the passage of light into the eye.
  • the pupil position is the position of the pupil in the face image frame.
  • the pupil position may specifically be the position of the pupil center point in the face image frame.
  • the center point of the pupil is the center point of the pupil area.
  • the computer device may acquire the face image frame frame by frame according to the acquisition timing of the face image frame, and recognize the face region in the face image frame by the face detection algorithm every time the image frame is acquired, and in the person The pupil position is determined in the face area.
  • the face detection algorithm can be customized according to needs, such as the OpenCV face detection algorithm, the face detection algorithm that comes with the IOS or Android system, or the face detection algorithm.
  • the face detection algorithm can return whether the face image frame includes a face, a specific face area, and a pupil position in the face area.
  • the face image frame acquired by the time sequence may be an image frame acquired by a computer device by calling a built-in camera, or may be an image frame acquired by an external camera connected to the computer device by the computer device.
  • the multi-frame face image frame may be a face image frame of an arbitrary frame number of more than one frame.
  • the pupil movement path is a path formed by the movement of the pupil.
  • the starting position of the pupil movement path is the position of the pupil when the object to be controlled is started to be controlled by the pupil movement.
  • the end position of the pupil movement path is the position at which the pupil is controlled when the object to be controlled is controlled by the pupil movement.
  • the path formed by the front face image frame and the pupil position in the late face image frame may be a pupil movement path.
  • the pupil position in the face image frame with the highest timing is the start position of the pupil movement path, and the pupil position in the face image frame with the lower timing is the end position of the pupil movement path.
  • the computer device may previously set a start trigger condition for starting control of the object to be controlled by the pupil movement, and ending an end trigger condition for controlling the object to be controlled by the pupil movement.
  • the computer device determines that the face image frame satisfies the start trigger condition, the pupil position in the face image frame is determined as the start position of the pupil movement path; and when the face image frame is determined to satisfy the end trigger condition, the face is determined The pupil position in the image frame is determined as the end position of the pupil movement path.
  • the start trigger condition and the end trigger condition may be trigger conditions set for the face feature in the face image frame, for example, the face image frame includes a closed face or a mouth feature.
  • the start trigger condition and the end trigger condition may also be trigger conditions set for the pupil position in the face image frame, such as the pupil position in the face image frame at a specific position.
  • the start trigger condition and the end trigger condition may be the same trigger condition or different trigger conditions.
  • the same trigger condition for example, when the computer device first determines that the face image frame satisfies the trigger condition, the pupil position in the face image frame is determined as the start position of the pupil movement path; and the face image frame is determined to satisfy the trigger condition again. At this time, the pupil position in the face image frame is determined as the end position of the pupil movement path, and alternately cycles in sequence.
  • S108 Determine an object control type according to the start position and the end position.
  • the object control type is the type to which the change made by the control object belongs.
  • Object control types such as object translation control type, object rotation control type, or object scaling control type. It can be understood that different object control types correspond to different location features, and the computer device can set corresponding location features of different object control types in advance.
  • the computer device may continue to determine the position features of the start position and the end position, and compare the determined position feature with the position feature corresponding to each object control type, and determine The object control type corresponding to the position feature of the position feature matching is the object control type corresponding to the determined start position and end position.
  • the position feature may be a relative position feature of the pupil position in the eye region.
  • the computer device controls the object to be controlled according to the object control type. For example, when the object control type is the object translation type, the object to be controlled is translated; when the object control type is the object rotation type, the object to be controlled is rotated.
  • control parameter by which the computer device controls the object to be controlled may be a default parameter corresponding to each object control type set in advance.
  • the default parameter corresponding to the object translation control type may be the default translation distance, and the computer device translates the object to be controlled according to the default translation distance each time the object to be controlled is translated.
  • the computer device controls the control parameters upon which the object to be controlled is based, which may be control parameters associated with the pupil movement path. It can be understood that different pupil movement paths can be subject to different control parameters.
  • the computer device may preset a manner of calculating a control parameter according to the pupil movement path. When the computer device specifically controls the object to be controlled, the current control parameter is calculated according to the current pupil movement path, so as to control the object to be controlled according to the control parameter.
  • the object control method automatically determines the pupil position in the face image frame acquired in time series after determining the target to be controlled, and then presses the corresponding face image frame in the pupil position determined in the multi-frame face image frame.
  • the acquisition timing determines the start position and the end position of the pupil movement path, respectively, so that the object control type is determined according to the movement path of the pupil to control the target according to different control types. In this way, the target is controlled by the eye, and the problem of the target occlusion caused by the hand or the finger during the manual control is avoided, and the object control accuracy is improved.
  • S104 includes: acquiring an image frame in time series; determining and buffering a pupil position in the image frame when the captured image frame includes a face region; and detecting an image frame including a face region in a next frame
  • S106 includes: taking the cached pupil position as the starting position of the pupil movement path; acquiring the pupil position of the image frame including the face region of the next frame as the end position of the pupil movement path.
  • the computer device can be run with an image processing application that can be configured with a pupil movement mode.
  • the pupil movement mode is a mode of controlling the object to be controlled by the movement of the pupil.
  • the pupil movement mode may be configured to be turned on when the image processing application is started, or may be configured to be turned on according to a specific trigger instruction after the image processing application is run.
  • Figure 2 shows an interface diagram of an image processing application in one embodiment.
  • the image processing application is a photographing application.
  • the user can set the standing open pupil movement mode in advance; or enter the camera to trigger the movement mode (such as blinking, gesture, etc.) by a specific action.
  • the user can also enter the camera to select the image processing material 210, press and hold the material to directly drag the material to the finger (compatible with the finger movement mode), long press the image processing material 210 will pop up the quick floating layer 220, and quickly click into the pupil movement mode.
  • the pupil movement mode is a mode of controlling the material through the pupil movement.
  • the computer device can acquire the image frames according to the timing according to a fixed or dynamic frame rate, and obtain the image frames acquired in time series.
  • the computer device can also acquire image frames acquired in time series according to the timing of the camera under the current field of view of the camera through a built-in or externally connected camera. Among them, the field of view of the camera may vary due to changes in the posture and position of the computer device.
  • the computer device can also provide an AR (Augmented Reality) shooting mode through an image processing application, and after selecting the AR shooting mode, acquire an image frame from a real scene and acquire the acquired image frame.
  • AR Augmented Reality
  • the computer device identifies, by the face detection algorithm, whether the face region is included in the captured image frame every time an image frame is acquired. Specifically, the computer device may extract image data included in the acquired image frame by using a face detection algorithm, and detect whether the image data includes facial feature data. If the computer device detects that the facial feature data is included in the image data, it is determined that the image frame includes a face region, and then the pupil position is determined in the face region according to the extracted image data. The computer device thereby acquires and caches the pupil position in the currently acquired image frame.
  • determining and buffering a pupil position in the image frame includes: determining a face deflection angle corresponding to the image frame when the captured image frame includes a face region And determining and buffering the pupil position in the image frame when the face deflection angle matches the reference deflection angle representing the face image.
  • the face deflection angle is the angle at which the face is absolutely deflected toward the camera. It can be understood that it is difficult to control the face in the real scene to be absolutely facing the camera, so that the captured face image within a certain deflection angle can be used as a face image within the range allowed by the error.
  • the specific deflection angle here is the reference deflection angle.
  • the computer device may calculate a face deflection angle corresponding to the face image in the image frame by using a face detection algorithm, and match the face deflection angle with a reference deflection angle representing the front face image, when the face is When the deflection angle does not exceed the reference deflection angle indicating the front face image, that is, it is determined that the face deflection angle corresponding to the image frame matches the reference deflection angle indicating the front face image, thereby determining and buffering the pupil position in the image frame.
  • the face deflection angle may be an Euler angle of a three-dimensional coordinate system of the face region based on the image frame.
  • the three-dimensional coordinate system of the image frame is a three-dimensional coordinate system including a horizontal axis (X axis), a vertical axis (Y axis), and a vertical axis (Z axis).
  • the axial direction of the horizontal coordinate system of the image frame is along the horizontal direction of the image frame
  • the axial direction of the vertical axis is along the vertical direction of the image frame
  • the axial direction of the vertical axis is along the direction perpendicular to the image frame.
  • the computer device can calculate a reference deflection angle corresponding to each direction.
  • the angle of the face region deflected about the X axis may be Pitch
  • the angle of deflection about the Y axis may be Yaw
  • the angle of deflection about the Z axis may be Roll.
  • the reference deflection angle corresponding to the X-axis direction may be 15 degrees
  • the corresponding reference deflection angle in the Y-axis direction may be 15 degrees
  • the Z-axis direction is not limited. Then, when the Pitch is less than 15 degrees and the Yaw is less than 15 degrees, it can be determined that the face area included in the acquired image frame is a face face area.
  • the pupil position is determined in the image frame, and then the object to be controlled is controlled according to the pupil position of the image frame, so that the positive face is determined.
  • the pupil position is more accurate, making subsequent object control more accurate.
  • the computer device caches the pupil position in the currently detected image frame when detecting that the image frame includes a face image every time an image frame is detected.
  • the computer device continues to detect the collected image frame frame by frame according to the acquisition timing until the next frame includes the image frame of the face region, obtains the buffer pupil position as the starting position of the pupil movement path, and obtains the current face including the face
  • the pupil position of the image frame of the region serves as the end position of the pupil movement path.
  • the computer device can control the object to be controlled according to the object control type corresponding to the pupil position in the face image frames of the two frames after each successive determination of two frames of the image frame including the face region.
  • the face image of the collected face image frame is detected in real time, and when the currently collected image frame includes the face region, the pupil position of the face region is cached, and Continue to detect the face image acquired in the next frame.
  • the object is controlled according to the change of the pupil position in the image frames of the two frames in real time, and the object control efficiency is improved.
  • S106 includes: extracting a face feature of each face image frame according to a collection timing of the multi-frame face image frame; and acquiring a face feature extraction when the face feature conforms to the initial moving feature;
  • the pupil position of the face image frame is used as the starting position of the pupil moving path; and when the face feature conforms to the ending moving feature, the pupil position of the face image frame from which the face feature is extracted is obtained as the ending position of the pupil moving path.
  • the face feature is data reflecting a certain feature of the face.
  • the initial movement feature is a facial feature representing an image frame when the control object is started to move through the pupil.
  • the end movement feature is a face feature indicating that the image frame is ended when the control object is moved by the pupil. It will be appreciated that the computer device can implement setting the initial movement feature and the ending movement feature. End moving features such as closed eye features or open mouth features.
  • the computer device may extract the face features of each face image frame according to the collection timing of the multi-frame face image frame, and compare each face feature with the initial moving feature in turn.
  • the pupil position of the face image frame from which the face feature is extracted is obtained as the starting position of the pupil moving path.
  • the computer device can continue to compare the remaining facial features one by one with the ending moving feature.
  • the pupil position of the facial image frame from which the facial feature is extracted is obtained as the end of the pupil moving path. position.
  • the initial moving feature and the ending moving feature may be the same facial feature or different facial features.
  • the same facial feature for example, when the computer device first determines that the face image frame includes the facial feature, the pupil position in the face image frame is determined as the starting position of the pupil moving path; When the frame includes the face feature, the pupil position in the face image frame is determined as the end position of the pupil movement path, and alternately cycles in sequence.
  • the image frame when the object is controlled by the pupil and the image frame when the object is controlled by the pupil is determined by the face feature of the image frame, thereby clarifying which changes in the pupil position are effective for the object control.
  • S108 includes: determining a first eye edge region in a corresponding face image frame at the end position; and determining, when the end position is located in the first eye edge region, in the corresponding face image frame at the start position The two eye edge regions; and when the home position is located in the second eye edge region, the object control type corresponding to the determination of the start position and the end position is the object rotation control type.
  • the edge area of the eye is the edge area of the eye image.
  • the eye edge region may be an annular region formed by extending a certain distance inward from the contour of the eye image.
  • the eye edge region may also be an annular region at a certain distance from the boundary line of the eye image, wherein the boundary line of the eye image is a boundary line formed by the upper, lower, left, and right fixed points of the eye image.
  • Figure 3 shows a schematic diagram of dividing an edge region of an eye in one embodiment.
  • the schematic includes an outline 310 of the image of the eye, and an annular region 320 formed by extending a certain distance inwardly from the contour 310 can be defined as an edge region of the eye.
  • the outline 310 of the eye image may include a top 311 on the outline, a left 312 on the outline, a bottom 313 on the outline, and a right ide on the outline 314.
  • the contour vertices 311 and the contour left vertices 312, the contour left vertices 312 and the lower contour vertices 313, the lower contour vertices 313 and the contour right vertices 314, and the contour right vertices 314 and the contour vertices 311 are the boundary lines of the eye image. 330.
  • the annular region 340 at a distance from the boundary line of the eye image can also be defined as the eye edge region.
  • the computer device may determine the first eye edge region in the corresponding face image frame at the end position. When it is determined that the end position is located in the first eye edge region, then the second eye edge region is determined to be determined in the corresponding face image frame of the home position. When the starting position is located in the second eye edge region, it can be determined that the pupil positions in the two image frames are located in the eye edge region, thereby determining that the object control type corresponding to the start position and the end position is the object rotation control type.
  • the object rotation control type indicates that the type of the control object change belongs to rotation.
  • the pupil movement mode corresponding to the object rotation control type is provided, and specifically, the pupil movement feature of the pupil movement path is located in the edge region of the eye to reflect the rotation control object, thereby detecting This kind of pupil movement feature realizes the rotation control object.
  • controlling the object to be controlled according to the object control type includes: determining a position of the eye center point in each corresponding face image frame of the start position and the end position; according to the start position, the end position, and the eye center Point position, determine the rotation angle; and rotate the object to be controlled according to the rotation angle.
  • the eye center point position is the position of the center point of the eye image.
  • the computer device may pass the left vertices of the outline of the eye image, make a vertical line connecting the vertices of the outline with the vertices of the outline, and use the intersection of the perpendicular and the line as the center point of the eye.
  • the computer device can perform image size adjustment on the corresponding face image frames of the start position and the end position, so that the eye images in the two frame image frames overlap, thereby ensuring that the eye center point position is determined in the normalized scene. Thereby, it is ensured that the rotation angle is determined based on the normalized scene to rotate the object to be controlled according to the rotation angle.
  • the computer device can determine the pupil pixel position and the eye center point position in the respective face image frames of the start position and the end position, and then establish an eye reference coordinate system with the eye center point position as a reference position.
  • the pupil pixel position is then mapped to the coordinates in the eye reference coordinate system to obtain the pupil position.
  • the pupil pixel position is the position of the pixel point of the pupil center point in the image frame.
  • a pixel coordinate system is typically established for an image frame with the left vertex of the image frame as a reference point.
  • the pupil pixel position is the coordinate position in the pixel coordinate system.
  • the reference point here may specifically be the origin of the coordinate system. It can be understood that the coordinate system here is a two-dimensional coordinate system including a horizontal axis (X axis) and a vertical axis (Y axis).
  • the computer device can calculate a transformation matrix that changes the origin of the coordinate system from the left vertex of the image frame to the position of the center point of the eye, so that the pupil pixel position can be mapped to the coordinates in the eye reference coordinate system according to the transformation matrix, and the pupil is obtained. position.
  • the pixel coordinate system established with the left vertex as the reference point needs to be rotated and translated to change to the eye reference coordinate system established with the eye center point position as the reference position.
  • the angle of rotation is ⁇
  • the distance to be translated along the X axis is t x
  • the distance to be translated along the Y axis is t y . Transformation matrix among them,
  • For the rotation matrix To translate the matrix.
  • the pupil pixel position is (x 1 , y 1 )
  • the computer device can also normalize the pupil position in the eye reference coordinate system, map the pupil pixel position to the normalized coordinates in the eye reference coordinate system, thereby obtaining a normalized pupil position.
  • the distance from the center point of the eye to the left apex of the contour of the eye image may be the standard distance EyeW, and (x 2 , y 2 ) is divided by the standard distance EyeW to obtain normalized coordinates (x 3 , y 3 ).
  • the pupil position is determined in a unified coordinate system by coordinate system transformation, and the accuracy of the subsequent pupil position movement path is ensured.
  • the computer device can use the angle between the starting position and the position of the center point of the eye, and the line connecting the ending position and the position of the center point of the eye as a rotation angle, according to the starting position, the ending position, and the eye reference coordinate system.
  • the coordinates of the position of the center point of the eye, and the trigonometric function calculate the specific angle value of the rotation angle, and then select the object to be controlled according to the angle value.
  • the direction in which the object rotates is the same as the direction in which the pupil moves the path.
  • Figure 4 shows a schematic diagram of determining the angle of rotation in one embodiment.
  • the schematic diagram includes an eye center point position (O) 410, a start position (A) 420 of the pupil movement path, and an end position (B) 430 of the pupil movement path.
  • the rotation angle 440 is the angle ⁇ ab between OA and OB.
  • the computer device can randomly select the rotation angle of one of the pupils, or select the rotation angle of the larger rotation pupil, or the average value of the binocular rotation angle to rotate The object to be controlled.
  • the object to be controlled when the object to be controlled is controlled according to the object control type, the object to be controlled is dynamically rotated according to the angle at which the actual pupil moves relative to the center of the eye, so that the object to be controlled is more flexible when rotated.
  • the object control method further includes: when the end position is not in the first eye edge region, or when the end position is in the first eye edge region, and the starting position is not in the second eye edge region, The start position and the end position determine a pupil position change vector; and obtain a scalar value of the pupil position change vector, and determine an object control parameter corresponding to the object control type according to the scalar value.
  • the object to be controlled is controlled according to the object control type, including: controlling the object to be controlled according to the object control type and the object control parameter.
  • the object control parameter is a parameter on which the control object changes.
  • the object control parameter may specifically be a translation distance or a scaling ratio or the like.
  • the computer device may determine that the object control type may be a pan control type Or zoom control type. At this time, the computer device may further determine, according to the feature of the pupil position, whether the current pupil movement corresponds to a translation control type or a zoom control type.
  • the computer device can continue to detect whether the determined pupil position is a single pupil position or a dual purpose pupil position.
  • the determined pupil position is a single-purpose pupil position
  • it can be determined that the object control type corresponding to the start position and the end position is the object shift control type.
  • the computer device can determine the pupil position change vector according to the start position and the end position, calculate the scalar value of the pupil position change vector, thereby controlling the vector direction of the object to be controlled along the pupil position change vector, and shifting the scalar value the distance.
  • the object control method when the pupil position is a dual-purpose pupil position; the pupil position change vector is a dual-purpose pupil position change vector, the object control method further includes: determining a vector direction corresponding to each of the dual-purpose pupil position change vectors And when the vector directions corresponding to the pupil position change vectors of the dual purpose are the same, it is determined that the object control type corresponding to the start position and the end position is the object shift control type.
  • the object to be controlled is controlled according to the object control type and the object control parameter, including: panning the object to be controlled according to the object control parameter.
  • the computer device can respectively determine a vector direction corresponding to each of the dual-purpose pupil position change vectors, and calculate an angle between the two vector directions. When the angle is smaller than the preset angle, determine that the dual-purpose pupil position change vectors correspond to each other. When the vector directions are the same, it is determined that the object control type corresponding to the start position and the end position is the object shift control type.
  • the vector angle can be calculated by the following formula.
  • vectorMul leftIrisMoveVector2.x*rightIrisMoveVector2.x+leftIrisMoveVector2.y*rightIrisMoveVector2.y;
  • verctorLeftLen sqrt(leftIrisMoveVector2.x*leftIrisMoveVector2.x+leftIrisMoveVector2.y*leftIrisMoveVector2.y);
  • verctorRightLen sqrt(rightIrisMoveVector2.x*rightIrisMoveVector2.x+rightIrisMoveVector2.y*rightIrisMoveVector2.y);
  • leftIrisMoveVector2.x currentPosLeft.x-lastPosLeft.x;
  • vectorMul is the intermediate variable
  • leftIrisMoveVector2.x is the vector of the left eye pupil position along the X-axis direction
  • leftIrisMoveVector2.y is the left eye pupil position along the Y-axis direction change vector
  • rightIrisMoveVector2.x is the right eye pupil position along the X-axis direction
  • the vector, rightIrisMoveVector2.y is the vector of the right eye pupil position along the Y-axis.
  • verctorLeftLen is the scalar value of the left eye pupil position change vector
  • verctorRightLen is the scalar value of the right eye pupil position change vector.
  • (currentPosLeft.x, currentPosLeft.y) is the coordinates of the end position of the left eye
  • (lastPosLeft.x, lastPosLeft.y) is the coordinates of the starting position of the left eye
  • (currentPosright.x, currentPosright.y) is the coordinates of the ending position of the right eye
  • ( lastPosright.x, lastPosright.y) is the coordinates of the starting position of the right eye.
  • the computer device can calculate a scalar value of the dual-purpose pupil position change vector, and select a scalar value, or a scalar value corresponding to a larger value, or an average value of the two scalar values, thereby controlling
  • the control object translates the distance of the selected scalar value along the vector direction of the pupil position change vector.
  • FIG. 5 is a schematic diagram showing a case where the vector directions corresponding to the dual-purpose pupil position change vectors are identical in one embodiment. Assume that the preset angle is 90 degrees. Referring to FIG. 5, it can be seen that the angle of the corresponding vector direction of the dual-purpose pupil position change vector is 0 degree, and 0 degree is less than 90 degrees, then the object control type corresponding to the start position and the end position is the object translation. Control type to control object panning.
  • the pupil movement mode corresponding to the object translation control type is provided, and the pupil movement feature corresponding to the vector direction of each of the dual pupil position change vectors is reflected to reflect the translation object, so that the pupil is detected.
  • the object to be controlled is translated.
  • the object control method further includes: determining that the object control type corresponding to the start position and the end position is the object zoom control type when the vector directions corresponding to the dual-purpose pupil position change vectors are inconsistent.
  • the object to be controlled is controlled according to the object control type and the object control parameter, including: adjusting the size of the object to be controlled according to the object control parameter.
  • the computer device can respectively determine a vector direction corresponding to each of the dual-purpose pupil position change vectors, and calculate an angle between the two vector directions. When the angle reaches a preset angle, determine that the dual-purpose pupil position change vectors correspond to each other. When the vector directions are inconsistent, it is determined that the object control type corresponding to the start position and the end position is the object rotation control type.
  • the computer device can calculate the distance between the center point of the dual-purpose eye in the image frame corresponding to the starting position, twoEyeCenterW1, the distance between the two-purpose pupil positions, twoIrisCenterW1; in the image frame corresponding to the ending position, double
  • scale is the scaling ratio
  • aConstValue is the variable coefficient, which is used to adjust the sensitivity of the zoom.
  • FIG. 6 is a schematic diagram showing a case where the vector directions corresponding to the dual-purpose pupil position change vectors do not coincide in one embodiment.
  • the preset angle is 90 degrees.
  • the angle of the corresponding vector direction of the dual-purpose pupil position change vector is 180 degrees, and the 180 degree is greater than 90 degrees, then the object control type corresponding to the start position and the end position is the object zoom. Control the type to control object scaling.
  • the distance between the center point of the double eye is twoEyeCenterW1
  • the distance between the two pupil positions is twoIrisCenterW1.
  • the pupil movement mode corresponding to the object zoom control type is provided, and the zoom target is reflected by the pupil movement feature in which the vector orientations of the dual-purpose pupil position change vectors are inconsistent, so that the pupil is detected. Zooming on the object to be controlled is achieved when the feature is moved.
  • the magnitude of the change of the object to be controlled is dynamically controlled according to the scalar value of the pupil position change vector formed when the actual pupil moves, so that the object to be controlled is more flexible when controlling the object to be controlled.
  • FIG. 7 is a schematic flowchart of an object control method in a specific embodiment, where the object control method specifically includes the following steps:
  • step S704. Acquire an image frame according to time series, and detect whether the collected image frame includes a face region; if yes, go to step S706; if no, return to step S704.
  • step S706 determining whether the face deflection angle corresponding to the image frame matches the reference deflection angle indicating the front face image; if yes, the process proceeds to step S708; if not, the process returns to step S704.
  • S708 determining a dual-purpose pupil pixel position and an eye center point position in the image frame; establishing an eye reference coordinate system with the eye center point position as a reference position; mapping the dual-purpose pupil pixel position to a coordinate in the eye reference coordinate system, Get the dual purpose pupil position and cache.
  • step S712. Determine whether the end position is located in the first eye edge region in the corresponding face image frame; if yes, go to step S714; if no, return to step S720.
  • step S714. Determine whether the starting position is located in the second eye edge region in the corresponding face image frame; if yes, go to step S716; if no, go back to step S720.
  • step S722 Determine whether the vector directions corresponding to the dual-purpose pupil position change vectors are consistent; if yes, go to step S724; if no, return to step S728.
  • S726 Obtain a scalar value of the pupil position change vector; determine an object control parameter corresponding to the object control type according to the scalar value; and translate the object to be controlled according to the object control parameter.
  • S728 Determine an object control type corresponding to the start and end positions of the dual destination as the object zoom control type.
  • S730 Obtain a scalar value of the pupil position change vector; determine an object control parameter corresponding to the object control type according to the scalar value; and adjust a size of the object to be controlled according to the object control parameter.
  • the pupil position is automatically determined in the face image frame acquired in time series, and then the corresponding face image frame is pressed in the pupil position determined in the multi-frame face image frame.
  • the acquisition timing determines the start position and the end position of the pupil movement path, respectively, so that the object control type is determined according to the movement path of the pupil to control the target according to different control types. In this way, the target is controlled by the eye, and the problem of the target occlusion caused by the hand or the finger during the manual control is avoided, and the object control accuracy is improved.
  • an object control device 800 is provided.
  • the object control apparatus 800 includes an object determination module 801, a position determination module 802, a type determination module 803, and an object control module 804.
  • the various modules included in the object control device 800 may be implemented in whole or in part by software, hardware, or a combination thereof.
  • the object determination module 801 is configured to determine an object to be controlled.
  • a position determining module 802 configured to determine a pupil position in a face image frame acquired in time series; in a pupil position determined in the multi-frame face image frame, determine a pupil movement path according to a collection timing of the corresponding face image frame The starting position and ending position.
  • the type determination module 803 is configured to determine an object control type according to the start position and the end position.
  • the object control module 804 is configured to control the object to be controlled according to the object control type.
  • the object control device 800 automatically determines the pupil position in the face image frame acquired in time series after determining the target to be controlled, and then presses the corresponding face image frame in the pupil position determined in the multi-frame face image frame.
  • the acquisition timing determines the start position and the end position of the pupil movement path, respectively, so that the object control type is determined according to the movement path of the pupil to control the target according to different control types. In this way, the target is controlled by the eye, and the problem of the target occlusion caused by the hand or the finger during the manual control is avoided, and the object control accuracy is improved.
  • the location determining module 802 is further configured to acquire image frames in time series; when the captured image frame includes a face region, determine and cache a pupil position in the image frame; and detect a next frame including a face region In the image frame, the buffered pupil position is taken as the starting position of the pupil moving path; and the pupil position of the image frame including the face region in the next frame is acquired as the ending position of the pupil moving path.
  • the location determining module 802 is further configured to: when the captured image frame includes a face region, determine a face deflection angle corresponding to the image frame; and a reference deflection angle between the face deflection angle and the front face image When matching, the pupil position in the image frame is determined and cached.
  • the location determining module 802 is further configured to extract a facial feature of each facial image frame according to a collection timing of the multi-frame facial image frame; and acquire a human face when the facial feature conforms to the initial moving feature.
  • the pupil position of the face image frame from which the feature is extracted is taken as the starting position of the pupil moving path; and when the face feature conforms to the ending moving feature, the pupil position of the face image frame from which the face feature is extracted is obtained as the pupil moving path The end position.
  • the type determining module 803 is further configured to determine a first eye edge region in a corresponding face image frame at the end position; and a corresponding face image at the start position when the end position is located in the first eye edge region The second eye edge region is determined in the frame; and when the home position is located in the second eye edge region, the object control type corresponding to the determination of the start position and the end position is the object rotation control type.
  • the object control module 804 is further configured to determine an eye center point position in each of the corresponding face image frames of the start position and the end position; and determine the rotation according to the start position, the end position, and the eye center point position. Angle; and rotate the object to be controlled according to the angle of rotation.
  • the object control device 800 further includes a parameter determination module 805.
  • the parameter determining module 805 is configured to: when the end position is not in the first eye edge region, or when the end position is in the first eye edge region, and the starting position is not in the second eye edge region, according to the starting position and the ending position Determining the pupil position change vector; obtaining a scalar value of the pupil position change vector; and determining an object control parameter corresponding to the object control type according to the scalar value.
  • the object control module 804 is further configured to control the object to be controlled according to the object control type and the object control parameter.
  • the pupil position is a dual purpose pupil position; the pupil position change vector is a dual purpose pupil position change vector.
  • the type determining module 803 is further configured to determine a vector direction corresponding to each of the dual-purpose pupil position change vectors; and when the vector directions corresponding to the dual-purpose pupil position change vectors are identical, determine the objects corresponding to the start position and the end position.
  • the control type is the object pan control type.
  • the object control module 804 is further configured to translate the object to be controlled according to the object control parameter.
  • the type determining module 803 is further configured to determine that the object control type corresponding to the start position and the end position is the object zoom control type when the vector directions corresponding to the dual-purpose pupil position change vectors are inconsistent.
  • the object control module 804 is further configured to adjust the size of the object to be controlled according to the object control parameter.
  • the location determining module 802 is further configured to determine a pupil pixel position and an eye center point position in the face image frame acquired in time series; establish an eye reference coordinate system with the eye center point position as a reference position; and insert the pupil The pixel position is mapped to the coordinates in the eye reference coordinate system to obtain the pupil position.
  • Figure 10 is a diagram showing the internal structure of a computer device in one embodiment.
  • the computer device includes a processor, a memory, a network interface, a camera, an input device, and a display screen connected by a system bus.
  • the memory comprises a non-volatile storage medium and an internal memory.
  • the non-volatile storage medium of the computer device stores an operating system and can also store computer readable instructions that, when executed by the processor, cause the processor to implement the object control method.
  • the internal memory can also store computer readable instructions that, when executed by the processor, cause the processor to perform an object control method.
  • the display screen of the computer device may be a liquid crystal display or an electronic ink display screen.
  • the input device may be a touch layer covered on the display screen, or may be a button, a trackball or a touchpad provided on a computer device casing, or may be An external keyboard, trackpad, or mouse.
  • the computer device may be a user terminal, and the user terminal may specifically be at least one of a mobile phone, a tablet computer, a personal digital assistant, or a wearable device. It will be understood by those skilled in the art that the structure shown in FIG. 10 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation of the computer device to which the solution of the present application is applied.
  • the specific computer device may It includes more or fewer components than those shown in the figures, or some components are combined, or have different component arrangements.
  • the object control apparatus may be implemented in the form of a computer readable instruction executable on a computer device as shown in FIG. 10, non-volatile storage of the computer device
  • the medium may store various instruction modules constituting the object control device, for example, the object determination module 801, the position determination module 802, the type determination module 803, the object control module 804, and the like shown in FIG.
  • the computer readable instructions comprising the various instruction modules cause the processor to perform the steps in the object control method of various embodiments of the present application described in this specification.
  • the computer device shown in FIG. 10 can be used to determine an object to be controlled by the object determination module 801 in the object control device 800 as shown in FIG.
  • the position determining module 802 is configured to determine a pupil position in the face image frame acquired in time series; in the pupil position determined in the multi-frame face image frame, determine the pupil movement path according to the collection timing of the corresponding face image frame respectively The starting position and ending position.
  • the pass type determination module 803 is configured to determine an object control type based on the start position and the end position.
  • the object control module 804 is configured to control the object to be controlled according to the object control type.
  • a computer readable storage medium having stored thereon computer readable instructions that, when executed by a processor, cause the processor to perform the step of determining Controlling the object; determining the pupil position in the face image frame acquired in time series; determining the starting position of the pupil movement path according to the acquisition timing of the corresponding face image frame in the pupil position determined in the multi-frame face image frame And the end position; determining the object control type according to the start position and the end position; and controlling the object to be controlled according to the object control type.
  • determining a pupil position in a face image frame acquired in time series includes: acquiring an image frame in time series; and determining and buffering a pupil position in the image frame when the captured image frame includes a face region.
  • the start position and the end of the pupil movement path are respectively determined according to the acquisition timing of the corresponding face image frame.
  • the position includes: using the cached pupil position as a starting position of the pupil movement path; and acquiring a pupil position of the image frame including the face region in the next frame as the end position of the pupil movement path.
  • determining and buffering a pupil position in the image frame includes: determining a face deflection angle corresponding to the image frame when the captured image frame includes a face region And determining and buffering the pupil position in the image frame when the face deflection angle matches the reference deflection angle representing the face image.
  • the start position and the end position of the pupil movement path are respectively determined according to the acquisition timing of the corresponding face image frame, including: pressing the multi-frame face
  • the image frame collection timing is used to extract the face features of each face image frame; when the face feature conforms to the initial moving feature, the pupil position of the face image frame from which the face feature is extracted is obtained as the start of the pupil movement path. Position; and when the face feature conforms to the end moving feature, the pupil position of the face image frame from which the face feature is extracted is obtained as the end position of the pupil moving path.
  • determining the object control type according to the start position and the end position comprises: determining a first eye edge region in a corresponding face image frame at the end position; when the end position is located at the first eye edge region, starting from The second eye edge region is determined in the corresponding face image frame of the start position; and when the start position is located in the second eye edge region, the object control type corresponding to the determination start position and the end position is the object rotation control type.
  • controlling the object to be controlled according to the object control type includes: determining a position of the eye center point in each corresponding face image frame of the start position and the end position; according to the start position, the end position, and the eye center Point position, determine the rotation angle; and rotate the object to be controlled according to the rotation angle.
  • the processor when the computer readable instructions are executed by the processor, the processor further causes the processor to perform the following steps: when the end position is not in the first eye edge region, or when the end position is in the first eye edge region, and When the start position is not in the second eye edge region, the pupil position change vector is determined according to the start position and the end position; the scalar value of the pupil position change vector is acquired; and the object control parameter corresponding to the object control type is determined according to the scalar value.
  • the object to be controlled is controlled according to the object control type, including: controlling the object to be controlled according to the object control type and the object control parameter.
  • the pupil position is a dual purpose pupil position; the pupil position change vector is a dual purpose pupil position change vector.
  • the computer readable instructions are further executed by the processor, the processor further performing the steps of: determining a vector direction corresponding to each of the dual-purpose pupil position change vectors; and when the vector directions of the dual-purpose pupil position change vectors are identical, Then, it is determined that the object control type corresponding to the start position and the end position is the object shift control type.
  • the object to be controlled is controlled according to the object control type and the object control parameter, including: panning the object to be controlled according to the object control parameter.
  • the processor when the computer readable instructions are executed by the processor, the processor further causes the processor to perform the step of determining the start position and the end position when the respective vector directions of the dual purpose pupil position change vectors are inconsistent
  • the corresponding object control type is the object zoom control type.
  • the object to be controlled is controlled according to the object control type and the object control parameter, including: adjusting the size of the object to be controlled according to the object control parameter.
  • determining a pupil position in the face image frame acquired in time series includes: determining a pupil pixel position and an eye center point position in the face image frame acquired in time series; using the eye center point position as a reference position Establish an eye reference coordinate system; and map the pupil pixel position to the coordinates in the eye reference coordinate system to obtain the pupil position.
  • the storage medium After determining the target to be controlled, the storage medium automatically determines the pupil position in the face image frame acquired in time series, and then collects the corresponding face image frame in the pupil position determined in the multi-frame face image frame. Timing, respectively determining the starting position and ending position of the pupil moving path, thereby determining the object control type according to the moving path of the pupil to control the target according to different control types. In this way, the target is controlled by the eye, and the problem of the target occlusion caused by the hand or the finger during the manual control is avoided, and the object control accuracy is improved.
  • a computer apparatus comprising: a memory and a processor, wherein the memory stores computer readable instructions, and when the computer readable instructions are executed by the processor, causing the processor to perform the step of: determining an object to be controlled; Determining the pupil position in the face image frame acquired in time series; determining the start position and the end position of the pupil movement path according to the acquisition timing of the corresponding face image frame in the pupil position determined in the multi-frame face image frame
  • the object control type is determined according to the start position and the end position; and the object to be controlled is controlled according to the object control type.
  • determining a pupil position in a face image frame acquired in time series includes: acquiring an image frame in time series; and determining and buffering a pupil position in the image frame when the captured image frame includes a face region.
  • the start position and the end of the pupil movement path are respectively determined according to the acquisition timing of the corresponding face image frame.
  • the position includes: using the cached pupil position as a starting position of the pupil movement path; and acquiring a pupil position of the image frame including the face region in the next frame as the end position of the pupil movement path.
  • determining and buffering a pupil position in the image frame includes: determining a face deflection angle corresponding to the image frame when the captured image frame includes a face region And determining and buffering the pupil position in the image frame when the face deflection angle matches the reference deflection angle representing the face image.
  • the start position and the end position of the pupil movement path are respectively determined according to the acquisition timing of the corresponding face image frame, including: pressing the multi-frame face
  • the image frame collection timing is used to extract the face features of each face image frame; when the face feature conforms to the initial moving feature, the pupil position of the face image frame from which the face feature is extracted is obtained as the start of the pupil movement path. Position; and when the face feature conforms to the end moving feature, the pupil position of the face image frame from which the face feature is extracted is obtained as the end position of the pupil moving path.
  • determining the object control type according to the start position and the end position comprises: determining a first eye edge region in a corresponding face image frame at the end position; when the end position is located at the first eye edge region, starting from The second eye edge region is determined in the corresponding face image frame of the start position; and when the start position is located in the second eye edge region, the object control type corresponding to the determination start position and the end position is the object rotation control type.
  • controlling the object to be controlled according to the object control type includes: determining a position of the eye center point in each corresponding face image frame of the start position and the end position; according to the start position, the end position, and the eye center Point position, determine the rotation angle; and rotate the object to be controlled according to the rotation angle.
  • the processor when the computer readable instructions are executed by the processor, the processor further causes the processor to perform the following steps: when the end position is not in the first eye edge region, or when the end position is in the first eye edge region, and When the start position is not in the second eye edge region, the pupil position change vector is determined according to the start position and the end position; the scalar value of the pupil position change vector is acquired; and the object control parameter corresponding to the object control type is determined according to the scalar value.
  • the object to be controlled is controlled according to the object control type, including: controlling the object to be controlled according to the object control type and the object control parameter.
  • the pupil position is a dual purpose pupil position; the pupil position change vector is a dual purpose pupil position change vector.
  • the computer readable instructions are further executed by the processor, the processor further performing the steps of: determining a vector direction corresponding to each of the dual-purpose pupil position change vectors; and when the vector directions of the dual-purpose pupil position change vectors are identical, Then, it is determined that the object control type corresponding to the start position and the end position is the object shift control type.
  • the object to be controlled is controlled according to the object control type and the object control parameter, including: panning the object to be controlled according to the object control parameter.
  • the processor when the computer readable instructions are executed by the processor, the processor further causes the processor to perform the step of determining the start position and the end position when the respective vector directions of the dual purpose pupil position change vectors are inconsistent
  • the corresponding object control type is the object zoom control type.
  • the object to be controlled is controlled according to the object control type and the object control parameter, including: adjusting the size of the object to be controlled according to the object control parameter.
  • determining a pupil position in the face image frame acquired in time series includes: determining a pupil pixel position and an eye center point position in the face image frame acquired in time series; using the eye center point position as a reference position Establish an eye reference coordinate system; and map the pupil pixel position to the coordinates in the eye reference coordinate system to obtain the pupil position.
  • the computer device After determining the target to be controlled, the computer device automatically determines the pupil position in the face image frame acquired in time series, and then collects the corresponding face image frame in the pupil position determined in the multi-frame face image frame. Timing, respectively determining the starting position and ending position of the pupil moving path, thereby determining the object control type according to the moving path of the pupil to control the target according to different control types. In this way, the target is controlled by the eye, and the problem of the target occlusion caused by the hand or the finger during the manual control is avoided, and the object control accuracy is improved.
  • Non-volatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.
  • Volatile memory can include random access memory (RAM) or external cache memory.
  • RAM is available in a variety of formats, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronization chain.
  • SRAM static RAM
  • DRAM dynamic RAM
  • SDRAM synchronous DRAM
  • DDRSDRAM double data rate SDRAM
  • ESDRAM enhanced SDRAM
  • Synchlink DRAM SLDRAM
  • Memory Bus Radbus
  • RDRAM Direct RAM
  • DRAM Direct Memory Bus Dynamic RAM
  • RDRAM Memory Bus Dynamic RAM

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • General Physics & Mathematics (AREA)
  • Ophthalmology & Optometry (AREA)
  • General Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)
  • Processing Or Creating Images (AREA)

Abstract

一种对象控制方法，该对象控制方法包括：确定待控制对象；在按时序采集的人脸图像帧中确定瞳孔位置；在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置；根据所述起始位置和所述结束位置确定对象控制类型；按照所述对象控制类型对所述待控制对象进行控制。

Description

对象控制方法、装置、存储介质和计算机设备
本申请要求于2017年12月29日提交中国专利局，申请号为2017114839627，申请名称为“对象控制方法、装置、存储介质和计算机设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机技术领域，特别是涉及一种对象控制方法、装置、存储介质和计算机设备。
背景技术
随着互联网技术的发展，在互联网平台上对目标对象进行控制越来越普遍。比如在图像处理应用中处理图像时，控制图像处理应用所提供的素材；或者，在游戏应用中控制交互对象进行交互等。
传统技术中，在进行对象控制时，通常是通过人工手动进行控制。然而，在手动控制的过程中，手或者手指通常会造成遮挡影响对象控制的精度，从而导致对象控制的准确率低下。
发明内容
根据本申请提供的各种实施例，提供一种对象控制方法、装置、存储介质和计算机设备。
一种对象控制方法，由计算机设备执行，所述方法包括：
确定待控制对象；
在按时序采集的人脸图像帧中确定瞳孔位置；
在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置；
根据所述起始位置和所述结束位置确定对象控制类型；及
按照所述对象控制类型对所述待控制对象进行控制。
一种对象控制装置，包括：
对象确定模块，用于确定待控制对象；
位置确定模块，用于在按时序采集的人脸图像帧中确定瞳孔位置；在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置；
类型确定模块，用于根据所述起始位置和所述结束位置确定对象控制类型；及
对象控制模块，用于按照所述对象控制类型对所述待控制对象进行控制。
一种存储有计算机可读指令的非易失性存储介质，所述计算机可读指令被一个或多个处理器执行时，使得一个或多个处理器执行以下步骤：
确定待控制对象；
在按时序采集的人脸图像帧中确定瞳孔位置；
在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置；
根据所述起始位置和所述结束位置确定对象控制类型；及
按照所述对象控制类型对所述待控制对象进行控制。
一种计算机设备，包括存储器和处理器，所述存储器中存储有计算机可读指令，所述计算机可读指令被所述处理器执行时，使得所述处理器执行以下步骤：
确定待控制对象；
在按时序采集的人脸图像帧中确定瞳孔位置；
在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置；
根据所述起始位置和所述结束位置确定对象控制类型；及
按照所述对象控制类型对所述待控制对象进行控制。
本申请的一个或多个实施例的细节在下面的附图和描述中提出。本申请的其它特征、目的和优点将从说明书、附图以及权利要求书变得明显。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为一个实施例中对象控制方法的流程示意图；
图2为一个实施例中图像处理应用的界面示意图；
图3为一个实施例中划分眼睛边缘区域的示意图；
图4为一个实施例中确定旋转角度的示意图；
图5为一个实施例中双目的瞳孔位置变化矢量各自对应的矢量方向一致时的示意图；
图6为一个实施例中双目的瞳孔位置变化矢量各自对应的矢量方向不一致时的示意图；
图7为另一个实施例中对象控制方法的流程示意图；
图8为一个实施例中对象控制装置的模块结构图；
图9为另一个实施例中对象控制装置的模块结构图；及
图10为一个实施例中计算机设备的内部结构图。
具体实施方式
为了使本申请的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本申请进行进一步详细说明。应当理解，此处所描述的具体实施例仅仅用以解释本申请，并不用于限定本申请。
图1为一个实施例中对象控制方法的流程示意图。本实施例主要以该方法应用于计算机设备来举例说明。参照图1，该对象控制方法具体包括如下 步骤：
S102，确定待控制对象。
其中，待控制对象是待进行控制的目标。待控制对象可以是实体对象，也可以是虚拟对象。实体对象是实际存在的具有物理形态的对象，比如智能机器人或者智能家居等。虚拟对象是计算机设备可操作的数字化对象，比如图像处理应用所提供的素材或者游戏应用中的虚拟人物等。虚拟对象可以是二维虚拟对象，也可以是三维虚拟对象。
具体地，计算机设备可通过计算机界面展示供选择的对象，并检测选择指令，将选择指令所指定的对象确定为待控制对象。计算机设备具体可检测预定义的触发操作，在检测到该触发操作时触发相应的选择指令。触发操作可以是对计算机界面中控件的操作，比如对控件的触控操作或者光标点击操作等。触发操作也可以是对预定义的物理按钮的点击，或者在通过计算机界面展示供选择的对象时触发的晃动操作或者语音输入操作等。
在一个实施例中，计算机设备上可运行有图像处理应用，该图像处理应用可提供图像处理素材。图像处理素材比如二维贴纸或者三维虚拟模型等。图像处理应用可根据用户指令展示图像处理素材，并继续根据用户指令从展示的图像处理素材中选取用户选中的图像处理素材作为待控制对象。其中，图像处理应用是能够进行图像处理的应用。图像处理应用包括社交应用、游戏应用、直播应用或者拍照应用等。图像处理应用可以是增强现实应用。
举例说明，在拍照应用中，该拍照应用可提供图像处理素材(如贴纸)，用户可选取图像处理素材作为待控制对象，以通过瞳孔移动控制该待控制对象。在游戏应用中，该拍照应用可提供图像处理素材(如虚拟游戏形象)，用户可选取图像处理素材作为待控制对象，以通过瞳孔移动控制该待控制对象。其中，游戏应用可以是增强现实游戏应用。
S104，在按时序采集的人脸图像帧中确定瞳孔位置。
其中，瞳孔是眼睛内虹膜中心的小圆孔，为光线进入眼睛的通道。瞳孔位置是瞳孔在人脸图像帧中的位置。瞳孔位置具体可以是瞳孔中心点在人脸 图像帧中的位置。瞳孔中心点是瞳孔区域的中心点。
具体地，计算机设备可按人脸图像帧的采集时序逐帧获取人脸图像帧，在每获取到图像帧时，即通过人脸检测算法识别人脸图像帧中的人脸区域，并在人脸区域中确定瞳孔位置。人脸检测算法可根据需要自定义，如可为OpenCV人脸检测算法、IOS或Android系统自带的人脸检测算法或者优图人脸检测算法等。人脸检测算法可以返回人脸图像帧中是否包含人脸、具体的人脸区域以及人脸区域中的瞳孔位置。
其中，按时序采集的人脸图像帧可以是计算机设备调用内置的摄像头采集得到的图像帧，也可以是计算机设备调用与计算机设备连接的外部摄像头采集得到的图像帧。
S106，在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置。
其中，多帧人脸图像帧可以是多于一帧的任意帧数的人脸图像帧。瞳孔移动路径是瞳孔移动形成的路径。瞳孔移动路径的起始位置是开始通过瞳孔移动控制待控制对象时瞳孔的位置。瞳孔移动路径的结束位置是结束通过瞳孔移动控制待控制对象时瞳孔的位置。
在一个实施例中，任意相邻的两帧人脸图像帧中，时序靠前的人脸图像帧与时序靠后的人脸图像帧中瞳孔位置形成的路径可为瞳孔移动路径。时序靠前的人脸图像帧中的瞳孔位置为瞳孔移动路径的起始位置，时序靠后的人脸图像帧中的瞳孔位置为瞳孔移动路径的结束位置。
在一个实施例中，计算机设备可事先设置开始通过瞳孔移动控制待控制对象的开始触发条件，以及结束通过瞳孔移动控制待控制对象的结束触发条件。计算机设备在判定人脸图像帧满足开始触发条件时，将该人脸图像帧中的瞳孔位置确定为瞳孔移动路径的起始位置；在判定人脸图像帧满足结束触发条件时，将该人脸图像帧中的瞳孔位置确定为瞳孔移动路径的结束位置。
其中，开始触发条件与结束触发条件，可以是针对人脸图像帧中人脸特征设置的触发条件，比如人脸图像帧中包括闭眼或者张嘴等人脸特征。开始 触发条件与结束触发条件，也可以是针对人脸图像帧中瞳孔位置设置的触发条件，比如人脸图像帧中瞳孔位置在特定的位置等。开始触发条件与结束触发条件可以是相同的触发条件，也可以是不同的触发条件。相同的触发条件比如，计算机设备在首次判定人脸图像帧满足触发条件时，将该人脸图像帧中的瞳孔位置确定为瞳孔移动路径的起始位置；在再次判定人脸图像帧满足触发条件时，将该人脸图像帧中的瞳孔位置确定为瞳孔移动路径的结束位置，依次交替循环。
S108，根据起始位置和结束位置确定对象控制类型。
其中，对象控制类型是控制对象所进行变化所属的类型。对象控制类型比如对象平移控制类型、对象旋转控制类型或者对象缩放控制类型等。可以理解，不同的对象控制类型对应不同的位置特征，计算机设备可事先设置不同对象控制类型各自对应的位置特征。
具体地，计算机设备在确定起始位置和结束位置后，可继续确定该起始位置和结束位置的位置特征，将确定的位置特征与各对象控制类型各自对应的位置特征进行比较，将与确定的位置特征匹配的位置特征所对应的对象控制类型，作为与确定的起始位置和结束位置对应的对象控制类型。其中，位置特征可以是瞳孔位置处于眼睛区域的相对位置特征。
S110，按照对象控制类型对待控制对象进行控制。
具体地，计算机设备在确定对象控制类型后，即按照该对象控制类型控制待控制对象。比如，当对象控制类型为对象平移类型时，即平移待控制对象；当对象控制类型为对象旋转类型时，即旋转待控制对象等。
在一个实施例中，计算机设备控制待控制对象所依据的控制参数，可以是事先设置的各对象控制类型各自对应的默认参数。比如，对象平移控制类型对应的默认参数，可以是默认的平移距离，计算机设备在每次平移待控制对象时，均按照该默认的平移距离平移待控制对象。
在一个实施例中，计算机设备控制待控制对象所依据的控制参数，可以是与瞳孔移动路径相关的控制参数。可以理解，不同的瞳孔移动路径可对象 不同的控制参数。计算机设备可预先设置根据瞳孔移动路径计算控制参数的方式，计算机设备在具体控制待控制对象时，根据当前的瞳孔移动路径计算当前的控制参数，以按该控制参数控制待控制对象。
上述对象控制方法，在确定待控制的目标后，自动在按时序采集的人脸图像帧中确定瞳孔位置，进而在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置，从而根据瞳孔的移动路径确定对象控制类型以按照不同的控制类型控制目标。这样通过眼睛控制目标，避免了手动控制的过程中手或者手指造成目标遮挡的问题，提高了对象控制准确率。
在一个实施例中，S104包括：按时序采集图像帧；当采集的图像帧包括人脸区域时，确定并缓存图像帧中的瞳孔位置；及在检测到下一帧包括人脸区域的图像帧时，S106包括：将缓存的瞳孔位置作为瞳孔移动路径的起始位置；获取下一帧包括人脸区域的图像帧的瞳孔位置，作为瞳孔移动路径的结束位置。
具体地，计算机设备可运行有图像处理应用，该图像处理应用可配置有瞳孔移动模式。其中，瞳孔移动模式是通过瞳孔的移动来控制待控制对象的模式。具体地，瞳孔移动模式可以被配置为图像处理应用启动时即开启，也可以被配置为在运行图像处理应用后根据特定的触发指令开启。
举例说明，图2示出了一个实施例中图像处理应用的界面示意图。参考图2，假设图像处理应用为拍照应用。用户可以提前设置常备开启瞳孔移动模式；也可以进入相机后，通过特定动作触发进入该瞳孔移动模式(如眨眼、手势等)。用户还可以在进入相机选择图像处理素材210后，按住素材可直接对素材进行手指拖动(兼容手指移动模式)，长按图像处理素材210会弹出快捷浮层220，快速点击进入瞳孔移动模式。其中，瞳孔移动模式是通过瞳孔移动控制素材的模式。
进一步地，在图像处理应该开启瞳孔移动模式后，计算机设备可按照固定或动态的帧率，依时序采集图像帧，得到按时序采集的图像帧。计算机设 备也可通过内置或者外部连接的摄像头，在摄像头当前的视野下，依时序采集图像帧，得到按时序采集的图像帧。其中，摄像头的视野可因计算机设备的姿态和位置的变化而变化。
计算机设备还可以通过图像处理应用，提供AR(Augmented Reality，增强现实)拍摄模式，并在选定该AR拍摄模式后，从现实场景中采集图像帧，获取采集得到的图像帧。
更进一步地，计算机设备在每采集到一帧图像帧时，即通过人脸检测算法识别采集的图像帧中是否包含人脸区域。具体地，计算机设备可通过人脸检测算法提取采集的图像帧中包括的图像数据，并检测该图像数据是否包含人脸特征数据。若计算机设备检测到该图像数据中包含人脸特征数据，则判定该图像帧中包括人脸区域，进而根据提取的图像数据在人脸区域中确定瞳孔位置。计算机设备从而获取并缓存当前采集的图像帧中的瞳孔位置。
在一个实施例中，当采集的图像帧包括人脸区域时，确定并缓存图像帧中的瞳孔位置，包括：当采集的图像帧包括人脸区域时，确定图像帧所对应的人脸偏转角度；及在人脸偏转角度与表示正脸图像的参考偏转角度匹配时，确定并缓存图像帧中的瞳孔位置。
其中，人脸偏转角度是人脸绝对正对摄像头时偏转的角度。可以理解，现实场景中难以控制人脸绝对正对摄像头，因此在误差允许的范围内，可将特定偏转角度内的采集的人脸图像作为正脸图像。这里的特定偏转角度即为参考偏转角度。
在一个实施例中，计算机设备可通过人脸检测算法计算图像帧中人脸图像对应的人脸偏转角度，将该人脸偏转角度与表示正脸图像的参考偏转角度进行匹配，当该人脸偏转角度未超过表示正脸图像的参考偏转角度时，即判定图像帧所对应的人脸偏转角度与表示正脸图像的参考偏转角度匹配，进而确定并缓存图像帧中的瞳孔位置。
其中，人脸偏转角度可以是人脸区域基于图像帧的立体坐标系的欧拉角。其中，图像帧的立体坐标系，是包括横轴(X轴)、纵轴(Y轴)和竖轴(Z 轴)的三维坐标系。图像帧的立体坐标系的横轴的轴向是沿着图像帧的水平方向，纵轴的轴向是沿着图像帧的竖直方向，竖轴的轴向是沿着垂直于图像帧的方向。计算机设备可计算各方向对应的参考偏转角度。
举例说明，人脸区域绕X轴偏转的角度可为Pitch，绕Y轴偏转的角度可为Yaw，绕Z轴偏转的角度可为Roll。其中，X轴方向上对应的参考偏转角度可以为15度，Y轴方向上对应的参考偏转角度可以为15度，Z轴方向上不作限制。那么，当Pitch小于15度且Yaw小于15度时，即可确定采集的图像帧中包括的人脸区域为正脸人脸区域。
在本实施例中，在图像帧中包括的人脸图像满足正脸条件时，才在图像帧中确定瞳孔位置，继而根据该图像帧的瞳孔位置来控制待控制对象，使得在正脸时确定的瞳孔位置更准确，从而使得后续的对象控制更准确。
这样，计算机设备在每检测一帧图像帧，在检测到该图像帧包括人脸图像时，缓存当前检测的图像帧中的瞳孔位置。计算机设备再继续按采集时序逐帧检测采集的图像帧，直到检测出下一帧包括人脸区域的图像帧时，获取缓存的瞳孔位置作为瞳孔移动路径的起始位置，并获取当前包括人脸区域的图像帧的瞳孔位置，作为瞳孔移动路径的结束位置。这样，计算机设备可以在每连续确定两帧包括人脸区域的图像帧后，实时根据这两帧人脸图像帧中瞳孔位置所对应的对象控制类型来控制待控制对象。
上述实施例中，在采集人脸图像帧时，实时检测采集的人脸图像帧中是否包括人脸区域，并在当前采集的图像帧包括人脸区域时，缓存人脸区域的瞳孔位置，并继续检测下一帧采集的人脸图像。这样在连续检测到两帧包括人脸请区域的图像帧时，即实时根据这两帧图像帧中瞳孔位置的变化来控制对象，提高了对象控制效率。
在一个实施例中，S106包括：按多帧人脸图像帧的采集时序，提取各人脸图像帧的人脸特征；当人脸特征符合起始移动特征时，则获取人脸特征提取自的人脸图像帧的瞳孔位置作为瞳孔移动路径的起始位置；及当人脸特征符合结束移动特征时，则获取人脸特征提取自的人脸图像帧的瞳孔位置作为 瞳孔移动路径的结束位置。
其中，人脸特征是反映人脸某种特征的数据。起始移动特征是表示开始通过瞳孔移动控制对象时图像帧的人脸特征。结束移动特征是表示结束通过瞳孔移动控制对象时图像帧的人脸特征。可以理解，计算机设备可实现设置起始移动特征和结束移动特征。结束移动特征比如闭眼特征或者张嘴特征等。
具体地，计算机设备可按照多帧人脸图像帧的采集时序，提取各人脸图像帧的人脸特征，依次将各人脸特征与起始移动特征比较。当人脸特征符合起始移动特征时，则获取该人脸特征提取自的人脸图像帧的瞳孔位置作为瞳孔移动路径的起始位置。计算机设备可再继续将剩余的人脸特征逐个与结束移动特征比较，当人脸特征符合结束移动特征时，则获取该人脸特征提取自的人脸图像帧的瞳孔位置作为瞳孔移动路径的结束位置。
其中，起始移动特征与结束移动特征可以是相同的人脸特征，也可以是不同的人脸特征。相同的人脸特征，比如，计算机设备在首次判定人脸图像帧包括该人脸特征时，将该人脸图像帧中的瞳孔位置确定为瞳孔移动路径的起始位置；在再次判定人脸图像帧包括该人脸特征时，将该人脸图像帧中的瞳孔位置确定为瞳孔移动路径的结束位置，依次交替循环。
在本实施例中，通过图像帧的人脸特征来确定表示开始通过瞳孔控制对象时的图像帧和表示结束通过瞳孔控制对象时的图像帧，从而明确哪些瞳孔位置的变化是进行对象控制的有效瞳孔位置变化，进而根据有效的瞳孔位置变化来控制对象，保证对象控制的准确性。
在一个实施例中，S108包括：在结束位置相应的人脸图像帧中确定第一眼睛边缘区域；当结束位置位于第一眼睛边缘区域时，在起始位置相应的人脸图像帧中确定第二眼睛边缘区域；及当起始位置位于第二眼睛边缘区域时，判定起始位置和结束位置所对应的对象控制类型为对象旋转控制类型。
其中，眼睛边缘区域是眼睛图像的边缘区域。眼睛边缘区域可以是以眼睛图像的轮廓为起始向内扩展一定距离所形成的环状区域。眼睛边缘区域，也可以是离眼睛图像的边界线一定距离的环状区域，其中，眼睛图像的边界 线是眼睛图像的上下左右四定点形成的边界线。
举例说明，图3示出了一个实施例中划分眼睛边缘区域的示意图。参考图3，可看到该示意图包括眼睛图像的轮廓310，以轮廓310为起始向内扩展一定距离所形成的环状区域320可以定义为眼睛边缘区域。眼睛图像的轮廓310可包括轮廓上顶点(top)311、轮廓左顶点(left)312、轮廓下顶点(bottom)313和轮廓右顶点(right)314。轮廓上顶点311与轮廓左顶点312、轮廓左顶点312与轮廓下顶点313、轮廓下顶点313与轮廓右顶点314，及轮廓右顶点314和轮廓上顶点311的连线即为眼睛图像的边界线330。离眼睛图像的边界线一定距离的环状区域340也可以定义为眼睛边缘区域。
具体地，计算机设备可在结束位置相应的人脸图像帧中确定第一眼睛边缘区域。当判定结束位置位于第一眼睛边缘区域时，则继续在起始位置相应的人脸图像帧中确定第二眼睛边缘区域。当起始位置位于第二眼睛边缘区域时，可判定两帧图像帧中瞳孔位置均位于眼睛边缘区域，从而判定起始位置和结束位置所对应的对象控制类型为对象旋转控制类型。其中，对象旋转控制类型表示控制对象变化所属的类型为旋转。
在本实施例中，提供了对象旋转控制类型所对应的瞳孔移动方式，具体以瞳孔移动路径的起始位置和终点位置均位于眼睛边缘区域的瞳孔移动特征来反映旋转控制对象，从而在检测到这种瞳孔移动特征时即实现旋转控制对象。
在一个实施例中，按照对象控制类型对待控制对象进行控制，包括：在起始位置和结束位置各自相应的人脸图像帧中，确定眼睛中心点位置；根据起始位置、结束位置和眼睛中心点位置，确定旋转角度；及按照旋转角度旋转待控制对象。
其中，眼睛中心点位置是眼睛图像的中心点的位置。在一个实施例中，计算机设备可过眼睛图像的轮廓左顶点，作轮廓上顶点与轮廓下顶点连线的垂线，将垂线与连线的交点作为眼睛中心点。
具体地，计算机设备可对起始位置和结束位置各自相应的人脸图像帧进 行图像尺寸调整，使得两帧图像帧中眼睛图像重合，进而保证在归一化的场景下确定眼睛中心点位置，从而保证基于归一化的场景确定旋转角度，以按照旋转角度旋转待控制对象。
在一个实施例中，计算机设备可在起始位置和结束位置各自相应的人脸图像帧中，确定瞳孔像素位置以及眼睛中心点位置，进而以眼睛中心点位置为参考位置建立眼睛参考坐标系，再将瞳孔像素位置映射为眼睛参考坐标系中的坐标，得到瞳孔位置。
其中，瞳孔像素点位置是瞳孔中心点的像素点在图像帧中的位置。对于图像帧通常以该图像帧的左顶点为参考点建立像素坐标系。瞳孔像素点位置即为该像素坐标系下的坐标位置。这里的参考点具体可以是坐标系原点。可以理解，这里的坐标系均为二维坐标系，包括横轴(X轴)和纵轴(Y轴)。
具体地，计算机设备可计算将坐标系原点由图像帧的左顶点变化为眼睛中心点位置的转换矩阵，从而即可根据该转换矩阵将瞳孔像素位置映射为眼睛参考坐标系中的坐标，得到瞳孔位置。
举例说明，以左顶点为参考点建立的像素坐标系需通过旋转和平移，才能变化至以眼睛中心点位置为参考位置建立的眼睛参考坐标系。假设需要旋转的角度为θ，需要沿X轴平移的距离为t x，需要沿Y轴平移的距离为t y。那么转换矩阵
Figure PCTCN2018123062-appb-000001
其中，
Figure PCTCN2018123062-appb-000002
为标准变换矩阵，
Figure PCTCN2018123062-appb-000003
为旋转矩阵，
Figure PCTCN2018123062-appb-000004
为平移矩阵。此时，假设瞳孔像素点位置为(x 1,y 1)，那么(x 1,y 1)*A＝(x 2,y 2)即为眼睛参考坐标系中的瞳孔位置。
在一个实施例中，计算机设备还可将眼睛参考坐标系中的瞳孔位置进行 归一化处理，将瞳孔像素位置映射为眼睛参考坐标系中的归一化坐标，从而得到归一化的瞳孔位置。比如，可将眼睛中心点距离眼睛图像的轮廓左顶点的距离为标准距离EyeW，从而将(x 2,y 2)除以标准距离EyeW得到归一化的坐标(x 3,y 3)。
在本实施例中，在计算多帧图像帧的瞳孔位置时，通过坐标系变换，在统一的坐标系下确定瞳孔位置，保证后续瞳孔位置移动路径的准确性。
进一步地，计算机设备可将起始位置与眼睛中心点位置的连线，和结束位置与眼睛中心点位置的连线的夹角作为旋转角度，根据眼睛参考坐标系下起始位置、结束位置和眼睛中心点位置的坐标，以及三角函数计算旋转角度具体的角度值，再根据该角度值选择待控制对象。其中，对象旋转的方向与瞳孔移动路径的方向相同。
举例说明，图4示出了一个实施例中确定旋转角度的示意图。参考图4，该示意图包括眼睛中心点位置(O)410，瞳孔移动路径的起始位置(A)420和瞳孔移动路径的结束位置(B)430。那么，旋转角度440即为OA与OB的夹角θ ab。其中，θ ab可通过以下公式计算：θ ab＝(|OA| 2+|OB| 2-|AB| 2)/(2*|OA|*|OB|)。
在一个实施例中，瞳孔位置为双目的瞳孔位置时，计算机设备可随机选取其中一只瞳孔的旋转角度，或者选取旋转较大的瞳孔的旋转角度，或者双目旋转角度的平均值来旋转待控制对象。
上述实施例中，在按照对象控制类型对待控制对象进行控制时，根据实际瞳孔移动时相对于眼睛中心旋转的角度动态旋转待控制对象，使得旋转待控制对象时更灵活。
在一个实施例中，该对象控制方法还包括：当结束位置不在第一眼睛边缘区域时，或者，当结束位置位于第一眼睛边缘区域、且起始位置不在第二眼睛边缘区域时，则根据起始位置和结束位置确定瞳孔位置变化矢量；及获取瞳孔位置变化矢量的标量值，根据标量值确定与对象控制类型相应的对象控制参数。按照对象控制类型对待控制对象进行控制，包括：按照对象控制 类型和对象控制参数对待控制对象进行控制。
其中，对象控制参数是控制对象变化时所依据的参数。对象控制参数具体可以是平移距离或者缩放比例等。
具体地，当结束位置不在第一眼睛边缘区域时，或者，当结束位置位于第一眼睛边缘区域、且起始位置不在第二眼睛边缘区域时，计算机设备可判定对象控制类型可为平移控制类型或者缩放控制类型。此时，计算机设备可进一步根据瞳孔位置的特征确定当前瞳孔移动对应平移控制类型还是缩放控制类型。
计算机设备可继续检测确定的瞳孔位置是单目的瞳孔位置还是双目的瞳孔位置。当确定的瞳孔位置是单目的瞳孔位置时，即可判定起始位置和结束位置所对应的对象控制类型为对象平移控制类型。此时，计算机设备可根据起始位置和结束位置确定瞳孔位置变化矢量，计算瞳孔位置变化矢量的标量值，从而控制待控制对象沿着该瞳孔位置变化矢量的矢量方向，平移该标量值的距离。
在一个实施例中，当瞳孔位置为双目的瞳孔位置；瞳孔位置变化矢量为双目的瞳孔位置变化矢量时，该对象控制方法还包括：确定双目的瞳孔位置变化矢量各自对应的矢量方向；及当双目的瞳孔位置变化矢量各自对应的矢量方向一致时，则判定起始位置和结束位置所对应的对象控制类型为对象平移控制类型。按照对象控制类型和对象控制参数对待控制对象进行控制，包括：按照对象控制参数平移待控制对象。
具体地，计算机设备可分别确定双目的瞳孔位置变化矢量各自对应的矢量方向，计算两个矢量方向的夹角，当夹角小于预设角度时，则判定双目的瞳孔位置变化矢量各自对应的矢量方向一致时，从而判定起始位置和结束位置所对应的对象控制类型为对象平移控制类型。
在一个实施例中，矢量夹角可通过以下公式计算。
vectorMul＝leftIrisMoveVector2.x*rightIrisMoveVector2.x+leftIrisMoveVector2.y*rightIrisMoveVector2.y；
verctorLeftLen＝sqrt(leftIrisMoveVector2.x*leftIrisMoveVector2.x+leftIrisMoveVector2.y*leftIrisMoveVector2.y)；
verctorRightLen＝sqrt(rightIrisMoveVector2.x*rightIrisMoveVector2.x+rightIrisMoveVector2.y*rightIrisMoveVector2.y)；
leftIrisMoveVector2.x＝currentPosLeft.x-lastPosLeft.x；
leftIrisMoveVector2.y＝currentPosLeft.y-lastPosLeft.y；
rightIrisMoveVector2.x＝currentPosright.x-lastPosright.x；
rightIrisMoveVector2.x＝currentPosright.y-lastPosright.y。
其中，vectorMul为中间变量，leftIrisMoveVector2.x为左眼瞳孔位置沿X轴方向变化矢量，leftIrisMoveVector2.y为左眼瞳孔位置沿Y轴方向变化矢量，rightIrisMoveVector2.x为右眼瞳孔位置沿X轴方向变化矢量，rightIrisMoveVector2.y为右眼瞳孔位置沿Y轴方向变化矢量。verctorLeftLen为左眼瞳孔位置变化矢量的标量值，verctorRightLen为右眼瞳孔位置变化矢量的标量值。(currentPosLeft.x，currentPosLeft.y)为左眼结束位置坐标，(lastPosLeft.x，lastPosLeft.y)为左眼起始位置坐标，(currentPosright.x，currentPosright.y)为右眼结束位置坐标，(lastPosright.x，lastPosright.y)为右眼起始位置坐标。
由公式cos(angleValue)*verctorLeftLen*verctorRightLen＝vectorMul可得到双目瞳孔位置变化矢量的夹角angleValue＝acos(angleCosValue)。其中，angleCosValue＝vectorMul/(verctorLeftLen*verctorRightLen)。
可以理解，上述坐标为将瞳孔像素位置映射为眼睛参考坐标系中的坐标。
进一步地，计算机设备可计算双目的瞳孔位置变化矢量的标量值，从中任选一个标量值，或者对应数值较大的标量值，或者两个标量值的平均值，从而控制待控制对象沿着该瞳孔位置变化矢量的矢量方向，平移该选取的标量值的距离。
举例说明，图5示出了一个实施例中双目的瞳孔位置变化矢量各自对应的矢量方向一致时的示意图。假设预设角度为90度。参考图5，可以看出双 目的瞳孔位置变化矢量各自对应的矢量方向的夹角为0度，0度小于90度，那么此时起始位置和结束位置所对应的对象控制类型为对象平移控制类型，从而控制对象平移。
在本实施例中，提供了对象平移控制类型所对应的瞳孔移动方式，具体以双目的瞳孔位置变化矢量各自对应的矢量方向一致的瞳孔移动特征来反映平移对象，从而在检测到这种瞳孔移动特征时即实现平移待控制对象。
在一个实施例中，该对象控制方法还包括：当双目的瞳孔位置变化矢量各自对应的矢量方向不一致时，则判定起始位置和结束位置所对应的对象控制类型为对象缩放控制类型。按照对象控制类型和对象控制参数对待控制对象进行控制，包括：按照对象控制参数调整待控制对象的尺寸。
具体地，计算机设备可分别确定双目的瞳孔位置变化矢量各自对应的矢量方向，计算两个矢量方向的夹角，当夹角达到预设角度时，则判定双目的瞳孔位置变化矢量各自对应的矢量方向不一致时，从而判定起始位置和结束位置所对应的对象控制类型为对象旋转控制类型。
进一步地，计算机设备可计算起始位置所对应的图像帧中，双目的眼睛中心点之间的距离twoEyeCenterW1，双目的瞳孔位置之间的距离twoIrisCenterW1；结束位置所对应的图像帧中，双目的眼睛中心点之间的距离twoEyeCenterW2，双目的瞳孔位置之间的距离twoIrisCenterW2；进而计算scale＝aConstValue*twoIrisCenterW1*twoEyeCenterW2/(twoIrisCenterW2*twoEyeCenterW1)。其中，scale为缩放比例，aConstValue为可变系数，用于调节缩放的灵敏度。
举例说明，图6示出了一个实施例中双目的瞳孔位置变化矢量各自对应的矢量方向不一致时的示意图。假设预设角度为90度。参考图6，可以看出双目的瞳孔位置变化矢量各自对应的矢量方向的夹角为180度，180度大于90度，那么此时起始位置和结束位置所对应的对象控制类型为对象缩放控制类型，从而控制对象缩放。其中，双目的眼睛中心点之间的距离twoEyeCenterW1，双目的瞳孔位置之间的距离twoIrisCenterW1。
在本实施例中，提供了对象缩放控制类型所对应的瞳孔移动方式，具体以双目的瞳孔位置变化矢量各自对应的矢量方向不一致的瞳孔移动特征来反映缩放对象，从而在检测到这种瞳孔移动特征时即实现缩放待控制对象。
上述实施例中，在按照对象控制类型对待控制对象进行控制时，根据实际瞳孔移动时形成的瞳孔位置变化矢量的标量值动态控制待控制对象的变化幅度，使得控制待控制对象时更灵活。
图7为一个具体的实施例中对象控制方法的流程示意图，该对象控制方法具体包括以下步骤：
S702，确定待控制对象。
S704，按时序采集图像帧，检测采集到的图像帧是否包括人脸区域；若是，则跳转到步骤S706；若否，则返回步骤S704。
S706，判断该图像帧所对应的人脸偏转角度是否与表示正脸图像的参考偏转角度匹配；若是，则跳转到步骤S708；若否，则返回步骤S704。
S708，确定该图像帧中双目的瞳孔像素位置以及眼睛中心点位置；以眼睛中心点位置为参考位置建立眼睛参考坐标系；将双目的瞳孔像素位置映射为眼睛参考坐标系中的坐标，得到双目的瞳孔位置并缓存。
S710，在检测到下一帧包括人脸区域的图像帧时，将缓存的双目的瞳孔位置作为双目的瞳孔移动路径各自的起始位置；获取当前包括人脸区域的图像帧的双目的瞳孔位置，作为双目的瞳孔移动路径各自的结束位置。
S712，判断结束位置是否位于相应的人脸图像帧中的第一眼睛边缘区域；若是，则跳转到步骤S714；若否，则返回步骤S720。
S714，判断起始位置是否位于相应的人脸图像帧中的第二眼睛边缘区域；若是，则跳转到步骤S716；若否，则返回步骤S720。
S716，判定双目的起始位置和结束位置所对应的对象控制类型为对象旋转控制类型。
S718，根据起始位置、结束位置和眼睛中心点位置，确定旋转角度；按照旋转角度旋转待控制对象。
S720，根据双目的起始位置和结束位置确定双目的瞳孔位置变化矢量。
S722，判断双目的瞳孔位置变化矢量各自对应的矢量方向是否一致；若是，则跳转到步骤S724；若否，则返回步骤S728。
S724，判定双目的起始位置和结束位置所对应的对象控制类型为对象平移控制类型。
S726，获取瞳孔位置变化矢量的标量值；根据标量值确定与对象控制类型相应的对象控制参数；按照对象控制参数平移待控制对象。
S728，判定双目的起始位置和结束位置所对应的对象控制类型为对象缩放控制类型。
S730，获取瞳孔位置变化矢量的标量值；根据标量值确定与对象控制类型相应的对象控制参数；按照对象控制参数调整待控制对象的尺寸。
在本实施例中，在确定待控制的目标后，自动在按时序采集的人脸图像帧中确定瞳孔位置，进而在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置，从而根据瞳孔的移动路径确定对象控制类型以按照不同的控制类型控制目标。这样通过眼睛控制目标，避免了手动控制的过程中手或者手指造成目标遮挡的问题，提高了对象控制准确率。
而且，这种通过瞳孔移动控制对象的方式，对手脚不便无法通过手工操作来实现对象控制的用户，提供了很大操作便利。
应该理解的是，虽然上述各实施例的流程图中的各个步骤按照箭头的指示依次显示，但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明，这些步骤的执行并没有严格的顺序限制，这些步骤可以以其它的顺序执行。而且，上述各实施例中的至少一部分步骤可以包括多个子步骤或者多个阶段，这些子步骤或者阶段并不必然是在同一时刻执行完成，而是可以在不同的时刻执行，这些子步骤或者阶段的执行顺序也不必然是依次进行，而是可以与其它步骤或者其它步骤的子步骤或者阶段的至少一部分轮流或者交替地执行。
如图8所示，在一个实施例中，提供了一种对象控制装置800。参照图8，该对象控制装置800包括：对象确定模块801、位置确定模块802、类型确定模块803和对象控制模块804。对象控制装置800中包括的各个模块可全部或部分通过软件、硬件或其组合来实现。
对象确定模块801，用于确定待控制对象。
位置确定模块802，用于在按时序采集的人脸图像帧中确定瞳孔位置；在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置。
类型确定模块803，用于根据起始位置和结束位置确定对象控制类型。
对象控制模块804，用于按照对象控制类型对待控制对象进行控制。
上述对象控制装置800，在确定待控制的目标后，自动在按时序采集的人脸图像帧中确定瞳孔位置，进而在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置，从而根据瞳孔的移动路径确定对象控制类型以按照不同的控制类型控制目标。这样通过眼睛控制目标，避免了手动控制的过程中手或者手指造成目标遮挡的问题，提高了对象控制准确率。
在一个实施例中，位置确定模块802还用于按时序采集图像帧；当采集的图像帧包括人脸区域时，确定并缓存图像帧中的瞳孔位置；在检测到下一帧包括人脸区域的图像帧时，将缓存的瞳孔位置作为瞳孔移动路径的起始位置；及获取下一帧包括人脸区域的图像帧的瞳孔位置，作为瞳孔移动路径的结束位置。
在一个实施例中，位置确定模块802还用于当采集的图像帧包括人脸区域时，确定图像帧所对应的人脸偏转角度；及在人脸偏转角度与表示正脸图像的参考偏转角度匹配时，确定并缓存图像帧中的瞳孔位置。
在一个实施例中，位置确定模块802还用于按多帧人脸图像帧的采集时序，提取各人脸图像帧的人脸特征；当人脸特征符合起始移动特征时，则获取人脸特征提取自的人脸图像帧的瞳孔位置作为瞳孔移动路径的起始位置； 及当人脸特征符合结束移动特征时，则获取人脸特征提取自的人脸图像帧的瞳孔位置作为瞳孔移动路径的结束位置。
在一个实施例中，类型确定模块803还用于在结束位置相应的人脸图像帧中确定第一眼睛边缘区域；当结束位置位于第一眼睛边缘区域时，在起始位置相应的人脸图像帧中确定第二眼睛边缘区域；及当起始位置位于第二眼睛边缘区域时，判定起始位置和结束位置所对应的对象控制类型为对象旋转控制类型。
在一个实施例中，对象控制模块804还用于在起始位置和结束位置各自相应的人脸图像帧中，确定眼睛中心点位置；根据起始位置、结束位置和眼睛中心点位置，确定旋转角度；及按照旋转角度旋转待控制对象。
如图9所示，在一个实施例中，对象控制装置800还包括：参数确定模块805。
参数确定模块805，用于当结束位置不在第一眼睛边缘区域时，或者，当结束位置位于第一眼睛边缘区域、且起始位置不在第二眼睛边缘区域时，则根据起始位置和结束位置确定瞳孔位置变化矢量；获取瞳孔位置变化矢量的标量值；及根据标量值确定与对象控制类型相应的对象控制参数。对象控制模块804还用于按照对象控制类型和对象控制参数对待控制对象进行控制。
在一个实施例中，瞳孔位置为双目的瞳孔位置；瞳孔位置变化矢量为双目的瞳孔位置变化矢量。类型确定模块803还用于确定双目的瞳孔位置变化矢量各自对应的矢量方向；及当双目的瞳孔位置变化矢量各自对应的矢量方向一致时，则判定起始位置和结束位置所对应的对象控制类型为对象平移控制类型。对象控制模块804还用于按照对象控制参数平移待控制对象。
在一个实施例中，类型确定模块803还用于当双目的瞳孔位置变化矢量各自对应的矢量方向不一致时，则判定起始位置和结束位置所对应的对象控制类型为对象缩放控制类型。对象控制模块804还用于按照对象控制参数调整待控制对象的尺寸。
在一个实施例中，位置确定模块802还用于在按时序采集的人脸图像帧 中确定瞳孔像素位置以及眼睛中心点位置；以眼睛中心点位置为参考位置建立眼睛参考坐标系；及将瞳孔像素位置映射为眼睛参考坐标系中的坐标，得到瞳孔位置。
图10示出了一个实施例中计算机设备的内部结构图。如图10所示，该计算机设备包括通过系统总线连接的处理器、存储器、网络接口、摄像头、输入设备和显示屏。其中，存储器包括非易失性存储介质和内存储器。该计算机设备的非易失性存储介质存储有操作系统，还可存储有计算机可读指令，该计算机可读指令被处理器执行时，可使得处理器实现对象控制方法。该内存储器中也可储存有计算机可读指令，该计算机可读指令被处理器执行时，可使得处理器执行对象控制方法。计算机设备的显示屏可以是液晶显示屏或者电子墨水显示屏等，输入装置可以是显示屏上覆盖的触摸层，也可以是计算机设备外壳上设置的按键、轨迹球或触控板，也可以是外接的键盘、触控板或鼠标等。该计算机设备可以是用户终端，用户终端具体可以是手机、平板电脑、个人数字助理或者穿戴式设备等中的至少一种。本领域技术人员可以理解，图10中示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的计算机设备的限定，具体的计算机设备可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。
在一个实施例中，本申请提供的对象控制装置可以实现为一种计算机可读指令的形式，计算机可读指令可在如图10所示的计算机设备上运行，计算机设备的非易失性存储介质可存储组成该对象控制装置的各个指令模块，比如，图8所示的对象确定模块801、位置确定模块802、类型确定模块803和对象控制模块804等。各个指令模块组成的计算机可读指令使得处理器执行本说明书中描述的本申请各个实施例的对象控制方法中的步骤。
例如，图10所示的计算机设备可以通过如图8所示的对象控制装置800中的对象确定模块801用于确定待控制对象。通过位置确定模块802用于在按时序采集的人脸图像帧中确定瞳孔位置；在多帧人脸图像帧中确定的瞳孔 位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置。通过类型确定模块803用于根据起始位置和结束位置确定对象控制类型。通过对象控制模块804用于按照对象控制类型对待控制对象进行控制。
在一个实施例中，提供了一种计算机可读存储介质，该计算机可读存储介质上存储有计算机可读指令，该计算机可读指令被处理器执行时，使得处理器执行以下步骤：确定待控制对象；在按时序采集的人脸图像帧中确定瞳孔位置；在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置；根据起始位置和结束位置确定对象控制类型；及按照对象控制类型对待控制对象进行控制。
在一个实施例中，在按时序采集的人脸图像帧中确定瞳孔位置，包括：按时序采集图像帧；及当采集的图像帧包括人脸区域时，确定并缓存图像帧中的瞳孔位置。在检测到下一帧包括人脸区域的图像帧时，在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置，包括：将缓存的瞳孔位置作为瞳孔移动路径的起始位置；及获取下一帧包括人脸区域的图像帧的瞳孔位置，作为瞳孔移动路径的结束位置。
在一个实施例中，当采集的图像帧包括人脸区域时，确定并缓存图像帧中的瞳孔位置，包括：当采集的图像帧包括人脸区域时，确定图像帧所对应的人脸偏转角度；及在人脸偏转角度与表示正脸图像的参考偏转角度匹配时，确定并缓存图像帧中的瞳孔位置。
在一个实施例中，在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置，包括：按多帧人脸图像帧的采集时序，提取各人脸图像帧的人脸特征；当人脸特征符合起始移动特征时，则获取人脸特征提取自的人脸图像帧的瞳孔位置作为瞳孔移动路径的起始位置；及当人脸特征符合结束移动特征时，则获取人脸特征提取自的人脸图像帧的瞳孔位置作为瞳孔移动路径的结束位置。
在一个实施例中，根据起始位置和结束位置确定对象控制类型，包括：在结束位置相应的人脸图像帧中确定第一眼睛边缘区域；当结束位置位于第一眼睛边缘区域时，在起始位置相应的人脸图像帧中确定第二眼睛边缘区域；及当起始位置位于第二眼睛边缘区域时，判定起始位置和结束位置所对应的对象控制类型为对象旋转控制类型。
在一个实施例中，按照对象控制类型对待控制对象进行控制，包括：在起始位置和结束位置各自相应的人脸图像帧中，确定眼睛中心点位置；根据起始位置、结束位置和眼睛中心点位置，确定旋转角度；及按照旋转角度旋转待控制对象。
在一个实施例中，该计算机可读指令被处理器执行时，还使得处理器执行以下步骤：当结束位置不在第一眼睛边缘区域时，或者，当结束位置位于第一眼睛边缘区域、且起始位置不在第二眼睛边缘区域时，则根据起始位置和结束位置确定瞳孔位置变化矢量；获取瞳孔位置变化矢量的标量值；及根据标量值确定与对象控制类型相应的对象控制参数。按照对象控制类型对待控制对象进行控制，包括：按照对象控制类型和对象控制参数对待控制对象进行控制。
在一个实施例中，瞳孔位置为双目的瞳孔位置；瞳孔位置变化矢量为双目的瞳孔位置变化矢量。该计算机可读指令被处理器执行时，还使得处理器执行以下步骤：确定双目的瞳孔位置变化矢量各自对应的矢量方向；及当双目的瞳孔位置变化矢量各自对应的矢量方向一致时，则判定起始位置和结束位置所对应的对象控制类型为对象平移控制类型。按照对象控制类型和对象控制参数对待控制对象进行控制，包括：按照对象控制参数平移待控制对象。
在一个实施例中，该计算机可读指令被处理器执行时，还使得处理器执行以下步骤：当双目的瞳孔位置变化矢量各自对应的矢量方向不一致时，则判定起始位置和结束位置所对应的对象控制类型为对象缩放控制类型。按照对象控制类型和对象控制参数对待控制对象进行控制，包括：按照对象控制参数调整待控制对象的尺寸。
在一个实施例中，在按时序采集的人脸图像帧中确定瞳孔位置，包括：在按时序采集的人脸图像帧中确定瞳孔像素位置以及眼睛中心点位置；以眼睛中心点位置为参考位置建立眼睛参考坐标系；及将瞳孔像素位置映射为眼睛参考坐标系中的坐标，得到瞳孔位置。
上述存储介质，在确定待控制的目标后，自动在按时序采集的人脸图像帧中确定瞳孔位置，进而在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置，从而根据瞳孔的移动路径确定对象控制类型以按照不同的控制类型控制目标。这样通过眼睛控制目标，避免了手动控制的过程中手或者手指造成目标遮挡的问题，提高了对象控制准确率。
在一个实施例中，提供了一种计算机设备，包括存储器和处理器，存储器中储存有计算机可读指令，计算机可读指令被处理器执行时，使得处理器执行以下步骤：确定待控制对象；在按时序采集的人脸图像帧中确定瞳孔位置；在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置；根据起始位置和结束位置确定对象控制类型；及按照对象控制类型对待控制对象进行控制。
在一个实施例中，在按时序采集的人脸图像帧中确定瞳孔位置，包括：按时序采集图像帧；及当采集的图像帧包括人脸区域时，确定并缓存图像帧中的瞳孔位置。在检测到下一帧包括人脸区域的图像帧时，在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置，包括：将缓存的瞳孔位置作为瞳孔移动路径的起始位置；及获取下一帧包括人脸区域的图像帧的瞳孔位置，作为瞳孔移动路径的结束位置。
在一个实施例中，当采集的图像帧包括人脸区域时，确定并缓存图像帧中的瞳孔位置，包括：当采集的图像帧包括人脸区域时，确定图像帧所对应的人脸偏转角度；及在人脸偏转角度与表示正脸图像的参考偏转角度匹配时，确定并缓存图像帧中的瞳孔位置。
在一个实施例中，在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置，包括：按多帧人脸图像帧的采集时序，提取各人脸图像帧的人脸特征；当人脸特征符合起始移动特征时，则获取人脸特征提取自的人脸图像帧的瞳孔位置作为瞳孔移动路径的起始位置；及当人脸特征符合结束移动特征时，则获取人脸特征提取自的人脸图像帧的瞳孔位置作为瞳孔移动路径的结束位置。
在一个实施例中，根据起始位置和结束位置确定对象控制类型，包括：在结束位置相应的人脸图像帧中确定第一眼睛边缘区域；当结束位置位于第一眼睛边缘区域时，在起始位置相应的人脸图像帧中确定第二眼睛边缘区域；及当起始位置位于第二眼睛边缘区域时，判定起始位置和结束位置所对应的对象控制类型为对象旋转控制类型。
在一个实施例中，按照对象控制类型对待控制对象进行控制，包括：在起始位置和结束位置各自相应的人脸图像帧中，确定眼睛中心点位置；根据起始位置、结束位置和眼睛中心点位置，确定旋转角度；及按照旋转角度旋转待控制对象。
在一个实施例中，该计算机可读指令被处理器执行时，还使得处理器执行以下步骤：当结束位置不在第一眼睛边缘区域时，或者，当结束位置位于第一眼睛边缘区域、且起始位置不在第二眼睛边缘区域时，则根据起始位置和结束位置确定瞳孔位置变化矢量；获取瞳孔位置变化矢量的标量值；及根据标量值确定与对象控制类型相应的对象控制参数。按照对象控制类型对待控制对象进行控制，包括：按照对象控制类型和对象控制参数对待控制对象进行控制。
在一个实施例中，瞳孔位置为双目的瞳孔位置；瞳孔位置变化矢量为双目的瞳孔位置变化矢量。该计算机可读指令被处理器执行时，还使得处理器执行以下步骤：确定双目的瞳孔位置变化矢量各自对应的矢量方向；及当双目的瞳孔位置变化矢量各自对应的矢量方向一致时，则判定起始位置和结束位置所对应的对象控制类型为对象平移控制类型。按照对象控制类型和对象 控制参数对待控制对象进行控制，包括：按照对象控制参数平移待控制对象。
在一个实施例中，该计算机可读指令被处理器执行时，还使得处理器执行以下步骤：当双目的瞳孔位置变化矢量各自对应的矢量方向不一致时，则判定起始位置和结束位置所对应的对象控制类型为对象缩放控制类型。按照对象控制类型和对象控制参数对待控制对象进行控制，包括：按照对象控制参数调整待控制对象的尺寸。
在一个实施例中，在按时序采集的人脸图像帧中确定瞳孔位置，包括：在按时序采集的人脸图像帧中确定瞳孔像素位置以及眼睛中心点位置；以眼睛中心点位置为参考位置建立眼睛参考坐标系；及将瞳孔像素位置映射为眼睛参考坐标系中的坐标，得到瞳孔位置。
上述计算机设备，在确定待控制的目标后，自动在按时序采集的人脸图像帧中确定瞳孔位置，进而在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置，从而根据瞳孔的移动路径确定对象控制类型以按照不同的控制类型控制目标。这样通过眼睛控制目标，避免了手动控制的过程中手或者手指造成目标遮挡的问题，提高了对象控制准确率。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机可读指令来指令相关的硬件来完成，所述的程序可存储于一非易失性计算机可读取存储介质中，该程序在执行时，可包括如上述各方法的实施例的流程。其中，本申请所提供的各实施例中所使用的对存储器、存储、数据库或其它介质的任何引用，均可包括非易失性和/或易失性存储器。非易失性存储器可包括只读存储器(ROM)、可编程ROM(PROM)、电可编程ROM(EPROM)、电可擦除可编程ROM(EEPROM)或闪存。易失性存储器可包括随机存取存储器(RAM)或者外部高速缓冲存储器。作为说明而非局限，RAM以多种形式可得，诸如静态RAM(SRAM)、动态RAM(DRAM)、同步DRAM(SDRAM)、双数据率SDRAM(DDRSDRAM)、增强型SDRAM(ESDRAM)、同步链路(Synchlink)DRAM(SLDRAM)、存储器总线 (Rambus)直接RAM(RDRAM)、直接存储器总线动态RAM(DRDRAM)、以及存储器总线动态RAM(RDRAM)等。
以上实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。
以上实施例仅表达了本申请的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对发明专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干变形和改进，这些都属于本申请的保护范围。因此，本申请专利的保护范围应以所附权利要求为准。

Claims (15)

  1. 一种对象控制方法，由计算机设备执行，所述方法包括：
    确定待控制对象；
    在按时序采集的人脸图像帧中确定瞳孔位置；
    在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置；
    根据所述起始位置和所述结束位置确定对象控制类型；及
    按照所述对象控制类型对所述待控制对象进行控制。
  2. 根据权利要求1所述的方法，其特征在于，所述在按时序采集的人脸图像帧中确定瞳孔位置，包括：
    按时序采集图像帧；及
    当采集的图像帧包括人脸区域时，确定并缓存所述图像帧中的瞳孔位置；
    在检测到下一帧包括人脸区域的图像帧时，所述在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置，包括：
    将缓存的所述瞳孔位置作为瞳孔移动路径的起始位置；及
    获取所述下一帧包括人脸区域的图像帧的瞳孔位置，作为瞳孔移动路径的结束位置。
  3. 根据权利要求2所述的方法，其特征在于，所述当采集的图像帧包括人脸区域时，确定并缓存所述图像帧中的瞳孔位置，包括：
    当采集的图像帧包括人脸区域时，确定所述图像帧所对应的人脸偏转角度；及
    在所述人脸偏转角度与表示正脸图像的参考偏转角度匹配时，确定并缓存所述图像帧中的瞳孔位置。
  4. 根据权利要求1所述的方法，其特征在于，所述在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置，包括：
    按多帧人脸图像帧的采集时序，提取各人脸图像帧的人脸特征；
    当所述人脸特征符合起始移动特征时，则获取所述人脸特征提取自的人脸图像帧的瞳孔位置作为瞳孔移动路径的起始位置；及
    当所述人脸特征符合结束移动特征时，则获取所述人脸特征提取自的人脸图像帧的瞳孔位置作为瞳孔移动路径的结束位置。
  5. 根据权利要求1所述的方法，其特征在于，所述根据所述起始位置和所述结束位置确定对象控制类型，包括：
    在所述结束位置相应的人脸图像帧中确定第一眼睛边缘区域；
    当所述结束位置位于所述第一眼睛边缘区域时，在所述起始位置相应的人脸图像帧中确定第二眼睛边缘区域；及
    当所述起始位置位于所述第二眼睛边缘区域时，判定所述起始位置和所述结束位置所对应的对象控制类型为对象旋转控制类型。
  6. 根据权利要求5所述的方法，其特征在于，所述按照所述对象控制类型对所述待控制对象进行控制，包括：
    在所述起始位置和所述结束位置各自相应的人脸图像帧中，确定眼睛中心点位置；
    根据所述起始位置、所述结束位置和所述眼睛中心点位置，确定旋转角度；及
    按照所述旋转角度旋转所述待控制对象。
  7. 根据权利要求5所述的方法，其特征在于，所述方法还包括：
    当所述结束位置不在所述第一眼睛边缘区域时，或者，当所述结束位置位于所述第一眼睛边缘区域、且所述起始位置不在所述第二眼睛边缘区域时，
    则根据所述起始位置和所述结束位置确定瞳孔位置变化矢量；
    获取所述瞳孔位置变化矢量的标量值；及
    根据所述标量值确定与所述对象控制类型相应的对象控制参数；
    所述按照所述对象控制类型对所述待控制对象进行控制，包括：
    按照所述对象控制类型和所述对象控制参数对所述待控制对象进行控 制。
  8. 根据权利要求7所述的方法，其特征在于，所述瞳孔位置为双目的瞳孔位置；所述瞳孔位置变化矢量为双目的瞳孔位置变化矢量；所述方法还包括：
    确定双目的瞳孔位置变化矢量各自对应的矢量方向；及
    当双目的瞳孔位置变化矢量各自对应的矢量方向一致时，则判定所述起始位置和所述结束位置所对应的对象控制类型为对象平移控制类型；
    所述按照所述对象控制类型和所述对象控制参数对所述待控制对象进行控制，包括：
    按照所述对象控制参数平移所述待控制对象。
  9. 根据权利要求8所述的方法，其特征在于，所述方法还包括：
    当双目的瞳孔位置变化矢量各自对应的矢量方向不一致时，则判定所述起始位置和所述结束位置所对应的对象控制类型为对象缩放控制类型；
    所述按照所述对象控制类型和所述对象控制参数对所述待控制对象进行控制，包括：
    按照所述对象控制参数调整所述待控制对象的尺寸。
  10. 根据权利要求1所述的方法，其特征在于，所述在按时序采集的人脸图像帧中确定瞳孔位置，包括：
    在按时序采集的人脸图像帧中确定瞳孔像素位置以及眼睛中心点位置；
    以所述眼睛中心点位置为参考位置建立眼睛参考坐标系；及
    将所述瞳孔像素位置映射为所述眼睛参考坐标系中的坐标，得到瞳孔位置。
  11. 一种对象控制装置，包括：
    对象确定模块，用于确定待控制对象；
    位置确定模块，用于在按时序采集的人脸图像帧中确定瞳孔位置；在多帧人脸图像帧中确定的瞳孔位置中，按相应人脸图像帧的采集时序，分别确定瞳孔移动路径的起始位置和结束位置；
    类型确定模块，用于根据所述起始位置和所述结束位置确定对象控制类型；及
    对象控制模块，用于按照所述对象控制类型对所述待控制对象进行控制。
  12. 根据权利要求11所述的装置，其特征在于，所述类型确定模块还用于在所述结束位置相应的人脸图像帧中确定第一眼睛边缘区域；当所述结束位置位于所述第一眼睛边缘区域时，在所述起始位置相应的人脸图像帧中确定第二眼睛边缘区域；及当所述起始位置位于所述第二眼睛边缘区域时，判定所述起始位置和所述结束位置所对应的对象控制类型为对象旋转控制类型。
  13. 根据权利要求12所述的装置，其特征在于，所述类型确定模块还用于当所述结束位置不在所述第一眼睛边缘区域时，或者，当所述结束位置位于所述第一眼睛边缘区域、且所述起始位置不在所述第二眼睛边缘区域时，则根据所述起始位置和所述结束位置确定瞳孔位置变化矢量；获取所述瞳孔位置变化矢量的标量值；根据所述标量值确定与所述对象控制类型相应的对象控制参数；及
    所述对象控制模块还用于按照所述对象控制类型和所述对象控制参数对所述待控制对象进行控制。
  14. 一种存储有计算机可读指令的非易失性存储介质，所述计算机可读指令被一个或多个处理器执行时，使得一个或多个处理器执行如权利要求1至10中任一项所述的方法的步骤。
  15. 一种计算机设备，包括存储器和处理器，所述存储器中存储有计算机可读指令，所述计算机可读指令被所述处理器执行时，使得所述处理器执行如权利要求1至10中任一项所述的方法的步骤。
PCT/CN2018/123062 2017-12-29 2018-12-24 对象控制方法、装置、存储介质和计算机设备 WO2019128917A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201711483962.7A CN108090463B (zh) 2017-12-29 2017-12-29 对象控制方法、装置、存储介质和计算机设备
CN201711483962.7 2017-12-29

Publications (1)

Publication Number Publication Date
WO2019128917A1 true WO2019128917A1 (zh) 2019-07-04

Family

ID=62181338

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/123062 WO2019128917A1 (zh) 2017-12-29 2018-12-24 对象控制方法、装置、存储介质和计算机设备

Country Status (2)

Country Link
CN (1) CN108090463B (zh)
WO (1) WO2019128917A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111652807A (zh) * 2019-07-05 2020-09-11 广州虎牙科技有限公司 眼部的调整、直播方法、装置、电子设备和存储介质
CN112016508A (zh) * 2020-09-07 2020-12-01 杭州海康威视数字技术股份有限公司 人脸识别方法、装置、系统、计算设备及存储介质
CN114429656A (zh) * 2021-12-30 2022-05-03 苏州思英智能科技有限公司 人脸识别设备控制方法、装置、电子设备和介质

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108962182A (zh) * 2018-06-15 2018-12-07 广东康云多维视觉智能科技有限公司 基于眼球追踪的三维图像显示装置及其实现方法
CN111435546A (zh) * 2019-01-15 2020-07-21 北京字节跳动网络技术有限公司 模型动作方法、装置、带屏音箱、电子设备及存储介质
CN111514584B (zh) * 2019-02-01 2022-07-26 北京市商汤科技开发有限公司 游戏控制方法及装置、游戏终端及存储介质
CN113986407A (zh) * 2020-07-27 2022-01-28 华为技术有限公司 封面生成方法、装置及计算机存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101344919A (zh) * 2008-08-05 2009-01-14 华南理工大学 视线跟踪方法及应用该方法的残疾人辅助系统
CN102799277A (zh) * 2012-07-26 2012-11-28 深圳先进技术研究院 一种基于眨眼动作的人机交互方法及系统
US20150035746A1 (en) * 2011-12-27 2015-02-05 Andy Cockburn User Interface Device
CN106354264A (zh) * 2016-09-09 2017-01-25 电子科技大学 基于视线追踪的实时人机交互系统及其工作方法

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5850211A (en) * 1996-06-26 1998-12-15 Sun Microsystems, Inc. Eyetrack-driven scrolling
US9274598B2 (en) * 2003-08-25 2016-03-01 International Business Machines Corporation System and method for selecting and activating a target object using a combination of eye gaze and key presses
CN104216508B (zh) * 2013-05-31 2017-05-10 中国电信股份有限公司 通过眼动跟踪技术进行功能键操作的方法和装置
CN103440476A (zh) * 2013-08-26 2013-12-11 大连理工大学 一种人脸视频中瞳孔的定位方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101344919A (zh) * 2008-08-05 2009-01-14 华南理工大学 视线跟踪方法及应用该方法的残疾人辅助系统
US20150035746A1 (en) * 2011-12-27 2015-02-05 Andy Cockburn User Interface Device
CN102799277A (zh) * 2012-07-26 2012-11-28 深圳先进技术研究院 一种基于眨眼动作的人机交互方法及系统
CN106354264A (zh) * 2016-09-09 2017-01-25 电子科技大学 基于视线追踪的实时人机交互系统及其工作方法

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111652807A (zh) * 2019-07-05 2020-09-11 广州虎牙科技有限公司 眼部的调整、直播方法、装置、电子设备和存储介质
CN111652807B (zh) * 2019-07-05 2024-03-08 广州虎牙科技有限公司 眼部的调整、直播方法、装置、电子设备和存储介质
CN112016508A (zh) * 2020-09-07 2020-12-01 杭州海康威视数字技术股份有限公司 人脸识别方法、装置、系统、计算设备及存储介质
CN112016508B (zh) * 2020-09-07 2023-08-29 杭州海康威视数字技术股份有限公司 人脸识别方法、装置、系统、计算设备及存储介质
CN114429656A (zh) * 2021-12-30 2022-05-03 苏州思英智能科技有限公司 人脸识别设备控制方法、装置、电子设备和介质
CN114429656B (zh) * 2021-12-30 2023-01-13 苏州思英智能科技有限公司 人脸识别设备控制方法、装置、电子设备和介质

Also Published As

Publication number Publication date
CN108090463B (zh) 2021-10-26
CN108090463A (zh) 2018-05-29

Similar Documents

Publication Publication Date Title
WO2019128917A1 (zh) 对象控制方法、装置、存储介质和计算机设备
US10365713B2 (en) Method and apparatus for providing interface recognizing movement in accordance with user's view
JP7058760B2 (ja) 画像処理方法およびその、装置、端末並びにコンピュータプログラム
WO2019205850A1 (zh) 位姿确定方法、装置、智能设备及存储介质
TWI678099B (zh) 視頻處理方法、裝置和儲存介質
US9516214B2 (en) Information processing device and information processing method
US9437045B2 (en) Real-time mobile capture and application of photographic images as textures in three-dimensional models
WO2015180659A1 (zh) 图像处理方法和图像处理装置
JP4608326B2 (ja) 指示動作認識装置及び指示動作認識プログラム
WO2016134534A1 (zh) 自动调整摄像头的方法和电子设备
JP7026825B2 (ja) 画像処理方法及び装置、電子機器並びに記憶媒体
TW201404128A (zh) 基於運動之影像拼接
WO2017144899A1 (en) Depth of field processing
KR20160110070A (ko) 전자 장치 및 그 제어 방법
WO2019019372A1 (zh) 移动终端的图片操作控制方法、装置、移动终端及介质
CN114640833A (zh) 投影画面调整方法、装置、电子设备和存储介质
TWI676113B (zh) 虹膜識別過程中的預覽方法及裝置
KR101597915B1 (ko) 화상 처리 장치 및 화상 처리 방법
WO2016187752A1 (zh) 一种测量天线姿态的方法和装置
CN115278084A (zh) 图像处理方法、装置、电子设备及存储介质
JP2012160051A (ja) 情報端末装置
CN114697530B (zh) 一种智能取景推荐的拍照方法及装置
JP5805013B2 (ja) 撮像画像表示装置、撮像画像表示方法、プログラム
CN107952240B (zh) 利用自拍杆实现的游戏控制方法、装置及计算设备
TW201714074A (zh) 使用手勢的拍照方法、系統與電子裝置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18897700

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18897700

Country of ref document: EP

Kind code of ref document: A1