TW201916669A - Method and apparatus for gaze recognition and interaction - Google Patents

Abstract

A method and an apparatus for gaze recognition and interaction are provided. The method is adapted for an electronic apparatus having a camera and a steering gear configured to change a direction of the camera. The method includes following steps: capturing multiple video frames; detecting at least one face in a current video frame of the captured video frames and a rotated video frame that is generated by rotating the current video frame relative to a direction axis; recognizing each of the detected faces to determine whether there is a face gazing the camera by using a previously trained classifier; and if it is confirmed that there is a face gazing the camera, controlling the steering gear to turn the camera toward the face being recognized as gazing the camera according to a position of the face in the current video frame or a position of the face mapped from the rotated video frame to the current video frame.

Description

Gaze recognition and interaction method and device

The present invention relates to an interactive method and apparatus, and more particularly to a method and apparatus for gaze recognition and interaction.

Current interactive devices (such as electronic dolls, electronic pets, or smart robots) can interact with the user through limb movement or sound and light effects to achieve entertainment. For example, an electronic pet can detect the user's voice and correspondingly change the expression or respond. With the action of instant response, the effect of interacting with the user can be achieved.

However, the actions or responses of these interactive devices must be pre-defined, and in the process of interacting with the user, only simple responses can be made for specific instructions (such as pressing a button or making a sound), and cannot be based on The user's facial expression or body language responds appropriately and fails to reflect the effect of human interaction in the real scene.

In view of this, the present invention provides a gaze recognition and interaction method and apparatus, which can simulate the effect of the visual communication when a person interacts with a person in a real scene.

The gaze recognition and interaction method of the present invention is applicable to an electronic device having a camera and a steering gear, wherein the steering gear is used to steer the camera. The method includes the following steps: capturing a plurality of video frames by using a camera; detecting at least one person in a rotating video frame generated by rotating a current video frame of the video frame and the current video frame in a direction axis Face; using a pre-trained classifier to identify whether each detected face is facing the camera; and if the recognition result confirms that the face is facing, according to the position of the face in the current video frame or mapped back to the current by the rotating video frame The position of the video frame controls the steering gear to turn the camera to the face that is recognized as the opposite.

The gaze recognition and interaction device of the present invention includes a camera, a steering gear, a storage device, and a processor. The camera is used to capture multiple video frames. The steering gear is used to steer the camera. The storage device is used to store a plurality of modules. The processor is configured to access and execute a module stored in the storage device. These modules include a video frame rotation module, a face detection module, a visual recognition module, and a steering module. The video frame rotation module rotates the current video frame of the video frame to a rotating video frame. The face detection module detects at least one face of the current video frame and the rotated video frame. The visual recognition module uses a pre-trained classifier to identify whether each detected face is facing the camera. The steering module controls the steering gear to turn the camera according to the position of the face in the current video frame or the position of the current video frame by the rotating video frame when the face recognition is confirmed by the recognition module of the visual recognition module. The face that is recognized as the opposite.

Based on the above, the gaze recognition and interaction method and apparatus of the present invention can detect face detection by performing face detection on the video frame captured by the camera, and rotating the video frame according to different axial directions, and then performing face detection. Face. The face of the detected face is recognized by the pre-trained classifier, and it can be confirmed whether the detected face is facing the camera, and then the camera is controlled to turn to the face. In this way, the effect of the visual communication in the dialogue between people in a real situation can be simulated.

The above described features and advantages of the invention will be apparent from the following description.

The invention integrates technologies such as voice recognition, face detection and visual recognition into smart robots or other smart devices that can interact with people. When the user's voice is received, the robot will turn to the sounding direction, so that the camera disposed on the robot can capture the user's video frame. When the user looks at the robot, the robot can detect the face from the video frame, and use the pre-trained classifier to identify whether the detected face is facing the robot, and then turn the head of the robot to the center of the face (representative use) The eyes of the person) can be used to simulate the effect of the visual communication when people talk to each other in a real situation.

FIG. 1 is a block diagram of a gaze recognition and interaction apparatus according to an embodiment of the invention. Referring to FIG. 1 , the gaze recognition and interaction device 10 of the present embodiment is, for example, a smart robot or other electronic device that can interact with a person, including a camera 12 , a steering gear 14 , a storage device 16 , and a processor 18 . as follows:

The camera 12 is composed of, for example, a lens, a diaphragm, a shutter, an image sensor, and the like. Wherein, the lens comprises a plurality of optical lenses, which are driven, for example, by an actuator such as a stepping motor or a voice coil motor (VCM) to change the relative position between the lenses, thereby changing the focal length. The aperture is a ring-shaped opening made up of a plurality of metal blades, and the opening is opened or reduced according to the aperture value, thereby controlling the amount of light entering the lens. The shutter is used to control the length of time that light enters the lens. The combination of the aperture and the aperture affects the exposure of the image captured by the image sensor. The image sensor is composed of, for example, a Charge Coupled Device (CCD), a Complementary Metal-Oxide Semiconductor (CMOS) component, or other kinds of photosensitive elements, which can sense the light intensity entering the lens. To create a frame of the subject.

The steering gear 14 is, for example, a servo motor that can be disposed under or around the camera 12 and can push the camera 12 to change its position and/or angle in accordance with the control signal of the processor 18.

The storage device 16 can be any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, A hard disk drive (HDD), a solid state drive (SSD) or the like or a combination of the above elements. In this embodiment, the storage device 16 is configured to store the software programs of the face detection module 162, the video frame rotation module 164, the view recognition module 166, and the steering module 168.

The processor 18 is, for example, a central processing unit (CPU), or other programmable microprocessor (Microprocessor), a digital signal processor (DSP), a programmable controller, and a special application. Application Specific Integrated Circuit (ASIC) or other similar components or a combination of the above. In this embodiment, the processor 18 is configured to access and execute the modules stored in the storage device 16 to implement the gaze recognition and interaction method in the embodiment of the present invention.

2 is a flow chart of a method for gaze recognition and interaction according to an embodiment of the invention. Referring to FIG. 1 and FIG. 2 simultaneously, the method of the present embodiment is applicable to the above-mentioned gaze recognition and interaction device 10, and the following is a detailed description of the detailed process of the method of the present embodiment in conjunction with the components of the gaze recognition and interaction device 10 of FIG. .

First, the processor 12 controls the camera 12 to capture a plurality of video frames (step S202). Then, the video frame rotation module 164 is executed by the processor 18 to rotate the current video frame into a rotating video frame, and the face detection module 162 is executed to detect at least the current video frame and the rotating video frame. A face (step S204). The face detection module 162, for example, performs a Viola-Jones detection method or other face detection algorithms to instantly process the video frame or the rotated video frame captured by the camera 12, and Detect faces that appear in these video frames.

In particular, in an initial scene of interaction with a person, the face may not be facing the gaze recognition and interaction device 10, which may cause the face to be sideways or in the video frame captured by the camera 12. Looking at the recognition and interaction device 10. In this regard, the present embodiment facilitates the face detection module 162 to perform a human face by, for example, rotating the current video frame to a horizontal or vertical axis in a clockwise or counterclockwise direction. By repeating the steps of rotating the video frame and detecting the face, the face of the originally tilted video frame is turned positive, so that the face detection module 162 can smoothly detect the face.

For example, FIG. 3 is a schematic diagram of a rotating video frame according to an embodiment of the invention. Referring to FIG. 3, it is assumed that the x-axis, the y-axis, and the z-axis are three directional axes of three-dimensional space, wherein the xz plane is a horizontal plane and the xy plane is a vertical plane. The rotation from the z-axis to the x-axis (clockwise rotation to the y-axis) shown in FIG. 3 represents the rotation in the horizontal direction, and the rotation from the y-axis to the x-axis (counterclockwise rotation to the z-axis) in FIG. 3 represents the vertical direction. Rotation on. By rotating the video frame clockwise or counterclockwise for different directions, and performing face detection after rotation, the face can still be detected under various face poses.

It should be noted that since the same face may be detected in both the original video frame and the rotated video frame, it actually represents the same person. In this regard, the embodiment of the present invention provides a method for eliminating the same face by using an area ratio. If the effective area ratio of the face detected in other directions (ie, the direction of rotation) is greater than a certain threshold, the face is It is regarded as the same face, and the information for saving the face is discarded, and the face is not recognized by the face. The effective area ratio can be understood as a repeating area ratio, and if the face detected in the rotating video frame and the original video frame is repeated, and the repetition rate exceeds a certain threshold, only the original video frame is detected. The face of the person performs subsequent visual recognition without visually recognizing the face detected in the rotating video frame. In this way, it is ensured that only one face recognition verification is performed on the face of a video frame to avoid duplication. It should be noted that the above-mentioned face detection target is all the faces of the original video frame and the rotating video frame, and all the faces are performed separately and only once.

In detail, after detecting the face in the rotating video frame, the face detection module 162 further maps the face in the rotating video frame back to the current video frame and performs the face corresponding to the current video frame position. The comparison determines whether the ratio of the area of the face mapped back to the current video frame to the face of the face originally in the current video frame and the original area of the face originally in the current video frame is greater than a threshold. If the ratio is greater than the threshold, it means that the face detected in the rotating video frame belongs to the same person as the face detected in the current video frame, and the face detection module 162 will abandon the saved in the rotating video. The information of the face detected in the box, and the face will not be recognized by the face to avoid repeated recognition.

Then, the processor 18 executes the visual recognition module 166, and uses the pre-trained classifier to identify whether each face detected by the face detection module 162 is facing the camera 12 to confirm whether a face is facing (steps). S206). In detail, the visual recognition module 166, for example, pre-captures a large number of facial images, and the user judges whether the faces in the respective facial images are facing the camera, thereby marking the faces on the respective facial images. label. Thereby, the visual recognition module 166 can train a neural network by using these facial images and their corresponding visual tags to obtain a classifier for recognizing the face. The above neural network includes, for example, two layers of a convolutional layer, two layers of a fully connected layer, and one layer of an output layer using a softmax function, but is not limited thereto. Those skilled in the art can use a convolutional layer, a pooling, a fully connected layer, an output layer convolutional neural network or other kinds of neural networks including different numbers and different combinations, depending on actual needs.

Finally, when the face recognition is confirmed by the recognition result of the visual recognition module 166, the processor 18 executes the steering module 168 to rotate the video frame according to the position of the face in the current video frame or the face. Mapping back to the position of the current video frame, the control servo 14 turns the camera 12 to the face recognized as the view (step S208). In detail, for the face in the rotating video frame to be recognized as the viewing condition, the visual recognition module 166 first maps the position of the face back to the current video frame as a basis for controlling the steering of the camera 12. . For example, if the rotation angle of the rotating video frame is α, the detected face position ( x ₀ , y ₀ ), the width of the original video frame is w, and the height is h, then the position of the original video frame is mapped back ( x) , y ) is:

Anticlockwise rotation: ;

clockwise rotation: .

It should be noted that, in an embodiment, the steering module 168 divides the current video frame into multiple regions, for example, according to the position of the face in the current video frame or the current video frame according to the rotated video frame. The position deviates from the distance and direction of the central area of these areas, and the steering gear 14 is controlled to turn the camera 12 to the human face so that the face can be located in the central area of the video frame captured by the rear camera 12. In another embodiment, the direction and angle at which the camera should rotate can be calculated from the pixel difference of the face from the central region by matching the steering range of the camera with the width w of the video frame. In another embodiment, the camera can be panned and rotated at the same time, so that the face can be located in the center of the video frame captured by the camera 12 after translation and/or rotation. Set limits.

For example, FIG. 4 is a schematic diagram of controlling camera steering according to an embodiment of the invention. Referring to FIG. 4, it is assumed that the video frame 40 is a video frame captured by the camera, and the human face 42 is a face that is recognized as having an opposite view. As shown in FIG. 4, the video frame 40 is divided into nine areas in which the face 42 that is recognized to be in view is located in the lower right area 40b. According to the distance and direction of the position of the face 42 (for example, the center point position of the face 42) from the central area 40a (center point position), the camera can be controlled to turn in the opposite direction (in the embodiment, for example, facing to the lower right) The direction is turned so that the face 42 is located in the central area 40a in the video frame captured by the camera after steering. By holding the face 42 in the central area 40a of the video frame 40, the position at which the camera is turned to the view can be achieved.

By means of the above-described gaze recognition and interaction method, it is possible to recognize whether or not someone is looking at the gaze recognition and interaction device 10 of the present embodiment, and turn the gaze recognition and interaction device 10 toward the gaze face, thereby simulating the real scene. The effect of the communication when talking to people.

It should be noted that in the initial scene of interaction with the person, the face may not appear in the field of view of the device camera, even if the face appears in the field of view of the camera and looks at the camera, this may be just a gaze Not specifically watching. In this regard, the present invention provides another embodiment that can solve the above problems, thereby obtaining a better recognition effect.

In detail, FIG. 5 is a flow chart of a gaze recognition and interaction method according to an embodiment of the invention. Referring to FIG. 1 and FIG. 5 simultaneously, the method of the present embodiment is applicable to the above-mentioned gaze recognition and interaction device 10, and the following is a detailed description of the detailed process of the method of the present embodiment with the components of the gaze recognition and interaction device 10 of FIG. .

First, the processor 18 receives the audio using the sound pickup device and determines the source direction of the audio to control the servo 14 to turn the camera 12 to the source direction (step S502). The sound receiving device is, for example, a microphone, a directional microphone, a microphone array, or the like, which can recognize the direction of the sound source, and is not limited herein. By turning the camera 12 to the source direction of the audio, it is ensured that the camera 12 can capture the video frame containing the face from which the audio is emitted, thereby performing subsequent view recognition.

Receiving, the processor 18 controls the camera 12 to capture a plurality of video frames (step S504). The video frame rotation module 164 is executed by the processor 18 to rotate the current video frame into a rotating video frame, and the face detection module 162 is configured to detect at least one face in the current video frame and the rotating video frame. (Step S506), and executing the visual recognition module 166, using the pre-trained classifier to identify whether each face detected by the face detection module 162 is facing the camera 12 to confirm whether a face is facing (steps) S508). The above steps S504 to S508 are the same as or similar to the steps S202 to S206 of the foregoing embodiment, and thus the details thereof are not described herein again.

In the foregoing embodiment, as long as the current video frame recognizes the face view and confirms the face view, the embodiment needs to have a plurality of consecutive video frames to recognize the face view to confirm the face view. . According to this, when the visual recognition module 166 recognizes the current video frame face-to-face in step S510, it is determined whether the number of video frames of the face-to-face view is continuously determined to be greater than a preset number (step S510).

If it is determined that the number of the video frames of the face is not greater than the preset number, the flow returns to step S504, and the processor 18 controls the camera 12 to capture the next video frame, and the face detection module 162 continues to detect the next frame. A video frame and a rotated face of the rotated video frame, and the visual recognition module 166 identifies whether the detected individual faces are facing the camera 12 to determine whether the next video frame is face-to-face. If it is determined that the face is in the opposite direction, the number of video frames in which the faces of the faces are successively determined may be accumulated, and the process proceeds to step S510 to determine.

If it is determined that the number of the video frames of the face is greater than the preset number, that is, the face is confirmed, the processor 18 executes the steering module 168 to determine the position of the face in the current video frame or the person. The face is mapped back to the position of the current video frame by the rotating video frame, and the steering gear 14 is controlled to turn the camera 12 to the face recognized as the opposite view (step S512). The foregoing steering method has been disclosed in the foregoing embodiments, and thus the details thereof are not described herein again.

By turning the camera 12 to the source direction of the audio, it is ensured that the camera 12 can capture the video frame containing the face that emits the audio, and by continuously detecting whether the plurality of video frames are face-to-face, the user can be confirmed. Whether the intention is really gazing. Thereby, a better recognition effect can be obtained.

In summary, the gaze recognition and interaction method and device of the present invention can perform instant face detection and visual recognition by the background system when the camera captures the video frame, and automatically adjust and adjust the camera steering. Thereby, whenever a gaze is detected, the camera (or the head of the robot containing the camera) will immediately turn to look at it, thereby achieving the effect of gaze on the eyes when communicating with each other in a real scene.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

10‧‧‧ Gaze recognition and interaction device

12‧‧‧ camera

14‧‧‧Steering gear

16‧‧‧Storage device

18‧‧‧ processor

40‧‧‧Video frame

40a‧‧‧Central area

40b‧‧‧Bottom right area

42‧‧‧ faces

S202~S208, S502~S512‧‧‧ steps

FIG. 1 is a block diagram of a gaze recognition and interaction apparatus according to an embodiment of the invention. 2 is a flow chart of a method for gaze recognition and interaction according to an embodiment of the invention. FIG. 3 is a schematic diagram of a rotating video frame according to an embodiment of the invention. 4 is a schematic diagram of controlling camera steering according to an embodiment of the invention. FIG. 5 is a flow chart of a gaze recognition and interaction method according to an embodiment of the invention.

Claims (12)

A gaze recognition and interaction method is applicable to an electronic device having a camera and a steering gear, wherein the steering gear is used for steering the camera, and the method comprises the following steps: capturing a plurality of video frames by using the camera (video frame Detecting at least one face of the rotating video frame generated by the current video frame of the video frame and the current video frame rotating in a direction axis; identifying each of the detected faces by using a pre-trained classifier Whether to view the camera; and if the recognition result confirms that the face is facing, according to the position of the face in the current video frame or the position of the current video frame by the rotating video frame, control The steering gear turns the camera to the face that is identified as being viewed. The gaze recognition and interaction method of claim 1, wherein the electronic device further comprises a sound pickup device, and before the step of capturing the video frame by using the camera, the method further comprises: using the sound pickup device Receiving an audio and determining a source direction of the audio to control the steering gear to turn the camera to the source direction. The gaze recognition and interaction method according to claim 1, wherein the current video frame of the video frame and the rotating video frame generated by the current video frame after rotating the direction axis are detected. After the step of describing the face, the method further includes: determining an overlapping area of the face corresponding to the current video frame position after the face in the rotating video frame is mapped back to the current video frame Whether the ratio of the face to the original area of the current video frame is greater than a threshold; and if the ratio is greater than the threshold, the information of the face in the rotated video frame is discarded. The gaze recognition and interaction method of claim 1, wherein before the step of using the pre-trained classifier to identify whether each of the detected faces faces the camera, the method further comprises: collecting a large number of a face image, and according to whether the face in each of the face images is opposite to the view mark; and using the face image and the corresponding view tag to train the neural network to obtain Identify the classifier of the view. The gaze recognition and interaction method of claim 1, wherein after the step of using the pre-trained classifier to identify whether each of the detected faces faces the camera, the method further includes: Describe the next video frame of the current video frame and the rotated face of the rotated video frame, and identify whether each of the detected faces faces the camera to determine whether the next video frame is The face is viewed; and the above steps are repeated, and when the number of the video frames of the face-to-face is continuously determined to be greater than the preset number, the face is confirmed. The gaze recognition and interaction method according to claim 1, wherein the step of controlling the steering gear to turn the camera to the face recognized as an eye view comprises: dividing the current video frame into equal parts a plurality of areas, and depending on the position of the face in the current video frame or the position and direction of the current video frame mapped back to the central area of the area by the rotating video frame, the control center The steering gear turns the camera to the face such that the face is located in the central area of the video frame captured by the camera after steering. A gaze recognition and interaction device, comprising: a camera for capturing a plurality of video frames; a steering gear for steering the camera; a storage device for storing a plurality of modules; and a processor for accessing and executing the module The module includes: a video frame rotation module, rotating a current video frame of the video frame to a rotating video frame; and a face detection module detecting the current video frame and the rotating video frame At least one face; a visual recognition module, using a pre-trained classifier to identify whether each of the detected faces is facing the camera; and a steering module, confirming the identification result in the visual recognition module When the face is in view, according to the position of the face in the current video frame or the position of the current video frame mapped by the rotating video frame, the steering gear is controlled to recognize the camera as being The face of the face. The gaze recognition and interaction device of claim 7, further comprising: a sound receiving device that receives an audio, wherein the steering module further determines a source direction of the audio to control the servo to the camera Turn to the source direction. The gaze recognition and interaction device of claim 7, wherein the face detection module further determines that the face in the rotated video frame is mapped back to the current video frame and the current video frame. Whether the ratio of the overlapping area of the face corresponding to the position of the frame to the original area of the face in the current video frame is greater than a threshold value, and if the ratio is greater than the threshold value, the saving is discarded. Rotate the information of the face in the video frame. The gaze recognition and interaction device according to claim 7, wherein the visual recognition module further collects a large number of facial images, and according to whether the faces in the respective facial images are opposite to each other, And training the neural network with the face image and the corresponding view tag to obtain the classifier for identifying the view. The gaze recognition and interaction device of claim 7, wherein: the face detection module further detects the next video frame of the current video frame and the rotated person in the rotated video frame The face recognition module further recognizes whether each of the detected faces faces the camera to determine whether the next frame is face-to-face, and continuously determines the face of the face. When the number of the video frames is greater than the preset number, it is confirmed that the face is facing. The gaze recognition and interaction device of claim 7, wherein the steering module comprises dividing the current video frame into a plurality of regions, and according to the human face in the current video frame Positioning or shifting the position of the current video frame back from the rotating video frame to a distance and direction of a central area of the area, controlling the steering gear to turn the camera to the face, so that the face Located in the central area of the video frame captured by the camera after steering.