JP6744536B1 - Eye-gaze imaging method and eye-gaze imaging system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and a system for automatically photographing a subject's line-of-sight image with a higher probability. A line-of-sight imaging system 1 includes a determination unit 3 that determines whether a subject of a captured image is looking at a camera, an image storage unit 4 that temporarily stores an image of the camera looking, and a display unit that displays the image of the camera looking. 4 and 4. The determination unit 3 includes a face detector 7 that predicts whether or not a target face or a non-target face exists as an object in the input image, and if so, predicts the position. The determination unit further includes a line-of-sight classifier 8 that determines whether or not the target face image input from the face detector 7 is the line of sight of the camera. [Selection diagram] Figure 1

Description

The present invention relates to a method and system for automatically photographing a human, a cat, an animal such as a dog, and recording an image of a line of sight of the camera.

Generally, it is not easy for a photographer who is not a photographer to take a so-called “camera line of sight” image with good timing when taking a picture of a person's face or figure with a camera. Especially when the subject is an animal such as a cat or a dog, it is even more difficult to take a line-of-sight image even if the subject is his or her pet.

For example, in the image pickup apparatus described in Patent Document 1, after the shutter button is first operated and a sound that draws the subject's attention is output, the shutter button is further operated, so that a still image of the subject is displayed at predetermined intervals. Image recognition is performed on each of the still images that have been photographed and stored in, and it is detected whether or not the line of sight of the subject is the line of sight of the camera.

Detection of the line of sight of the camera in the patent document is performed by the following three methods. In the first method, the image portion of the eye is extracted from the still image by image recognition, its edge coordinates are detected, and the iris ratio apart from the center coordinates of the iris is calculated to determine the result. The second method further extracts a face image from the still image by image recognition, detects edge coordinates of the face image, and determines a ratio of the face facing obliquely from the distance from the center coordinate of the nose, which is the center of the face. A judgment is made based on the calculation result and the calculation result of the black eye ratio. In the third method, the iris ratio data when looking at the camera from a large number of face images is stored in advance as sample data, and the comparison is made with the data detected by image recognition from the still image to make a determination.

In recent years, with the improvement of computer performance, a deep learning device called a convolutional neural network (CNN) has been widely adopted in image recognition for identifying an object reflected in an image (for example, non-convolutional neural network (CNN)). See Patent Document 1). Generally, a CNN is composed of a plurality of convolutional layers and an output layer, performs extraction of a feature amount from an input image a plurality of times, and recognizes an object at the last output layer.

In addition, YOLO (You Only Look Once) is known as an object detection algorithm using CNN (for example, see Non-Patent Document 2). YOLO simultaneously classifies an object by dividing an image into a plurality of grids (for example, 7×7) and detects the position of the object to obtain the position coordinates of a rectangular area (bounding box) surrounding the object in parallel. By doing, and finally combining both results, the object is recognized.

JP, 2007-96440, A

Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton, "Image Net Classification with Deep Convolutional Neural Networks", Advances In Neural Information Processing Systems, Vol.25, pp.1106-1114, 2012. Joseph Redmon, Ali Farhadi, "YOLOv3: An Incremental Improvement", Cornell University, Technical Report, 2018.

However, in order to capture a still image for image recognition, the image pickup apparatus described in Patent Document 1 needs to call the attention of the subject person by the sound output by the first operation of the shutter button. Therefore, when the subject is an infant, a baby, or an animal such as a cat or a dog, it is not always possible to expect that the captured still image includes the camera line-of-sight image by just calling out the sound.

Further, in the third method described in Patent Document 1, it is necessary to store a large amount of sample data indicating a camera line-of-sight image in the ROM in advance in order to compare it with a captured still image. The larger the amount of data, the lower the possibility of erroneous detection, but the processing time becomes longer, or a high-performance processor or large-capacity memory is required for high-speed processing. If the amount of sampled data is limited, the detection process may be terminated immediately when a still image is captured, but the possibility of erroneous detection increases.

Therefore, the present invention has been made in view of the problems of the above-described conventional technology, and the object thereof is a person including an infant or a baby, and a cat looking image of an animal such as a cat or a dog with a higher probability. It is to provide a method and an apparatus capable of automatically photographing.

Further, the object of the present invention is sufficient for use not only in personal computers and high-performance electronic devices with relatively high degree of freedom of resources, but also in mobile electronic devices such as smartphones and tablets where resources are often limited. It is an object of the present invention to provide a method and system for photographing and recording an image of a subject's line of sight, which can operate at a very high processing speed.

The eye-gaze imaging system of the present invention is
A shooting unit that outputs the shot image,
A determination unit for determining whether or not the target of the image input from the image capturing unit is looking at the camera,
A storage unit that stores the image determined to be the camera's line of sight by the determination unit,
The determining unit detects a target face and a face other than the target in the input image, and a face detector that predicts the position thereof,
Wherein said target face image input from the face detector possess and determining eyes classifier whether the Waist,
The face detector includes a learning device that learns the target face by using image data obtained by capturing the target face as learning data.
The learner further implicitly learns an object other than the target face and the face other than the target as an object that is not the target face and the face other than the target, and explicitly identifies the face other than the target. Learn to
The face detector selects one image showing the target face from the learning data in order to learn the target face by using the image data obtained by photographing the target face as learning data. Performing a process of detecting the face of the target, an additional process of updating the parameter based on the detection error calculated from the detection result and the correct data, for the implicit learning, the learning device of the target When learning a face, it is performed with a predetermined probability set in advance .

In some actual forms, the face detector creates a mask image of the object relative to the target image before Symbol learning data, by using the mask image of the object, of the mask image original of the object By extracting only the part of the target from the image, generating a composite image placed on the background image in which an object other than the target is captured, and using the composite image as additional learning data, the learning device is implicitly to make learning.

In one embodiment, the eye gaze classifier sets a target value of 1 for a complete camera eye and 0 for a complete camera eye with respect to the target face image determined by the face detector. Based on, an evaluation value of 1 to 0 is given according to the degree of the line of sight of the camera, and when the evaluation value exceeds a preset threshold value, it is determined to be the line of sight of the camera.

In another embodiment, an image storage unit that temporarily stores an image that is determined by the determination unit to be the face image of the target from the viewpoint of the camera, and a display unit that displays a list of images stored in the image storage unit. And further.

In one embodiment, the target face is a cat face and the non-target face is a human face .

The eye-gaze imaging method of the present invention detects an image, a step of detecting a target face or a face other than the target from the captured image, and predicting a detection area thereof, and detecting the target face . From the image and the image in which a face other than the target has been detected, a step of extracting the face image of the target, a step of detecting the camera line of sight from the extracted face image of the target, and the target in which the camera line of sight is detected look including the step of storing the face image in order to detect the captured face other than the target of the face or the target from the image, wherein the subject's face image data obtained by photographing the face of the target as learning data Learning, further implicitly learning an object other than the target face and the face other than the target as an object that is not the target face and the face other than the target, an image in which the target face is detected And the step of explicitly learning the face other than the target in order to extract the face image of the target from the image in which the face other than the target is detected, the image data obtained by photographing the face of the target is learning data. In order to learn the target face, a process of selecting one image showing the target face from the learning data and detecting the target face is performed, and the detection result and correct answer data are obtained. It is characterized in that an additional process of updating the parameter based on the detection error calculated from is performed with a preset probability when the target face is learned in the implicit learning step .

In one embodiment, the step of implicitly learning creates a mask image of the target for an image of the face of the target, which is learning data for learning the face of the target, and generates the mask image of the target. By using the original image of the target of the mask image, only the target portion is extracted to generate a composite image placed on a background image showing an object other than the target, and the composite image is added. This is done by using it as learning data.

In one embodiment, the target face is a cat face and the non-target face is a human face .

According to the eye-gaze capturing system and method of the present invention, even in a mobile electronic device with limited resources, an image of the target animal such as a cat or a person looking at the camera can be captured at a processing speed that is sufficiently high for use. Can be recorded.

It is a block diagram which shows the structure of the eye gaze imaging system of this invention. It is explanatory drawing of the image used as learning data in a face detector. It is explanatory drawing of the image used as learning data in a face detector. (A) is an image of a cat used as learning data for the face detector, and (b) is its mask image. (A) is a background image used as learning data for the face detector, and (b) is a composite image in which a cat's face is superimposed on it. (A) is a face image of a cat looking at the camera (target value 1), and (b) is a face image of a cat not looking at the camera (target value 0). It is a block diagram showing the whole composition of an eye-gaze photographing device to which an eye-gaze photographing system is applied. It is a flowchart which shows the process of the eye-gaze photographing method by this invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this embodiment, a line-of-sight image capturing method, device, and system for recognizing a cat as a subject and automatically capturing and storing a camera line-of-sight image thereof will be described.

FIG. 1 schematically shows the configuration of a line-of-sight imaging system 1 of this embodiment. The line-of-sight imaging system 1 includes an input unit 2 for inputting an image of a subject, a determination unit 3 for identifying and identifying whether the subject of the input image is looking at the camera, an image storage unit 4 for temporarily storing the image of the camera looking, and And a display unit 5 for displaying an image of the line of sight of the camera.

In the present embodiment, a subject image is captured as a moving image and input to the input unit 2. The input unit 2 acquires frame images from the input moving image at preset constant intervals and sequentially sends the frame images to the determination unit 3. In another embodiment, the image of the subject can be captured as a still image. In that case, the input unit 2 sequentially sends the input still images one by one to the determination unit 3.

The determination unit 3 includes a face detector 7 and a line-of-sight classifier 8. The face detector 7 predicts whether or not a face of a cat or a person exists as an object in the frame image input from the input unit 2, and further, if it exists, the position thereof. If it is predicted that the cat's face does not exist, the frame image is deleted. In the present embodiment, the face detector 7 uses YOLOv3[1] which is a developed version of the object detection algorithm YOLO described above.

Specifically, YOLOv3[1] divides the input image I into a g×g grid G, as illustrated in FIG. 2, and sets a plurality of the following seven values for each cell of the grid G. Predict.
・X: X coordinate value of the center point of the object ・Y: Y coordinate value of the center point of the object ・W: Width of the object ・H: Height of the object ・O: Probability that the center of the object is within the cell ・C1 : Probability that the object is a cat face C2: Probability that the object is a human face

The face detector 7 generates a plurality of sets of the above seven values (X, Y, W, H, O, C1, C2). From the generated plurality of sets, only the prediction in which the probability O×C1 that a cat's face is present exceeds a preset threshold is adopted. Delete the remaining predictions.

Further, in prediction, it is not preferable that a plurality of regions detected for one target overlap in one image. In this embodiment, Non-Maximum Suppression (Paper: Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation", University of California, Berkeley, and the International Computer Science Institute , CVPR 2014), and remove the overlapping area from the detected area. As a result, as shown in FIG. 3, only one region B surrounding the face of one target is left in the input image I.

The face detector 7 uses the learning device 9 in order to reduce the probability of erroneously recognizing an object other than the face of a cat or a person (for example, a wall or a sofa) detected in the input image as a face of a cat or a person. Have. The learning device 9 implicitly learns objects other than cats and human faces. Here, “implicitly” means that the object other than the detected cat and human face is not specifically known, but for the time being, it is learned that it is not the detection target (cat face or human face). Is.

Of course, it is technically possible to explicitly detect every object that can be encountered when shooting a cat and ignore it to reduce false positives. However, in general, the detection capability of the object detector has an upper limit, and there is a problem that the detection accuracy for each object decreases as the number of detection targets increases.

By increasing the parameters of the object detector, it is possible to suppress the deterioration of detection accuracy. However, if the parameter of the object detector is increased, the processing time is increased accordingly, and it becomes difficult to detect in real time from the image being captured. In particular, in the case of mobile electronic devices whose resources are relatively limited, such as smartphones, GPU support cannot always be obtained, and real-time detection processing is more difficult.

The implicit learning of objects other than cats and human faces in the learning device 9 of the present embodiment will be specifically described below. The face detector 7 previously learns the texture pattern of the cat's face by using a large number of image data of the image of the cat's face as learning data. For implicit learning by the learning device 9, image segmentation is performed in advance on the cat image data used for the learning data, and a large number of cat mask images are created. FIG. 4A shows an original cat image I1 used for the learning data, and FIG. 4B shows a cat mask image I2 represented by a black and white binary image.

Further, for the implicit learning, the following additional processing is performed when the learning device 9 learns the face of a cat. In the learning of the face of the cat in the learning device 9, one image showing the cat is selected from the learning data, the face of the cat is detected, and the detection result and the correct answer data (position information of the face of the cat) are selected. Then, a series of learning processing is performed in which the detection error is calculated from and the parameters are updated based on the detection error.

The additional processing is performed with a predetermined probability set in advance when one image including a cat is selected. In the present embodiment, it is selected that the probability is 50%, but the accuracy of the additional processing required, the use of the hardware for executing the eye-gaze photographing system of the present invention, and the like can be changed according to various conditions. Can be set.

Prior to the addition processing, as illustrated in FIG. 5A, a plurality of background images BI including objects other than cats and people are prepared. Next, using the mask image of the cat, only the cat portion is extracted from the original cat image. From the image of only the part of the cat, only the face part of the cat is extracted using the position information of the face of the cat in the image of the original cat which is the correct answer data. One background image is randomly selected from the plurality of background images prepared previously, and the face image of the cat extracted previously is placed on the background image to generate a plurality of composite images CI illustrated in FIG. 5B. To do.

According to the present embodiment, the learning device 9 is trained by using the plurality of composite images thus obtained as additional learning data. As a result, various objects other than the faces of cats and humans appearing in the background image can be learned implicitly, and false detection of the learning device 9 can be reduced.

Furthermore, the learning device 9 explicitly learned the texture pattern of the human face. This reduces erroneous detection of objects other than cats and human faces, and eliminates erroneous detection of human faces as cat faces.

The line-of-sight classifier 8 is a two-class image classifier using CNN. The line-of-sight classifier 8 is configured to determine whether the face image of the cat input from the face detector 7 is a face image of a cat looking at the camera or a face image of a cat not looking at the camera. In the present embodiment, a target value of 1 is assigned to a face image of a cat that is evaluated to be a perfect camera line of sight, and a target value of 0 is assigned to a face image of a cat that is not evaluated to be a camera line of sight. For example, it is evaluated that FIG. 6A is a complete camera line of sight, and FIG. 6B is not a camera line of sight at all. Target values between 1 and 0 are given to the face images of cats other than these depending on the degree of the line of sight of the camera.

Evaluation of the camera line of sight and determination/assignment of the target value for the face image of the cat are performed by a learning executor who trains the line of sight classifier 8. In this way, the eye-line classifier 8 is trained using the face images of a large number of cats to which the target values are assigned as learning data. Based on this learning result, the line-of-sight classifier 8 evaluates the degree of the line of sight of the camera with respect to the face image of the cat input from the face detector 7, and assigns the same evaluation value between 1 and 0 as the target value. Output. In order to determine that the face image of the cat is looking at the camera, a predetermined threshold value is set in advance as the evaluation value. When the evaluation value output from the eye-line classifier 8 exceeds the threshold value, the determination unit 3 determines that the face image of the cat is the line of sight of the camera.

The image storage unit 4 sequentially and temporarily stores the face images of the cat looking at the camera output from the determination unit 3. When the user inputs the end of the process of automatically capturing and storing the cat's camera line-of-sight image by the above-described line-of-sight image capturing system 1, the cat's face image of the line of sight of the camera temporarily stored in the image storage unit 4 is displayed in the list. The image data is sent to the display unit 4.

The determination unit 3 also outputs the determined face image of the cat looking at the camera to the image storage unit 4 together with the score given according to the degree of the camera looking. In the present embodiment, as the score, the evaluation value given by the eye-line classifier 8 based on the target value of the image data used as learning data in the machine learning of the camera eye can be used as it is. It is also possible to arrange the evaluation values assigned by the eye-line classifier 8 in descending order and use that order as the score of the image.

The display unit 4 displays a list of face images of cats looking from the camera, which are output from the image storage unit 4. In addition, the display unit 4 can rearrange and display the list of the face images of the cat looking at the camera in the descending order of the score assigned to each image. The user can select a face image of the cat looking at the camera, which he or she likes, from the list. The face image of the cat selected by the user can be output from the image storage unit 4 to a storage unit (not shown) of the line-of-sight imaging system 1 or an external storage unit and stored.

The line-of-sight imaging system 1 can be applied to mobile electronic devices such as smartphones and personal computers. FIG. 7 shows the configuration of the eye-gaze photographing device 11 of the present embodiment, which is a smartphone to which the eye-gaze photographing system 1 is applied.

The line-of-sight image capturing device 11 includes an image capturing device 12 including a camera that captures a moving image and/or a still image of a subject, a storage device 13 that stores captured image data, and a GPU (Graphics Processing Unit) that processes the image data. The calculation processing unit 14 includes a control unit 15 including a CPU (Central Processing Unit) that controls the operations of the imaging device, the storage device, and the calculation processing unit, an input device 16, and a display device 17. The eye-gaze photographing program for executing the eye-gaze photographing method of the present embodiment in the eye-gaze photographing device 11 is stored in the ROM (not shown) or the storage device 13 included in the arithmetic processing unit 14.

The input device 16 is, for example, a keyboard, a mouse, a touch panel, a ten-key pad, and the like, which is originally included in a smartphone or the like, and is operated by a user. The display device 17 includes a display device such as a liquid crystal panel that is originally included in a smartphone or the like. Further, the line-of-sight photographing device 11 can include a communication device (not shown) for bidirectionally communicating image data and other data with the outside in a wireless or wired manner. The eye-gaze photographing device 11 can store image data in an external storage device via the communication device.

FIG. 8 is a flow chart showing a processing process by the eye-gaze photographing method of the present embodiment, which is executed using the eye-gaze photographing device 11 of FIG. When the user has activated the eye-gaze photographing program via the input device 16, the control unit 15 reads the eye-gaze photographing program from the ROM or the storage device 13 and executes the program. First, in step S1, the control unit 15 controls the imaging device 12 to start capturing and recording a moving image of a subject (cat) by the imaging device.

Next, the control unit 14 acquires frames from image data (moving images) that are continuously output from the image pickup apparatus 12 at preset constant intervals, and determines the frames from the determination unit 4 including the arithmetic processing unit 14. To the face detector 7 (step S2). At the same time, the control unit 14 causes the storage device 13 to continuously output and store the acquired frames. The moving image output from the imaging device 12 can be stored in the storage device 13 as it is.

The face detector 7 causes the arithmetic processing unit 14 to perform the above-described process of detecting the face of the cat on the input frame. As a result, when the face of the cat is not detected in the frame (No in step S3), the frame is discarded and the next frame is waited for. When the face detector 7 detects a cat face in the frame (Yes in step S3), the detection area (B in FIG. 4) is extracted from the frame and sent to the eye-line classifier 8.

The line-of-sight classifier 8 causes the arithmetic processing unit 14 to perform the above-described process of detecting the line of sight of the camera from the face of the cat in the detection area of the face of the cat sent from the line-of-sight classifier 8. When the cat's camera line of sight is not recognized in the detection area (No in step S4, the frame is discarded and the next frame is waited). When the line-of-sight classifier 8 recognizes the cat's camera line-of-sight in the detection area (Yes in step S4), the control unit 15 sets the frame as the cat's camera line-of-sight image and the image storage unit 5 including the storage device 13. (Step S5).

The line-of-sight photographing program provides an end button for ending photographing and recording by the image pickup device 12. After step S5, the control unit 15 confirms whether or not the user has operated the input device 16 and pressed the end button (step S6). If the end button has not been pressed, the process returns to step S and the series of processes S2 to S6 described above is repeatedly executed.

When the end button is pressed in step S6, the control unit 15 causes the display device 17 included in the display unit 6 to display a list of the camera line-of-sight images of the cat temporarily stored in the image storage unit 5 (step S6). S7). When the user selects a desired image from the list of images displayed on the display device 17, the control unit 15 stores the selected camera line-of-sight image of the cat in the storage device 13. In another embodiment, the user's line-of-sight image of the cat selected by the user may be transmitted via the communication device of the line-of-sight photographing device 11 and stored in an external storage device.

Although the present invention has been described above with reference to the preferred embodiments, the present invention is not limited to the above embodiments, and various modifications or variations can be made within the technical scope thereof. .. For example, the above-described embodiment has been described with respect to a cat as a case of shooting and recording a cat's line-of-sight image of the cat. Can be similarly applied.

1 Eye-gaze imaging system 2 Input section 3 Judgment section 4 Image storage section 5 Display section 7 Face detector 8 Eye-gaze classifier 9 Learner

Claims (8)

A shooting unit that outputs the shot image,
A determination unit for determining whether or not the target of the image input from the image capturing unit is looking at the camera,
A storage unit that stores the image determined to be the camera's line of sight by the determination unit,
The determining unit detects a target face and a face other than the target in the input image, and a face detector that predicts the position thereof,
And a line-of-sight classifier that determines whether or not the target face image input from the face detector is a camera line of sight,
The face detector includes a learning device that learns the target face by using image data obtained by capturing the target face as learning data.
The learner further implicitly learns an object other than the target face and the face other than the target as an object that is not the target face and the face other than the target, and explicitly identifies the face other than the target. Learn to
The face detector selects one image showing the target face from the learning data in order to learn the target face by using the image data obtained by photographing the target face as learning data. Performing a process of detecting the face of the target, an additional process of updating the parameter based on the detection error calculated from the detection result and the correct data, for the implicit learning, the learning device of the target When learning a face, it is performed with a preset probability.
Eye-gaze photography system. The face detector creates a mask image of the target for the target image of the learning data, and using the mask image of the target, removes only the target portion from the original target image of the mask image. The method according to claim 1, wherein the learner is implicitly learned by generating a composite image that is extracted and placed on a background image including an object other than the target, and using the composite image as additional learning data. The line-of-sight imaging system described. The line-of-sight classifier is based on a target value that is 1 for a complete camera line of sight and 0 for no face of the camera for the target face image determined by the face detector, based on a target value. The eye-gaze photographing according to claim 1 or 2 , wherein an evaluation value of 1 to 0 is given according to the degree of the eye-gaze, and when the evaluation value exceeds a preset threshold value, it is determined to be a camera eye-gaze. system. An image storage unit that temporarily stores an image that is determined by the determination unit to be the face image of the target looking at the camera, and a display unit that displays a list of images stored in the image storage unit. The eye-gaze photographing system according to any one of claims 1 to 3 . The eye gaze photographing system according to any one of claims 1 to 4 , wherein the target face is a cat face and the non-target face is a human face. A step of taking an image,
Detecting a target face or a face other than the target from the captured image, and predicting the detection area,
Extracting the face image of the target from the image in which the face of the target is detected and the image in which a face other than the target is detected,
Detecting the camera line of sight from the extracted target face image,
Saving a face image of the object from which a camera line of sight is detected,
In order to detect a target face or a face other than the target from the captured image, the target face is learned using image data of the target face captured as learning data, and the target face and the target are further learned. An object other than a face other than the step of implicitly learning as an object that is not the target face and the face other than the target,
To retrieve the target face image from an image a face has been detected other than the subject's face to detect the image and the object, further saw including a step of explicitly learn the face of other than the target,
In order to learn the target face by using the image data of the image of the target face as learning data, one image showing the target face is selected from the learning data to detect the target face. The additional processing of updating the parameter based on the detection error calculated from the detection result and the correct answer data is performed, and when learning the target face in the step of implicitly learning, a predetermined preset value is set. With the probability of
How to take a look. The step of implicitly learning creates a mask image of the target for an image of the face of the target, which is learning data for learning the face of the target, and uses the mask image of the target to generate the mask image. By extracting only the target portion from the original target image of, to generate a composite image placed on a background image in which an object other than the target is captured, and using the composite image as additional learning data. The eye-gaze photographing method according to claim 6, which is performed. The eye-gaze photographing method according to claim 6 or 7 , wherein the target face is a cat face, and the non-target face is a human face.