JP2021077333A - Line-of-sight detection method, line-of-sight detection device, and control program - Google Patents

Abstract

To further improve line-of-sight detection accuracy.SOLUTION: Image data of the face of a person is acquired. A left-eye region including the left eye of the person and a right-eye region including the right eye of the person are detected from the facial image data. When only one eye region, that is, the right-eye region or the left-eye region, is detected, an ear region including an ear of the person and/or a hair region including the hair of the person is detected from the facial image data, information indicating the orientation of the face is detected on the basis of information indicating the position of the one eye included in the one eye region and information indicating at least one of the position of the ear included in the ear region and the position of the hair included in the hair region, and information indicating the line of sight of the person is detected on the basis of the information indicating the position of the one eye and the information indicating the orientation of the face.SELECTED DRAWING: Figure 2

Description

The present disclosure relates to a technique for detecting a person's line of sight.

The line-of-sight detection technique is used in various applications such as estimation of a person's interest target, estimation of a person's state such as drowsiness, and a user interface for inputting to a device by the line of sight. The line-of-sight detection technology is roughly divided into a head-mounted type and a non-contact type. The head-mounted line-of-sight detection technique is a technique for detecting a line-of-sight by attaching a device such as a head-mounted display to a person's head. The non-contact type line-of-sight detection technology is a technology for detecting the line-of-sight without attaching a device to a person. Unlike the head-mounted line-of-sight detection technology, the non-contact type line-of-sight detection technology does not require a device to be attached to the head, and has the advantages of not bothering the person and not obstructing the person's field of vision. .. Therefore, this disclosure focuses on a non-contact line-of-sight detection technique.

Here, the non-contact type eye-gaze detection technique is roughly classified into two methods, a pupillary corneal reflex method and a three-dimensional eyeball model method. The pupillary corneal reflex method is, for example, as described in Patent Document 1, in which light is irradiated to the human eye by a light emitting element, the position of the reflected image projected on the eye, the irradiation angle of the light, and the position of the reflected image. This is a method of detecting the line of sight based on the amount of movement of the light.

The method based on the three-dimensional eyeball model is a method of estimating the line-of-sight direction based on the information about the eyeball obtained by analyzing the facial image. For example, Patent Document 2 discloses a method of estimating the line-of-sight direction from the rotation angle of the eyeball and the direction of the face obtained by analyzing the face image. In addition, the three-dimensional vector connecting the center position of the eyeball and the center position of the iris obtained by analyzing the face image is estimated as the line-of-sight direction, and the normal line of the center position of the pupil with respect to the tangent plane is estimated as the line-of-sight direction. The method is also known.

Japanese Unexamined Patent Publication No. 61-172552 Japanese Unexamined Patent Publication No. 2003-271932

However, in the non-contact type line-of-sight detection technology, when a face image showing one side of the face is used, in other words, when only one eye can be detected from the face image, it is difficult to accurately detect the line-of-sight direction. It becomes. Therefore, for example, there is a problem that it becomes difficult to determine the degree of gaze and the presence or absence of aside from an object installed on the side of the pedestrian's traveling direction based on the pedestrian's line-of-sight information. In the non-contact type line-of-sight detection technique disclosed in Patent Documents 1 and 2, the method of detecting the line-of-sight when only one eye can be detected is not specifically considered.

The present disclosure has been made to solve such a problem, and an object of the present disclosure is to further improve the detection accuracy of the line of sight when only one eye can be detected from the face image.

The first aspect of the present disclosure is a line-of-sight detection method in a line-of-sight detection device, in which image data of a person's face is acquired, and the left eye region including the left eye of the person and the right eye of the person are obtained from the image data of the face. When the right eye region including the right eye region is detected and only one eye region of the right eye region or the left eye region is detected, at least the ear region including the human ear and the hair region including the human hair are detected from the image data of the face. One is detected, and information indicating the position of one eye included in the one eye region and information indicating the position of at least one of the position of the ear included in the ear region and the position of the hair included in the hair region. Based on, the information indicating the direction of the face is detected, and the information indicating the line of sight of the person is detected based on the information indicating the position of one of the eyes and the information indicating the direction of the face.

A second aspect of the present disclosure is a line-of-sight detection method in a line-of-sight detection device, in which image data of a person's face is acquired, and the left eye region including the left eye of the person and the right eye of the person are obtained from the image data of the face. When the right eye region including the right eye region is detected and the right eye region and the left eye region are detected, the information indicating the orientation of the face is detected from the image data of the face, and the information indicating the position of the right eye included in the right eye region and the said Information indicating the line of sight of the person is detected by the first process using the information indicating the position of the left eye and the information indicating the orientation of the face included in the left eye region, and one eye of the right eye region or the left eye region is detected. When only a region is detected, information indicating the orientation of the face is detected based on the information indicating the position of one eye included in the one eye region and the information indicating the position of a portion different from the eyes on the face. Then, the information indicating the line of sight of the person is detected by the second process using the information indicating the position of the one eye and the information indicating the direction of the face.

According to the present disclosure, it is possible to further improve the detection accuracy of the line of sight when only one eye can be detected from the face image.

It is a figure which shows an example of the whole structure of the image processing system which concerns on Embodiment 1 of this disclosure. It is a block diagram which shows an example of the detailed structure of the image processing system which concerns on Embodiment 1. FIG. It is a figure which shows an example of a right eye region and a left eye region. It is a block diagram which shows an example of the detailed structure of the one-sided visual line detection part. It is a figure which shows an example of an ear region. It is a figure which shows an example of a face area. It is a figure which shows an example of the binarized face area. It is a figure which shows another example of the binarized face area. It is a schematic diagram which shows an example of the degree of face orientation of a person seen from above. It is a figure which shows the viewing angle of a person. It is explanatory drawing of the detection method of the line-of-sight direction. It is a flowchart which shows an example of the operation of the image processing apparatus which concerns on Embodiment 1. FIG. It is a flowchart which shows an example of the operation of the one-sided visual line detection part which concerns on Embodiment 1. FIG. It is a block diagram which shows an example of the detailed structure of the image processing system which concerns on Embodiment 2. FIG.

(Findings underlying this disclosure)
In the conventional method of detecting the line of sight by the three-dimensional eyeball model, information on the eyeballs of both eyes is detected by analyzing the face image, and the line of sight direction is estimated based on the detected information. However, when the person to be detected is looking aside, a face image containing only one eye may be used for line-of-sight detection. In this case, there was no choice but to presume that one eye that can be detected from the face image and one eye that cannot be detected from the face image both face substantially the same direction.

As described above, in the conventional method of detecting the line of sight using the three-dimensional eyeball model, it is difficult to accurately detect the direction of the line of sight when only a face image showing one side of the face can be detected. If the line-of-sight direction cannot be detected accurately, for example, it is difficult to determine the degree of gaze and the presence or absence of aside from an object installed on the side of the pedestrian's traveling direction based on the pedestrian's line-of-sight information. become.

As a result of conducting a detailed study on such a problem, the present inventor can improve the accuracy of line-of-sight detection by using information on a part different from the eyes when only one eye can be detected from the face image. Based on the findings, we came up with the following aspects.

The line-of-sight detection method according to the first aspect of the present disclosure is a line-of-sight detection method in a line-of-sight detection device, in which image data of a human face is acquired, and the left eye region including the left eye of the person and the left eye region including the left eye of the person are obtained from the image data of the face. When the right eye region including the right eye of the person is detected and only one eye region of the right eye region or the left eye region is detected, the ear region including the human ear and the hair of the person are obtained from the image data of the face. At least one of the information indicating the position of one eye included in the one eye region, the position of the ear included in the ear region, and the position of the hair included in the hair region by detecting at least one of the including hair regions. Information indicating the orientation of the face is detected based on the information indicating the position, and information indicating the line of sight of the person is obtained based on the information indicating the position of the one eye and the information indicating the orientation of the face. To detect.

According to this configuration, when only one eye region of the right eye region or the left eye region is detected from the face image data, at least one of the ear region and the hair region is detected from the face image data. Then, based on the information indicating the position of one eye included in the one eye region and the information indicating the position of the ear or hair included in at least one of the detected ear region and hair region, the face Information indicating the orientation is detected. Then, based on the detected information indicating the orientation of the face and the information indicating the position of one of the eyes, the information indicating the line of sight of a person is detected.

As a result, in this configuration, even if only the image data of the face including only the right eye or the left eye can be acquired, the orientation of the face is based on the detected positional relationship between one eye and the ear or hair instead of the positional relationship between both eyes. Can be detected accurately. As a result, this configuration can accurately identify the orientation of the face even when only one eye can be detected from the face image, and can improve the detection accuracy of the line of sight.

In the above aspect, when only the one eye region is detected, the ear region is detected from the image data of the face, and when the ear region can be detected, the information indicating the position of the one eye and the ear The information indicating the orientation of the face may be detected based on the information indicating the position.

According to this configuration, when only one eye region of the right eye region or the left eye region is detected from the face image data, the ear region is detected from the face image data, and one eye included in the one eye region is detected. Information indicating the orientation of the face is detected based on the information indicating the position and the information indicating the position of the ear included in the detected ear region. As a result, even if only one eye can be detected from the face image, the present configuration can accurately detect the orientation of the face based on the positional relationship between one eye and the ear instead of the positional relationship between both eyes. it can.

In the above aspect, when the ear region cannot be detected, the hair region is detected from the image data of the face, and the hair region is detected based on the information indicating the position of the one eye and the information indicating the position of the hair. Information indicating the orientation of the face may be detected.

According to this configuration, when only one eye region of the right eye region or the left eye region is detected from the face image data, the hair region is detected from the face image data, and one eye included in the one eye region is detected. Information indicating the orientation of the face is detected based on the information indicating the position and the information indicating the position of the hair included in the detected hair area. As a result, even if only one eye can be detected from the face image, the present configuration can accurately detect the orientation of the face based on the positional relationship between one eye and the hair instead of the positional relationship between both eyes. it can.

In the above aspect, based on the image data of the face and the information indicating the position of the one eye, on the contour line of the face existing on the inner side of the eye of the one eye with respect to the position of the outer corner of the eye of the one eye. A contour point indicating a point is detected, and the face is based on the first distance from the position of the one eye to the contour point and the second distance from the position of the one eye to the position of the ear. Information indicating the direction of is may be detected.

According to this configuration, the first distance from the position of the one eye to the contour point indicating a point on the contour line of the face existing on the inner side of the eye of the one eye rather than the position of the outer corner of the eye of the one eye. , Information indicating the orientation of the face is detected based on the second distance from the position of one eye to the position of the ear. As a result, in this configuration, the width of the face on the inner side of the eye from the detected one eye, which is indicated by the first distance, and the width of the face on the ear side, that is, the outer side of the eye, which is indicated by the second distance. Based on the above, the orientation of the face can be detected accurately.

In the above aspect, based on the image data of the face and the information indicating the position of the one eye, on the contour line of the face existing on the inner side of the eye of the one eye with respect to the position of the outer corner of the eye of the one eye. The contour point indicating a point is detected, and the first distance from the position of the one eye to the contour point and the position of the one eye are located closer to the outer corner of the eye than the position of the inner corner of the one eye. Information indicating the orientation of the face may be detected based on the third distance to the point in the hair area.

According to this configuration, the first distance from the position of the one eye to the contour point indicating a point on the contour line of the face existing on the inner side of the eye of the one eye rather than the position of the outer corner of the eye of the one eye. , Information indicating the orientation of the face is detected based on the third distance from the position of the one eye to the point in the hair region located on the outer corner side of the eye with respect to the position of the inner corner of the eye. .. Thereby, this configuration is based on the width of the face on the inner side of the eye than the detected one eye indicated by the first distance and the width of the face on the outer corner of the eye than the detected one eye indicated by the third distance. The orientation of the face can be detected accurately.

In the above aspect, when the right eye region and the left eye region are detected, information indicating the orientation of the face is detected from the image data of the face, and the position of the right eye included in the right eye region and the left eye included in the left eye region are detected. Information indicating the line of sight of the person may be detected based on the information indicating the position of the person and the information indicating the direction of the face.

According to this configuration, when the right eye region and the left eye region are detected from the face image data, the information indicating the direction of the face is detected from the face image data. Then, based on the detected information indicating the orientation of the face and the information indicating the positions of both eyes, the information indicating the line of sight of the person is detected. Thereby, in the present configuration, when both eyes can be detected from the face image, the line of sight can be detected from the positions of both eyes and the direction of the face obtained from the image data of the face.

The line-of-sight detection method according to the second aspect of the present disclosure is a line-of-sight detection method in a line-of-sight detection device, which acquires image data of a person's face, and from the image data of the face, a left eye region including the left eye of the person and When the right eye region including the right eye of the person is detected and the right eye region and the left eye region are detected, information indicating the orientation of the face is detected from the image data of the face, and the position of the right eye included in the right eye region is detected. The information indicating the line of sight of the person is detected by the first processing using the information indicating the left eye, the information indicating the position of the left eye included in the left eye region, and the information indicating the orientation of the face, and the right eye region or the left eye. When only one eye region of the region is detected, the orientation of the face is based on the information indicating the position of one eye included in the one eye region and the information indicating the position of a portion of the face different from the eyes. The information indicating the line of sight of the person is detected by a second process using the information indicating the position of one eye and the information indicating the direction of the face.

According to this configuration, when the right eye region and the left eye region are detected from the face image data, the information indicating the direction of the face is detected from the face image data. Then, the information indicating the line of sight of a person is detected by the first process using the detected information indicating the orientation of the face and the information indicating the positions of both eyes. As a result, when both eyes can be detected from the face image, the present configuration can detect the line of sight from the positions of both eyes and the orientation of the face by performing the first process.

On the other hand, when only one eye region of the right eye region or the left eye region is detected from the image data of the face, the information indicating the position of one eye included in the one eye region and the position of the portion different from the eyes on the face Based on the information indicating the direction of the face, the information indicating the orientation of the face is detected. Then, the information indicating the line of sight of a person is detected by the second process using the detected information indicating the orientation of the face and the information indicating the position of one of the eyes. As a result, when only one eye can be detected from the face image, the present configuration detects the orientation of the face based on the positional relationship between the part different from the eye and the detected one eye, and performs the second process. Therefore, the line of sight can be detected from the detected position of one eye and the detected orientation of the face.

As described above, in this configuration, the process can be switched according to the number of eyes detected from the face image, and the line of sight can be appropriately detected.

In the above embodiment, the different part may be an ear or hair.

According to this configuration, when only one eye region of the right eye region or the left eye region is detected from the image data of the face, the information indicating the position of one eye included in the one eye region and the position of the ear or hair Information indicating the orientation of the face is detected based on the information indicating the orientation of the face. As a result, the present configuration can accurately detect the orientation of the face based on the positional relationship between the detected one eye and the ear or hair even when only one eye can be detected from the face image.

In the above aspect, when only the one eye region is detected, the ear region including the human ear is detected from the image data of the face, and when the ear region can be detected, the position of the one eye is indicated. The information indicating the orientation of the face may be detected based on the information and the information indicating the position of the ear included in the ear region.

According to this configuration, when only one eye region of the right eye region or the left eye region is detected from the face image data, the ear region is detected from the face image data, and one eye included in the one eye region is detected. Information indicating the orientation of the face is detected based on the information indicating the position and the information indicating the position of the ear included in the detected ear region. As a result, even if only one eye can be detected from the face image, the present configuration can accurately detect the orientation of the face based on the positional relationship between one eye and the ear instead of the positional relationship between both eyes. it can.

In the above aspect, when the ear region cannot be detected, the hair region including the human hair is detected from the image data of the face, and the information indicating the position of the one eye and the hair included in the hair region are detected. The information indicating the orientation of the face may be detected based on the information indicating the position.

According to this configuration, when only one eye region of the right eye region or the left eye region is detected from the face image data, the hair region is detected from the face image data, and one eye included in the one eye region is detected. Information indicating the orientation of the face is detected based on the information indicating the position and the information indicating the position of the hair included in the detected hair area. As a result, even if only one eye can be detected from the face image, the present configuration can accurately detect the orientation of the face based on the positional relationship between one eye and the hair instead of the positional relationship between both eyes. it can.

In the above aspect, based on the image data of the face and the information indicating the position of the one eye, on the contour line of the face existing on the inner side of the eye of the one eye with respect to the position of the outer corner of the eye of the one eye. A contour point indicating a point is detected, and the orientation of the face is based on the first distance from the position of the one eye to the contour point and the second distance from the position of the one eye to the position of the ear. Information indicating that may be detected.

According to this configuration, the first distance from the position of the one eye to the contour point indicating a point on the contour line of the face existing on the inner side of the eye of the one eye rather than the position of the outer corner of the eye of the one eye. , Information indicating the orientation of the face is detected based on the second distance from the position of one eye to the position of the ear. As a result, in this configuration, the width of the face on the inner side of the eye from the detected one eye, which is indicated by the first distance, and the width of the face on the ear side, that is, the outer side of the eye, which is indicated by the second distance. Based on the above, the orientation of the face can be detected accurately.

In the above aspect, based on the image data of the face and the information indicating the position of the one eye, on the contour line of the face existing on the inner side of the eye of the one eye with respect to the position of the outer corner of the eye of the one eye. The contour point indicating a point is detected, and the first distance from the position of the one eye to the contour point and the position of the one eye are located closer to the outer corner of the eye than the position of the inner corner of the one eye. Information indicating the orientation of the face may be detected based on the third distance to the point in the hair area.

According to this configuration, the first distance from the position of the one eye to the contour point indicating a point on the contour line of the face existing on the inner side of the eye of the one eye rather than the position of the outer corner of the eye of the one eye. , Information indicating the orientation of the face is detected based on the third distance from the position of the one eye to the point in the hair region located on the outer corner side of the eye with respect to the position of the inner corner of the eye. .. Thereby, this configuration is based on the width of the face on the inner side of the eye than the detected one eye indicated by the first distance and the width of the face on the outer corner of the eye than the detected one eye indicated by the third distance. The orientation of the face can be detected accurately.

The present disclosure can also be realized as a control program that causes a computer to execute each characteristic configuration included in such a line-of-sight detection method, or as a line-of-sight detection device that operates by this control program. Needless to say, such a control program can be distributed via a computer-readable non-temporary recording medium such as a CD-ROM or a communication network such as the Internet.

It should be noted that all of the embodiments described below show a specific example of the present disclosure. The numerical values, shapes, components, steps, order of steps, etc. shown in the following embodiments are examples, and are not intended to limit the present disclosure. Further, among the components in the following embodiments, the components not described in the independent claims indicating the highest level concept are described as arbitrary components. Moreover, in all the embodiments, each content can be combined.

(Embodiment 1)
FIG. 1 is a diagram showing an example of the overall configuration of the image processing system 1 according to the first embodiment of the present disclosure. The image processing system 1 is a system that photographs a person 400 and detects line-of-sight information indicating the line-of-sight of the person 400 from the obtained image data of the person 400. In the example of FIG. 1, the image processing system 1 specifies which object 301 the person 400 is gazing at among the plurality of objects 301 displayed on the display device 300. However, this is an example, and the image processing system 1 may specify not only the object 301 displayed on the display screen of the display device 300 but also the object 301 that the person 400 gazes at in the real space.

In the example of FIG. 1, the image processing system 1 is applied to a digital signage system. Therefore, the object 301 displayed on the display device 300 becomes an image of signage such as an advertisement.

The image processing system 1 includes an image processing device 100, a camera 200, and a display device 300. The image processing device 100 is connected to the camera 200 and the display device 300 via a predetermined communication path. The predetermined communication path is, for example, a wired communication path such as a wired LAN, or a wireless communication path such as a wireless LAN and Bluetooth (registered trademark). The image processing device 100 is composed of, for example, a computer installed around the display device 300. However, this is an example, and the image processing device 100 may be configured by a cloud server. In this case, the image processing device 100 is connected to the camera 200 and the display device 300 via the Internet. The image processing device 100 detects the line-of-sight information of the person 400 from the image data of the person 400 captured by the camera 200 and outputs the information to the display device 300. Further, the image processing device 100 may be incorporated as hardware in the camera 200 or the display device 300. Further, the camera 200 or the display device 300 may include a processor, and the image processing device 100 may be incorporated as software.

The camera 200 acquires image data of a person 400 located around the display device 300, for example, by photographing the environment around the display device 300 at a predetermined frame rate. The camera 200 sequentially outputs the acquired image data to the image processing device 100 at a predetermined frame rate. The camera 200 may be a visible light camera or an infrared light camera. Further, the camera 200 may be, for example, a visible light camera or an infrared light camera functionally provided in an electronic device such as a display device 300 or a smartphone.

The display device 300 is composed of a display device such as a liquid crystal panel or an organic EL panel. In the example of FIG. 1, the display device 300 is a signage display. In the example of FIG. 1, the image processing system 1 has been described as including the display device 300, but this is an example, and another device may be adopted instead of the display device 300. For example, if the image processing system 1 is used as a user interface that accepts input to the device by the line of sight, the image processing system 1 may be replaced with, for example, the display device 300 by a home appliance such as a refrigerator, a television, and a washing machine. It may be adopted. For example, if the image processing system 1 is mounted on a vehicle, a vehicle such as an automobile may be adopted instead of the display device 300. In this case, the camera 200 may be a visible light camera or an infrared light camera functionally provided in a home electric appliance, a vehicle, or the like. Further, a storage device such as a hard disk drive or a solid state drive may be adopted instead of the display device 300.

FIG. 2 is a block diagram showing an example of a detailed configuration of the image processing system 1 according to the first embodiment. The image processing device 100 includes a processor 110. The processor 110 is an electric circuit such as a CPU and an FPGA. The processor 110 includes a line-of-sight detection device 120 and an output unit 130.

The line-of-sight detection device 120 includes an image acquisition unit 121, an eye detection unit 122, both visual line detection units 123, and a single visual line detection unit 124. Each block included in the processor 110 may be realized by the processor 110 executing a control program that causes the computer to function as an image processing device, or may be configured by a dedicated electric circuit.

The image acquisition unit 121 acquires the image data captured by the camera 200. Here, the acquired image data includes the faces of a person 400 (an example of a person) around the display device 300. The image data acquired by the image acquisition unit 121 may be, for example, image data posted on a website or image data stored by an external storage device.

The eye detection unit 122 detects a left eye region including the left eye of the person 400 and a right eye region including the right eye of the person 400 from the image data acquired by the image acquisition unit 121. Hereinafter, when the left eye area and the right eye area are collectively referred to, they are referred to as an eye area. Specifically, the eye detection unit 122 may detect the eye region using a classifier created in advance to detect the eye region. The classifier used here is, for example, a Haar-shaped cascade classifier created in advance for detecting an eye region in an open source image processing library.

The eye area is a rectangular area having a size obtained by adding a predetermined margin to the size of the eyes. However, this is an example, and the shape of the eye region may be other than a rectangle, for example, a triangle, a pentagon, a hexagon, an octagon, or the like. In addition, the position where the boundary of the eye region is set with respect to the eye depends on the performance of the classifier.

FIG. 3 is a diagram showing an example of a right eye region 50R and a left eye region 50L. As shown in FIG. 3, in the present embodiment, the eye refers to a region including a white eye and a colored portion such as a black eye, which is surrounded by the boundary 53 of the upper eyelid and the boundary 54 of the lower eyelid. As shown in FIG. 3, the pupil refers to a colored portion including the pupil 55 and the donut-shaped iris 56 surrounding the pupil 55. Further, in the present embodiment, for convenience of explanation, the right eye refers to the eye on the right side when the person 400 is viewed from the front, and the left eye refers to the eye on the left side when the person 400 is viewed from the front. FIG. 3 shows an example in which the eye detection unit 122 detects the right eye region 50R including the right eye and the left eye region 50L including the left eye. However, this is an example, and the eye on the right side when viewed from the person 400 may be the right eye, and the eye on the left side when viewed from the person 400 may be the left eye. Further, in the present embodiment, the right side direction of the paper surface is the right side, and the left side direction of the paper surface is the left side.

When the eye detection unit 122 detects the right eye region 50R and the left eye region 50L, both visual line detection units 123 provide information indicating the face orientation of the person 400 from the image data acquired by the image acquisition unit 121 (hereinafter, face orientation information). ) Is detected. Then, both visual line detection units 123 perform the first processing using the information indicating the position of the right eye included in the right eye region 50R, the information indicating the position of the left eye included in the left eye region 50L, and the detected face orientation information. , Information indicating the line of sight of the person 400 (hereinafter referred to as line-of-sight information) is detected.

Specifically, a known line-of-sight detection process for detecting the line-of-sight by a three-dimensional eyeball model may be applied to the detection of face orientation information and the first process performed by both line-of-sight detection units 123. Both visual line detection units 123 may detect face orientation information from the image data acquired by the image acquisition unit 121 according to a known line-of-sight detection process applied to the detection of face orientation information. The face orientation information includes, for example, an angle indicating the front direction of the face with respect to the optical axis of the camera 200. Further, both visual line detection units 123 provide information indicating the positions of the right eye and the left eye from the right eye region 50R and the left eye region 50L detected by the eye detection unit 122 according to the known eye line detection process applied as the first process. You just have to get it. The information indicating the position of the right eye includes, for example, the positions of the pupil of the right eye, the pupil of the right eye, the inner corner of the right eye, the outer corner of the right eye, and the center of gravity of the right eye. Similarly, the information indicating the position of the left eye includes, for example, the positions of the pupil of the left eye, the pupil of the left eye, the inner corner of the left eye, the outer corner of the left eye, and the center of gravity of the left eye.

The line-of-sight information may include a vector that three-dimensionally indicates the direction of the line of sight of the person 400, or may include coordinate data of the gazing point on a predetermined target surface (for example, the display device 300). The gazing point is, for example, a position where the target surface and the vector indicating the line of sight intersect.

When the eye detection unit 122 detects only one eye area 50 of the right eye area 50R or the left eye area 50L, the one-sided visual line detection unit 124 indicates information indicating the position of one eye included in the one eye area 50 and a person. Face orientation information is detected based on information indicating the position of a portion different from the eyes on the face of 400. Then, the one-sided visual line detection unit 124 detects the line-of-sight information by a second process using the information indicating the position of the one eye and the detected face orientation information. The portion different from the eyes includes, for example, ears or hair. Details of the one-sided visual line detection unit 124 will be described later.

The output unit 130 outputs the line-of-sight information detected by both the line-of-sight detection units 123 and the one-sided line-of-sight detection unit 124 to the display device 300. The output unit 130 acquires the information of the object 301 displayed on the display device 300, identifies the object 301 (hereinafter referred to as the gaze object) to be gazed by the person 400 from the acquired information and the coordinate data of the gazing point, and identifies the object 301. The result may be output to the display device 300.

Since the camera 200 has been described with reference to FIG. 1, the description thereof will be omitted here.

The display device 300 displays, for example, a marker indicating line-of-sight information output from the output unit 130. The display device 300 may display, for example, a marker indicating an object 301 to be watched by the person 400 output from the output unit 130.

For example, it is assumed that the coordinate data of the gazing point is output to the display device 300 as the line-of-sight information. In this case, the display device 300 performs a process of superimposing a marker indicating the line-of-sight position on the displayed image and displaying it at a position corresponding to the coordinate data. For example, it is assumed that the specific result of the gaze object is output to the display device 300. In this case, the display device 300 may perform a process of superimposing a marker indicating the gaze object on the screen being displayed and displaying the marker.

When the image processing system 1 is composed of home appliances instead of the display device 300, the home appliances accept the input of the person 400 from the line-of-sight information. Further, when the image processing system 1 is composed of a storage device instead of the display device 300, the storage device stores the line-of-sight information. In this case, the storage device may store the line-of-sight information in association with a time stamp.

Next, the details of the one-sided visual line detection unit 124 will be described. FIG. 4 is a block diagram showing an example of a detailed configuration of the one-sided visual line detection unit 124. As shown in FIG. 4, the one-sided visual line detection unit 124 includes an ear detection unit 41 (an example of a site detection unit), a hair detection unit 42 (an example of a site detection unit), a face orientation detection unit 43, and a one-sided face line-of-sight detection unit. 44 (an example of a line-of-sight detection unit) is included.

The ear detection unit 41 detects the right ear region including the right ear of the person 400 and the left ear region including the left ear of the person 400 from the image data acquired by the image acquisition unit 121. Specifically, the ear detection unit 41 detects the right ear region using a right ear classifier created in advance to detect the region including the right ear of the person 400, and the region including the left ear of the person 400. The left ear region may be detected using a pre-made left ear classifier in order to detect. The right-ear and left-ear classifiers used here are, for example, Haar-shaped cascade classifiers created in advance to detect each of the right-ear region and the left-ear region in an open source image processing library. is there. Hereinafter, when the right ear region and the left ear region are collectively referred to, they are referred to as ear regions.

The ear region is a rectangular region having a size about the size of an ear. However, this is an example, and the shape of the ear region may be other than a rectangle, for example, a triangle, a pentagon, a hexagon, an octagon, or the like. In addition, the position where the boundary of the ear region is set with respect to the ear depends on the performance of the classifier.

FIG. 5 is a diagram showing an example of the ear region 30. As shown in FIG. 5, in the present embodiment, the ear refers to the region indicating the pinna taken in by the broken line 31. Further, in the present embodiment, for convenience of explanation, the right ear refers to the ear on the right side when the person 400 is viewed from the front, and the left ear refers to the ear on the left side when the person 400 is viewed from the front. Point to. FIG. 5 shows an example in which the ear detection unit 41 detects only the left ear region 30L using the left ear classifier without detecting the right ear region 30R (not shown) using the right ear classifier. Shown. However, this is an example, and the ear on the right side when viewed from the person 400 may be the right ear, and the ear on the left side when viewed from the person 400 may be the left ear.

The hair detection unit 42 detects a hair region including the hair of the person 400 from the image data acquired by the image acquisition unit 121. Hereinafter, the details of the process for detecting the hair region will be described.

First, the hair detection unit 42 detects a face region including at least a part of the face of the person 400 from the image data acquired by the image acquisition unit 121. Specifically, the hair detection unit 42 may detect the face region using a classifier created in advance to detect the face region. The classifier used here is, for example, a Haar-shaped cascade classifier created in advance for detecting a face region in an open source image processing library.

FIG. 6 is a diagram showing an example of the face region 60. As shown in FIG. 6, the face area 60 is, for example, a rectangular area having a size that includes the entire face. However, this is an example, and the shape of the face region 60 may be, for example, a triangle, a pentagon, a hexagon, or an octagon other than a rectangle. Further, the hair detection unit 42 may detect the face region 60 by pattern matching, for example.

Next, the hair detection unit 42 binarizes the detected face region 60. FIG. 7 is a diagram showing an example of the binarized face region 60. Here, for example, a method called binarization of Otsu is adopted. Further, the portion where the brightness is lower than the threshold value is represented by white, and the portion where the brightness is higher than the threshold value is represented by black.

Next, the hair detection unit 42 performs a labeling process of assigning a labeling number to the white islands appearing in the binarized face area 60. In the example of FIG. 7, the hair is regarded as one island 71, and the island 71 is given a labeling number of "1". Further, the left eye is regarded as one island 72, and the island 72 is given a labeling number of "2". Further, the left hole of the nose is regarded as one island 73, and the island 73 is given a labeling number of "3". In addition, the lips are regarded as one island 74, and the island 74 is given a labeling number of "4".

Then, the hair detection unit 42 detects the island having the largest area among the labeled islands as the hair region. In the example of FIG. 7, the island 71 is detected as a hair region.

The face orientation detection unit 43 includes information indicating the position of one eye included in one eye region 50 of the right eye region 50R or the left eye region 50L detected by the eye detection unit 122, and the ear detected by the ear detection unit 41. The face orientation information is detected based on the information indicating the position of the ears included in the region 30 or the information indicating the position of the hair included in the hair region detected by the hair detecting unit 42. The face orientation information includes information indicating whether the face of the person 400 is facing right or left, and the degree to which the person 400 tilts the face to the left or right with respect to the optical axis of the camera 200 (hereinafter, the face orientation information). Face orientation) etc. are included. Hereinafter, the details of the process of detecting the face orientation information will be described.

The face orientation detection unit 43 first determines whether the face of the person 400 is facing right or facing left. Specifically, when the left ear region 30L is detected by the ear detection unit 41, the face orientation detection unit 43 determines that the person 400 is facing to the right and determines that the face is facing to the right. .. On the other hand, when the right ear region 30R is detected by the ear detection unit 41, the face orientation detection unit 43 determines that the person 400 is facing left, and determines that the face is facing left. In the example of FIG. 5, since the ear detection unit 41 has detected the left ear region 30L, it is determined that the face is facing right.

The ear detection unit 41 may erroneously detect the right ear region 30R and the left ear region 30L. In consideration of the false detection, the face orientation detection unit 43 may be configured as follows. That is, after the ear detection unit 41 detects the ear region 30, the face orientation detection unit 43 may detect the face region 60 from the image data acquired by the image acquisition unit 121 in the same manner as the hair detection unit 42. .. Then, when the position of the eye region 50 detected by the eye detection unit 122 in the detected face region 60 by the face orientation detection unit 43 is on the right side of the ear region 30 detected by the ear detection unit 41. It may be determined that the face orientation is rightward, and when the position of the eye region 50 is on the left side of the ear region 30, it may be determined that the face orientation is leftward.

Further, the face orientation detection unit 43 states that the person 400 is facing to the right when the position of the center of gravity of the island detected as the hair region by the hair detection unit 42 is in the left half region of the face region 60. Judge and judge that the face is facing right. On the other hand, when the position of the center of gravity of the island detected as the hair region by the hair detection unit 42 exists in the right half region of the face region 60, the face orientation detection unit 43 states that the person 400 is facing to the left. Judge and judge that the face is facing left. In the example of FIG. 7, since the position P71 of the center of gravity of the island 71 detected as the hair region by the hair detection unit 42 exists in the left half region of the face region 60, it is determined that the face orientation is to the right. ..

Next, the face orientation detection unit 43 is based on the image data acquired by the image acquisition unit 121 and the information indicating the position of the eyes included in one of the eye regions 50 detected by the eye detection unit 122. A contour point indicating a point on the contour line of the face of the person 400 existing on the inner corner side of the eye rather than the position of the outer corner of the eye is detected. Hereinafter, details of the process of detecting the contour point when it is determined that the face direction is rightward will be described. The process of detecting the contour point when the face orientation is determined to be left is the same as that when the face orientation is determined to be right, and thus the description thereof will be omitted.

First, the face orientation detection unit 43 detects the face region 60 from the image data acquired by the image acquisition unit 121 and binarizes it in the same manner as the hair detection unit 42. Next, the face orientation detection unit 43 performs a labeling process of assigning a labeling number to the white islands appearing in the binarized face area 60. FIG. 8 is a diagram showing another example of the binarized face region 60.

In the example of FIG. 8, the hair is regarded as one island 81, and the island 81 is given a labeling number of “1”. The left eye is regarded as one island 82, and the island 82 is given a labeling number of "2". The left hole of the nose is regarded as one island 83, and the island 83 is given a labeling number of "3". The lips are regarded as one island 84, and the island 84 is given a labeling number of "4". The left eyebrow is regarded as one island 85, and the island 85 is given a labeling number of "5". A part of the right eye is regarded as one island 86, and the island 86 is given a labeling number of "6".

Next, the face orientation detection unit 43 is an island having an area equal to or less than a predetermined area among the labeled islands, and is included in the one eye area 50 from the one eye area 50 detected by the eye detection unit 122. Detects the island farthest from the inner corner of the eye. The predetermined area is set to, for example, an area of about the area of one eye region 50 detected by the eye detection unit 122. In the example of FIG. 8, the face orientation detection unit 43 determines that the face orientation is rightward. Therefore, the face orientation detection unit 43 determines that the eye region 50 detected by the eye detection unit 122 is the left eye region 50L, and the island 82 included in the left eye region 50L indicates the left eye. Next, the face orientation detection unit 43 detects the island 86 having an area equal to or less than a predetermined area and being farthest from the left eye region 50L to the right, which is the inner corner side of the left eye.

Next, the face orientation detection unit 43 considers that the detected island has the contour point, and detects the contour point. In the example of FIG. 8, the face orientation detection unit 43 detects the inner corner side of the left eye, that is, the right end portion P81 of the detected island 86 as the contour point 61.

Next, the face orientation detection unit 43 detects the position of the center of gravity of the eye and the position of the inner corner of the eye from the eye area 50 detected by the eye detection unit 122 as information indicating the position of the eye included in the eye area 50. In the example of FIG. 8, since the face orientation detection unit 43 determines that the face orientation is to the right, the eye region 50 detected by the eye detection unit 122 is the left eye region 50L, and the island included in the left eye region 50L. It is determined that 82 indicates the left eye. Therefore, the face orientation detection unit 43 detects the position P82 of the center of gravity of the island 82 as the position 62 of the center of gravity of the left eye. Further, the face orientation detection unit 43 detects the inner corner side of the left eye, that is, the right end P83 on the island 82, as the position 63 of the inner corner of the left eye.

Next, when the ear region 30 is detected by the ear detection unit 41, the face orientation detection unit 43 extends from the position of the inner corner of the eye included in one eye region 50 detected by the eye detection unit 122 to the contour point 61. The first distance is calculated, and the second distance from the position of the inner corner of the eye to the position of the ear included in the ear region 30 detected by the ear detection unit 41 is calculated. Then, the face orientation detection unit 43 detects the face orientation information based on the first distance and the second distance.

In the example of FIG. 8, the face orientation detection unit 43 calculates the horizontal distance L1 from the position 63 of the inner corner of the left eye included in the left eye region 50L to the contour point 61 as the first distance (hereinafter, the first distance L1). To do. The face orientation detection unit 43 sets the inner side of the left eye, that is, the right end of the left ear region 30L detected by the ear detection unit 41 as the position of the ear included in the ear region 30. Then, the face orientation detection unit 43 calculates the horizontal distance L2 from the position 63 of the inner corner of the left eye to the position of the ear as the second distance (hereinafter, the second distance L2).

FIG. 9 is a schematic view showing an example of the degree of face orientation of the person 400 when viewed from above. As the degree to which the person 400 tilts the face in the left-right direction toward the camera 200 increases, as shown in FIG. 9, an angle indicating the front direction 92 of the face of the person 400 with respect to the optical axis 91 of the camera 200 (hereinafter, the posture). Angle) 93 becomes larger. Further, as the degree of face orientation increases, the first distance L1 from the position of the inner corner of the eye to the contour point imaged by the camera 200 becomes shorter, and the second distance L2 from the position of the inner corner of the eye to the position of the ear becomes longer. .. Therefore, the posture angle 93 indicating the degree of face orientation and the ratio of the second distance L2 to the first distance L1 (= L2 / L1) or the ratio of the first distance L1 to the second distance L2 (= L1 / L2). The function indicating the relationship (hereinafter referred to as the first function) can be predetermined based on experimental values and the like.

Therefore, the face orientation detection unit 43 calculates the ratio of the second distance L2 to the first distance L1 or the ratio of the first distance L1 to the second distance L2. Then, the face orientation detection unit 43 inputs the calculated ratio into the first function, and acquires the posture angle 93 corresponding to the input ratio as the face orientation degree. The face orientation detection unit 43 detects the degree of face orientation thus obtained as face orientation information. The method of obtaining the degree of face orientation is not limited to this. For example, a function indicating the relationship between the difference between the first distance L1 and the second distance L2 and the posture angle 93 may be determined in advance, and the degree of face orientation may be obtained using the function.

The ear region 30 cannot be detected from the image data of the face of the person 400 whose ears are covered with hair. Therefore, in this case, the face orientation detection unit 43 uses the hair detection unit 42 to detect the hair region from the image data of the face of the person 400. Next, the face orientation detection unit 43 calculates the first distance L1 and moves from the position of the eye included in one eye region 50 detected by the eye detection unit 122 to the outer corner of the eye with respect to the position of the inner corner of the eye. The third distance to the point in the hair area where it is located is calculated. Then, the face orientation detection unit 43 detects the face orientation information based on the first distance L1 and the third distance.

Hereinafter, the process of detecting the face orientation information from the hair region will be described with reference to FIG. 7. In the example of FIG. 7, the face orientation detection unit 43 determines that the island 72 included in the left eye region 50L indicates the left eye, as in the example of FIG. Then, the face orientation detection unit 43 has an area equal to or less than a predetermined area, and has an end portion P75 on the inner side of the left eye on the island 73 that is farthest to the right, which is the inner side of the left eye, from the left eye region 50L. , Detected as contour point 61. Further, the face orientation detection unit 43 detects the position P72 of the center of gravity of the island 72 indicating the left eye as the position 62 of the center of gravity of the left eye, and sets the end P73 on the inner side of the left eye on the island 72 as the position 63 of the inner corner of the left eye. To detect. Then, the face orientation detection unit 43 calculates the first distance L1, which is the horizontal distance from the detected position 63 of the inner corner of the left eye to the detected contour point 61.

Further, the face orientation detection unit 43 detects the position P71 of the center of gravity of the hair region detected by the hair detection unit 42 as a point 64 in the hair region. The point 64 in the hair region is not limited to this, and on the contour line of the island 71 indicating the hair region, from the position 63 of the inner corner of the left eye included in the left eye region 50L to the left side of the outer corner of the left eye. It may be the farthest point P74. The face orientation detection unit 43 calculates the horizontal distance L3 from the position 63 of the inner corner of the left eye to the point 64 in the hair region as the third distance (hereinafter, the third distance L3).

Then, the face orientation detection unit 43 calculates the ratio of the third distance L3 to the first distance L1 or the ratio of the first distance L1 to the third distance L3 in the same manner as in the example of FIG. Then, the face orientation detection unit 43 inputs the calculated ratio into the second function similar to the first function, and acquires the posture angle 93 corresponding to the input ratio as the face orientation degree. The second function is a function showing the relationship between the posture angle 93 (FIG. 9) and the ratio of the third distance L3 to the first distance L1 or the ratio of the first distance L1 to the third distance L3, and is an experiment. It is predetermined based on the value and the like. The face orientation detection unit 43 detects the degree of face orientation thus obtained as face orientation information. The method of obtaining the degree of face orientation is not limited to this. For example, a function indicating the relationship between the difference between the first distance L1 and the third distance L3 and the posture angle 93 may be determined in advance, and the degree of face orientation may be obtained using the function.

The one-sided facial line-of-sight detection unit 44 is based on the information indicating the position of the eyes included in the one eye region 50 detected by the eye detection unit 122 and the face orientation information detected by the face orientation detection unit 43. The second process of detecting the line-of-sight information is performed. The details of the second process will be described below.

FIG. 10 is a diagram showing a viewing angle 94 of the person 400. Generally, it is said that the viewing angle 94 of the eyes of the person 400 falls within the range of 0 degrees or more and 60 degrees or less to the left and right with respect to the front direction 92 of the face of the person 400. Therefore, in the present embodiment, it is assumed that the viewing angle 94 is predetermined to be 60 degrees.

In the second process, the one-sided facial line-of-sight detection unit 44 first detects the position of the outer corner of the eye included in the one eye region 50 from the one eye region 50 detected by the eye detection unit 122.

FIG. 11 is an explanatory diagram of a detection method in the line-of-sight direction. FIG. 11 shows an enlarged view of the left eye region 50L shown in FIG. In the example of FIG. 11, the one-sided face line-of-sight detection unit 44 detects the left end portion P85 of the island 82 determined to indicate the left eye by the face orientation detection unit 43 as the position 65 of the outer corner of the left eye.

The one-sided facial line-of-sight detection unit 44 is a horizontal distance from the position 62 of the center of gravity of the eye included in the one eye region 50 detected by the eye detection unit 122 to the position 65 of the outer corner of the eye (hereinafter, the outer corner of the center of gravity). Distance) is obtained. Further, the one-sided facial line-of-sight detection unit 44 is a horizontal distance from the position 62 of the center of gravity of the eye included in the detected one eye region 50 to the position 63 of the inner corner of the eye (hereinafter, the distance between the inner corners of the center of gravity). ) To get. Then, the one-sided facial line-of-sight detection unit 44 indicates an angle indicating the direction of the line of sight with respect to the front direction 92 of the face based on the acquired distance between the outer corners of the center of gravity and the inner corner of the center of gravity and a predetermined viewing angle 94 (hereinafter, Line-of-sight angle) is calculated.

Specifically, the closer the position 62 of the center of gravity of the eyes included in one eye region 50 detected by the eye detection unit 122 is to the position 63 of the inner corner of the eyes, the closer the line of sight of the person 400 is to the front direction 92 of the face. It is thought that it is facing a direction close to. On the other hand, the closer the position 62 of the center of gravity of the eye is to the position 65 of the outer corner of the eye, the more the line of sight of the person 400 is in the direction away from the front direction 92 of the face, that is, the viewing angle with respect to the front direction 92 of the face. It is considered that the direction is closer to the left or right by 94 minutes.

Therefore, in the example of FIG. 11, the one-sided facial line-of-sight detection unit 44 is located in the horizontal direction from the position 62 of the center of gravity of the left eye included in the left eye region 50L detected by the eye detection unit 122 to the position 65 of the outer corner of the left eye. The distance K1 between the outer corners of the center of gravity, which is the distance, is acquired. Further, the one-sided facial line-of-sight detection unit 44 acquires the distance K2 between the inner corners of the center of gravity, which is the horizontal distance from the position 62 of the center of gravity of the left eye to the position 63 of the inner corner of the left eye. Then, the one-sided facial line-of-sight detection unit 44 uses the acquired distance K1 between the outer corners of the center of gravity and K2 between the inner corners of the center of gravity and the predetermined value α (= 60 °) of the viewing angle 94 according to the following equation. , Calculate the line-of-sight angle β. β = α × (K2 / K1).

The one-sided face line-of-sight detection unit 44 calculates the value obtained by adding the calculated line-of-sight direction angle β to the face orientation degree detected by the face direction detection unit 43 as an angle indicating the direction of the line-of-sight with respect to the optical axis 91 of the camera 200. To do. The one-sided facial line-of-sight detection unit 44 detects the calculated angle indicating the direction of the line of sight with respect to the optical axis 91 of the camera 200 as line-of-sight information.

Next, the operation of the image processing device 100 will be described. FIG. 12 is a flowchart showing an example of the operation of the image processing device 100 according to the first embodiment.

The image acquisition unit 121 acquires image data of the face of the person 400 from the camera 200 (step S1). Next, the eye detection unit 122 detects the right eye region 50R and the left eye region 50L from the image data by inputting the image data acquired in step S1 into the classifier for detecting the eye region 50 (step S2). ).

In step S2, it is assumed that the right eye region 50R and the left eye region 50L are detected as in the example of FIG. 3 (YES in step S3). In this case, both visual line detection units 123 include information indicating the position of the right eye included in the right eye region 50R, information indicating the position of the left eye included in the left eye region 50L, and face orientation information detected from the image data acquired in step S1. The line-of-sight information is detected by the first process using and (step S4).

On the other hand, as in the examples of FIGS. 5 and 6, it is assumed that only one eye region 50 of the right eye region 50R or the left eye region 50L is detected in step S2 (NO in step S3). In this case, the one-sided visual line detection unit 124 detects the face orientation information based on the information indicating the position of the eyes included in the one eye region 50 and the information indicating the position of a portion different from the eyes on the face of the person 400. To do. Then, the one-sided visual line detection unit 124 detects the line-of-sight information by a second process using the information indicating the position of the one eye and the detected face orientation information (step S5). Details of the operation of the one-sided visual line detection unit 124 in step S5 will be described later.

After step S4 or step S5, the output unit 130 outputs the line-of-sight information detected in step S4 or step S5 to the display device 300 (step S6).

Next, the operation of the one-sided visual line detection unit 124 in step S5 will be described. FIG. 13 is a flowchart showing an example of the operation of the one-sided visual line detection unit 124 according to the first embodiment.

When step S5 is started, the ear detection unit 41 inputs the image data acquired in step S1 into the classifier for detecting the right ear region and the left ear region, so that the ear region 30 is obtained from the image data. Is detected (step S51). For example, in step S51, the ear detection unit 41 detects the left ear region 30L from the image data acquired in step S1 as shown in FIG.

When the ear region 30 is detected in step S51 (YES in step S52), the face orientation detection unit 43 includes information indicating the position of the eyes included in one of the eye regions 50 detected in step S2, and in step S51. Face orientation information is detected based on the information indicating the position of the ear included in the detected ear region 30 (step S53).

For example, in step S51, it is assumed that the left ear region 30L is detected as shown in FIG. In this case, in step S53, the face orientation detection unit 43 determines that the face orientation is right because the left ear region 30L was detected in step S51.

In this case, as shown in FIG. 8, the face orientation detection unit 43 is based on the image data acquired in step S1 and the position 63 of the inner corner of the left eye included in the left eye region 50L detected in step S2. , The first distance L1, which is the horizontal distance from the position 63 of the inner corner of the left eye to the contour point 61, is calculated. Further, the face orientation detection unit 43 sets the end portion of the left ear region 30L detected in step S51 on the inner corner side of the left eye as the position of the ear included in the ear region 30, and the position 63 of the inner corner of the left eye of the ear. The second distance L2, which is the horizontal distance to the position, is calculated.

Then, the face orientation detection unit 43 calculates the ratio of the second distance L2 to the first distance L1 or the ratio of the first distance L1 to the second distance L2, and inputs the calculated ratio into the first function. As a result, the face orientation detection unit 43 acquires the posture angle 93 (FIG. 9) corresponding to the input ratio as the face orientation degree. The face orientation detection unit 43 detects the degree of face orientation thus obtained as face orientation information.

On the other hand, when the ear region 30 is not detected in step S51 (NO in step S52), the face orientation detection unit 43 uses the hair detection unit 42 to detect the hair region from the image data acquired in step S1. (Step S54). For example, as shown in FIG. 7, in step S54, the hair detection unit 42 detects the face area 60 from the image data acquired in step S1, and determines the largest area in the binarized area of the face area 60. The island 71 to have is detected as a hair area.

Then, the face orientation detection unit 43 includes information indicating the position of the eyes included in one eye region 50 detected in step S2, information indicating the position of the hair included in the hair region detected in step S54, and information indicating the position of the hair. The face orientation information is detected based on (step S55).

For example, in step S54, as shown in FIG. 7, it is assumed that the island 71 is detected as a hair region. In this case, in step S55, as described above, the face orientation detection unit 43 has the position P71 of the center of gravity of the island 71 detected as the hair region in the left half region of the face region 60, so that the face orientation Is determined to be facing right.

In this case, as shown in FIG. 7, the face orientation detection unit 43 is based on the image data acquired in step S1 and the position 63 of the inner corner of the left eye included in the left eye region 50L detected in step S2. , The first distance L1, which is the horizontal distance from the position 63 of the inner corner of the left eye to the contour point 61, is calculated. Further, the face orientation detection unit 43 further sets the position P71 of the center of gravity of the hair region detected in step S54 as a point 64 in the hair region, and extends from the position 63 of the inner corner of the left eye to the point 64 in the hair region. The third distance L3, which is the distance in the horizontal direction, is calculated.

Then, the face orientation detection unit 43 calculates the ratio of the third distance L3 to the first distance L1 or the ratio of the first distance L1 to the third distance L3, and inputs the calculated ratio into the second function. As a result, the face orientation detection unit 43 acquires the posture angle 93 (FIG. 9) corresponding to the input ratio as the face orientation degree. The face orientation detection unit 43 detects the degree of face orientation thus obtained as face orientation information.

After step S53 or step S55, the one-sided facial line-of-sight detection unit 44 includes information indicating the position of the eyes included in one eye region 50 detected in step S2, and facial orientation information detected in step S53 or step S55. The line-of-sight information is detected by the second process using and (step S56).

For example, in step S56, as shown in FIG. 11, the one-sided facial line-of-sight detection unit 44 extends from the position 62 of the center of gravity of the left eye included in the left eye region 50L detected in step S2 to the position 65 of the outer corner of the left eye. The distance K1 between the outer corners of the center of gravity, which is the distance in the horizontal direction, is acquired. Further, the one-sided facial line-of-sight detection unit 44 acquires the distance K2 between the inner corners of the center of gravity, which is the horizontal distance from the position 62 of the center of gravity of the left eye to the position 63 of the inner corner of the left eye. Then, the one-sided facial line-of-sight detection unit 44 uses the acquired distance K1 between the outer corners of the center of gravity and K2 between the inner corners of the center of gravity, and a predetermined viewing angle 94 value α (= 60 °), as described above (β). = Α × (K2 / K1)) to calculate the line-of-sight angle β. The one-sided facial line-of-sight detection unit 44 detects as line-of-sight information a value obtained by adding the calculated line-of-sight direction angle β to the degree of face direction detected as face-direction information in step S53 or step S55.

As described above, according to the present embodiment, when only one eye region 50 of the right eye region 50R or the left eye region 50L is detected from the face image data, the ear region 30 and the hair region are detected from the face image data. At least one of them is detected. Then, based on the information indicating the position of the eyes included in the one eye region 50 and the information indicating the position of the ear or the hair included in at least one of the detected ear region 30 and the hair region, the face orientation Information is detected. Then, the line-of-sight information is detected based on the detected face orientation information and the information indicating the position of the eyes.

As a result, even if only the image data of the face including only the right eye or the left eye can be acquired, the present configuration provides face orientation information based on the positional relationship between the ear or hair and the detected one eye instead of the positional relationship between both eyes. Can be detected accurately. As a result, this configuration can accurately identify the orientation of the face even when only one eye can be detected from the face image, and can improve the detection accuracy of the line of sight.

It is also possible that the hair region cannot be detected in step S54. In this case, the output unit 130 may output an error message indicating that the line-of-sight information cannot be detected to the display device 300. Alternatively, it is assumed that one eye that can be detected from the face image and one eye that cannot be detected from the face image both face substantially the same direction, and the processes after step S4 (FIG. 12) may be performed.

(Embodiment 2)
The second embodiment estimates the degree of interest of the person 400. FIG. 14 is a block diagram showing an example of a detailed configuration of the image processing system 1A according to the second embodiment. In the present embodiment, the same components as those in the first embodiment are designated by the same reference numerals, and the description thereof will be omitted. Further, in FIG. 14, a block having the same name as that in FIG. 2 but having a different function is given a reference numeral A at the end.

The line-of-sight detection device 120A further includes a feature point detection unit 140 and an interest level estimation unit 150.

Similar to the hair detection unit 42 (FIG. 3), the feature point detection unit 140 uses a classifier created in advance to detect a face region from the image data acquired by the image acquisition unit 121 to detect the face of the person 400. The face area 60 including at least a part of the above is detected. Then, the feature point detection unit 140 detects the feature points of the face from the detected face area 60. The facial feature points are one or more points at characteristic positions in each of the plurality of parts constituting the face such as the outer corners of the eyes, the inner corners of the eyes, the contours of the face, the nose muscles, the corners of the mouth, and the eyebrows. Characteristic points are also called landmarks. The feature point detection unit 140 may detect the feature points of the face by, for example, executing a landmark detection process using a model file of a machine learning framework.

The interest level estimation unit 150 estimates the interest level of the person 400 by the following processing. First, the interest level estimation unit 150 detects eyebrows and corners of the mouth from the face region 60 using the facial feature points detected by the feature point detection unit 140. Here, the degree of interest estimation unit 150 identifies the feature points of the face detected by the feature point detection unit 140 with landmark point numbers corresponding to the eyebrows and the corners of the mouth, thereby eyebrows. And the corner of the mouth may be detected.

Next, the interest level estimation unit 150 estimates the interest level of the person 400 based on the line-of-sight information detected by both visual line detection units 123 or the single visual line detection unit 124, and the positions of the detected eyebrows and the corners of the mouth. , Output to the display device 300. Specifically, the interest level estimation unit 150 describes in advance the standard positions of eyebrows and corners of the mouth when, for example, a person has various facial expressions such as joy, surprise, anger, sadness, and expressionlessness. Pattern data is acquired from, for example, a memory (not shown). Then, the interest level estimation unit 150 collates the detected positions of the eyebrows and the corners of the mouth of the person 400 with the pattern data, and estimates the facial expression of the person 400. Then, the interest level estimation unit 150 uses the estimated facial expression of the person 400 and the line of sight indicated by the line-of-sight information, and when the line of sight of the person 400 is in which direction or the gazing point of the person 400 is in which position, the person Identify what the 400 looks like. That is, the interest level estimation unit 150 specifies the data in which the line-of-sight information and the facial expression of the person 400 are associated with each other as the interest level of the person 400. Here, the degree of interest estimation unit 150 has been described as estimating the degree of interest based on the eyebrows and the angle of the mouth, but this is an example, and the degree of interest may be estimated based on one of the eyebrows and the angle of the mouth. ..

As described above, according to the present embodiment, the degree of interest of the person 400 is estimated by further using the eyebrows and the corners of the mouth in addition to the line-of-sight information. The degree of interest can be estimated with high accuracy.

(Modification example)
(1) When an infrared light camera is adopted as the camera 200, the infrared light camera uses infrared light in a predetermined second wavelength band in which the spectral intensity of sunlight is attenuated from the predetermined first wavelength. It may be configured with an infrared light camera. The predetermined first wavelength is, for example, 850 nm. The predetermined second wavelength is, for example, 940 nm. The band of the second wavelength does not include, for example, 850 nm, and is a band having a predetermined width with 940 nm as a reference (for example, the center). As an infrared light camera that captures near-infrared light, one that uses infrared light of 850 nm is known. However, since the spectral intensity of sunlight is not sufficiently attenuated at 850 nm, it may not be possible to detect line-of-sight information with high accuracy outdoors where the spectral intensity of sunlight is strong. Therefore, the present disclosure employs, for example, a camera that uses infrared light in the band of 940 nm as an infrared light camera. As a result, it is possible to detect line-of-sight information with high accuracy even outdoors where the spectral intensity of sunlight is strong. Here, the predetermined second wavelength is set to 940 nm, but this is an example and may be a wavelength slightly deviated from 940 nm. An infrared light camera using infrared light having a second wavelength is, for example, a camera including a floodlight that irradiates infrared light having a second wavelength.

(2) In the above embodiment, the line-of-sight information has been described as including coordinate data indicating the gazing point, but the present disclosure is not limited to this. For example, the line-of-sight information may include coordinate data indicating a gaze surface that is a region of a predetermined size (for example, a circle, a quadrangle, etc.) with a gaze point as a reference (for example, the center). Thereby, the gaze object can be appropriately determined without depending on the distance between the person 400 and the gaze object or the size of the gaze object.

Since the present disclosure can detect line-of-sight information with high accuracy, it is useful in estimation of a person's interest target using line-of-sight information, estimation of a person's state, user interface using line-of-sight, and the like.

30: Ear area 30L: Left ear area (ear area)
30R: Right ear area (ear area)
41: Ear detection unit (site detection unit)
42: Hair detection unit (site detection unit)
43: Face orientation detection unit 44: One-sided face line-of-sight detection unit (line-of-sight detection unit)
50: Eye area 50L: Left eye area 50R: Right eye area 61: Contour points 120, 120A: Line-of-sight detection device 121: Image acquisition unit 122: Eye detection unit 123: Both visual line detection units 124: One-line visual line detection unit 400: Person (person) )
L1: First distance L2: Second distance L3: Third distance

Claims (6)

It is a line-of-sight detection method in a line-of-sight detection device.
Get image data of human face,
The left eye region including the left eye of the person and the right eye region including the right eye of the person are detected from the image data of the face.
When the right eye region and the left eye region are detected, information indicating the orientation of the face is detected from the image data of the face, information indicating the position of the right eye included in the right eye region, and information indicating the position of the right eye included in the left eye region and the left eye included in the left eye region. The information indicating the line of sight of the person is detected by the first process using the information indicating the position and the information indicating the direction of the face.
When only one eye region of the right eye region or the left eye region is detected, information indicating the position of one eye included in the one eye region and the one eye rather than the position of the inner corner of the one eye. Based on the information indicating the position of the predetermined portion of the face existing on the outer corner of the eye, the information indicating the orientation of the face is detected, and the information indicating the position of one of the eyes and the information indicating the orientation of the face are The information indicating the person's line of sight is detected by the second process using
In the second process, (1) the predetermined portion is detected from the image data of the face, and (2) the one eye is based on the image data of the face and the information indicating the position of the one eye. A contour point indicating a point on the contour line of the face existing on the inner corner side of the eye with respect to the position of the outer corner of the eye is detected, and (3) the first distance from the position of the one eye to the contour point. , Information indicating the orientation of the face is detected based on the second distance from the position of the one eye to the position of the predetermined portion, and (4) the information indicating the position of the one eye and the face. This is a process of detecting information indicating the line of sight of the person based on the information indicating the direction of the person.
Line-of-sight detection method. The predetermined part is an ear or hair.
The line-of-sight detection method according to claim 1. In the second process,
The ear region including the human ear is detected from the image data of the face, and the ear region is detected.
When the ear region can be detected, the information indicating the orientation of the face is detected with the distance from the position of the one eye to the position of the ear included in the ear region as the second distance.
The line-of-sight detection method according to claim 2. In the second process,
When the ear region could not be detected, the hair region including the person's hair was detected from the image data of the face, and the hair region was detected.
The line-of-sight detection method according to claim 3, wherein the distance from the position of one eye to the position of the hair included in the hair region is defined as the second distance, and information indicating the orientation of the face is detected. An image acquisition unit that acquires image data of a human face,
An eye detection unit that detects a left eye region including the left eye of the person and a right eye region including the right eye of the person from the image data of the face.
When the right eye region and the left eye region are detected, information indicating the orientation of the face is detected from the image data of the face, information indicating the position of the right eye included in the right eye region, and information indicating the position of the right eye included in the left eye region and the left eye included in the left eye region. Both visual line detection units that detect information indicating the line of sight of the person by the first process using the information indicating the position and the information indicating the direction of the face, and
When only one eye region of the right eye region or the left eye region is detected, information indicating the position of one eye included in the one eye region and the one eye rather than the position of the inner corner of the one eye. Based on the information indicating the position of the predetermined portion of the face existing on the outer corner of the eye, the information indicating the orientation of the face is detected, and the information indicating the position of one of the eyes and the information indicating the orientation of the face are It is provided with a one-sided visual line detection unit that detects information indicating the person's line of sight by a second process using the above.
In the second process, (1) the predetermined portion is detected from the image data of the face, and (2) the one eye is based on the image data of the face and the information indicating the position of the one eye. A contour point indicating a point on the contour line of the face existing on the inner corner side of the eye with respect to the position of the outer corner of the eye is detected, and (3) the first distance from the position of the one eye to the contour point. , Information indicating the orientation of the face is detected based on the second distance from the position of the one eye to the position of the predetermined portion, and (4) the information indicating the position of the one eye and the face. This is a process of detecting information indicating the line of sight of the person based on the information indicating the direction of the person.
Line-of-sight detector. It is a control program of the line-of-sight detection device.
The computer included in the line-of-sight detection device
An image acquisition unit that acquires image data of a human face,
An eye detection unit that detects a left eye region including the left eye of the person and a right eye region including the right eye of the person from the image data of the face.
When the right eye region and the left eye region are detected, information indicating the orientation of the face is detected from the image data of the face, information indicating the position of the right eye included in the right eye region, and information indicating the position of the right eye included in the left eye region and the left eye included in the left eye region. Both visual line detection units that detect information indicating the line of sight of the person by the first process using the information indicating the position and the information indicating the direction of the face, and
When only one eye region of the right eye region or the left eye region is detected, information indicating the position of one eye included in the one eye region and the one eye rather than the position of the inner corner of the one eye. Based on the information indicating the position of the predetermined portion of the face existing on the outer corner of the eye, the information indicating the orientation of the face is detected, and the information indicating the position of one of the eyes and the information indicating the orientation of the face are By the second process using the above, it is made to function as a one-sided visual line detection unit that detects information indicating the person's line of sight.
In the second process, (1) the predetermined portion is detected from the image data of the face, and (2) the one eye is based on the image data of the face and the information indicating the position of the one eye. A contour point indicating a point on the contour line of the face existing on the inner corner side of the eye with respect to the position of the outer corner of the eye is detected, and (3) the first distance from the position of the one eye to the contour point. , Information indicating the orientation of the face is detected based on the second distance from the position of the one eye to the position of the predetermined portion, and (4) the information indicating the position of the one eye and the face. This is a process of detecting information indicating the line of sight of the person based on the information indicating the direction of the person.
Control program.

Images