JP7103443B2 - Information processing equipment, information processing methods, and programs - Google Patents

Description

The present disclosure relates to a line-of-sight estimation device and the like.

The line of sight (eye orientation) of a person can be an important clue in analyzing the behavior or intention of the person. Therefore, a technique for estimating information on the human line of sight, particularly a technique for estimating the line of sight based on an image including a human face (hereinafter, also referred to as a "face image") has been widely studied.

As a technique for estimating the line of sight based on a face image, for example, there are techniques described in Patent Documents 1 to 3 and Non-Patent Documents 1 and 2. Patent Document 1 discloses an example of feature-based methods using feature points (image feature points) included in an image. Further, Patent Document 2 and Non-Patent Document 2 disclose an example of appearance-based methods that utilize the appearance of an object. Non-Patent Document 1 discloses a method of estimating the line of sight by approximating the shape of the iris of the pupil with an ellipse.

Japanese Patent No. 4829141 Japanese Unexamined Patent Publication No. 2009-059257 Japanese Patent No. 5772821

J. Wang, E. Sung, and R. Venkateswarlu, "Eye Gaze Estimation from a Single Image of One Eye," Proc. IEEE ICCV 2003, pp.I-136-143, 2003. X. Zhang, Y. Sugano, M. Fritz and A. Bulling, "Appearance-Based Gaze Estimation in the Wild," Proc. IEEE CVPR 2015, pp. 4511-4520, 2015.

As mentioned above, various methods are used to estimate the line of sight, and each has its own characteristics. However, in either method, the accuracy of estimation may decrease when the orientation of the face, the brightness of the illumination, or the like is a specific condition.

An exemplary object of the present disclosure is to provide techniques for improving the accuracy of image-based gaze estimation.

The information processing apparatus of one aspect of the present disclosure uses a learning result of learning a plurality of images having different conditions to estimate a plurality of lines of sight of a target included in the target image, the conditions thereof, and conditions related to the target image. Based on the above, a determination unit for determining the line of sight of the target is provided.

In one aspect of the information processing method of the present disclosure, a computer estimates a plurality of lines of sight of a target included in a target image by using learning results obtained by learning a plurality of images having different conditions, and the conditions and the target image are related. The line of sight of the target is determined based on the conditions.

The program of one aspect of the present disclosure is based on a process of estimating a plurality of lines of sight of a target included in a target image by using learning results obtained by learning a plurality of images having different conditions, the conditions, and conditions related to the target image. Then, the computer is made to execute the process of determining the line of sight of the target.

According to the present disclosure, the accuracy of line-of-sight estimation based on images is improved.

FIG. 1 is a block diagram showing an example of the configuration of the line-of-sight estimation device. FIG. 2 is a flowchart showing an example of the line-of-sight estimation method. FIG. 3 is a block diagram showing an example of the configuration of the data processing device. FIG. 4 is a flowchart showing an operation example of the data processing device. FIG. 5 is a diagram showing an example of a face image. FIG. 6 is a diagram showing an example of the eye region. FIG. 7 is a conceptual diagram for explaining the imaging conditions of the image in the eye region. FIG. 8 is a diagram showing an example of the effect of the embodiment. FIG. 9 is a block diagram showing an example of the hardware configuration of the computer device.

[First Embodiment]
FIG. 1 is a block diagram showing a configuration of a line-of-sight estimation device 100 according to an embodiment. The line-of-sight estimation device 100 is a device for estimating the line-of-sight included in the face image. The line-of-sight estimation device 100 includes at least an estimation unit 110 and a determination unit 120. However, the line-of-sight estimation device 100 may include other components as needed.

Here, the face image means an image including a part or all of a human face. The face image is an image captured by an imaging device (surveillance camera, built-in camera of electronic device, etc.). The face image may be the captured image itself, or may be a part of the captured image, that is, an image in which a region corresponding to the face is extracted from the captured image.

The estimation unit 110 estimates the line of sight of the face included in the face image. For example, the estimation unit 110 estimates the line of sight, that is, the direction (more accurately, the direction) that the human eye is looking at by estimating the area of the eyes included in the face image. The method of estimating the line of sight by the estimation unit 110 may be any well-known method. For example, the estimation unit 110 can estimate the line of sight by using machine learning such as supervised learning. Specifically, the estimation unit 110 may learn the relationship between the face image and the line of sight using the face image collected in advance.

The estimation unit 110 estimates the line of sight of the face included in the face image with a plurality of estimators. In other words, the estimation unit 110 estimates the line of sight for a single face image using a plurality of estimation methods. The line of sight estimated by a plurality of estimators may have different directions. Therefore, there are a plurality of lines of sight estimated by the estimation unit 110.

Each of the plurality of estimators estimates the line of sight of the face included in the face image based on a predetermined algorithm. The plurality of estimators may be realized by different circuits, or may be realized by a single circuit. The plurality of estimators may be implemented using software.

When the line-of-sight estimation is performed by machine learning, the difference in the estimator can be caused by the difference in the data used for the prior learning. That is, the estimation unit 110 may estimate the line of sight based on learning using a certain data set and learning using another data set. If the data set used for the prior learning is different, the estimation result of the line of sight based on the data set may be different.

The determination unit 120 determines the line of sight of the face included in the face image. Specifically, the determination unit 120 determines the line of sight based on the estimation result of the line of sight by the estimation unit 110. In other words, the determination unit 120 determines a single direction based on the plurality of lines of sight (that is, a plurality of directions) estimated by the estimation unit 110.

More specifically, the determination unit 120 determines the line of sight based on the plurality of lines of sight estimated by the estimation unit 110, the first condition information, and the second condition information. The first condition information includes at least a condition relating to the acquisition of a facial image. In other words, the first condition information includes information indicating how the facial image was captured by the imaging device. The first condition information may represent such a condition by a numerical value representing a physical quantity or the like.

As an example, the first condition information may be information indicating the relative positional relationship between the image pickup apparatus and the person who is the subject. Specifically, the first condition information may indicate the distance between the image pickup device and the person and the height of the image pickup device based on the height of the face of the person. Alternatively, the first condition information may be information indicating the performance of the image pickup apparatus. Specifically, the first condition information may indicate parameters (angle of view, etc.) of the optical system of the image pickup apparatus.

Further, the first condition information may indicate the installation angle of the imaging device. Here, the installation angle of the image pickup device means an angle formed by the direction of the face of the person to be imaged and the direction of the optical axis of the image pickup device. The direction of the face referred to here may be calculated based on the face image or may be determined in advance. For example, when an unspecified number of people passing through a certain passage are imaged by an imaging device, the direction of the face may be the average or typical face direction of the person passing through the passage. In this case, the direction of the face is likely to coincide with the direction of travel of the passage. The installation angle may be represented by a horizontal angle and an elevation / depression angle (also referred to as a vertical angle), or may be represented only by the horizontal angle without the vertical angle.

On the other hand, the second condition information includes at least the conditions corresponding to each of the plurality of estimators of the estimation unit 110. The condition represented by the second condition information can be compared with the condition represented by the first condition information. For example, when the line of sight is estimated based on machine learning based on a data set of face images collected in advance, the condition represented by the second condition information is the imaging when the face image included in the data set is imaged. It may be the distance between the device and the person, the installation angle or the angle of view of the image pickup device (or the average value of any of these).

The determination unit 120 can determine the line of sight by comparing the first condition information and the second condition information. For example, the determination unit 120 includes a condition when the face image is captured and a plurality of conditions corresponding to the plurality of lines of sight estimated by the estimation unit 110 (in other words, a plurality of conditions used for estimating the plurality of line of sights). Compare with multiple conditions) corresponding to the estimator of. The determination unit 120 determines the line of sight based on these comparison results.

Specifically, the determination unit 120 brings the condition represented by the second condition information closer to the line of sight represented by the condition represented by the first condition information among the plurality of lines of sight estimated by the estimation unit 110. To determine the line of sight. For example, the determination unit 120 gives weights to a plurality of lines of sight estimated by the estimation unit 110 according to the comparison result between the first condition information and the second condition information (weighted addition, weighted addition, The line of sight may be determined by performing a weighted average, etc.). The determination unit 120 may compare the first condition information with the second condition information, exclude the estimation result that does not satisfy a certain criterion, and then execute the weighted operation.

FIG. 2 is a flowchart showing a line-of-sight estimation method according to the present embodiment. The line-of-sight estimation device 100 can estimate the line-of-sight of the face included in the face image by executing the process according to this flowchart.

In step S11, the estimation unit 110 estimates a plurality of lines of sight based on the face image. More specifically, the estimation unit 110 calculates a plurality of lines of sight as an estimation result by applying a plurality of estimators to one face image. In other words, it can be said that the estimation unit 110 estimates the line of sight by a plurality of methods.

In step S12, the determination unit 120 determines one line of sight based on the plurality of lines of sight estimated in step S11. More specifically, the determination unit 120 determines the line of sight corresponding to the face image used for the estimation in step S11 based on the first condition information and the second condition information.

As described above, the line-of-sight estimation device 100 of the present embodiment has a configuration in which the line-of-sight of the face included in the face image is estimated by a plurality of estimators and one line of sight is determined based on the estimated plurality of lines of sight. This configuration can reduce the possibility that the accuracy of the estimation will decrease as compared with the case where the line of sight is estimated using a single estimator. Therefore, according to the line-of-sight estimation device 100, it is possible to improve the accuracy of the line-of-sight estimation.

The accuracy of line-of-sight estimation can vary due to a variety of factors. For example, the accuracy of line-of-sight estimation may vary depending on the conditions for capturing a facial image. Specifically, the accuracy of the line-of-sight estimation may vary depending on the relative positional relationship (face orientation, etc.) between the person who is the subject and the imaging device. In addition, the accuracy of line-of-sight estimation may vary depending on the performance of the imaging device itself, lighting conditions such as brightness, and the like. In addition, the accuracy of line-of-sight estimation may decrease under certain conditions depending on the estimation method.

The line-of-sight estimation device 100 can suppress a decrease in accuracy due to the use of a single estimator by determining the line-of-sight based on a plurality of lines of sight estimated using a plurality of estimators. Is. Therefore, according to the line-of-sight estimation device 100, it is possible to obtain a robust estimation result with respect to the condition in which the face image is captured. In other words, the line-of-sight estimation device 100 can realize good line-of-sight estimation for face images captured under various conditions.

[Second Embodiment]
FIG. 3 is a block diagram showing a configuration of a data processing device 200 according to another embodiment. The data processing device 200 corresponds to an example of the line-of-sight estimation device 100 of the first embodiment. The data processing device 200 includes an image acquisition unit 210, a condition acquisition unit 220, an area extraction unit 230, a line-of-sight estimation unit 240, an integration unit 250, and an output unit 260.

The data processing device 200 is a device for estimating the line of sight based on an image. The image referred to here may be either a still image or a moving image. For example, when estimating the line of sight based on a moving image, the facial image may be included in one period of the moving image and may not be included in another period of the moving image. In such a case, the data processing device 200 is configured to estimate the line of sight for the image during the period including the face image and not to estimate the line of sight (do not output the estimation result) for the image during the period not including the face image. You may.

The image acquisition unit 210 acquires an image. For example, the image acquisition unit 210 acquires an image by receiving image data from another device. The other device referred to here may be an imaging device such as a surveillance camera, or a storage device such as a database in which a plurality of image data are recorded. The image acquisition unit 210 supplies image data to the area extraction unit 230.

The image data referred to here is data in which the image is represented by the luminance values of a plurality of pixels. The number of pixels, the number of colors (the number of color components), the number of gradations, and the like of image data are not limited to specific numerical values. The number of pixels and the number of colors of the image data acquired by the image acquisition unit 210 may or may not be predetermined. For convenience of explanation, in the following, the image data acquired by the image acquisition unit 210 is also referred to as “input image data”.

For convenience of explanation, in the following, it is assumed that one image data may include only one face image and does not include a plurality of face images. However, when one image data includes a plurality of face images, the data processing device 200 suffices to execute the processing described later for each of the plurality of face images.

The image acquisition unit 210 may supply the input image data to the area extraction unit 230 as it is, or may process the input image data and then supply the input image data to the area extraction unit 230. For example, the image acquisition unit 210 detects a human face from an image represented by the image data, generates image data representing a face image that is a part of the image, and extracts the generated image data as an area. It may be supplied to the unit 230.

Alternatively, the image acquisition unit 210 may convert the image data so that the number of colors and the number of gradations of the image become predetermined numerical values, and then supply the image data to the area extraction unit 230. For example, the image acquisition unit 210 converts image data representing a color image by a plurality of color components such as R (red), G (green), and B (blue) into image data representing a single component grayscale image. You may.

The condition acquisition unit 220 acquires camera information. The camera information is data including imaging conditions of an image acquired by the image acquisition unit 210. The imaging condition referred to here is, for example, the installation angle of the imaging device. In addition, the imaging conditions may include parameters (angle of view, etc.) of the lens of the imaging apparatus and an estimated range of the line of sight at the time of imaging. The camera information corresponds to an example of the first condition information of the first embodiment.

The camera information may be input together with the image data. For example, the camera information may be described as metadata included in the image data. Alternatively, the camera information may be input by a user operation. In this case, the condition acquisition unit 220 accepts the user's operation via the keyboard or the touch screen display.

The area extraction unit 230 extracts a specific area from the image data. The area extraction unit 230 extracts a region necessary for the line-of-sight estimation by the line-of-sight estimation unit 240. In the present embodiment, the region extraction unit 230 extracts a region around the eyes in particular from the face image. In the following, the region extracted by the region extraction unit 230 is referred to as an “eye region”. The eye area is, for example, a rectangle of a predetermined size that includes both human eyes.

The area extraction unit 230 can extract the eye area based on the image features peculiar to a general face image. The region extraction unit 230 can extract the eye region by detecting, for example, the iris (so-called pupil), sclera (so-called white eye), inner eye angle (so-called inner corner of the eye), outer eye angle (so-called outer corner of the eye), eyebrows, and the like. .. For the extraction of the eye region, a well-known feature point detection method such as the method described in Patent Document 3 can be used.

The region extraction unit 230 may execute preprocessing according to the method of estimating the line of sight. For example, the region extraction unit 230 arranges that the right eye and the left eye are positioned horizontally when the extracted eye region is not horizontal, that is, when the height of the center of the right eye and the height of the center of the left eye in the eye region do not match. You may rotate the image. Further, the region extraction unit 230 may enlarge or reduce the image so that the size of the eye region becomes a constant size. Well-known image processing can be applied to image rotation processing, enlargement processing (that is, interpolation processing), and reduction processing (that is, thinning processing). When such image processing is executed, it is not necessary to learn the scale and inclination of the eye region by stabilizing them, so that it is possible to improve the estimation accuracy of the line of sight.

The line-of-sight estimation unit 240 estimates the line-of-sight of the face included in the face image. More specifically, the line-of-sight estimation unit 240 includes line-of _- sight estimators 241, 241, ..., _{241 n .} The value of n here, that is, the total number of line-of-sight estimators is not limited to a specific numerical value as long as it is "2" or more. In the following, the line-of _- sight estimators 241, 241, ..., 241 _n are collectively referred to as "line-of-sight estimators 241" when they do not need to be distinguished from each _other . The line-of-sight estimation unit 240 corresponds to an example of the estimation unit 110 of the first embodiment.

The line-of-sight estimator 241 estimates the line of sight using the eye region extracted by the region extraction unit 230. In the present embodiment, the line-of-sight estimator 241 is configured to learn the line-of-sight of the eyes included in the face image in advance by machine learning and estimate the line-of-sight using the learning result.

The line-of _- sight estimators 241, 241, ..., 241 _{n have} different line-of-sight estimation methods. _{For example, the line-of-sight estimators 241, 241, ..., 241 n have different} facial images used as samples in machine learning. Alternatively, the line-of _- sight estimators 241, 241, ..., 241 _{n may} have different machine learning algorithms.

The integration unit 250 integrates the estimation results estimated by the line-of-sight estimation unit 240, more specifically, the line-of _- sight estimators 241, 241, ..., _{241 n .} In other words, the integration unit 250 determines the line of sight in a single direction based _on the plurality of lines of sight estimated by the line of sight _{estimators 241, 241, ..., 241 n .} The integration unit 250 corresponds to an example of the determination unit 120 of the first embodiment.

The integration unit 250 integrates a plurality of lines of sight based on camera information and learning information. Here, the learning information is data including conditions related to learning of the line-of _- sight estimators 241, 241, ..., 241 _n , _respectively . The learning information represents, for example, the imaging conditions of the imaging apparatus used for learning each of the line-of _- sight _{estimators 241, 241, ..., 241 n .} It is assumed that the learning information is stored in the data processing device 200. The learning information corresponds to an example of the second conditional information of the first embodiment.

The integration unit 250 _{performs a weighted operation using weights determined for each of the line-of-sight estimators 241, 241, ..., 241 n , and} the line _- of _- sight estimators 241, 241, ..., Combine multiple gazes estimated by each of 241 _n . At this time, the integration unit 250 can determine the weight for each line of sight by using the camera information and the learning information. The weighted operation by the integration unit 250 will be described in detail in an operation example described later.

The output unit 260 outputs data indicating the line of sight integrated by the integration unit 250 (hereinafter, also referred to as “line of sight data”). The line-of-sight data represents, for example, the line-of-sight integrated by the integration unit 250, in other words, the direction determined by the integration unit 250 according to a predetermined rule. The output by the output unit 260 may be to supply the line-of-sight data to another device such as a display device, or to write the line-of-sight data to a storage medium included in the data processing device 200.

The configuration of the data processing device 200 is as described above. Under this configuration, the data processing device 200 estimates the line of sight based on the image data. The data processing device 200 operates, for example, as in the following operation example. However, the specific operation of the data processing device 200 is not limited to this operation example.

FIG. 4 is a flowchart showing an operation example of the data processing device 200. The data processing device 200 can execute the process shown in FIG. 4 at an appropriate timing such as a timing specified by the user or a timing when image data is transmitted from another device. In this example, the image represented by the image data is assumed to include a face image. Further, it is assumed that the camera information and the learning information are the installation angles of the imaging device. Further, the coordinates of the image referred to here are represented by a Cartesian coordinate system having a predetermined position as the origin.

In step S21, the image acquisition unit 210 acquires image data. In step S22, the condition acquisition unit 220 acquires camera information. The processes of steps S21 and S22 may be executed in the reverse order of FIG. 4, or may be executed at the same time (that is, in parallel).

In step S23, the area extraction unit 230 extracts the eye area using the image data acquired in step S21. In this example, the region extraction unit 230 identifies the coordinates of the center of the iris of the right eye and the coordinates of the center of the iris of the left eye. The region extraction unit 230 determines the eye region based on these coordinates. For convenience of explanation, in the following, the coordinates of the center of the iris of the right eye are also referred to as "center coordinates of the right eye", and the coordinates of the center of the iris of the left eye are also referred to as "center coordinates of the left eye".

Specifically, the region extraction unit 230 uses the midpoint of the line segment connecting the center coordinates of the right eye and the center coordinates of the left eye as the center of the eye region. The area extraction unit 230 has a width of the eye region that is twice the length of the line segment connecting the center coordinates of the right eye and the center coordinates of the left eye (hereinafter, also referred to as “interpupillary distance”). The height of the eye region is 0.75 times the interpupillary distance. The region extraction unit 230 cuts out a rectangular region defined by the center, width, and height thus determined as an eye region from the image.

In addition, the region extraction unit 230 may execute preprocessing for correcting the inclination, width, and height of the eye region so that subsequent processing can be facilitated. More specifically, if the center coordinates of the right eye and the center coordinates of the left eye are not horizontal, the area extraction unit 230 makes these coordinates horizontal, and the number of pixels in the width direction and the height direction of the eye area is a predetermined number of pixels. If not, the eye area is enlarged or reduced.

FIG. 5 is a diagram showing an example of a face image. FIG. 6 is a diagram showing an example of an eye region extracted from this face image. The eye area 600 shown in FIG. 6 corresponds to a part of the face image 500 shown in FIG. Specifically, the eye region 600 corresponds to the region 510 surrounded by the broken line in the face image 500. However, the number of pixels and the inclination of the eye area 600 do not always match the area 510 when the above-mentioned preprocessing is executed.

In step S24, the line-of-sight estimation unit 240 estimates the line of sight based on the eye region extracted in step S23. The line-of-sight estimation unit 240 estimates the line-of-sight using the line-of-sight estimators 241 ₁ to 241 _n learned in advance. In this example, the line-of-sight estimators 241 ₁ to 241 _n estimate the line of sight based on the amount of image features detected from the eye region.

The image feature amount in this example is a feature amount related to the gradient of the brightness of the image. As a feature amount related to the gradient of brightness, for example, a HOG (Histograms of Oriented Gradients) feature amount is known. The image feature amount in this example indicates the direction and magnitude of the change in brightness in the eye region by a predetermined number of dimensions (for example, hundreds to thousands). In the following, this image feature amount is also represented by a column vector f having a predetermined number of elements.

The line-of-sight estimators 241 ₁ to 241 _n calculate the line-of-sight (g _x , _gy ) using the following equation (1). Here, the line of sight (g _x , _gy ) indicates the direction of the line of sight with respect to the direction of the face by a horizontal angle and an elevation / depression angle. Of these, g _x represents the horizontal angle and satisfies −90 ≦ g _x ≦ 90 (unit is [deg]). Further, _gy represents an elevation / depression angle and satisfies −90 ≦ _gy ≦ 90 (unit is [deg]).

The line of sight (g _x , _gy ) is based on the case where (g _x , _gy ) = (0,0), that is, the line of sight is facing directly in front of the face, and the line of sight is deviated from the front. Is represented by the horizontal angle and the elevation / depression angle. For example, (g _x , _gy ) = (0, +90) when the line of sight is directly upward, and (g _x , _gy ) = (0, -90) when the line of sight is directly downward. ). Further, when the line of sight is directed to the side (right), (g _x , gy) = (+90,0), and when the line of sight is directed to the side (left), (g _x , _gy ) ₌ . (-90,0).

The front orientation referred to here depends on the orientation of the face represented by the face image. That is, the front surface referred to here changes according to the direction of the face. Therefore, the direction actually seen by the photographed person is not specified only by the line of sight (g _x , _gy ), but is specified by the line of sight (g _x , _gy ) and the direction of the person's face. Will be done.

In equation (1), u _x and u _y are weight vectors. Each of the weight vectors u _x and u _y is a row vector having the same number of elements as the image feature amount f, and the inner product with the image feature amount f can be calculated. The weight vectors u _x and u _y can be different for each line-of-sight estimator 241 ₁ to 241 _n . The weight vectors u _x and u _y can be learned in advance by well-known methods such as support vector regression and linear regression by the least squares method. For learning in the line-of-sight estimators 241 ₁ to 241 _n , in general, a large number of sets of an image of the eye region extracted as in step S23 and information indicating the actual line-of-sight of the image (that is, correct answer information) are prepared. Is executed.

In this example, the line-of-sight estimators 241 ₁ to 241 _n perform learning using images of eye regions having different imaging conditions. Specifically, for learning the line-of-sight estimators 241 ₁ to 241 _n , images of the eye region captured by imaging devices having different installation angles are used.

FIG. 7 is a conceptual diagram for explaining the imaging conditions of the image in the eye region. Here, the number of the line-of-sight estimator 241 (that is, the value of n) is assumed to be "4". In the example of FIG. 7, the cameras 710, 720, 730, and 740 are image pickup devices that capture a facial image of a person 700. The camera 710 captures a face image from the upper right. The camera 720 captures a face image from the upper left. The camera 730 captures a face image from the lower right. The camera 740 captures a face image from the lower left. The person 700 may be a different person for each image, or may be the same person in any of the images. Further, it is assumed that the person 700 faces the same direction (front) at the time of imaging.

The line-of _- sight estimator 2411 uses the face image captured by the camera 710 for learning. The line-of _- sight estimator 2412 uses the face image captured by the camera 720 for learning. The line-of _- sight estimator 2413 uses the face image captured by the camera 730 for learning. The line-of _- sight estimator 2414 uses the face image captured by the camera 740 for learning. Then, the line-of _- sight estimators 241 ₁ to 2414 have different installation angles of the image pickup devices corresponding to the face images used for learning.

Since the line-of-sight estimators 241 ₁ to 241 _n have different machine learning conditions (here, the imaging conditions of the face image used for learning), the estimation result is obtained even if the line-of-sight is estimated using images of the same eye region. Can be different. In other words, in the line-of-sight estimators 241 ₁ to 241 _n , since the weight vectors ux and _yy in the equation (1) can be different from each other, the line-of-sight (g _x , _{gy )} even if the values of the image feature values f are the same. ) May be different. In the following, the line of sight estimated by the line _- of-sight estimator 2411 is (g ( ₁ ⁾ _x , g ⁽¹⁾ _y ), and the line of sight estimated by the line-of-sight estimator 2412 is (g ⁽²⁾ _x , g ⁽ 2). ⁾ _Y ), ..., The line of sight estimated by the line of sight estimator 241 _n is also referred to as (g ⁽ⁿ⁾ _x , g ⁽ⁿ⁾ _y ).

In step S25, the integrating unit 250 integrates the line of sight (g ⁽¹⁾ _x , g ⁽¹⁾ _y ) to (g ⁽ⁿ⁾ _x , g ⁽ⁿ⁾ _y ) estimated in step S24. That is, the integrating unit 250 calculates a single line of sight based on the line of sight (g ⁽¹⁾ _x , g ⁽¹⁾ _y ) to (g ⁽ⁿ⁾ _x , g ⁽ⁿ⁾ _y ) estimated in step S24. do. Here, the integration unit 250 calculates the weight based on the camera information and the learning information. The camera information and learning information here are the installation angles of the imaging device.

The integration unit 250 calculates the weight _wi corresponding to the line-of-sight estimator 241 _i using the following equation (2). Here, c _i , c _j , and c _t are all vectors representing the installation angles of the image pickup apparatus. c _i (or c _j ) is the direction of each of the faces represented by the plurality of face images used for learning the line-of-sight estimator 241 _i (or 241 _j ) and the optical axis direction of the image pickup device that captured the face image. Shows the average value of the angles formed by. On the other hand, _ct indicates an angle formed by the direction of the face represented by the face image included in the input image data and the direction of the optical axis of the image pickup device that captured the face image. c _i and c _j are examples of learning information. On the other hand, _ct is an example of camera information. Also, α is an appropriate coefficient greater than 0.

For example, assuming that n = 2, that is, the number of line-of-sight estimators 241 is _two , the weights w1 and w2 can be expressed by the following equations ( ₃ ) and (4). The weight w _i increases as the difference between the learning information c _i and the camera information c _t becomes smaller.

After calculating the weight wi in this way, the integrating unit 250 calculates the line of sight (G _x , G _{y )} according to the following equation (5). As shown in equation (5), the line of sight (G _x , G _y ) is the weighted average of the line of sight (g ⁽¹⁾ _x , g ⁽¹⁾ _y ) to (g ⁽ⁿ⁾ _x , g ⁽ⁿ⁾ _y ). be. The denominator on the right side of equation (5) is "1" here (see equation (2)).

In step S26, the output unit 260 outputs line-of-sight data indicating the line-of-sight (G _x , G _y ) calculated by the integration unit 250. This line-of-sight data is visualized by, for example, a display device. The line of sight indicated by the line-of-sight data may be displayed numerically, or an arrow indicating the line of sight may be superimposed on the face image.

FIG. 8 is a diagram showing an example of the effect of the present embodiment. In this example, the number of line-of-sight estimators 241 is two. This example is an example in which the line of sight is estimated using a moving image of one subject looking at two gazing points in order. The learning information (installation angle) corresponding to the line _- of-sight estimator 2411 is (+2.3 [deg], +5.5 [deg]). The learning information (installation angle) corresponding to the line _- of-sight estimator 2412 is (+1.2 [deg], -22.7 [deg]). The camera information (installation angle) is (0 [deg], 0 [deg]). The coefficient α is “0.04” here.

In FIG. 8, graph 810 represents the line of sight (g ⁽¹⁾ _x , g ⁽¹⁾ _{y )} estimated by the line of sight estimator 2411. Graph 820 represents the line of sight (g ⁽²⁾ _x , g ⁽²⁾ _{y )} estimated by the line of sight estimator 2412. Graph 830 represents the line of sight (G _x , G _y ) integrated by the integration unit 250. Graph 840 represents the actual line of sight of the subject.

As shown in FIG. 8, the line of sight (G _x , G _y ) integrated by the integration unit 250 is the line of sight (g ⁽¹⁾ _x , g ⁽¹⁾ _{y )} and the line of sight estimated by the line of sight estimator 2411. Compared with the line of sight (g ⁽²⁾ _x , g ⁽²⁾ _{y )} estimated by the estimator 2412, the error from the actual line of sight is small. Therefore, it can be said that the data processing device 200 has improved the accuracy of the line-of-sight estimation as compared with the case of using the single line-of-sight estimator 241.

In this example, comparing graph 810, i.e. line of sight (g ⁽¹⁾ _x , g ⁽¹⁾ _y ) with graph 820, i.e. line of sight (g ⁽²⁾ _x , g ⁽²⁾ _y ), gaze (g ⁽ 1) x, g (2) y) ¹⁾ It can be said that _x , g ⁽¹⁾ _y ) is an estimation result closer to the actual line of sight (graph 840). Here, when the camera information and the learning information corresponding to the line _- of-sight estimators 241 ₁ and 2412 are compared, it can be said that the learning information corresponding to the line-of-sight estimator 2411 has _a smaller difference from the camera information. According to the weighted addition of the present embodiment (see equations (2) to (5)), the weight _wi becomes larger as the line-of-sight estimator 241 has a smaller difference between the learning information and the camera information. Therefore, the line of sight represented by the line-of-sight data, that is, the final estimation result, approaches the line of sight estimated by the line-of-sight estimator 241 whose installation angle included in the imaging conditions is closer.

In this example, it can be said that the estimation accuracy of the line-of-sight estimator 241 depends on the imaging conditions in the prior learning and the imaging conditions of the face image to be estimated by the data processing device 200, that is, the facial image represented by the input image data. More specifically, the estimation accuracy of the line-of-sight estimator 241 is the relative positional relationship (installation angle) between the face image and the imaging device in the prior learning, and the imaging of the face image to be estimated and the face image. It can be said that it depends on the degree of approximation to the relative positional relationship (installation angle) with the device. However, the relative positional relationship between the face image to be estimated and the imaging device is not always constant, and may vary depending on the imaging method.

According to the line-of-sight estimation method of the present embodiment, by executing the weighted addition as in the equations (2) to (5), the estimation results by the plurality of line-of-sight estimators 241 learned using different imaging conditions can be obtained. Can be integrated. Therefore, according to the line-of-sight estimation method of the present embodiment, even if the image pickup condition of the face image represented by the input image data and the image pickup condition in the prior learning of the plurality of line-of-sight estimators 241 do not match, the line-of-sight with good accuracy It can be estimated. In addition, according to the line-of-sight estimation method of the present embodiment, it is possible to perform line-of-sight estimation with good accuracy even if the relative positional relationship between the face image represented by the input image data and the imaging device is not constant.

As described above, the data processing device 200 of the present embodiment has a configuration in which the line of sight of the face included in the face image is estimated by the plurality of line of sight estimators 241 and the estimated plurality of line of sight are integrated. With this configuration, the data processing device 200 can exert the same effects as the line-of-sight estimation device 100 of the first embodiment.

Further, the data processing device 200 has a configuration in which the line of sight is integrated by executing a weighted operation according to a weight determined according to the learning information representing the imaging condition and the camera information. This configuration makes it possible to give weights according to the imaging conditions to the plurality of lines of sight estimated by the plurality of line-of-sight estimators 241. Therefore, the data processing device 200 can improve the accuracy of the line-of-sight estimation as compared with the case where such a weight is not applied.

Further, the data processing device 200 has a configuration for determining the weight according to the comparison result of the learning information representing the imaging condition and the camera information. More specifically, the data processing device 200 is a face image in which the installation angle of the image pickup device at the time of learning of the line-of-sight estimator 241 is represented by input image data with respect to a plurality of lines of sight estimated by the plurality of line-of-sight estimators 241. The closer to the installation angle of the image pickup device that captured the image, the heavier the weight. With such a configuration, the data processing device 200 can bring the line of sight represented by the output line-of-sight data closer to the line of sight estimated by the line-of-sight estimator 241 having a closer installation angle.

[Modification example]
For example, the following modifications can be applied to the above-described embodiment. These modifications can be combined as needed.

(Modification example 1)
The determination unit 120 can estimate the direction of the face by using a well-known face orientation estimation technique. The determination unit 120 may calculate the installation angle of the image pickup device based on the angle formed by the face direction estimated in this way and the optical axis direction of the image pickup device.

(Modification 2)
The camera information and the learning information may include information indicating the type of the imaging device used for capturing the facial image. The type of the image pickup apparatus referred to here represents, for example, the model of the image pickup apparatus and the wavelength band of light having the sensitivity of the image pickup apparatus.

For example, the image pickup apparatus may include a visible light camera that captures images with visible light and a near-infrared light camera that captures images with near-infrared light. In such a case, when the image pickup device used for learning the line-of-sight estimator 241 also includes a visible light camera and a near-infrared light camera, the image pickup device used for capturing the input face image and learning Differences may occur with the imaging device used. For example, if the image pickup device used to capture the input face image is a near-infrared light camera, the estimation result by the line-of-sight estimator 241 whose image pickup device used for learning is a near-infrared light camera is better. It can be said that it is likely to be reliable (that is, accuracy is guaranteed).

In such a case, the integration unit 250 increases the weight _wi corresponding to the line-of-sight estimator 241 _i in which the type of the image pickup device used for capturing the input face image and the type of the image pickup device used for learning match. However, the weight _wi corresponding to the line-of-sight estimator 241 _i , which is not the case, is reduced. In this way, it is possible to more strongly reflect the estimation result of the line-of-sight estimator 241 _i , which is used for learning by the same type of image pickup device as the image pickup device used for capturing the input face image, in the line-of-sight data. be.

Further, the camera information and the learning information may be parameters of the optical system of the imaging device. For example, the camera information and the learning information may include the horizontal and vertical angles of view of the lens as parameters. In this case, the integration unit 250 can calculate the weight by the same calculation as in the equation (2) by using the vector having such a parameter as an element as the camera information and the learning information.

(Modification example 3)
The method of weighted operation in the second embodiment is not limited to the above-mentioned operation example. For example, the integration unit 250 does not use a part of the weight wi calculated by the equation (2), and the line of sight ( _g ⁽¹⁾ _x , g ⁽¹⁾ _y ) to (g ⁽ⁿ⁾ _x , g ⁽ n). ⁾ _Y ) may be integrated. Specifically, the integration unit 250 may replace weights wi that are not equal to or greater than a predetermined threshold value (or a predetermined number in descending order of value) with “ ₀ ”. This substitution corresponds to truncating the weights _wi that have little effect on the final estimation result. Further, in this case, the integration unit 250 may recalculate the ratio of each weight (denominator of the equation (2)) so that the sum of the weights _wi after truncation becomes “1”.

Further, in the equation (2), the integration unit 250 is different from exp (-α || c _{i -ct} ||), which decreases monotonically with the increase of || c _{i -ct} ||. You may use a function. For example, the integration unit 250 may calculate the weight _wi using the following equation (6). Here, max (a, b) represents a function that returns a larger value of a and b. Further, β is a constant of 0 or more.

Alternatively, the integrating unit 250 cuts off a part of the line of sight (g ⁽¹⁾ _x , g ⁽¹⁾ _y ) to (g ⁽ⁿ⁾ _x , g ⁽ⁿ⁾ _y ), and then the line of sight (G) of the equation (5). _x , G _y ) may be calculated. For example, the integration unit 250 excludes outliers when the line of sight (g ⁽¹⁾ _x , g ⁽¹⁾ _y ) to (g ⁽ⁿ⁾ _x , g ⁽ⁿ⁾ _y ) contains outliers. The calculation of equation (5) may be executed. This is because the line of sight corresponding to the outliers is considered to be the line of sight that failed to be estimated. The outliers referred to here are values that are significantly different from the other values of the line of sight (g ⁽¹⁾ _x , g ⁽¹⁾ _y ) to (g ⁽ⁿ⁾ _x , g ⁽ⁿ⁾ _y ). Outliers are specified, for example, based on the Euclidean distance between the lines of sight when the line of sight (g ⁽¹⁾ _x , g ⁽¹⁾ _y ) to (g ⁽ⁿ⁾ _x , g ⁽ⁿ⁾ _y ) is regarded as a vector. Will be done.

(Modification example 4)
As mentioned above, the learning information and the camera information may include an estimated range of line of sight. The line-of-sight range referred to here indicates the range of the line-of-sight to be estimated in the camera information, and indicates the range of the line-of-sight used for learning in the line-of-sight estimator 241 in the learning information. As for the range of the line of sight, for example, the deviation from the front is represented by a numerical value in the range of −90 to +90 [deg], similarly to the line of sight (g _x , _gy ). The learning information and the camera information may represent the range of the line of sight by both the horizontal angle and the elevation / depression angle, or may be represented by one of them.

When such learning information and camera information are used, the integration unit 250 can calculate the weight based on the ratio of overlapping line-of-sight ranges (hereinafter, also referred to as “overlap rate”). Here, the duplication rate represents the ratio of the line-of-sight range included in at least one of the learning information and the camera information to the line-of-sight range included in both the learning information and the camera information.

For example, when the line-of-sight range represented by the learning information and the line-of-sight range represented by the camera information completely match, the duplication rate is "1.0". On the other hand, when the line-of-sight range represented by the learning information and the line-of-sight range represented by the camera information do not match at all, the duplication rate is "0". More specifically, the range of the horizontal line of sight represented by the learning information of the line _- of-sight estimator 2411 is -10 to +5 [deg], and the range of the horizontal line of sight represented by the camera information is -10 to +10 [deg]. ], The horizontal overlap rate is "0.75 (= 15/20)".

When such learning information and camera information are used, the integration unit 250 can use horizontal and vertical overlap _rates as the learning information c _i , c _j and camera information ct of the equation (2). The integration unit 250 may use the overlap rate instead of the installation angle of the imaging angle, or may use the overlap rate in addition to the installation angle of the imaging angle. For example, the learning information c _i , c _j and the camera information c _t have four components (horizontal component and vertical component of the installation angle and horizontal component of the overlap rate) when both the installation angle and the overlap rate of the imaging angle are used. And the vertical component) vector.

(Modification 5)
The area extraction unit 230 does not have to specify the center coordinates of the right eye and the left eye and the eye area by calculation. For example, the center coordinates of the right eye and the left eye and the eye area may be input by the user. In this case, the data processing device 200 can specify the center coordinates of the right eye and the left eye and the eye area based on the input of the user.

(Modification 6)
The shape of the eye region is not necessarily limited to a rectangle. For example, the region extraction unit 230 may exclude a region (for example, a nose region) that does not directly affect the estimation of the line of sight from the above-mentioned eye region (see FIG. 6). Also, the eye area does not necessarily have to include both eyes. For example, the region extraction unit 230 may extract as a region and an eye region that include one of the right eye or the left eye and does not include the other.

(Modification 7)
The learning by the line-of-sight estimation unit 240 is not limited to the above-mentioned example. For example, the line-of-sight estimation unit 240 may learn a non-linear function for estimating the line-of-sight by a group learning algorithm such as Random Forest.

(Modification 8)
The use of the line of sight estimated by the line-of-sight estimation device 100 (or the data processing device 200) is not particularly limited. For example, the line-of-sight estimation device 100 may be applied to a system that estimates the line-of-sight of a person captured by a surveillance camera installed in a retail store such as a convenience store and detects a suspicious person. Further, the line-of-sight estimation device 100 may be applied to a system that estimates the user's interest / interest based on the user's line of sight with respect to the screen on which the information is displayed. Alternatively, the line-of-sight estimation device 100 may be applied to an electronic device that can be operated by the movement of the line of sight, or to support driving of an automobile or the like.

(Modification 9)
The specific hardware configuration of the device (line-of-sight estimation device 100 or data processing device 200) according to the present disclosure includes various variations and is not limited to a specific configuration. For example, the apparatus according to the present disclosure may be realized by using software, or may be configured to share various processes by using a plurality of hardware.

FIG. 9 is a block diagram showing an example of the hardware configuration of the computer device 300 that realizes the device according to the present disclosure. The computer device 300 includes a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, a storage device 304, a drive device 305, a communication interface 306, and an input / output interface. It is configured to include 307. The apparatus according to the present disclosure can be realized by the configuration (or a part thereof) shown in FIG.

The CPU 301 executes the program 308 using the RAM 303. The program 308 may be stored in the ROM 302. Further, the program 308 may be recorded on a recording medium 309 such as a memory card and read by the drive device 305, or may be transmitted from an external device via the network 310. The communication interface 306 exchanges data with an external device via the network 310. The input / output interface 307 exchanges data with peripheral devices (input device, display device, etc.). The communication interface 306 and the input / output interface 307 can function as components for acquiring or outputting data.

The components of the apparatus according to the present disclosure may be composed of a single circuit (processor or the like) or a combination of a plurality of circuits. The circuitry referred to here may be either dedicated or general purpose. For example, the apparatus according to the present disclosure may be partially realized by a dedicated processor and the other part may be realized by a general-purpose processor.

The configuration described as a single device in the above-described embodiment may be distributed to a plurality of devices. For example, the line-of-sight estimation device 100 may be realized by the collaboration of a plurality of computer devices by using cloud computing technology or the like. Further, the line-of-sight estimators 241 ₁ to 241 _n may be realized by different computer devices.

As described above, the present invention has been described as a model example of the above-described embodiments and modifications. However, the present invention is not limited to these embodiments and modifications. The present invention may include embodiments within the scope of the present invention to which so-called variations or applications that can be grasped by those skilled in the art are applied. In addition, the present invention may include embodiments in which the matters described in the present specification are appropriately combined or replaced as necessary. For example, the matters described using a particular embodiment may be applied to other embodiments as long as they do not cause inconsistency.

[Additional Notes]
Part or all of this disclosure may also be described as in the appendix below. However, the present disclosure is not necessarily limited to this additional aspect.
(Appendix 1)
Estimating means for estimating the line of sight of the face included in the face image with multiple estimators,
The first condition information including the condition relating to the imaging of the face image, the plurality of second condition information including the condition each corresponding to any of the plurality of estimators, and the estimated plurality of lines of sight. A line-of-sight estimation device including a determination means for determining the line-of-sight of the face based on the above.
(Appendix 2)
The line-of-sight estimation device according to Appendix 1, wherein the conditions include imaging conditions by an imaging means.
(Appendix 3)
The line-of-sight estimation device according to Appendix 1 or Appendix 2, wherein the condition includes an estimated range of the line-of-sight.
(Appendix 4)
The determination means is a weight determined for each of the plurality of lines of sight estimated by the plurality of estimators, and includes the second condition information and the first condition information corresponding to the estimator. The line-of-sight estimation device according to any one of Supplementary note 1 to Supplementary note 3, which executes a weighted operation according to a weight determined according to.
(Appendix 5)
The line-of-sight estimation device according to Appendix 4, wherein the determination means determines the weight based on a comparison result between the second condition information and the first condition information.
(Appendix 6)
The line-of-sight estimation device according to Appendix 5, wherein the determination means increases the weight as the second condition information is closer to the first condition information.
(Appendix 7)
The line-of-sight estimation device according to any one of Supplementary note 1 to Supplementary note 6, wherein the plurality of estimators are learned based on facial images whose conditions are different from each other.
(Appendix 8)
The first acquisition means for acquiring the face image and
A second acquisition means for acquiring the first condition information and
An extraction means for extracting the area around the eyes from the acquired face image,
Further provided with an output means for outputting line-of-sight information indicating the line-of-sight determined by the determination means.
The line-of-sight estimation device according to any one of Supplementary note 1 to Supplementary note 7, wherein the estimation means estimates the line-of-sight of the face using the region of the face image.
(Appendix 9)
Estimate the line of sight of the face included in the face image with multiple estimators,
The first condition information including the condition relating to the imaging of the face image, the plurality of second condition information including the condition each corresponding to any of the plurality of estimators, and the estimated plurality of lines of sight. A line-of-sight estimation method for determining the line-of-sight of the face based on.
(Appendix 10)
The line-of-sight estimation method according to Appendix 9, wherein the first condition information and the second condition information include information indicating imaging conditions by the imaging means.
(Appendix 11)
On the computer
Processing to estimate the line of sight of the face included in the face image with multiple estimators,
The first condition information including the condition relating to the imaging of the face image, the plurality of second condition information including the condition each corresponding to any of the plurality of estimators, and the estimated plurality of lines of sight. A computer-readable program recording medium that records a program for executing the process of determining the line of sight of the face based on the above.
(Appendix 12)
The program recording medium according to Appendix 11, wherein the first condition information and the second condition information include information indicating imaging conditions by the imaging means.

100 Line-of-sight estimation device 110 Estimating unit 120 Determining unit 200 Data processing device 210 Image acquisition unit 220 Condition acquisition unit 230 Area extraction unit 240 Line-of-sight estimation unit 241 Line-of-sight estimation unit 250 Integration unit 260 Output unit 300 Computer device

Claims (7)

An estimation means for estimating a plurality of lines of sight of an object included in a target image by using a plurality of estimators that have learned a plurality of images under different conditions, and an estimation means.
A weighted operation is executed according to the weights determined for each of the plurality of lines of sight estimated by the plurality of estimators and determined according to the conditions and the conditions relating to the target image. Then , the determination means for determining the line of sight of the object and
Information processing device equipped with. The plurality of estimators include a first estimator learned under the first condition and a second estimator learned under the second condition.
The estimation means estimates a plurality of lines of sight of the target by using the first estimator and the second estimator.
The information processing device according to claim 1. The first estimator is trained using an image captured from the first image pickup device, and is trained.
The second estimator is learned using an image captured by the second image pickup device.
The information processing device according to claim 2. The first imaging device and the second imaging device have different installation angles.
The information processing device according to claim 3. The first imaging device and the second imaging device have different performances.
The information processing apparatus according to claim 3 or 4. The computer
Using multiple estimators that have learned multiple images with different conditions, multiple gazes of the target included in the target image are estimated.
A weighted operation is executed according to the weights determined for each of the plurality of lines of sight estimated by the plurality of estimators and determined according to the conditions and the conditions relating to the target image. Then , the line of sight of the target is determined.
Information processing method. A process of estimating multiple lines of sight of a target included in a target image using a plurality of estimators that have learned a plurality of images under different conditions, and
A weighted operation is executed according to the weights determined for each of the plurality of lines of sight estimated by the plurality of estimators and determined according to the conditions and the conditions relating to the target image. Then , the process of determining the line of sight of the target and
A program that causes a computer to run.

Images