KR20240019030A - Learning method and learning device, and testing method and testing device for gaze detection model based on deep learning - Google Patents

Abstract

The present invention provides a method for learning a deep learning-based gaze direction detection model for detecting a person's gaze direction. (a) When at least one first learning image is acquired, the learning device converts the first learning image into a body. At least one first body that is input to a convolutional layer and causes the body convolutional layer to perform a convolutional operation on the first training image at least once to extract body features of the first person included in the first training image. A feature map is generated, and the first body feature map is input to a body FC (Fully Connected) layer so that the body FC layer performs an FC operation on the first body feature map at least once to determine the front of the body of the first person. Output at least one predicted body direction information that predicts the direction it is facing, and use labeled body direction information included in the predicted body direction information and first ground truth information corresponding to the first learning image. Generating at least one body direction loss and learning the body FC layer and the body convolutional layer using the body direction loss; and (b) when at least one second learning image is acquired, the learning device inputs the second learning image into the body convolutional layer to cause the body convolutional layer to obtain at least the second learning image. Convolutional operation is performed once to generate at least one second body feature map extracting the body features of a second person included in the second learning image, and the second learning image is input to the head convolutional layer to generate the second body feature map. A head convolutional layer performs a convolutional operation on the second learning image at least once to generate at least one first head feature map from which head features of the second person are extracted, and the second body feature map and the Concatenate the first head feature map to create a first integrated feature map, input the first integrated feature map to the head FC layer, and have the head FC layer perform FC operation on the first integrated feature map at least once. To output at least one first predicted head direction information predicting the direction in which the front of the second person's head is facing, and second ground truth corresponding to the first predicted head direction information and the second learning image. generating at least one head direction loss using labeled head direction information included in and learning the head FC layer and the head convolutional layer using the head direction loss; It relates to a method of including .

Description

A learning method and learning device for learning a deep learning-based gaze direction detection model that detects the gaze direction, and a test method and test device using the same {LEARNING METHOD AND LEARNING DEVICE, AND TESTING METHOD AND TESTING DEVICE FOR GAZE DETECTION MODEL BASED ON DEEP LEARNING }

The present invention relates to a method of learning a deep learning-based gaze direction detection model for detecting gaze direction, and more specifically, deep learning to detect a person's gaze direction using a person's body direction information and head direction information. It relates to a learning method and learning device for learning a based gaze direction detection model, and a testing method and testing device using the same.

Gaze information can be used in various fields. For example, in the marketing field, gaze direction information can be used to analyze whether advertising is effective.

In the past, the user's facial image was captured through a camera mounted on the user terminal, and the user's gaze direction information was obtained from the user's facial image.

However, there was a problem in that the gaze direction detection method described above could only be used in situations with extremely limited conditions (for example, a situation where a user was watching specific content through a mobile phone equipped with a camera).

Another conventional gaze direction detection method detects the pupil from a facial image of a person's face or a facial image of a person's face, then detects the light reflection point within the pupil, and detects the detected light reflection point within the pupil. Since this method detects the direction of gaze using lighting reflection points, there was a problem that it was difficult to apply to images in which the pupil was not captured.

Therefore, improvement measures to solve the above problems are required.

The purpose of the present invention is to solve all of the above-mentioned problems.

Additionally, another purpose of the present invention is to accurately detect the gaze direction.

Additionally, another purpose of the present invention is to accurately detect the gaze direction using head direction information and body direction information.

Another purpose of the present invention is to support effective advertising to consumers by accurately detecting the gaze direction.

In order to achieve the object of the present invention as described above and realize the characteristic effects of the present invention described later, the characteristic configuration of the present invention is as follows.

According to an embodiment of the present invention, in a method of learning a deep learning-based gaze direction detection model for detecting a person's gaze direction, (a) when at least one first learning image is acquired, the learning device, The first training image is input to the body convolutional layer, and the body convolutional layer performs a convolutional operation on the first training image at least once to extract the body features of the first person included in the first training image. At least one first body feature map is generated, and the first body feature map is input to a body FC (Fully Connected) layer to cause the body FC layer to perform FC operation on the first body feature map at least once. 1 Output at least one piece of predicted body direction information that predicts the direction in which the front of the human body faces, and labeled information included in the predicted body direction information and first ground truth information corresponding to the first learning image. Generating at least one body direction loss using body direction information, and learning the body FC layer and the body convolutional layer using the body direction loss; and (b) when at least one second learning image is acquired, the learning device inputs the second learning image into the body convolutional layer to cause the body convolutional layer to obtain at least the second learning image. Convolutional operation is performed once to generate at least one second body feature map extracting the body features of a second person included in the second learning image, and the second learning image is input to the head convolutional layer to generate the second body feature map. A head convolutional layer performs a convolutional operation on the second learning image at least once to generate at least one first head feature map from which head features of the second person are extracted, and the second body feature map and the Concatenate the first head feature map to create a first integrated feature map, input the first integrated feature map to the head FC layer, and have the head FC layer perform FC operation on the first integrated feature map at least once. To output at least one first predicted head direction information predicting the direction in which the front of the second person's head is facing, and second ground truth corresponding to the first predicted head direction information and the second learning image. generating at least one head direction loss using labeled head direction information included in and learning the head FC layer and the head convolutional layer using the head direction loss; A method including is provided.

In the embodiment, in step (b), the learning device trains the head FC layer and the head convolutional layer by adding a loss weight to the head direction loss, and the head direction loss is If the loss weight is less than the set threshold, “0” can be applied as the loss weight, and if the head direction loss is greater than the threshold value, a preset real number greater than “0” can be applied as the loss weight.

In the above embodiment, in step (b), the learning device causes the head FC layer to: (i) determine which of the preset head direction classes the direction in which the front of the second person's head is facing; The predicted direction information is one of (ii) classification information that classifies whether the second person's head is facing, and (ii) regression information that classifies which direction the front of the second person's head faces corresponds to among the continuous direction candidates. It can be output as .

In one embodiment, the predicted direction information may be a prediction of the direction in which the front of the second person's head faces in either a two-dimensional plane or a three-dimensional space corresponding to the second learning image.

In the above embodiment, the first learning image or the second learning image is, in a photographed or cropped person image, (i) each of the person's body direction and gaze direction is preset in a two-dimensional plane or three-dimensional space; labeling each of the body direction classes and preset gaze direction classes with a specific body direction class and a specific gaze direction class, or (ii) labeling each of the body direction and gaze direction of the person in question on a two-dimensional plane or 3 It may be generated by labeling each of the body direction vector and gaze direction vector in a dimensional space.

In the embodiment, the first learning image or the second learning image is a corresponding image obtained using sensing information of the gyroscope sensor at the time of shooting in a human image of a person wearing a gyroscope sensor. Using a person's sensed body direction information and sensed gaze direction information, (i) the sensed body direction information and the Label each of the sensed gaze direction information with a specific body direction class and a specific gaze direction class, or (ii) the sensed body direction information and the sensed gaze direction in a two-dimensional plane or three-dimensional space. It may be generated by labeling each piece of information with each of the person's body direction vector and gaze direction vector.

In the embodiment, (c) the learning device inputs at least one evaluation image to the body convolutional layer and causes the body convolutional layer to perform a convolutional operation on the evaluation image at least once to perform the evaluation. At least one third body feature map is generated by extracting the body features of a third person included in the image, and the evaluation image is input to the head convolutional layer to cause the head convolutional layer to generate the evaluation image. A convolutional operation is performed at least once to generate at least one second head feature map from which the head features of the third person are extracted, and the third body feature map and the second head feature map are concatenated to perform a second integration. Create a feature map, input the second integrated feature map to the head FC layer, and have the head FC layer calculate the second integrated feature map at least once to determine the direction in which the front of the third person's head faces. At least one second predicted head direction information is output, and the body convolutional layer and the head convolution layer are generated with reference to the second predicted head direction information and a third ground truth corresponding to the evaluation image. Evaluating the gaze direction detection model including a sional layer and the head FC layer; It may further include.

In the embodiment, the learning device calculates accuracy through the following equation using the second predicted head direction information and the third ground truth, and detects the gaze direction using the calculated accuracy. The model can be evaluated. , in the above equation, N is the total number of the second predicted head direction information used for evaluation, # of predicted soft corrections is the number of the second predicted head direction information that did not correctly predict the labeled correct answer, # of predicted corrects may be the number of correctly predicted labeling correct answers.

According to another embodiment of the present invention, in a method of learning a deep learning-based gaze direction detection model for detecting a person's gaze direction, (a) when at least one learning image is acquired, the learning device Input to the body convolutional layer to cause the body convolutional layer to perform a convolutional operation on the learning image at least once to generate at least one body feature map extracting the body features of the person included in the learning image, and , inputting the learning image to a head convolutional layer to cause the head convolutional layer to perform a convolutional operation on the learning image at least once to generate at least one head feature map from which the head features of the person are extracted. ; (b) The learning device inputs the body feature map to a body FC layer and causes the body FC layer to perform FC operation on the body feature map at least once to predict the direction in which the front of the human body faces. Output the predicted body direction information, input the integrated feature map obtained by concatenating the body feature map and the head feature map to the head FC layer, and have the head FC layer perform FC operation on the integrated feature map at least once. outputting at least one piece of predicted head direction information predicting a direction in which the front of the person's head is facing; and (c) the learning device generates at least one body direction loss using the predicted body direction information and labeled body direction information included in ground truth information corresponding to the learning image, and the predicted head Generating at least one head direction loss using direction information and labeled head direction information included in the ground truth, learning the body FC layer and the body convolutional layer using the body direction loss, Learning the head FC layer and the head convolutional layer using head direction loss; A method including is provided.

In the other embodiment, the learning image is, in a photographed or cropped person image, (i) each of the person's body direction and gaze direction is divided into preset body direction classes and preset gaze in a two-dimensional plane or three-dimensional space; Label each of the direction classes with a specific body direction class and a specific gaze direction class, or (ii) label each of the person's body direction and gaze direction as a body direction vector and gaze direction in a two-dimensional plane or three-dimensional space. It may be generated by labeling each direction vector.

In another embodiment, the learning image includes the person's sensed body direction information obtained using the sensing information of the gyroscope sensor at the time of shooting in a person image of a person wearing a gyroscope sensor, and Using the sensed gaze direction information, (i) corresponding to each of the sensed body direction information and the sensed gaze direction information among preset body direction classes and preset gaze direction classes in a two-dimensional plane or three-dimensional space; labeling each of a specific body direction class and a specific gaze direction class, or (ii) labeling each of the sensed body direction information and the sensed gaze direction information in a two-dimensional plane or three-dimensional space with the body of the person in question. It may be generated by labeling each of the direction vector and gaze direction vector.

According to another embodiment of the present invention, in a learning device for learning a deep learning-based gaze direction detection model for detecting a person's gaze direction, a deep learning-based gaze direction detection model for detecting a person's gaze direction is learned. a memory in which instructions for doing so are stored; and a processor that performs an operation to learn the gaze direction detection model according to the instructions stored in the memory. Including, wherein the processor: (I) When at least one first training image is obtained, input the first training image to the body convolutional layer to cause the body convolutional layer to generate the first training image At least one first body feature map is generated by extracting the body features of the first person included in the first learning image by performing a convolutional operation at least once, and the first body feature map is used as a body FC (Fully Connected) layer. input to cause the body FC layer to perform an FC operation on the first body feature map at least once and output at least one predicted body direction information predicting the direction in which the front of the first person's body faces, and the prediction At least one body direction loss is generated using the labeled body direction information and the labeled body direction information included in the first ground truth information corresponding to the first learning image, and the body FC layer is generated using the body direction loss. and a process of training the body convolutional layer, and (II) when at least one second training image is obtained, inputting the second training image to the body convolutional layer to cause the body convolutional layer to Perform a convolutional operation on the second learning image at least once to generate at least one second body feature map extracting body features of a second person included in the second learning image, and convert the second learning image into a head control image. Input to a translational layer to cause the head convolutional layer to perform a convolutional operation on the second training image at least once to generate at least one first head feature map from which the head features of the second person are extracted, A first integrated feature map is generated by concatenating the second body feature map and the first head feature map, and the first integrated feature map is input to the head FC layer to cause the head FC layer to generate the first integrated feature map. FC-operates the map at least once to output at least one first predicted head direction information predicting the direction in which the front of the second person's head is facing, and the first predicted head direction information and the second learning image are combined. A process of generating at least one head direction loss using labeled head direction information included in the second ground truth corresponding to and learning the head FC layer and the head convolutional layer using the head direction loss. A learning device that performs is provided.

In another embodiment, in the process (II), the processor trains the head FC layer and the head convolutional layer by adding a loss weight to the head direction loss, and the head direction loss is If the loss weight is less than the set threshold, “0” can be applied as the loss weight, and if the head direction loss is greater than the threshold value, a preset real number greater than “0” can be applied as the loss weight.

In another embodiment, the processor, in process (II), causes the head FC layer to: (i) determine which of the preset head direction classes the direction in which the front of the second person's head is facing; The predicted direction information is one of (ii) classification information that classifies whether the second person's head is facing, and (ii) regression information that classifies which direction the front of the second person's head faces corresponds to among the continuous direction candidates. It can be output as .

In another embodiment, the predicted direction information may be a prediction of the direction in which the front of the second person's head faces in either a two-dimensional plane or a three-dimensional space corresponding to the second learning image. .

In another embodiment, the first learning image or the second learning image is, in a photographed or cropped person image, (i) each of the person's body direction and gaze direction is based on a two-dimensional plane or three-dimensional space; Label each of the set body direction classes and the preset gaze direction classes with a specific body direction class and a specific gaze direction class, or (ii) label each of the body direction and gaze direction of the person in question on a two-dimensional plane or It may be generated by labeling each of the body direction vector and gaze direction vector in three-dimensional space.

In another embodiment, the first learning image or the second learning image is obtained using sensing information of the gyroscope sensor at the time of shooting from a human image of a person wearing a gyroscope sensor. Using the person's sensed body direction information and sensed gaze direction information, (i) the sensed body direction information among preset body direction classes and preset gaze direction classes in a two-dimensional plane or three-dimensional space, and Label each of the sensed gaze direction information with a specific body direction class and a specific gaze direction class, or (ii) the sensed body direction information and the sensed gaze in a two-dimensional plane or three-dimensional space. It may be generated by labeling each piece of direction information with each of the person's body direction vector and gaze direction vector.

In another embodiment, the processor: (III) inputs at least one evaluation image to the body convolutional layer and causes the body convolutional layer to perform a convolutional operation on the evaluation image at least once to perform the evaluation. At least one third body feature map is generated by extracting the body features of a third person included in the image, and the evaluation image is input to the head convolutional layer to cause the head convolutional layer to generate the evaluation image. A convolutional operation is performed at least once to generate at least one second head feature map from which the head features of the third person are extracted, and the third body feature map and the second head feature map are concatenated to perform a second integration. Create a feature map, input the second integrated feature map to the head FC layer, and have the head FC layer calculate the second integrated feature map at least once to determine the direction in which the front of the third person's head faces. At least one second predicted head direction information is output, and the body convolutional layer and the head convolution layer are generated with reference to the second predicted head direction information and a third ground truth corresponding to the evaluation image. A processor that evaluates the gaze direction detection model including a sional layer and the head FC layer may be further performed.

In another embodiment, the processor calculates accuracy through the following equation using the second predicted head direction information and the third ground truth, and detects the gaze direction using the calculated accuracy. The model can be evaluated. , in the above equation, N is the total number of the second predicted head direction information used for evaluation, # of predicted soft corrections is the number of the second predicted head direction information that did not correctly predict the labeled correct answer, # of predicted corrects may be the number of correctly predicted labeling correct answers.

According to another embodiment of the present invention, in a learning device for learning a deep learning-based gaze direction detection model for detecting a person's gaze direction, a deep learning-based gaze direction detection model for detecting a person's gaze direction is learned. a memory in which instructions for doing so are stored; and a processor that performs an operation to learn the gaze direction detection model according to the instructions stored in the memory. It includes: (I) when at least one training image is obtained, the processor inputs the training image to a body convolutional layer to perform a convolutional operation on the training image at least once generate at least one body feature map extracting the human body features included in the learning image, and input the learning image to a head convolutional layer to cause the head convolutional layer to extract the learning image at least once A process of generating at least one head feature map from which the head features of the person are extracted through convolutional calculation, (II) inputting the body feature map to a body FC layer to cause the body FC layer to extract the body feature map at least FC calculation is performed once to output at least one predicted body direction information predicting the direction in which the front of the human body faces, and an integrated feature map obtained by concatenating the body feature map and the head feature map is generated as a head FC layer. A process of inputting to the head FC layer to perform an FC operation on the integrated feature map at least once to output at least one predicted head direction information predicting the direction in which the front of the person's head is facing, and (III) the At least one body direction loss is generated using predicted body direction information and labeled body direction information included in ground truth information corresponding to the learning image, and labeling included in the predicted head direction information and ground truth is used. Generate at least one head direction loss using the head direction information, learn the body FC layer and the body convolutional layer using the body direction loss, and train the head FC layer using the head direction loss. And a learning device that performs a process of learning the head convolutional layer is provided.

In another embodiment, the learning image is, in a photographed or cropped person image, (i) each of the person's body direction and gaze direction is classified into preset body direction classes and preset body direction classes in a two-dimensional plane or three-dimensional space; Label each of the gaze direction classes with a specific body direction class and a specific gaze direction class, or (ii) label each of the person's body direction and gaze direction with a body direction vector and a body direction vector in a two-dimensional plane or three-dimensional space. It may be generated by labeling each gaze direction vector.

According to another embodiment, the learning image is a person image of a person wearing a gyroscope sensor, and the sensed body direction of the person obtained using the sensing information of the gyroscope sensor at the time of shooting. Using information and sensed gaze direction information, (i) the sensed body direction information and the sensed gaze direction information, respectively, among preset body direction classes and preset gaze direction classes in a two-dimensional plane or three-dimensional space; or (ii) label each of the sensed body direction information and the sensed gaze direction information in a two-dimensional plane or three-dimensional space with a specific body direction class and a specific gaze direction class corresponding to the corresponding person. It may be generated by labeling with each of the body direction vector and gaze direction vector of .

In addition, a computer-readable recording medium for recording a computer program for executing the method of the present invention is further provided.

The present invention has the effect of accurately detecting the gaze direction.

Additionally, the present invention has the effect of accurately detecting the gaze direction using head direction information and body direction information.

In addition, the present invention has the effect of supporting effective advertising to consumers by accurately detecting the gaze direction.

The following drawings attached for use in explaining embodiments of the present invention are only some of the embodiments of the present invention, and to those skilled in the art (hereinafter “those skilled in the art”), the invention Other drawings can be obtained based on these drawings without further work being done.
Figure 1 schematically shows a learning device for learning a deep learning-based gaze direction detection model for detecting the gaze direction according to the present invention;
Figure 2 is a diagram for explaining a learning method for learning a deep learning-based gaze direction detection model for detecting the gaze direction according to the present invention;
Figure 3 is a diagram illustrating a learning method for learning a deep learning-based gaze direction detection model for detecting the gaze direction according to the present invention;
Figure 4 schematically shows a test device for testing a deep learning-based gaze direction detection model for detecting the gaze direction according to the present invention;
Figure 5 is a diagram illustrating a test method for testing a deep learning-based gaze direction detection model for detecting the gaze direction according to the present invention;
Figure 6 schematically shows a learning image used for learning a deep learning-based gaze direction detection model that detects the gaze direction according to the present invention.

The detailed description of the present invention described below refers to the accompanying drawings, which show by way of example specific embodiments in which the present invention may be practiced to make clear the objectives, technical solutions and advantages of the present invention. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention.

Additionally, throughout the description and claims, the word “comprise” and variations thereof are not intended to exclude other technical features, attachments, components or steps. Other objects, advantages and features of the invention will appear to those skilled in the art, partly from this description and partly from practice of the invention. The examples and drawings below are provided by way of example and are not intended to limit the invention.

Moreover, the present invention encompasses all possible combinations of the embodiments shown herein. It should be understood that the various embodiments of the present invention are different from one another but are not necessarily mutually exclusive. For example, specific shapes, structures and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description that follows is not intended to be taken in a limiting sense, and the scope of the invention is limited only by the appended claims, together with all equivalents to what those claims assert, if properly described. Similar reference numbers in the drawings refer to identical or similar functions across various aspects.

The title or summary of the present invention provided herein is provided for convenience only and does not limit or construe the scope or meaning of these embodiments.

In the following description, the case of detecting the gaze of a pedestrian is taken as an example, but the present invention is not limited to this, and the present invention can be applied even to cases where pedestrians are not pedestrians.

Hereinafter, in order to enable those skilled in the art to easily practice the present invention, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

Figure 1 schematically shows a learning device 1000 for learning a deep learning-based gaze direction detection model for detecting the gaze direction according to an embodiment of the present invention. The learning device 1000 is a gaze direction detection model. It may include a memory 1001 in which instructions for learning are stored and a processor 1002 that performs an operation for learning a gaze direction detection model in response to the instructions stored in the memory 1001. At this time, the gaze direction detection model may include a body convolutional layer, a head convolutional layer, and a head FC (Fully-Connected) layer, which will be described in detail in the following learning method.

Specifically, learning device 1000 is typically a computing device (e.g., a device that may include a computer processor, memory, storage, input and output devices, and other components of a traditional computing device; electronic devices such as routers, switches, etc. the desired system using a combination of a communication device (electronic information storage systems, such as network attached storage (NAS) and storage area network (SAN)) and computer software (i.e., instructions that cause a computing device to function in a particular way) Performance may be achieved.

Additionally, the processor of the computing device may include hardware components such as a Micro Processing Unit (MPU) or Central Processing Unit (CPU), cache memory, and data bus. Additionally, the computing device may further include an operating system and a software component of an application that performs a specific purpose.

However, this does not exclude the case where the computing device includes an integrated processor in which a medium, processor, and memory for implementing the present invention are integrated.

A learning method for learning a gaze direction detection model for detecting the gaze direction using the learning device 1000 configured as described above will be described with reference to FIGS. 2 and 3 as follows.

For reference, although each component is described below in the singular, this does not exclude the possibility of plurality.

For reference, it is generally easier to predict which direction a pedestrian's body is moving than to predict which direction the pedestrian's face is pointing. This is because it is possible to predict which direction the pedestrian's body is moving based on abundant information such as the pedestrian's arms, legs, hands, and feet.

Therefore, in order to increase learning efficiency, the learning device 1000 according to an embodiment of the present invention uses the parameters of the body convolutional layer 1100 to extract features related to the direction in which the pedestrian's body moves. After first learning, with the parameters of the body convolutional layer 1100 fixed, the parameters of the head convolutional layer and head FC layer, which extract features related to which direction the pedestrian's face is pointing, can be learned. there is.

However, the present invention is not limited to this, and the learning device 1000 according to the present invention may simultaneously learn the parameters of the body convolutional layer 1100, the head convolutional layer, and the head FC layer.

For convenience of explanation, below, the parameters of the body convolutional layer 1100, which extracts features related to the direction in which the pedestrian's body moves, are first learned, and then the parameters of the body convolutional layer 1100 are fixed. In this state, the case of learning the parameters of the head convolutional layer and head FC layer, which extract features related to which direction the pedestrian's face is pointing, will first be described, and the body convolutional layer 1100 and head control layer will be described first. The case of simultaneously learning the parameters of the volutional layer and the head FC layer will be described next.

Referring to FIG. 2, when at least one first learning image (e.g., pedestrian image) is acquired, the learning device 1000 inputs the first learning image into the body convolutional layer 1100 to create a body convolutional image. At least one first body feature map can be generated by having the layer 1100 apply a convolutional operation to the first learning image. That is, the learning device 1000 causes the body convolutional layer 1100 to perform a convolutional operation on the first learning image at least once to extract at least one body feature of the first person included in the first learning image. You can create a body feature map.

For reference, the first learning image and the second learning image, which will be described later, may be separate learning images, but are not limited thereto, and may be the same learning image.

Additionally, the first learning image, the second learning image to be described later, and the test image to be described later may be images obtained through CCTV, but are not limited thereto.

Next, the learning device 1000 inputs the body feature map to the body FC (fully-connected) layer 1200 and causes the body FC layer 1200 to apply FC operation to the body feature map to make at least one prediction. body direction information can be output. That is, the learning device 1000 causes the body FC layer 1200 to perform FC operations on the first body feature map at least once and output at least one piece of predicted body direction information predicting the direction in which the front of the first person's body faces. You can do it.

At this time, the learning device 1000 may cause the body FC layer 1200 to output classification information, that is, information with a discontinuous value, as predicted body direction information. , but is not limited to this, and any one of regression information, that is, information with continuous values, may be output.

That is, the learning device 1000 causes the body FC layer 1200 to provide classification information classifying which class among preset body direction classes the direction in which the front of the first person's body faces corresponds, and first One of the regression information obtained by regressing which direction among the continuous direction candidates the direction in which the front of the human body faces corresponds can be output as predicted body direction information.

As an example, the predicted body direction information is centered on the body of the first person, that is, a pedestrian, with the front of the pedestrian's body facing the camera as a reference (e.g., South), and the front of the pedestrian's body facing. It may be classification information that classifies which class the direction corresponds to among eight preset body direction classes (i.e., S, SE, E, NE, N, NW, W, SW) on a two-dimensional plane.

As another example, the predicted body direction information is based on the body of the first person, that is, the pedestrian, when the front of the pedestrian's body is facing the camera (i.e., 0 degrees), and the front of the pedestrian's body is This may be regression information obtained by regressing which direction (for example, 150.1482 degrees) out of 360 degrees on a two-dimensional plane is headed.

Meanwhile, although it has been described above that the predicted body direction information is collision information or regression information in a two-dimensional plane, it may also be collision information or regression information in a three-dimensional space.

That is, the predicted body direction information is centered on the body of the first person, that is, the pedestrian, and is based on when the front of the pedestrian's body faces the camera, and the front of the pedestrian's body is radial in three-dimensional space. In addition to the 26 preset body direction classes (i.e., the 8 preset body direction classes on the two-dimensional plane, 8 body direction classes in the upper direction, 8 body direction classes in the lower direction, upper Classification information that classifies which class it corresponds to (including direction classes and lower direction classes), or league that regresses which direction it faces among directions with continuous values according to a three-dimensional spherical coordinate system. It may be lesion information.

As a result, the predicted body direction information is, depending on the settings of the body FC layer, collision information in a 2-dimensional plane, regression information in a 2-dimensional plane, collision information in a 3-dimensional space, and 3-dimensional It may be any one of regression information in space.

Again, referring to FIG. 2, the learning device 1000 uses at least one piece of predicted body direction information and labeled body direction information included in the first ground truth corresponding to the first learning image. Parameters of at least some of the body convolutional layer 1100 and the body FC layer 1200 can be learned by generating at least one body direction loss and performing backpropagation using the body direction loss. That is, the learning device 1000 can learn the body convolutional layer 1100 and the body FC layer 1200 using the body direction loss.

As above, after the parameters of the body convolutional layer 1100 and the body FC layer 1200 are learned, the learning device 1000 performs the head control while fixing the parameters of the body convolutional layer 1100. Parameters of at least some of the volutional layer and the head FC layer can be learned.

For reference, as previously explained, the pedestrian's eyes (pupils) are often not photographed, but according to one embodiment of the present invention, the direction in which the pedestrian's face faces is regarded as the direction in which the pedestrian's gaze is directed, so that the image It is possible to accurately predict the direction of a pedestrian's gaze, regardless of the various environments in which it is filmed.

Referring to FIG. 3, when at least one second learning image (e.g., pedestrian image) is acquired, the learning device 1000 divides the second learning image into a head convolutional layer 1300 and a body convolutional layer ( 1100) to cause each of the head convolutional layer 1300 and the body convolutional layer 1100 to apply a convolutional operation to the second learning image, thereby creating at least one first head feature map and at least one Each of the second body feature maps can be generated.

That is, the learning device 1000 inputs the second learning image into the body convolutional layer 1100 and causes the body convolutional layer 1100 to perform a convolutional operation on the second learning image at least once to perform the second learning image. At least one second body feature map is created by extracting the body features of the second person included in the image, and the second learning image is input to the head convolutional layer 1300 to be converted to the head convolutional layer 1300. It is possible to generate at least one first head feature map from which head features of a second person are extracted by performing a convolutional operation on the second learning image at least once.

For reference, the body FC layer 1200 explained through FIG. 2 is only used in the process of learning the parameters of the body convolutional layer 1100, and as shown in FIG. 3, the head convolutional layer ( 1300) and may not be used in the process of learning the head FC layer 1400. That is, the gaze direction detection model according to the present invention may be composed of a body convolutional layer 1100, a head convolutional layer 1300, and a head FC layer 1400, and the body FC layer 1200 is a gaze direction detection model. It may be used only in the learning process of the direction detection model.

Next, the learning device 1000 concatenates the head feature map and the body feature map to generate a first integrated feature map, and inputs the first integrated feature map to the head FC (fully-connected) layer to head FC The layer 1400 may perform an FC operation on the first integrated feature map at least once to output at least one piece of first predicted head direction information that predicts the direction in which the front of the second person's head is facing.

At this time, the learning device 1000 may cause the head FC layer 1400 to output classification information, that is, information with a discontinuous value, as predicted head direction information. , but is not limited to this, and any one of regression information, that is, information with continuous values, may be output.

That is, the learning device 1000 causes the head FC layer 1400 to provide classification information classifying which class among preset body direction classes the direction in which the front of the second person's head faces corresponds, and the second person's head One of the regression information obtained by regressing which direction among the continuous direction candidates the direction in which the front of the person's head faces corresponds can be output as predicted head direction information.

As an example, the predicted head direction information is centered on the head of the second person, that is, the pedestrian, and is based on when the front of the pedestrian's head is facing the camera (e.g., South), and the front of the pedestrian's head is toward the camera. It may be classification information that classifies which class the direction corresponds to among eight preset head direction classes (i.e., S, SE, E, NE, N, NW, W, SW) on a two-dimensional plane. At this time, the predicted head direction information is a class for a class in which the direction in which the front of the pedestrian's head faces does not correspond to any of the eight head direction classes, that is, in the case where the pedestrian is looking at the pedestrian himself. It may be classified information that further includes a corresponding default direction class.

As another example, the predicted head direction information is based on the head of the second person, that is, the pedestrian, and the front of the pedestrian's head, that is, when the pedestrian's face is facing the camera (i.e., 0 degrees), This may be regression information obtained by regressing whether the front of the head is facing in one of the 360-degree directions on a two-dimensional plane (for example, 150.1482 degrees). At this time, the predicted head direction information is a class for a direction in which the direction in which the front of the pedestrian's head faces does not correspond to any of the 360-degree directions, that is, the default corresponding to the case where the pedestrian is facing the pedestrian himself. There is regression information that further includes direction.

Meanwhile, although it has been described above that the predicted head direction information is collision information or regression information in a two-dimensional plane, it may also be collision information or regression information in a three-dimensional space.

In other words, the predicted head direction information is based on the head of the second person, that is, the pedestrian, when the front of the pedestrian's head is facing the camera, and the front of the pedestrian's head is radial in three-dimensional space. 26 preset head direction classes (i.e., in addition to the 8 preset head direction classes on a two-dimensional plane, 8 head direction classes in the upper direction, 8 head direction classes in the lower direction, upper Classification information that classifies which class it corresponds to (including direction classes and lower direction classes), or league that regresses which direction it faces among directions with continuous values according to a three-dimensional spherical coordinate system. It may be lesion information.

As a result, the predicted head direction information is, depending on the settings of the head FC layer, collision information in a 2-dimensional plane, regression information in a 2-dimensional plane, collision information in a 3-dimensional space, and 3-dimensional It may be any one of regression information in space.

Again, referring to FIG. 3, the learning device 1000 generates at least one head direction loss with reference to at least one first predicted head direction information and at least one GT head direction information corresponding thereto, and generates a head direction loss. By performing backpropagation using loss, the parameters of at least some of the head convolutional layer 1300 and the head FC layer 1400 can be learned. That is, the learning device 1000 generates at least one head direction loss using the labeled head direction information included in the first predicted head direction information and the second ground truth corresponding to the second learning image, and The head FC layer 1400 and the head convolutional layer 1300 can be learned using the loss.

At this time, the learning device 1000 may use cross entropy as a loss function for generating the head direction loss, but the present invention is not limited thereto and generates the head direction loss using various loss functions. You may.

Meanwhile, in the above, the learning device 1000 uses the head direction loss as is to learn the parameters of at least some of the head convolutional layer 1300 and the head FC layer 1400. However, unlike this, the head direction loss Additional loss weights may be added to learn the parameters of at least some of the head FC layer 1400 and the head convolutional layer 1300.

For example, if the head direction loss is less than a preset threshold, "0" is applied as the loss weight, and if the head direction loss is more than the threshold, a preset real number greater than "0" is applied as the loss weight. You can.

As another example, if the correct answer where the predicted head direction information is actually True is predicted to be False (FN: False negative), “0” is applied as the loss weight, and the correct answer where the predicted head direction information is actually False is called true. If it is a prediction (FP: False Positive), a preset real number greater than "0" can be applied as the loss weight.

Next, the learning device 1000 determines the gaze direction during or after learning of the gaze direction detection model including the body convolutional layer 1100, the head convolutional layer 1300, and the head FC layer 1400. The performance of the detection model can be evaluated.

As an example, the learning device 1000 inputs at least one evaluation image into the body convolutional layer 1100 and causes the body convolutional layer 1100 to perform a convolutional operation on the evaluation image at least once and include it in the evaluation image. At least one third body feature map is generated by extracting the body features of a third person, and the evaluation image is input to the head convolutional layer 1300 to cause the head convolutional layer 1300 to extract the evaluation image at least. At least one second head feature map from which the head features of a third person are extracted can be generated by performing a convolutional operation once. Then, the learning device 1000 concatenates the third body feature map and the second head feature map to generate a second integrated feature map, and inputs the second integrated feature map to the head FC layer 1400 to create a head FC The layer 1400 may perform an FC operation on the second integrated feature map at least once to output at least one second predicted head direction information predicting the direction in which the front of the third person's head is facing. Thereafter, the learning device 1000 uses the body convolutional layer 1100, the head convolutional layer 1300, and the head FC layer with reference to the second predicted head direction information and the third ground truth corresponding to the evaluation image. A gaze direction detection model including (1400) can be evaluated.

At this time, the learning device 1000 calculates accuracy through the following equation using the second predicted head direction information and the third ground truth, and can evaluate the gaze direction detection model using the calculated accuracy.

In the above equation, N is the total number of second predicted head direction information used for evaluation, # of predicted soft corrections is the number of second predicted head direction information that did not correctly predict the labeled correct answer, and # of predicted corrects may be the number of correctly predicted labeling answers.

As an example, each of the second predicted head direction information used for evaluation is True Positive (TP), which predicts the correct answer that is actually True as True (correct answer), and False Positive (TP), which predicts (incorrect answer) the correct answer that is actually False as True. Assuming that it includes FP), False Negative (FN), which predicted (incorrect answer) the correct answer that was actually True as False, and True Negative (TN) that predicted (correct answer) that the correct answer that was actually False was False, N is TP+FP +FN+TN, # of predicted corrects can be TP+TN, and # of predicted soft corrects can be FN.

Meanwhile, the above described the process of learning the body convolutional layer first, and then learning the head convolutional layer and head FC layer. However, unlike this, the body convolutional layer, head convolutional layer, and head FC layer The process of learning is explained as follows. In the description below, detailed description of parts that can be easily understood from the description referring to FIGS. 2 and 3 will be omitted.

When at least one learning image is acquired, the learning device inputs the learning image to the body convolutional layer and causes the body convolutional layer to perform a convolutional operation on the learning image at least once to determine the human body features included in the learning image. At least one extracted body feature map is generated, and the learning image is input to the head convolutional layer, and the head convolutional layer performs a convolutional operation on the learning image at least once to extract at least one human head feature. You can create a head feature map.

Then, the learning device inputs the body feature map to the body FC layer and causes the body FC layer to perform FC operations on the body feature map at least once to generate at least one predicted body direction information that predicts the direction in which the front of the human body faces. You can have it printed.

In addition, the learning device inputs an integrated feature map obtained by concatenating the body feature map and the head feature map into the head FC layer, and causes the head FC layer to perform FC calculation on the integrated feature map at least once to determine the direction in which the front of the human head is facing. At least one predicted head direction information predicted can be output.

And, the learning device generates at least one body direction loss using the labeled body direction information included in the predicted body direction information and ground truth information corresponding to the learning image, and uses the predicted head direction information and ground truth At least one head direction loss can be generated using the included labeled head direction information.

Afterwards, the learning device can learn the body FC layer and the body convolutional layer using the body direction loss, and the head FC layer and the head convolutional layer using the head direction loss.

In this way, while the learning device 1000 has learned the parameters of at least some of the body convolutional layer 1100, the head convolutional layer 1300, and the head FC layer 1400, a test image (e.g., a pedestrian image) ) is obtained, the operation of the test device will be described with reference to FIGS. 4 and 5.

First, with reference to FIG. 4, a test device 2000 for testing a gaze direction detection model that detects the gaze direction will be described.

The test device 2000 has a memory 2001 storing instructions for testing a gaze direction detection model that detects the gaze direction, and performs an operation to test the gaze direction detection model according to the instructions stored in the memory 2001. It may include a processor 2002.

Specifically, test device 2000 is typically a computing device (e.g., a device that may include a computer processor, memory, storage, input and output devices, and other components of a traditional computing device; electronic devices such as routers, switches, etc. the desired system using a combination of a communication device (electronic information storage systems, such as network attached storage (NAS) and storage area network (SAN)) and computer software (i.e., instructions that cause a computing device to function in a particular way) Performance may be achieved.

Additionally, the processor of the computing device may include hardware components such as a Micro Processing Unit (MPU) or Central Processing Unit (CPU), cache memory, and data bus. Additionally, the computing device may further include an operating system and a software component of an application that performs a specific purpose.

However, this does not exclude the case where the computing device includes an integrated processor in which a medium, processor, and memory for implementing the present invention are integrated.

At this time, the test device 2000 may be the same device as the learning device 1000 shown in FIG. 1, or may be a separate device.

Below, a process in which the test device 2000 tests the gaze direction detection model will be described in detail with reference to FIG. 5 .

For reference, overlapping descriptions of content that is the same/similar to the content described about the learning device 1000 will be omitted.

First, with the parameters of at least some of the body convolutional layer 1100, the head convolutional layer 1300, and the head FC layer 1400 learned by the learning device 1000, the test device 2000 A test image can be obtained.

Then, the test device 2000 inputs the test image into the head convolutional layer 1300 and the body convolutional layer 1100, respectively, to create the head convolutional layer 1300 and the body convolutional layer 1100. At least one head feature map for testing and at least one body feature map for testing can be generated by having each of them apply a convolutional operation to the test image at least once.

Then, the test device 2000 generates an integrated feature map for testing by concatenating the head feature map for testing and the body feature map for testing, and inputs the integrated feature map for testing to the head FC (fully-connected) layer. By having the head FC layer 1400 apply FC operation to the integrated feature map for testing, at least one piece of predicted head direction information for testing can be output.

Meanwhile, the learning image, the first learning image, the second learning image used in the process of learning the gaze direction detection model described above, and the evaluation image used in the process of evaluating the gaze direction detection model are each labeled with ground truths. As a result, the method of labeling each ground truth in a human image is explained as follows.

First, in order to generate a learning image, at least one photographed or cropped human image may be collected.

At this time, a person image is obtained by taking a picture of a single person, or information on people is detected by object detection in an image with multiple people, and each bounding box is used to detect each person in an image. Human images can be acquired, and the direction in which the front of the person's body faces and the direction in which the front of the person's head faces can be labeled in the human image.

However, the present invention is not limited to this, and the direction in which the front of the person's body faces and the direction in which the front of the person's head faces can be labeled on a video of a person.

As an example, each of the person's body direction and gaze direction in a person image is matched to a specific body direction class and a specific gaze direction class among preset body direction classes and preset gaze direction classes in a two-dimensional plane or three-dimensional space. Each gaze direction class can be labeled. At this time, if the specific body direction class and the specific gaze direction class are the same, only one direction class may be labeled.

For example, in a two-dimensional plane, based on when the front of a person's body or head is facing the camera (e.g., South), the directions in which the front of a person's body or head faces are set in eight directions on the two-dimensional plane. The human image can be labeled with a corresponding specific direction class among the classes (i.e., S, SE, E, NE, N, NW, W, SW). Alternatively, the human image may be labeled with a corresponding specific direction class among each direction class that divides the 360-degree direction centered on the human body or head into unit angles. In other words, when 360 degrees are divided into 10-degree increments, 36 discrete direction classes can be set, and when 360 degrees are divided into 1-degree increments, 360 discrete direction classes can be set, and the set direction classes are Among them, the corresponding specific direction class can be labeled on the human image. Additionally, the human image may be labeled by further including a default direction class corresponding to a case where the front of the person's head is facing the person himself.

Meanwhile, in the above description, positive samples were described that accurately labeled the direction in which the front of the human body or head was facing, but negative samples were labeled in a direction different from the direction in which the front of the human body or head was facing. You can also label human images.

And, taking the three-dimensional space as an example, there are 26 head direction classes preset in the radial direction in the three-dimensional space, that is, eight direction classes for eight directions in the reference plane corresponding to the height of the person, 8 direction classes in the upward direction, pointing in 8 directions in the upward direction, 8 direction classes in the downward direction, pointing in 8 directions in the downward direction, an upper direction class in the upward direction, and lower in the downward direction. Among the direction classes, the corresponding specific direction class can be labeled in the human image. Alternatively, the human image may be labeled with a corresponding specific direction class among each direction class that divides the 3D spherical coordinate system into unit coordinates. Additionally, the human image may be labeled by further including a default direction class corresponding to a case where the front of the person's head is facing the person himself.

Meanwhile, in the above description, positive samples were described that accurately labeled the direction in which the front of the human body or head was facing, but negative samples were labeled in a direction different from the direction in which the front of the human body or head was facing. You can also label human images.

As another example, in a person image, each of the person's body direction and gaze direction may be labeled as a body direction vector and a gaze direction vector, respectively, in a two-dimensional plane or three-dimensional space.

For example, in a two-dimensional plane, a person image may be labeled with a specific direction vector corresponding to one direction in a continuous 360-degree direction centered on the person's body or head. Additionally, a default direction vector corresponding to a case where the front of the person's head is facing the person himself may be labeled in the person image.

Meanwhile, in the above description, positive samples were described that accurately labeled the direction in which the front of the human body or head was facing, but negative samples were labeled in a direction different from the direction in which the front of the human body or head was facing. You can also label human images.

And, taking three-dimensional space as an example, labeling a human image with a specific direction vector for a specific coordinate corresponding to the direction in which the front of the human body or head is facing among continuous coordinates in a spherical coordinate system corresponding to three-dimensional space. can do. Additionally, the human image may be labeled by further including a default direction vector corresponding to a case where the front of the person's head is facing the person himself.

Meanwhile, in the above description, positive samples were described that accurately labeled the direction in which the front of the human body or head was facing, but negative samples were labeled in a direction different from the direction in which the front of the human body or head was facing. You can also label human images.

In addition, in the above, the human image was directly labeled with ground truth, but differently from this, if there is continuous direction information about the direction in which the front of the human body or head is facing, the continuous direction information is used to label the human image. You may.

As an example, in a human image of a person wearing a gyroscope sensor, the person's sensed body direction information and sensed gaze direction information obtained using the sensing information of the gyroscope sensor at the time of shooting are used to obtain 2 One specific body direction class and a specific gaze direction class corresponding to each of the sensed body direction information and the sensed gaze direction information among preset body direction classes and preset gaze direction classes in a dimensional plane or three-dimensional space. Alternatively, the sensed body direction information and the sensed gaze direction information in a two-dimensional plane or three-dimensional space may be labeled as the person's body direction vector and gaze direction vector, respectively.

Below, a method of labeling ground truth using a gyroscope sensor will be described in more detail with reference to FIG. 6. In the description below, the process of labeling head direction information is explained. Since the process of labeling body direction information is also similar, detailed description thereof will be omitted.

Figure 6 shows a learning image of a pedestrian looking at a second advertisement among the first advertisement posted on the first pillar 610 on the left side of the pedestrian and the second advertisement posted on the second pillar 620. It is shown schematically.

At this time, (i) the direction 630 in which the pedestrian's body actually moves, i.e., GT body direction information, may be a vector corresponding to the (x1, y1, z1) component, and (ii) the pedestrian's face part actually faces. The direction 640, that is, GT head direction information, may be a vector corresponding to the (x2, y2, z2) component.

For reference, as shown in FIG. 6, z1 corresponding to GT body direction information in a situation where a pedestrian is walking on level ground has a value of 0 (e.g., (x1, y1, z1) is (-1, 1). , 0), z1 will have a very small value compared to x1 and/or y1 (e.g., if (x1, y1, z1) is (-1, 1, 0.01). On the other hand, although not shown in FIG. 6, z1 corresponding to GT body direction information in a situation where a pedestrian walks on an inclined section (e.g., stairs) has a non-zero value (e.g., (x1, y1, z1) (1, 2, 1) will have).

Meanwhile, as shown in FIG. 6, pedestrians can not only turn their heads left and right, but also turn up and down, so z2 corresponding to GT head direction information has various values, regardless of which section the pedestrian is walking. (For example, if (x2, y2, z2) is (-1, -1, 1)).

For reference, as described above, ground truth may correspond to information obtained from a gyroscope sensor mounted on the head of a pedestrian.

However, as shown in FIG. 6, when the first pillar 610 and the second pillar 620 are located at a close distance (or the first pillar and the second pillar are at a preset close viewing angle or close viewing angle of the pedestrian) (if located within the frustum), only the information obtained from the gyroscope determines whether the pedestrian is looking at the first advertisement posted on the first pillar 610, or the second advertisement posted on the second pillar 620. It can be difficult to accurately label what you are looking at.

Therefore, for more accurate labeling, pedestrian assistance information acquired within a predetermined time period from the time the pedestrian was imaged (e.g., content corresponding to the second advertisement using a pedestrian terminal within 60 seconds from the time the pedestrian was imaged) Information obtained from a gyroscope sensor (such as search information indicating that a user has searched for or history information of visiting a second advertisement-related product store or purchasing the product) may be used together.

Meanwhile, unlike learning the gaze direction detection model as described in FIGS. 2 and 3 using ground truth-labeled learning images according to the above description, sensing information sensed from a gyroscope sensor without generating a learning image You can also use as ground truth to learn the gaze direction detection model in real time.

As an example, the learning device can acquire a video of a pedestrian wearing a gyroscope sensor walking. At this time, it can be assumed that the pedestrian's body direction information obtained from the gyroscope sensor is (-1, 1, 0) and the head direction information is (-1, -1, 1).

Then, the learning device inputs a cropped image of the area where the pedestrian was captured into the gaze direction detection model learned as described above to predict the direction in which the front of the pedestrian's head is facing. Predicted head direction information (e.g., (-1.1, -1, 0.8)) can be output.

In addition, the learning device uses predicted head direction information (e.g., (-1.1, -1, 0.8)) and the pedestrian's actual head direction information (e.g., (-1, -) obtained from information sensed by the gyroscope sensor. A gaze direction detection model can be learned by generating head direction loss by referring to 1, 1)) and performing backpropagation using the head direction loss.

Additionally, the embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and usable by those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specifically configured to store and perform program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the invention and vice versa.

In the above, the present invention has been described with specific details such as specific components and limited embodiments and drawings, but this is only provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , a person skilled in the art to which the present invention pertains can make various modifications and variations from this description.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all modifications equivalent to or equivalent to the scope of the claims fall within the scope of the spirit of the present invention. They will say they do it.

1000: learning device,
1001: memory,
1002: processor,
2000: Test device,
2001: Memory,
2002: Processor

Claims (22)

In a method of learning a deep learning-based gaze direction detection model for detecting a person's gaze direction,
(a) When at least one first learning image is acquired, the learning device inputs the first learning image into a body convolutional layer and causes the body convolutional layer to convolve the first learning image at least once. At least one first body feature map is generated by extracting the body features of the first person included in the first learning image by performing a shunt operation, and the first body feature map is input to the body FC (Fully Connected) layer. Have the body FC layer perform FC operations on the first body feature map at least once to output at least one piece of predicted body direction information that predicts the direction in which the front of the first person's body faces, and the predicted body direction information and generate at least one body direction loss using labeled body direction information included in first ground truth information corresponding to the first learning image, and use the body direction loss to determine the body FC layer and the body control. Learning a volutional layer; and
(b) When at least one second learning image is acquired, the learning device inputs the second learning image to the body convolutional layer to cause the body convolutional layer to display the second learning image at least once. A convolutional operation is performed to generate at least one second body feature map that extracts body features of a second person included in the second learning image, and the second learning image is input to a head convolutional layer to generate the head Have a convolutional layer perform a convolutional operation on the second learning image at least once to generate at least one first head feature map from which head features of the second person are extracted, and the second body feature map and the 1 Concatenate the head feature map to create a first integrated feature map, input the first integrated feature map to the head FC layer, and have the head FC layer perform FC operation on the first integrated feature map at least once. Output at least one first predicted head direction information predicting the direction in which the front of the second person's head is facing, and output a second ground truth corresponding to the first predicted head direction information and the second learning image. Generating at least one head direction loss using the included labeled head direction information, and learning the head FC layer and the head convolutional layer using the head direction loss;
How to include . According to paragraph 1,
In step (b) above,
The learning device trains the head FC layer and the head convolutional layer by adding a loss weight to the head direction loss,
If the head direction loss is less than a preset threshold, “0” is applied as the loss weight,
A method of applying a preset real number greater than “0” as the loss weight when the head direction loss is greater than the threshold. According to paragraph 1,
In step (b) above,
The learning device causes the head FC layer to provide (i) classification information classifying which class among preset head direction classes the direction in which the front of the second person's head faces corresponds, and (ii) A method of outputting one of the regression information obtained by regressing which direction among the continuous direction candidates the direction in which the front of the second person's head faces corresponds as the predicted direction information. According to paragraph 2,
The predicted direction information predicts the direction in which the front of the second person's head faces in either a two-dimensional plane or a three-dimensional space corresponding to the second learning image. According to paragraph 1,
The first learning image or the second learning image is, in a photographed or cropped person image, (i) each of the person's body direction and gaze direction is classified into preset body direction classes and groups in a two-dimensional plane or three-dimensional space; Label each of the corresponding specific body direction classes and specific gaze direction classes among the set gaze direction classes, or (ii) label each of the person's body direction and gaze direction as a body direction vector in a two-dimensional plane or three-dimensional space. and a method generated by labeling each of the gaze direction vectors. According to paragraph 1,
The first learning image or the second learning image is the sensed body direction of the person obtained using the sensing information of the gyroscope sensor at the time of shooting in an image of a person wearing a gyroscope sensor. Using information and sensed gaze direction information, (i) the sensed body direction information and the sensed gaze direction information, respectively, among preset body direction classes and preset gaze direction classes in a two-dimensional plane or three-dimensional space; or (ii) label each of the sensed body direction information and the sensed gaze direction information in a two-dimensional plane or three-dimensional space with a specific body direction class and a specific gaze direction class corresponding to the corresponding person. A method that is generated by labeling with each of the body direction vector and the gaze direction vector. According to paragraph 1,
(c) The learning device inputs at least one evaluation image into the body convolutional layer, causes the body convolutional layer to perform a convolutional operation on the evaluation image at least once, and calculates the third evaluation image included in the evaluation image. At least one third body feature map is generated by extracting human body features, and the evaluation image is input to the head convolutional layer to perform a convolutional operation on the evaluation image at least once. generate at least one second head feature map by extracting the head features of the third person, and generate a second integrated feature map by concatenating the third body feature map and the second head feature map, Inputting the second integrated feature map to the head FC layer to cause the head FC layer to perform an FC operation on the second integrated feature map at least once to predict the direction in which the front of the third person's head is facing 2 Output predicted head direction information, and refer to the second predicted head direction information and the third ground truth corresponding to the evaluation image to output the body convolutional layer, the head convolutional layer, and the head. Evaluating the gaze direction detection model including an FC layer;
How to include more. In clause 7,
The learning device calculates accuracy through the following equation using the second predicted head direction information and the third ground truth, and evaluates the gaze direction detection model using the calculated accuracy.

In the above equation, N is the total number of the second predicted head direction information used for evaluation, # of predicted soft corrections is the number of the second predicted head direction information that did not correctly predict the labeled correct answer, # of predicted corrects is the number of correctly predicted labeling correct answers. In a method of learning a deep learning-based gaze direction detection model for detecting a person's gaze direction,
(a) When at least one learning image is acquired, the learning device inputs the learning image to a body convolutional layer and causes the body convolutional layer to perform a convolutional operation on the learning image at least once to generate the learning image. generate at least one body feature map by extracting the human body features included in the learning image, and input the learning image into a head convolutional layer to perform a convolutional operation on the learning image at least once. generating at least one head feature map from which head features of the person are extracted;
(b) The learning device inputs the body feature map to a body FC layer and causes the body FC layer to perform FC operation on the body feature map at least once to predict the direction in which the front of the human body faces. Output the predicted body direction information, input the integrated feature map obtained by concatenating the body feature map and the head feature map to the head FC layer, and have the head FC layer perform FC operation on the integrated feature map at least once. outputting at least one piece of predicted head direction information predicting a direction in which the front of the person's head is facing; and
(c) the learning device generates at least one body direction loss using the predicted body direction information and labeled body direction information included in ground truth information corresponding to the learning image, and the predicted head direction Generate at least one head direction loss using information and labeled head direction information included in the ground truth, learn the body FC layer and the body convolutional layer using the body direction loss, and learn the body FC layer and the body convolutional layer. Learning the head FC layer and the head convolutional layer using a direction loss;
How to include . According to clause 9,
The learning image is, in a photographed or cropped person image, (i) each of the person's body direction and gaze direction corresponds to the corresponding person among preset body direction classes and preset gaze direction classes in a two-dimensional plane or three-dimensional space; By labeling each of the person's body direction and gaze direction with a specific body direction class and a specific gaze direction class, or (ii) labeling each of the person's body direction and gaze direction with a body direction vector and a gaze direction vector, respectively, in a two-dimensional plane or three-dimensional space. How it was created. According to clause 9,
The learning image is a person image taken of a person wearing a gyroscope sensor, and the person's sensed body direction information and sensed gaze direction information obtained using the sensing information of the gyroscope sensor at the time of shooting Using, (i) any one specific body corresponding to each of the sensed body direction information and the sensed gaze direction information among preset body direction classes and preset gaze direction classes in a two-dimensional plane or three-dimensional space labeling each of a direction class and a specific gaze direction class, or (ii) labeling each of the sensed body direction information and the sensed gaze direction information in a two-dimensional plane or three-dimensional space as a body direction vector and a gaze direction vector of the person in question. A method created by labeling each. In a learning device for learning a deep learning-based gaze direction detection model for detecting a person's gaze direction,
A memory storing instructions for learning a deep learning-based gaze direction detection model for detecting a person's gaze direction; and
a processor that performs an operation to learn the gaze direction detection model according to the instructions stored in the memory;
Including,
The processor, (I) when at least one first training image is obtained, inputs the first training image to a body convolutional layer and causes the body convolutional layer to convolute the first training image at least once At least one first body feature map is generated by extracting the body features of the first person included in the first learning image by performing a shunt operation, and the first body feature map is input to the body FC (Fully Connected) layer. Have the body FC layer perform FC operations on the first body feature map at least once to output at least one piece of predicted body direction information that predicts the direction in which the front of the first person's body faces, and the predicted body direction information and generate at least one body direction loss using labeled body direction information included in first ground truth information corresponding to the first learning image, and use the body direction loss to determine the body FC layer and the body control. A process of training a translational layer, and (II) once at least one second training image is obtained, inputting the second training image to the body convolutional layer to cause the body convolutional layer to train the second learning image. Perform a convolutional operation on the image at least once to generate at least one second body feature map extracting body features of a second person included in the second training image, and apply the second training image to a head convolutional layer. input to cause the head convolutional layer to perform a convolutional operation on the second learning image at least once to generate at least one first head feature map from which the head feature of the second person is extracted, and the second body feature A first integrated feature map is generated by concatenating the map and the first head feature map, and the first integrated feature map is input to the head FC layer to cause the head FC layer to generate the first integrated feature map at least once. FC calculation is performed to output at least one first predicted head direction information predicting the direction in which the front of the second person's head is facing, and a first predicted head direction information corresponding to the first predicted head direction information and the second learning image is output. 2 Learning to generate at least one head direction loss using labeled head direction information included in ground truth, and to perform a process of learning the head FC layer and the head convolutional layer using the head direction loss. Device. According to clause 12,
The processor,
In the process (II), additional loss weights are added to the head direction loss to learn the head FC layer and the head convolutional layer,
If the head direction loss is less than a preset threshold, “0” is applied as the loss weight,
A learning device that applies a preset real number greater than “0” as the loss weight when the head direction loss is greater than the threshold. According to clause 12,
The processor,
In the process (II), the head FC layer is provided with (i) classification information classifying which class among preset head direction classes the direction in which the front of the second person's head faces corresponds, and ( ii) A learning device that outputs as the predicted direction information one of the regression information obtained by regressing which direction among the continuous direction candidates the direction in which the front of the second person's head faces corresponds. According to clause 13,
The predicted direction information predicts the direction in which the front of the second person's head faces in either a two-dimensional plane or a three-dimensional space corresponding to the second learning image. According to clause 12,
The first learning image or the second learning image is, in a photographed or cropped person image, (i) each of the person's body direction and gaze direction is classified into preset body direction classes and groups in a two-dimensional plane or three-dimensional space; Label each of the corresponding specific body direction classes and specific gaze direction classes among the set gaze direction classes, or (ii) label each of the person's body direction and gaze direction as a body direction vector in a two-dimensional plane or three-dimensional space. and a learning device generated by labeling each of the gaze direction vectors. According to clause 12,
The first learning image or the second learning image is the sensed body direction of the person obtained using the sensing information of the gyroscope sensor at the time of shooting in an image of a person wearing a gyroscope sensor. Using information and sensed gaze direction information, (i) the sensed body direction information and the sensed gaze direction information, respectively, among preset body direction classes and preset gaze direction classes in a two-dimensional plane or three-dimensional space; or (ii) label each of the sensed body direction information and the sensed gaze direction information in a two-dimensional plane or three-dimensional space with a specific body direction class and a specific gaze direction class corresponding to the corresponding person. A learning device created by labeling with each of the body direction vector and the gaze direction vector. According to clause 12,
The processor, (III) inputs at least one evaluation image to the body convolutional layer, causes the body convolutional layer to perform a convolutional operation on the evaluation image at least once to determine a third person included in the evaluation image. At least one third body feature map is generated by extracting body features, and the evaluation image is input to the head convolutional layer to perform a convolutional operation on the evaluation image at least once. generate at least one second head feature map by extracting the head features of the third person, and generate a second integrated feature map by concatenating the third body feature map and the second head feature map, Inputting a second integrated feature map to the head FC layer to cause the head FC layer to perform an FC operation on the second integrated feature map at least once to predict the direction in which the front of the third person's head is facing Predicted head direction information is output, and the body convolutional layer, the head convolutional layer, and the head FC are generated with reference to the second predicted head direction information and a third ground truth corresponding to the evaluation image. A learning device further performing a processor that evaluates the gaze direction detection model including the layer. According to clause 18,
The processor calculates accuracy through the following equation using the second predicted head direction information and the third ground truth, and evaluates the gaze direction detection model using the calculated accuracy.

In the above equation, N is the total number of the second predicted head direction information used for evaluation, # of predicted soft corrections is the number of the second predicted head direction information that did not correctly predict the labeled correct answer, # of predicted corrects is the number of correctly predicted labeling correct answers. In a learning device for learning a deep learning-based gaze direction detection model for detecting a person's gaze direction,
A memory storing instructions for learning a deep learning-based gaze direction detection model for detecting a person's gaze direction; and
a processor that performs an operation to learn the gaze direction detection model according to the instructions stored in the memory;
Includes,
The processor, (I) when at least one learning image is acquired, inputs the learning image to a body convolutional layer and causes the body convolutional layer to perform a convolutional operation on the learning image at least once to generate the learning image generate at least one body feature map by extracting the human body features included in the learning image, and input the learning image into a head convolutional layer to perform a convolutional operation on the learning image at least once. A process of generating at least one head feature map from which the head features of the person are extracted, (II) inputting the body feature map to a body FC layer and causing the body FC layer to perform FC operation on the body feature map at least once Output at least one predicted body direction information predicting the direction in which the front of the human body faces, and input an integrated feature map obtained by concatenating the body feature map and the head feature map to the head FC layer. A process of causing a head FC layer to perform an FC operation on the integrated feature map at least once to output at least one predicted head direction information predicting the direction in which the front of the person's head is facing, and (III) the predicted body direction. At least one body direction loss is generated using information and labeled body direction information included in ground truth information corresponding to the learning image, and the predicted head direction information and labeled head direction information included in the ground truth are generated. Generate at least one head direction loss using, learn the body FC layer and the body convolutional layer using the body direction loss, and learn the head FC layer and the head control using the head direction loss. A learning device that performs the process of learning volutional layers. According to clause 20,
The learning image is, in a photographed or cropped person image, (i) each of the person's body direction and gaze direction corresponds to the corresponding person among preset body direction classes and preset gaze direction classes in a two-dimensional plane or three-dimensional space; By labeling each of the person's body direction and gaze direction with a specific body direction class and a specific gaze direction class, or (ii) labeling each of the person's body direction and gaze direction with a body direction vector and a gaze direction vector, respectively, in a two-dimensional plane or three-dimensional space. A learning device that has been created. According to clause 20,
The learning image is a person image taken of a person wearing a gyroscope sensor, and the person's sensed body direction information and sensed gaze direction information obtained using the sensing information of the gyroscope sensor at the time of shooting Using, (i) any one specific body corresponding to each of the sensed body direction information and the sensed gaze direction information among preset body direction classes and preset gaze direction classes in a two-dimensional plane or three-dimensional space labeling each of a direction class and a specific gaze direction class, or (ii) labeling each of the sensed body direction information and the sensed gaze direction information in a two-dimensional plane or three-dimensional space as a body direction vector and a gaze direction vector of the person in question. A learning device created by labeling each.