KR20210085270A - Apparatus and method for detecting interest in gazed objects - Google Patents

Abstract

An apparatus and a method for detecting a gaze target interest are disclosed. A gaze target interest detection apparatus according to an embodiment of the present invention comprises one or more processors and an execution memory for storing at least one or more programs executed by the one or more processors. The at least one program receives an image captured by a user gazing at a display from at least one image sensor. Tag information inserted into tag-based content related to the target audience output through the display is input. It is possible to detecting, from the image, facial feature information of the user looking at the gaze target, and gaze information tracking the gaze of the user looking at the gaze target. The user's degree of interest in the gaze target is detected based on the facial feature information, the gaze information, and the tag information.

Description

Apparatus and method for detecting the interest of the target audience {APPARATUS AND METHOD FOR DETECTING INTEREST IN GAZED OBJECTS}

The present invention relates to a technology for detecting an interest in an object, and more particularly, to a technology for detecting an interest in an object that a user gazes at.

A conventional method of detecting a user's interest in an object is fixedly attached to the top or bottom of the monitor while sitting on a chair arranged so that the monitor can be viewed from a close distance, such as Tobii's eye tracking system. It uses an image sensor system that can capture the face of As a preprocessing process, a user-monitor calibration process is performed through the detection of the user's gaze direction gazing at a specific location of the monitor, and the user is able to see where the user looks on the screen in a line that does not deviate from the position of the face during calibration. A method of detecting an object in which the gaze point stays is mainly used by detecting the gaze point. This method requires a calibration process, and after calibration, the user's movement is extremely limited, and there is a limitation in detecting only the gaze on the target. If you go out of the calibration position or turn your head a lot, it is difficult to detect the gaze point.

Another method is to attach an image sensor that can directly photograph the user's eyes by attaching it to eyeware equipment such as HMD or AR Eye-glass to detect the user's gaze movement and gaze point. In addition, there is a method of detecting where the user's gaze is gazing by attaching an image sensor that captures the space in the direction in which the user is looking to the outside of the eyewear. This method is mainly used to provide virtual or augmented reality services to users through VR or AR devices, and the eye-gazing image sensor always looks at the eye at a certain point in time and uses active light such as infrared to facilitate pupil detection. This has the advantage of precisely detecting the gaze, but it is inconvenient to wear eyewear equipped with a separate image sensor, and from the standpoint of a smart terminal that provides services such as contents, it is difficult to provide information or users without eyewear. The disadvantage is that there is no way to find out the degree of interest of the rejecting user.

Meanwhile, Korean Patent Registration No. 10-1647969, “A device for detecting a user's gaze, a method thereof, and a computer program for executing the method,” uses a depth camera to photograph a user, and the photographed depth Disclosed are an apparatus and method for detecting a user's gaze that can effectively detect a user's gaze using a map.

An object of the present invention is to detect a user's level of interest and interactively provide content reflecting the user's level of interest.

In addition, an object of the present invention is to maximize an advertisement effect by detecting a user's level of interest.

Another object of the present invention is to provide diagnostic information based on a gaze characteristic of a gaze by detecting a user's level of interest.

Another object of the present invention is to provide an interactive interaction service by detecting a user's level of interest.

To achieve the above object, an apparatus for detecting a gaze target interest according to an embodiment of the present invention includes one or more processors and an execution memory storing at least one or more programs executed by the one or more processors, and the at least one or more The program receives an image of a user gazing at a display from at least one or more image sensors, receives tag information inserted in tag-based content about a gazing target output through the display, and selects the gazing target from the image Detecting facial characteristic information of the looking user, gaze information tracking the gaze of the user looking at the gaze target, and the user's interest in the gaze target based on the facial feature information, the gaze information, and the tag information can be detected.

In this case, the tag information may include at least one of a location, an area, a size, an attribute, a product category, and a material of the gaze target output on the display.

At this time, when the gaze point of the user is maintained for a preset time based on the gaze information, the at least one program detects a change in the user's facial features for the preset time based on the facial feature information, It is possible to detect the user's interest level.

In this case, the user's level of interest may be detected as at least one of indifference, liking, and dislike according to the change in the facial feature.

In this case, the at least one program may detect the degree of interest of the user based on at least one of a change in the feature point of the user's mouth and a change in the feature point of the eyebrow among the changes in the user's facial features.

In addition, in the gaze target interest detection method according to an embodiment of the present invention for achieving the above object, the gaze target interest detection method of the gaze target interest detection apparatus, photographing a user gazing at a display from at least one image sensor receiving an image, receiving tag information inserted in tag-based content about the gaze target output through the display, and facial feature information of a user looking at the gaze target from the image, looking at the gaze target and detecting gaze information tracking the user's gaze, and detecting the user's interest in the gaze target based on the facial feature information, the gaze information, and the tag information.

In this case, the tag information may include at least one of a location, an area, a size, an attribute, a product category, and a material of the gaze target output on the display.

In this case, the detecting of the user's level of interest may include, when the user's gaze point is maintained for a preset time based on the gaze information, detects a change in the user's facial features for the preset time based on the facial feature information. It is possible to detect the user's level of interest by detecting.

The user's level of interest may be detected as at least one of indifference, liking, and dislike according to the change in the facial feature.

In this case, the detecting of the user's degree of interest may include detecting the user's degree of interest based on at least one of a change in the feature point of the user's mouth and a change in the feature point of the eyebrow among the changes in the user's facial features.

The present invention can detect the user's interest level and interactively provide content reflecting the user's interest level.

In addition, the present invention can maximize the advertising effect by detecting the user's interest.

In addition, the present invention can provide diagnostic information based on the gaze characteristic of the gaze by detecting the user's level of interest.

In addition, the present invention can provide an interactive interaction service by detecting a user's level of interest.

1 is a block diagram illustrating an apparatus for detecting a gaze target's interest according to an embodiment of the present invention.
FIG. 2 is a detailed block diagram illustrating an example of a gaze target interest level detector illustrated in FIG. 1 .
3 and 4 are diagrams illustrating an apparatus for detecting a gaze target's interest using a smart device and a base-type display apparatus according to an embodiment of the present invention.
5 is an operation flowchart illustrating a method for detecting a gaze target interest according to an embodiment of the present invention.
6 is a diagram illustrating a computer system according to an embodiment of the present invention.

The present invention will be described in detail with reference to the accompanying drawings as follows. Here, repeated descriptions, well-known functions that may unnecessarily obscure the gist of the present invention, and detailed descriptions of configurations will be omitted. The embodiments of the present invention are provided in order to more completely explain the present invention to those of ordinary skill in the art. Accordingly, the shapes and sizes of elements in the drawings may be exaggerated for clearer description.

Throughout the specification, when a part "includes" a certain element, it means that other elements may be further included, rather than excluding other elements, unless otherwise stated.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating an apparatus for detecting a gaze target interest according to an embodiment of the present invention. FIG. 2 is a detailed block diagram illustrating an example of a gaze target interest level detector illustrated in FIG. 1 . 3 and 4 are diagrams illustrating an apparatus for detecting a gaze target's interest using a smart device and a base-type display apparatus.

Referring to FIG. 1 , an apparatus for detecting a gaze target interest according to an embodiment of the present invention includes an image sensor input unit 110 , a content input unit 120 , a gaze target interest level detection unit 130 , an interaction application interworking unit 140 , and a database. part 150 .

The image sensor input unit 110 may receive an image captured by a user gazing at the display from at least one image sensor.

The content input unit 120 may receive tag information inserted into the tag-based content related to the gaze target output through the display.

In this case, the content input unit 120 may receive tag-based content including tag information including the type or product name of an object displayed on the screen for each frame in the time series within the content, and the location and area on the screen. .

In addition, the content input unit 120 may receive a 3D map for the arrangement and shape of a space and a 3D map including tag information containing location and area information of an object constituting the space as tag-based content.

In this case, the tag information may include at least one of a location, an area, a size, an attribute, a product category, and a material of the gaze target output on the display.

The gaze target interest detection unit 130 detects, from the image, facial feature information of the user looking at the gaze target, gaze information tracking the gaze of the user looking at the gaze target, the facial feature information, the gaze information and The user's interest in the gaze target may be detected based on the tag information.

At this time, the gaze target interest detection unit 130 may receive the input image and tag information, and when the user gazes at an object in the content or space, identifies the gaze target and detects the interest in the gaze target.

In this case, the gaze target interest detection unit 130 detects a change in the user's facial features for the preset time based on the facial feature information when the gaze point of the user is maintained for a preset time based on the gaze information Thus, the user's level of interest can be detected.

In this case, the user's level of interest may be detected as at least one of indifference, liking, and dislike according to the change in the facial feature.

In this case, the gaze target interest level detection unit 130 may detect the user's level of interest based on at least one of a change in the feature point of the user's mouth and a change in the feature point of the eyebrow among the changes in the user's facial features.

Referring to FIG. 2 , the gaze target interest detection unit 130 includes a facial feature detection unit 131 , a time series tag parser unit 132 , an interest level detection unit 133 , a gaze gaze tracking unit 134 , a gaze target detection unit) and an interest recorder (136).

The facial feature detection unit 131 may detect facial feature information for each image input from the image sensor input unit 110 . In this case, the facial feature information used may include eyes, nose, mouth, eyebrows, ears, jaw line, etc. constituting the face, and is defined as a facial landmark composed of several or dozens of feature points. Alternatively, for a 3D model of a face composed of a triangular or quadrangular mesh, it may be defined as a feature point of a face corresponding to a vertex constituting the mesh.

The feature point of the face may include features of elements constituting the face.

In this case, the facial feature detection unit 131 may utilize a facial feature information detection technology such as PRNet. However, in PRNet, the 3D model provided as an output can correspond to the scaled 3D model expressed in the normalized 3D space without the definition of scale.

In this case, the facial feature detection unit 131 may remove the scale information by using the correspondence between the feature points of the scaled 3D model input through the individual image sensor in order to detect feature information of the face corresponding to the physical space.

In this case, the facial feature detector 131 may detect a 3D facial feature corresponding to the size of an actual face based on a 3D spatial coordinate system obtained through camera correction after removing scale information from the scaled 3D model.

In this case, the facial feature detector 131 may perform a visibility test on the 3D model obtained for each image sensor by using a plurality of image sensors in consideration of the viewpoint of each image sensor and the location of the user.

In this case, the facial feature detector 131 may selectively collect 3D model information obtained from an image having maximum visibility.

In this case, the facial feature detection unit 131 may integrate information not seen in each image sensor based on one 3D spatial coordinate system.

In this case, the facial feature detector 131 may consistently express all information of the multi-camera and detect the facial feature from the integrated information.

At this time, the facial feature detector 131 determines that the absolute value of the dot product between the normal vector of the mesh and the vector viewed by the camera is negative on the basis of individual triangular or quadrangular meshes constituting the 3D model. By recording the information of the largest image in the corresponding mesh, it is possible to obtain a 3D model with maximized visibility.

At this time, the facial feature detection unit 131 unwraps the acquired user's 3D shape and color onto the 3D model in the same way as a uv-Map used to express a human-like character in the field of computer graphics. (Same as a picture of a globe unfolded in two dimensions) processing can be performed.

At this time, the facial feature detection unit 131 learns the unwrapped image through deep learning techniques, and can detect all facial features of the user at the same time even when the user turns his/her head to the side by about 90 degrees based on the display screen. .

In this case, the facial feature detection unit 131 can detect the user's face even when the user's face does not stare at the front, and can efficiently detect the user's facial features looking around multiple displays or spaces.

In this case, the facial feature detector 131 may detect a facial feature from an individual image and then detect one facial feature using a correspondence between the features.

At this time, the facial feature detection unit 131 actively uses an image acquired using an image sensor composed only of a color camera such as a conventional webcam, a pattern such as Kinect, or TOF (Time of Flight), Active Stereo, etc. In the case of a sensor capable of obtaining information, a single image sensor or a configuration of a much smaller number of sensors can achieve the same results as in the embodiment configured with only a color camera.

The eye gaze tracking unit 134 may use the 3D feature information of the face found by the facial feature detection unit 131 to identify the positions of the left and right eyes in the image and to find the image region including the user's eye region. have.

In this case, the eye gaze tracking unit 134 may identify an image sensor having the highest eye visibility using 3D feature information detected from a plurality of image sensors.

At this time, the gaze tracking unit 134 determines that the absolute value of the dot product between the normal vector perpendicular to the surface of the eye obtained using 3D information of the eye region and the vector viewed by the camera is negative. can be identified by selecting the largest image sensor.

At this time, the eye gaze tracking unit 134 may detect the position of the pupil by identifying the information of the iris and the pupil in the image through learning using a deep learning technique for the left and right eye image regions that have the most visibility. .

In this case, the gaze tracking unit 134 may find the position of the 3D pupil by referring to the 3D model of the user's face obtained by the facial feature detecting unit 131 for the position of the found pupil.

In this case, since the found 3D pupil position is the position of the pupil, which is the surface of the actual user's spherical eye, the gaze gaze tracking unit 134 may detect the gaze direction in 3D using the 3D model of the eyeball.

The 3D model of the eyeball can be simplified to a spherical model having a central point and a certain radius, and more complex real eyeball models can be simulated in 3D.

In addition, in the case of the spherical eye model, when precision such as a mouse replacement of a monitor is not required, the radius may be determined based on the average anatomical size of the human body in consideration of the face size or age.

In this case, the eye gaze tracking unit 134 may set the center point by predicting the user's anatomical skeleton based on the facial features obtained through the facial feature detecting unit 131 .

At this time, the gaze tracking unit 134 finds the 3D gaze intersection linking the 3D gaze directions of the left and right eyes, and thus the 3D position of the gaze point, which is the 3D gaze intersection point centered on the 3D spatial coordinate system obtained through camera calibration, and It is possible to find a gaze candidate region in consideration of an error range occurring in the gaze tracking process.

The error may be caused by the pupil detection, the center point and radius of the eye model, and the detection uncertainty of the 3D face model. In this case, the eye gaze tracking unit 134 takes into account the size of the object identified in the content on the screen in 3D space. range can be determined.

In this case, the gaze tracking unit 134 may provide the set 3D error range and the 3D gaze point together as a 3D elliptical model.

The time series tag parser 132 may interpret tag information inserted in the content input from the tag-based content input unit 120 .

In this case, the time series tag parser 132 may provide 3D information of the gaze target displayed for each frame displayed on the display screen to the gaze target detection unit 135 .

At this time, the time series tag parser 132 calculates the size and location of the gaze target in 3D space in consideration of the 3D spatial arrangement information of the display displayed around the 3D spatial coordinate system, and generates 3D information of the gaze target. have.

In addition, the time series tag parser 132 analyzes the location, size, and attribute information of an object having tag information on a digital 3D map obtained through spatial 3D scan in advance when a base such as a real store is targeted and gazes at it. It may be provided to the target detection unit 135 .

The gaze target detection unit 135 uses the tag information of the target currently displayed on the display screen obtained through the time series tag parser unit 132 and the 3D information of the gaze target detected by the gaze tracking unit 134 to make the most of the interaction with the user. An adjacent gaze target may be detected.

In this case, the gaze target detector 135 may detect the gaze target closest to the 3D gaze point in distance from the 3D gaze point among the objects within the error range in consideration of the 3D gaze point and the 3D error range in the 3D information of the gaze target.

In this case, the gaze target detection unit 135 may provide the 3D information of the gaze target and the detected gaze target and the corresponding parser information to the interest level recorder 136 .

As a result of 3D gaze tracking of the user's gaze based on the 3D gaze point information of the gaze gaze tracking unit 134 , the interest detection unit 133 determines that the user's gaze point is maintained for a predetermined time or longer in the time series. of interest can be detected.

Also, the interest detection unit 133 may selectively detect the user's interest only when the gaze point is maintained for a predetermined time or longer for computational efficiency and the gaze target detection unit 135 detects the gaze target.

In this case, when the condition is activated, the interest detection unit 133 may examine a change in facial features input in chronological order within a time range in which the gaze point is maintained by the facial feature detection unit 131 .

In this case, the interest level detector 133 may detect the user's level of interest based on the change in facial features.

Interest can be broadly defined as liking, indifference, and dislike.

In this case, the interest level detection unit 133 may detect the three categories of interest by analyzing the change in facial features.

In this case, the interest level detection unit 133 may detect it through the overall motion analysis of the face.

For example, the user may nod his head consciously or unconsciously when he has a liking for the gaze target.

Through this, the interest level detection unit 133 may detect the level of interest as a crush by detecting a movement in which a change in a facial feature repeatedly moves up and down in a predetermined range with respect to the gaze target.

Conversely, in case of unfavorable feeling, the user may move the head left and right with respect to the gaze target.

In this case, the interest level detection unit 133 may detect the level of interest as an unfavorable feeling by detecting a movement in which a change in a facial feature moves left and right with respect to the gaze target.

In the case of indifference, the user may not change the facial feature with respect to the gaze target.

At this time, when the change in the facial feature does not change with respect to the gaze target, the interest level detector 133 may detect the interest level with indifference.

In this case, the interest detection unit 133 may also detect the movement of the feature point of the mouth.

At this time, in the case of a favorable feeling, the interest detection unit 133 may identify by analyzing whether the feature points at both ends of the lips move relatively in the upward direction of the face based on the central feature point of the lips.

In this case, the interest detection unit 133 may identify the degree of liking by analyzing the relative movement ranges of the feature points of the mouth in the smiling direction.

At this time, the interest detection unit 133, in the case of liking, 'Oh!, Wow!' It can also be identified by detecting the shape of the mouth when exclamation such as etc.

In this case, in the case of dislike, the interest detection unit 133 may identify the degree of dislike through analysis of the movement range of the mouth feature points in the opposite direction to the favorable feeling.

In this case, in the case of indifference, the interest detection unit 133 may identify by analyzing the movement range of individual feature points in the arrangement of the feature points of the mouth corresponding to the expressionless state.

In this case, the interest level detection unit 133 may also detect the level of interest through movement of features constituting the eyebrows.

In this case, the interest detection unit 133 may identify the unfavorable feeling by detecting movement of facial features that occur when the forehead is narrowed and the face is frowned.

In this case, the interest level detection unit 133 may identify by detecting movements of facial features that occur in the case of favorable surprise in the case of favorable impression.

In this case, the interest level detection unit 133 may also define the level of interest through various facial feature changes for each individual.

The above-described changes in facial features according to the degree of interest may occur singly or in combination.

At this time, the interest detection unit 133 generates as learning data the change of facial features in chronological order and the interest value at that time when viewing content of liking, dislike, and indifference by various users to detect interest by learning through a deep learning technique. .

The output value of interest has a positive value that increases according to the degree of liking, zero in case of indifference, and a negative value that increases according to dislike, and can be normalized to a value of [-1, 1]. Also, the output value of the degree of interest may include the gaze time together.

The interest level recording unit 136 receives the interest maintenance time and normalized interest value input from the interest level detection unit 133 and the parser information of the gaze target inputted from the gaze target detection unit 135, and together with the database unit 150 can be recorded

Also, the database unit 150 may provide recorded values to the interaction application interworking unit 140 .

The database unit 150 may store the user's interest in association with the tag information.

The interaction application interworking unit 140 may provide a customized service suitable for the user based on the user's level of interest.

In this case, the interaction application linkage unit 140 may include the gaze target interest level detection unit 130 , and the user selectively views the tag-based content to identify the user's interest level through the gaze target interest level detection unit 130 . It can be expressed on a display device.

Also, the interaction application interworking unit 140 may provide content satisfying the user's interest in an interactive manner according to the identified user's interest.

In this case, the interaction application interworking unit 140 may provide information or content in a user's desired direction by smartly identifying the user's interest through the application through the interaction application.

Referring to FIG. 3 , it can be seen that an embodiment of an apparatus and method for detecting a gaze target interest through a smart device such as a TV is shown. Referring to FIG. 4 , an apparatus and method for detecting an gaze target interest through a base such as a shopping store It can be seen that the examples are shown.

As in the embodiment through a smart device or a base, the image sensor input unit 131 may include a single display and a single or multiple image sensors, and may include a plurality of displays and a plurality of image sensor units. have.

In addition, the tag-based content input to the tag-based content input unit 120 is the location and area of the gaze target displayed by time on the display unit of the smart device as shown in the example through the smart device of FIG. It may include tag information including attributes or the like, or tag information including the location, size, and attributes of an object disposed in space in the space as an example through a base. In addition to this, the information included in the tag may include various information expressing properties of the object, such as a product category and a material.

Referring to FIGS. 3 and 4 , the image sensor input unit 110 may include a single or multiple sensors, and may include multiple composite image sensors such as a stereo camera or a Kinect inside a single sensor. These sensors are placed on a smart device or space, and in general, in the field of computer vision, a universal camera calibration that finds out internal properties such as the relative placement in space and the focus of the camera with respect to a specific coordinate system between multiple cameras. Through this process, data can be exchanged around one 3D spatial coordinate system. In addition, in the camera calibration process, relative arrangement information about the 3D space coordinate system of the display unit and the relative arrangement information about the 3D space coordinate system for each object in space can also be secured.

If a single point in space can be photographed without occlusion by multiple image sensors through this camera calibration process, the relative 3D position centered on the 3D spatial coordinate system that became the standard in the camera calibration process for a single point in space is It can be calculated using common computer vision techniques.

The configuration of the present invention has been described through the above specific methodologies and examples.

The present invention can explain the facial feature analysis-based gaze object interest detection technology, which records information on the target corresponding to the gaze direction through the existing simple gaze tracking, and provides the information first in future searches [Reference 3 ], it is possible to provide the user with content or information that more satisfies the user's preference by providing the user's interest in the subject.

5 is an operation flowchart illustrating a method for detecting a gaze target interest according to an embodiment of the present invention.

Referring to FIG. 5 , in the method for detecting an interest of a gaze target according to an embodiment of the present invention, an image and content may be first received ( S210 ).

That is, in step S210, an image captured by a user gazing at the display may be received from at least one image sensor.

In this case, in step S210, tag information inserted into the tag-based content related to the gaze target output through the display may be received.

In this case, in step S210, tag-based content including tag information including the type or product name of an object displayed on the screen for each frame in the time series within the content, and the location and area on the screen may be input.

In this case, in step S210, a 3D map for the arrangement and shape of a space and a 3D map including tag information containing location and area information of an object constituting the space may be input as tag-based content.

In this case, the tag information may include at least one of a location, an area, a size, an attribute, a product category, and a material of the gaze target output on the display.

Also, the method for detecting the gaze target interest level according to an embodiment of the present invention may detect the gaze target interest level ( S220 ).

That is, step S220 detects, from the image, facial feature information of the user looking at the gaze target, gaze information tracking the gaze of the user looking at the gaze target, and the facial feature information, the gaze information and the The user's interest in the gaze target may be detected based on the tag information.

In this case, in step S220 , when the user gazes at an object in the content or space by receiving the input image and tag information, the gaze target may be identified and the degree of interest in the gaze target may be detected.

In this case, in step S220, when the gaze point of the user is maintained for a preset time based on the gaze information, a change in the user's facial features is detected for the preset time based on the facial feature information, and the user of interest can be detected.

In this case, the user's level of interest may be detected as at least one of indifference, liking, and dislike according to the change in the facial feature.

In this case, in step S220, the user's degree of interest may be detected based on at least one of a change in the feature point of the user's mouth and a change in the feature point of the eyebrow among the changes in the user's facial features.

In this case, in step S220 , facial feature information may be detected for each image input from the image sensor input unit 110 . In this case, the facial feature information used may include eyes, nose, mouth, eyebrows, ears, jaw line, etc. constituting the face, and is defined as a facial landmark composed of several or dozens of feature points. Alternatively, for a 3D model of a face composed of a triangular or quadrangular mesh, it may be defined as a feature point of a face corresponding to a vertex constituting the mesh.

The feature point of the face may include features of elements constituting the face.

In this case, step S220 may utilize a facial feature information detection technology such as PRNet. However, in PRNet, the 3D model provided as an output can correspond to the scaled 3D model expressed in the normalized 3D space without the definition of scale.

In this case, in step S220 , scale information may be removed by using the correspondence between the feature points of the scaled 3D model input through the individual image sensor in order to detect feature information of the face corresponding to the physical space.

In this case, in step S220 , after removing the scale information from the scaled 3D model, 3D facial features corresponding to the size of the actual face may be detected based on the 3D spatial coordinate system obtained through camera correction.

In this case, in step S220 , a visibility test may be performed on the 3D model obtained for each image sensor using a plurality of image sensors in consideration of the viewpoint of each image sensor and the location of the user.

In this case, in step S220, 3D model information obtained from an image having maximum visibility may be selectively collected.

In this case, in step S220 , information that has not been seen in each image sensor may be integrated based on one 3D spatial coordinate system.

In this case, in step S220, all information of multiple cameras may be consistently expressed and facial features may be detected from the integrated information.

At this time, in step S220, the absolute value of the dot product between the normal vector of the mesh and the vector viewed by the camera on the basis of the individual triangular or square mesh constituting the 3D model is the most negative in the negative direction. A 3D model with maximized visibility can be obtained by recording large image information in the corresponding mesh.

At this time, step (S220) is unwrap (2) on the 3D model in the same way as the uv-Map used to express the human body character in the computer graphics field using the acquired 3D shape and color of the user. (same as a dimensionally unfolded globe photo) processing can be performed.

At this time, step S220 learns the unwrapped image through deep learning techniques, and it is possible to simultaneously detect all facial features of the user even in a situation where the user turns his/her head to the side by about 90 degrees based on the display screen.

In this case, in step S220, the user's face can be detected even when the user's face does not stare at the front, and facial features of the user looking around the multiple displays or space can be efficiently detected.

In this case, in step S220, one facial feature may be detected by using the correspondence between the features after detecting the facial features in the individual images.

At this time, step S220 is an image acquired using an image sensor composed only of a color camera such as a conventional webcam, a pattern such as Kinect, or TOF (Time of Flight), Active Stereo, etc. In the case of an obtainable sensor, the same result as in the embodiment comprising only a color camera can be obtained only with a single image sensor or with a much smaller number of sensors.

In this case, in step S220, the position of the left and right eyes in the image may be identified using the 3D feature information of the face found by the facial feature detection unit 131, and the image region including the user's eye region may be found. have.

In this case, in step S220, an image sensor having the highest eye visibility may be identified using 3D feature information detected from a plurality of image sensors.

At this time, in step S220, the absolute value of the dot product between the normal vector perpendicular to the surface of the eye obtained using 3D information of the eye region and the vector viewed by the camera is the largest in the negative direction. It can be identified by selecting an image sensor.

At this time, in step S220, information on the iris and pupil in the image may be identified through learning using a deep learning technique for the left and right eye image regions in which visibility is most secured, and the position of the pupil may be detected.

In this case, in step S220 , the position of the 3D pupil may be found by referring to the 3D model of the user's face obtained by the facial feature detection unit 131 for the position of the found pupil.

At this time, in step S220, since the found 3D pupil position is the position of the pupil, which is the surface of the actual user's spherical eyeball, the gaze direction of the gaze can be detected in 3D using the 3D model of the eyeball.

The 3D model of the eyeball can be simplified to a spherical model having a central point and a certain radius, and more complex real eyeball models can be simulated in 3D.

In addition, in the case of the spherical eye model, when precision such as a mouse replacement of a monitor is not required, the radius may be determined based on the average anatomical size of the human body in consideration of the face size or age.

In this case, in step S220 , the central point may be set by predicting the user's anatomical skeleton based on the facial features obtained through the facial feature detection unit 131 .

At this time, in step S220, the 3D gaze tracking point and the 3D gaze point that is the 3D gaze intersection point centered on the 3D spatial coordinate system obtained through camera calibration by finding the 3D gaze intersection linking the 3D gaze directions of the left and right eyes It is possible to find the candidate area for the examination taking into account the error range that occurs in the process.

The error may be caused by the pupil detection, the center point and radius of the eye model, and the detection uncertainty of the 3D face model. In this case, in step S220, the error range is determined by considering the 3D spatial size of the object to be identified in the content on the screen. can

In this case, step S220 may provide the set 3D error range and 3D gaze point together as a 3D elliptical model.

In this case, in step S220 , tag information inserted in the content input by the tag-based content input unit 120 may be interpreted.

In this case, in step S220 , 3D information of the gaze target shown for each frame displayed on the display screen may be provided to the gaze target detection unit 135 .

In this case, in step S220, the size and position of the gaze target in 3D space may be calculated in consideration of the 3D spatial arrangement information of the display displayed based on the 3D spatial coordinate system, and 3D information of the gaze target may be generated.

At this time, in step S220, when a base such as an actual store is a target, the gaze target detection unit analyzes the location, size, and attribute information of the target having tag information on the digital 3D map obtained through spatial 3D scan in advance. (135) may be provided.

In this case, step S220 is performed with the user using the tag information of the object displayed on the current display screen obtained through the time series tag parser unit 132 and 3D information of the gaze target detected by the gaze tracking unit 134 . The closest gaze target may be detected.

In this case, in step S220 , the gaze target closest to the 3D gaze point in distance from the 3D gaze point may be detected among the objects within the error range by considering the 3D gaze point and the 3D error range in the 3D information of the gaze target.

In this case, in step S220 , the 3D information of the gaze target may be provided to the interest level recorder 136 with the detected gaze target and corresponding parser information.

At this time, in step S220, as a result of 3D gaze tracking of the user's gaze based on the 3D gaze point information of the gaze tracking unit 134, when the user's gaze point is maintained for a predetermined time or more in the time series, the gaze target is It is possible to detect the user's interest in

In this case, in step S220, the user's interest may be detected only when the gaze point is maintained for a predetermined time or longer for computational efficiency and the gaze target detection unit 135 detects the gaze target.

In this case, in step S220 , when the condition is activated, the facial feature detection unit 131 may examine changes in facial features input in chronological order within a time range in which the gaze point is maintained.

In this case, in step S220, the user's interest level may be detected based on the change in facial features.

Interest can be broadly defined as liking, indifference, and dislike.

In this case, in step S220, the degree of interest of the three categories may be detected by analyzing the change in the facial feature.

In this case, step S220 may be detected through the overall motion analysis of the face.

For example, the user may nod his head consciously or unconsciously when he has a liking for the gaze target.

In this case, in step S220 , the degree of interest may be detected as a crush by detecting a movement in which the change of the facial feature repeatedly moves up and down with respect to the gaze target in a certain range.

Conversely, in case of unfavorable feeling, the user may move the head left and right with respect to the gaze target.

In this case, in step S220 , the degree of interest may be detected as unfavorable by detecting a movement in which the change of the facial feature moves left and right with respect to the gaze target.

In the case of indifference, the user may not change the facial feature with respect to the gaze target.

In this case, in step S220, when the change in the facial feature does not change with respect to the gaze target, the degree of interest may be detected with indifference.

In this case, step S220 may be detected through the movement of the feature point of the mouth.

In this case, in step S220 , in the case of a favorable feeling, it is possible to identify by analyzing whether the feature points at both ends of the lips move relatively in the upward direction of the face based on the central feature point of the lips.

In this case, in step S220, the degree of liking may be identified by analyzing the relative movement ranges of the feature points of the mouth in the smiling direction.

At this time, step S220 is a case of liking, 'Oh!, Wow!' It can also be identified by detecting the shape of the mouth when exclamation such as etc.

In this case, in step S220 , in the case of unfavorable feeling, the degree of unfavorable feeling may be identified through analysis of the movement range of the mouth feature points in the opposite direction to the favorable feeling.

In this case, in step S220 , in the case of indifference, the movement range of individual feature points in the arrangement of the feature points of the mouth corresponding to the expressionless state may be analyzed and identified.

In this case, in step S220, the degree of interest may also be detected through movement of features constituting the eyebrows.

In this case, in step S220, in the case of unfavorable feeling, the facial features may be identified by detecting movements of facial features that occur when the forehead is narrowed and the face is frowned.

In this case, in step S220, on the contrary, in the case of a favorable feeling, it can be identified by detecting the movement of the facial features that occur in the case of a favorable surprise.

In this case, in step S220, the degree of interest may be defined through various facial feature changes for each individual.

The above-described changes in facial features according to the degree of interest may occur singly or in combination.

At this time, in step S220, the degree of interest can be detected by generating as learning data the change of facial features in chronological order and the interest value at that time when viewing content of liking, dislike, and indifference by various users, and learning through a deep learning technique.

The output value of interest has a positive value that increases according to the degree of liking, zero in case of indifference, and a negative value that increases according to dislike, and can be normalized to a value of [-1, 1]. Also, the output value of the degree of interest may include the gaze time together.

At this time, in step S220, the interest maintenance time and normalized interest value input from the interest level detection unit 133 and the parser information of the gaze target inputted from the gaze target detection unit 135 are received together with the database unit 150 ) can be recorded.

In addition, the method for detecting a gaze target interest according to an embodiment of the present invention may provide a user-customized service (S230).

That is, in step S230, a customized service suitable for the user may be provided based on the user's level of interest.

In this case, in step S230 , the database unit 150 may provide the recorded values to the interaction application interworking unit 140 .

In this case, in step S230, the database unit 150 may store the user's interest in association with the tag information.

In this case, in step S230 , the tag-based content may be selectively displayed on the display device viewed by the user in order to identify the user's interest level through the gaze target interest level detection unit 130 .

In this case, in step S230, content satisfying the user's interest may be provided to the user in an interactive manner according to the identified user's interest.

In this case, in step S230, the user through the interaction application can smartly identify the user's interest in the application and provide information or content in the user's desired direction.

6 is a diagram illustrating a computer system according to an embodiment of the present invention.

Referring to FIG. 6 , the apparatus for detecting a gaze target's interest according to an embodiment of the present invention may be implemented in a computer system 1100 such as a computer-readable recording medium. As shown in FIG. 6 , the computer system 1100 includes one or more processors 1110 , a memory 1130 , a user interface input device 1140 , and a user interface output device 1150 that communicate with each other via a bus 1120 . and storage 1160 . In addition, the computer system 131 may further include a network interface 1170 coupled to the network 1180 . The processor 1110 may be a central processing unit or a semiconductor device that executes processing instructions stored in the memory 1130 or the storage 1160 . The memory 1130 and the storage 1160 may be various types of volatile or non-volatile storage media. For example, the memory may include a ROM 1131 or a RAM 1132 .

A gaze target interest detection apparatus according to an embodiment of the present invention includes one or more processors 1110; and an execution memory 1130 for storing at least one or more programs executed by the one or more processors 1110, wherein the at least one or more programs display an image captured by a user gazing at a display from at least one or more image sensors. Receive input, receive tag information inserted in tag-based content related to the gaze target output through the display, and receive facial feature information of the user looking at the gaze target from the image, and the user's gaze looking at the gaze target may detect gaze information that has been tracked, and detect the user's interest in the gaze target based on the facial feature information, the gaze information, and the tag information.

In this case, the tag information may include at least one of a location, an area, a size, an attribute, a product category, and a material of the gaze target output on the display.

At this time, when the gaze point of the user is maintained for a preset time based on the gaze information, the at least one program detects a change in the user's facial features for the preset time based on the facial feature information, It is possible to detect the user's interest level.

In this case, the user's level of interest may be detected as at least one of indifference, liking, and dislike according to the change in the facial feature.

In this case, the at least one program may detect the degree of interest of the user based on at least one of a change in the feature point of the user's mouth and a change in the feature point of the eyebrow among the changes in the user's facial features.

The apparatus and method for detecting the subject's interest level according to an embodiment of the present invention are actually used in viewing content such as education or advertisement through a smartphone or smart pad, monitor, TV, etc., shopping at a store such as a department store, viewing an exhibition hall/experience hall, etc. It is possible to provide a method of finding a user's gaze object for objects in a space or a screen, and automatically detecting the degree of interest in the user's liking, dislike, and indifference toward the object using a non-contact image sensor.

In addition, the apparatus and method for detecting the gaze target's interest according to an embodiment of the present invention is expressed on a screen on a smart terminal such as a smartphone, monitor, advertisement display, kiosk, etc. capable of photographing a user's face using a learning-based image analysis technology The user's level of interest in the user's gaze target is detected through analysis of the user's facial features in the image, and the user's level of interest in the target is determined using a non-contact image. It can be obtained through the sensor.

In addition, the apparatus and method for detecting a target interest level according to an embodiment of the present invention provides intelligent user-customized content or provides intelligent user-customized content by researching or analyzing customer interest in content, advertisements, and exhibits using the user's interest level automatically detected in real time. It can be used for services that provide intelligent interaction, etc., based on the user's interest, such as interactive services between computer and computer.

In addition, the apparatus and method for detecting the target interest level according to an embodiment of the present invention provides a content viewing environment such as education or advertisement through a smart phone or smart pad, monitor, TV, etc., in which it is difficult for a user to observe a user from a predetermined front view of a predetermined location In an environment such as shopping at stores such as department stores or department stores, viewing exhibition halls/experience halls, etc., it is a method to find a user's gaze object for objects in a real space or screen and automatically detect the user's interest in the object using a non-contact image sensor. A user-friendly UX service can be provided by interactively providing content that reflects the user's interest by the computer.

In addition, when applied to a smart advertisement, the apparatus and method for detecting the target interest in the gaze target according to an embodiment of the present invention identify the target in the advertisement and the degree of interest at that time when people passing the billboard pass by and turn their heads to stare at the big data stored By analyzing the advertisement, advertisements suitable for the level of interest of the floating population passing the advertisement board are exposed by the installation location and time of the advertisement board, and the user's gaze is actively drawn to the advertisement board, thereby maximizing the advertisement effect. In addition, if you stop and stare to see the advertisement, the advertisement exposure effect can be maximized by naturally additionally expressing an advertisement that will show additional interest according to the identification of the user's interest. By combining the user's interest level identified according to the interactively provided advertisement, such as age, body type, etc., to build big data, it is possible to minimize exposure to unfavorable advertisements, provide customized advertisements that easily lead to purchases, and collect user trends in real time. can

In addition, when the apparatus and method for detecting the target interest level according to an embodiment of the present invention are applied to store shopping such as a department store, big data on the location and the target where the eyes of consumers who shop i-shopping mainly stay, and the degree of interest of the target The analysis can be used to analyze the characteristics of the consumer group or to more efficiently arrange and operate products in the store. In the case of a store, in combination with the example of the billboard above, it identifies the customer's gender, age, and body type information obtained through the image sensor installed on the billboard, and exposes the products that the customer may prefer to customers in real time to induce customers to the store and lead to purchase. effect can be maximized. In addition, when the present invention is applied to an exhibition hall, the user's interest in viewing the exhibits can be combined with age, gender, etc. to develop the composition of the exhibits in a direction to induce the users' interest.

In addition, when the apparatus and method for detecting the gaze target interest according to an embodiment of the present invention are applied to educational facilities such as kindergartens, children passing through the display identify content elements that are mainly interested, so that the children naturally become interested in the display. It can be used to early screening for diseases related to children's development, such as autism, by analyzing the gaze characteristics of children's gaze when they see an object of interest on the screen.

In addition, when applied to a robot, the gaze target interest detection apparatus and method according to an embodiment of the present invention combine information such as gender, age, etc. obtained through analysis of a user's facial features with the accumulated interest information of an existing user or by time zone. Based on the identified user's immediate interest information, it can be utilized for a service that provides a natural interactive interaction in a direction preferred by the user.

As described above, in the apparatus and method for detecting a gaze target interest according to an embodiment of the present invention, the configuration and method of the embodiments described above are not limitedly applicable, but various modifications may be made to the embodiments. All or part of each embodiment may be selectively combined and configured.

110: image sensor input unit 120: content input unit
130: target interest level detection unit
131: facial feature detection unit 132: time series tag parser unit
133: interest detection unit 134: eye gaze tracking unit
135: gaze target detection unit 136: interest record
140: interaction application linkage
150: database unit
1100: computer system 1110: processor
1120: bus 1130: memory
1131: rom 1132: ram
1140: user interface input device
1150: user interface output device
1160: storage 1170: network interface
1180: network

Claims (10)

one or more processors; and
an execution memory for storing at least one or more programs executed by the one or more processors;
including,
the at least one program
Receive an image of a user gazing at a display from at least one image sensor, and receive tag information embedded in tag-based content about a gazing target output through the display,
Detects facial characteristic information of the user looking at the gaze target from the image, gaze information tracking the gaze of the user looking at the gaze target, and the gaze based on the facial feature information, the gaze information, and the tag information A gaze target interest detection apparatus, characterized in that detecting the user's interest in the target. The method according to claim 1,
The tag information is
The gaze target interest detection apparatus, characterized in that it includes at least one of a location, area, size, attribute, product category, and material of the gaze target output on the display. 3. The method according to claim 2,
the at least one program
When the user's gaze point is maintained for a preset time based on the gaze information, the user's degree of interest is detected by detecting a change in the user's facial features for the preset time based on the facial feature information. A device for detecting the interest of the gaze target. 4. The method according to claim 3,
The user's interest
The gaze target interest detection apparatus, characterized in that the detection of at least one of indifference, liking, and dislike according to the change in the facial feature. 5. The method according to claim 4,
the at least one program
and detecting the degree of interest of the user based on at least one of a change in a feature point of the user's mouth and a change in a feature point of the eyebrow among the changes in the user's facial features. In the gaze target interest detection method of the gaze target interest level detection device,
receiving an image of a user gazing at a display from at least one image sensor, and receiving tag information inserted into tag-based content related to a gazing target output through the display; and
Detects facial characteristic information of the user looking at the gaze target from the image, gaze information tracking the gaze of the user looking at the gaze target, and the gaze based on the facial feature information, the gaze information, and the tag information detecting the user's interest in the object;
Gaze target interest detection method comprising a. 7. The method of claim 6,
The tag information is
The gaze target interest detection method, characterized in that it includes at least one of a location, area, size, attribute, product category, and material of the gaze target output on the display. 8. The method of claim 7,
The step of detecting the user's interest
When the user's gaze point is maintained for a preset time based on the gaze information, the user's degree of interest is detected by detecting a change in the user's facial features for the preset time based on the facial feature information. A method of detecting the interest of the gaze target. 9. The method of claim 8,
The user's interest
The gaze target interest detection method, characterized in that the detection of at least one of indifference, liking, and dislike according to the change in the facial feature. 10. The method of claim 9,
The step of detecting the user's interest
and detecting the degree of interest of the user based on at least one of a change in a feature point of the user's mouth and a change in a feature point of the eyebrow among the changes in the user's facial features.

Images