KR20220090940A - System and method for analyzing speaker-listener recognition and gaze interaction through tracking the eyes of the characters in story-based visual media - Google Patents

Abstract

Disclosed are a speaker-listener recognition and gaze interaction analysis system and method through character gaze tracking in a story-based video medium. A gaze interaction analysis method performed by a gaze interaction analysis system according to an embodiment includes: receiving image information from a story-based video medium; tracking the gaze information of the object using a gaze tracking model using a depth map restored from the received image information and object information recognized from the received image information; tracking gaze information of a counterpart object from gaze information of the tracked object as the gaze information of the tracked object is input to the gaze tracking model; and classifying gaze interaction information based on gaze information of the tracked object and gaze information of the tracked counterpart object.

Description

SYSTEM AND METHOD FOR ANALYZING SPEAKER-LISTENER RECOGNITION AND GAZE INTERACTION THROUGH TRACKING THE EYES OF THE CHARACTERS IN STORY-BASED VISUAL MEDIA

The description below relates to the technique of analyzing the gaze interaction between the speaker and the listener recognized through the gaze tracking of the character.

Common gaze recognition technology required hardware devices and sensors, such as eye trackers. Recently, a technique for estimating the position and angle of the pupil using only images captured by a camera using deep learning technology or estimating a point inside the screen of a device such as a smart phone has been reported.

However, this technique could not obtain information on what object the person gazes at. Although the latest technology for gaze following has been reported as an alternative technology, image media such as dramas and movies that cannot be shot with RGB-D cameras so far lose depth information, so the existing state-of-the-art technology is utilized. Therefore, there is a limitation in that the accuracy significantly decreases when tracking the gaze of a character.

Using a CNN-based learning model, it automatically detects the speaker in the story-based video medium and recognizes the listener by tracking the gaze information, and collects the gaze interaction relationship between the speaker and the listener by extracting the recognized gaze information. Methods and systems for analyzing may be provided.

The relationship of gaze interaction between the speaker and the listener by tracking the gaze information of the speaker recognized through the depth map generated from the video information and the video information in the story-based video medium, and tracking the gaze information of the listener from the tracked gaze of the speaker A method and system for collecting and analyzing can be provided.

The gaze interaction analysis method performed by the gaze interaction analysis system includes the steps of: receiving image information from a story-based video medium; tracking the gaze information of the object using a gaze tracking model using a depth map restored from the received image information and object information recognized from the received image information; tracking gaze information of a counterpart object from gaze information of the tracked object as the gaze information of the tracked object is input to the gaze tracking model; and classifying gaze interaction information based on gaze information of the tracked object and gaze information of the tracked counterpart object.

The object means a speaker among the characters in the story-based video medium, and the step of tracking the gaze information of the object includes receiving the image information into a depth map generation model, and generating the depth map generation model. It may include generating a depth map from the image information using the image information.

The object means a speaking speaker among the characters in the story-based video medium, and the step of tracking gaze information of the object includes receiving the image information into a speaker recognition model and using the speaker recognition model The method may include obtaining speaker recognition information from the image information.

The tracking of the gaze information of the object may include storing the acquired speaker recognition information in a database.

The tracking of the gaze information of the object may include storing a tracking result of the gaze information of the tracked object in a database.

The counterpart object refers to a listener who is a target of staring at a speaker or speaking with the speaker among the characters in the story-based video medium, and the step of tracking gaze information of the counterpart object includes: It may include the step of storing the tracking result in a database.

The classifying the gaze interaction information includes gaze interaction including no interaction, mutual gaze, gaze avoidance, gaze tracking, and mutual interest using gaze information of the tracked object and gaze information of the tracked counterpart object. classifying the gaze interaction based on the action criterion.

Classifying the gaze interaction information may include storing gaze interaction classification data corresponding to mutual gaze among the classified gaze interaction in a database.

The classifying the gaze interaction information includes: the information of the recognized object, the tracking result of the gaze information of the tracked object, the tracking result of the gaze information of the tracked counterpart object, and the gaze interaction classification corresponding to the mutual gaze The method may include providing search information related to gaze interaction through a search of a database including data.

The classifying the gaze interaction information includes the character's disposition, conflict, dialogue discrimination, dialogue positive or negative relationship, and main dialogue in the story based on the interaction relationship between the speaker and the listener through the classified gaze interaction information. It may include analyzing a person and a main object.

The gaze interaction analysis system includes: an input unit for receiving image information from a story-based image medium; a speaker gaze tracking unit that tracks gaze information of an object using a gaze tracking model based on a depth map restored from the received image information and object information recognized from the received image information; a listener recognition unit that tracks gaze information of a counterpart object from gaze information of the tracked object as the gaze information of the tracked object is input to the gaze tracking model; and a gaze interaction classification unit configured to classify gaze interaction information based on gaze information of the tracked object and gaze information of the tracked counterpart object.

The speaker gaze tracking unit receives the image information into a depth map generation model, generates a depth map from the image information using the depth map generation model, receives the image information into a speaker recognition model, and recognizes the speaker Speaker recognition information may be obtained from the image information using a model.

The gaze interaction classification unit, based on gaze interaction criteria including no interaction, mutual gaze, gaze avoidance, gaze tracking, and mutual interest, using gaze information of the tracked object and gaze information of the tracked counterpart object Thus, the gaze interaction can be classified.

The gaze interaction analysis system stores, in the database, information on the recognized object, a tracking result of gaze information of the tracked object, a tracking result of gaze information of the tracked counterpart object, and gaze interaction classification data corresponding to mutual gaze in a database. It may further include a storage unit for storing.

The gaze interaction classification unit, based on the interaction relationship between the speaker and the listener through the classified gaze interaction information, the character's disposition, conflict, dialogue discrimination, dialogue positive or negative relationship, main dialogue person in the story, main object can be analyzed.

It is possible to automatically detect a dialogue scene in a story-based video medium, and to collect information about the speaker-listener's gaze and gaze interaction information.

By using a deep learning model that automatically generates a depth map from RGB image frames of video media such as dramas and movies, it is possible to recognize gaze with high accuracy in drama and movie media without additional hardware equipment, sensors, or camera equipment. In other words, it is possible to automatically collect meaningful information such as who is watching what and who is talking with whom from a simple RGB image medium.

A search service for gaze-related scenes can be devised based on the collected speaker-listener recognition, gaze information, and gaze interaction information. This is possible. Analyzing the gaze interaction relationship along with dialogue, facial expression, and behavior can be utilized for the analysis of drama stories and social relationships between characters, which can be applied to academic research and related commercial services.

1 is an example for comparing eye tracking, according to an embodiment.
2 is a diagram for explaining an operation of a gaze interaction analysis system according to an embodiment.
3 is a block diagram illustrating a configuration of a gaze interaction analysis system according to an embodiment.
4 is an example for explaining a process of recognizing a listener by the speaker's gaze and classifying the gaze interaction, according to an embodiment.
5 is a flowchart illustrating a gaze interaction analysis method according to an embodiment.
6 is an example for explaining a gaze interaction classification system according to an embodiment.
7 is an example for explaining a learning model for recognizing a speaker and a listener, according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

In the embodiment, the operation of automatically detecting the speaker in a story-based video medium such as a movie, recognizing the listener by tracking the gaze, and collecting and analyzing the gaze interaction relationship by extracting the gaze information of the recognized listener. do it with Through this, the gaze interaction relationship between the conversation participants can be automatically collected and analyzed by extracting the gaze person, the gaze object, the gaze direction, etc. of the conversation participant.

In the embodiment, a 'shot' is a portion taken during the time from when the camera starts to operate and ends, and a plurality of shots is defined as a scene. In a dialogue scene, the speaking character (object) is defined as the 'speaker', and the target staring at the speaker or the conversational target (opposite object), that is, the character listening to the speaker is called the 'listener'. to define

1 is an example for comparing eye tracking, according to an embodiment.

1 is a comparison between the prior art and the technique proposed in Examples. If the existing technology is used, since there is no depth information in the drama video image, it is often inferred that the viewer gazes at an object (for example, a character or an object, etc.) that exists behind the video (for example, behind the listener). However, since the technique proposed in the embodiment automatically generates a depth map, the problems of the prior art can be effectively prevented.

4 is an example for explaining a process of recognizing a listener by the speaker's gaze and classifying the gaze interaction, according to an embodiment.

A speaker recognized in the image information may be located at (x1, y1). A listener stared at from the recognized speaker may be located at (x2, y2). Also, the listener may gaze at the speaker located at (x1, y1). In this case, when the speaker and the listener are staring at the same time, it may be determined that they are staring at each other.

2 is a diagram for explaining an operation of a gaze interaction analysis system according to an embodiment.

The gaze interaction analysis system may receive image information from a story-based image medium. In other words, the gaze interaction analysis system may receive the input image (image) 201 .

The gaze interaction analysis system may receive the image information 201 into the depth map generation model 210 . The gaze interaction analysis system may generate a depth map from the image information 201 using the depth map generation model 210 . In this case, the depth map generation model 210 may be a deep learning-based model configured to generate a depth map.

Also, the gaze interaction analysis system may receive the image information 201 into the speaker recognition model 220 . The gaze interaction analysis system may obtain speaker recognition information from the image information 201 using the speaker recognition model 220 . In this case, the speaker recognition model 220 may be a deep learning-based model configured to recognize the speaker.

In this case, the gaze interaction analysis system may simultaneously or asynchronously perform an operation of generating a depth map and an operation of recognizing a speaker, respectively.

The gaze interaction analysis system may store the acquired speaker recognition information in the database 260 . The gaze interaction analysis system may store the acquired speaker recognition result in the database 260 .

The gaze interaction analysis system uses a depth map restored from the input image information 201 and speaker information recognized from the input image information 201 to obtain gaze information of an object through the first gaze tracking model 230 . can be tracked. In this case, the first gaze tracking model 230 may be a deep learning-based model configured to track the speaker's gaze.

The gaze interaction analysis system may store the tracking result of the tracked speaker's gaze information in the database 260 . For example, when user A and user B exist, A may gaze at B existing on the right. Accordingly, as a result of tracking the speaker's gaze, it may be determined that the speaker is gazing at the right side, and the determined information (right gaze) may be stored in the database. In addition, various information related to the determined information (eg, the location of an object, an object name, etc.) may be stored in the database. A user present on the right side of the gaze from the speaker may be recognized as the listener.

The gaze interaction analysis system may track the listener's gaze information from the tracked speaker's gaze information as the tracked speaker's gaze information is input to the second gaze tracking model 240 . In this case, the second gaze tracking model 240 may be a deep learning-based model configured to track the gaze of the listener. Also, the first gaze tracking model 230 and the second gaze tracking model 240 may be configured separately to track the gaze of the speaker and the listener, or to track the gaze of an object including the speaker and the listener. It may be a single model configured for In other words, the first eye tracking model 230 and the second eye tracking model 240 may be the same model, and through this, the eyes of the speaker and the listener may be tracked.

The gaze interaction analysis system may store the tracking result of the tracked listener's gaze information in the database 260 . For example, when user A and user B exist, B may gaze at A existing to the left. Accordingly, as a result of tracking the listener's gaze, it may be determined that the listener is gazing at the left side, and the determined information (left gaze) may be stored in the database. In addition, various information related to the determined information (eg, the location of an object, an object name, etc.) may be stored in the database. A user present on the left side, gazed from the listener, may be recognized as the speaker.

The gaze interaction analysis system may classify the gaze interaction information 250 based on the tracked gaze information of the speaker and the tracked gaze information of the listener. The gaze interaction analysis system uses the tracked speaker's gaze information and the tracked listener's gaze information to perform gaze interaction based on gaze interaction criteria including no interaction, mutual gaze, gaze avoidance, gaze tracking, and mutual interest. can be classified. The gaze interaction analysis system may store gaze interaction information corresponding to mutual gaze among the classified gaze interactions in a database.

The gaze interaction analysis system uses a search result of a database including a result of the recognized speaker, a tracking result of the tracked speaker's gaze information, a tracking result of the tracked listener's gaze information, and gaze interaction data corresponding to mutual gaze. You can provide information related to the interaction. The gaze interaction analysis system analyzes the character's disposition, conflict, dialogue discrimination, dialogue positive or negative relationship, the main dialogue person in the story, and the main object based on the interaction relationship between the speaker and the listener through the classified gaze interaction information. can do.

Through this, through a search of a database including the recognized speaker's result, the tracked speaker's gaze information tracking result, the tracked listener's gaze information tracking result, and gaze interaction data corresponding to mutual gaze, gaze interaction and Relevant search information may be provided. For example, search information related to gaze interaction may be provided through a database search by a user. In story-based video media such as dramas and movies, dialogue scenes can be automatically retrieved through dialogue scene recognition, speaker-listener recognition, and gaze interaction collection. In addition, it can be applied to a social relationship analysis service between characters based on the result of eye tracking.

3 is a block diagram illustrating a configuration of a gaze interaction analysis system according to an embodiment.

The gaze interaction analysis system includes the input unit 310 , the speaker recognizer 320 , the speaker gaze tracking unit 330 , the listener recognizer 340 , the gaze interaction classification unit 350 , and the gaze interaction related analysis unit 360 . ) may be included.

The input unit 310 may receive image information from a story-based image medium. For example, time information may be set by the user in the image medium, and image information corresponding to the set time information may be input.

The speaker recognition unit 320 may detect a speaker of a conversation scene. The speaker recognition unit 320 may detect the action of the person speaking in the video medium and recognize the person who is speaking as the speaker.

The speaker's gaze tracking unit 330 may automatically generate a depth map to determine where the speaker's gaze is gazing. The speaker's gaze tracking unit 330 uses a CNN-based deep learning model that takes the recognized speaker's face part and the entire scene as input, and outputs the coordinate data (x, y) that the speaker gazes at to track the gaze. can The speaker's gaze tracker 330 may generate a depth map from the image information through a depth map generation model in order to restore depth information lost in the image medium in the process of inferring the gaze point. The gaze information that can be collected by the speaker gaze tracking unit 330 includes a person gaze, an object gaze, no gaze, and a gaze direction.

For example, the speaker's gaze tracking unit 330 tracks the gaze of a specific character in a drama or video medium to focus on the inner side without gazing at what type of object or mainly gazing at what type of person he is gazing at and talking to. The state of gaze behavior, such as information, may be automatically collected in units of episodes of the drama.

The listener recognizer 340 may detect the listener based on the tracked gaze point. The listener recognizer 340 may identify an object located at a corresponding point based on the inferred coordinate data of the gaze point. If the object is a person, the listener recognizer 340 may recognize the person as a listener that is a target to which the recognized speaker's utterance is transmitted. In this case, the listener recognizer 340 may collect and store data even when there is nothing in the estimated coordinate data of the gaze point or a general object other than a person is recognized.

The gaze interaction classification unit 350 may classify the interaction relationship of the gaze through the gaze direction and the degree of interaction between the speaker and the listener. The gaze interaction classifying unit 350 may classify one of a plurality of (eg, five) gaze interactions in units of shots according to the extracted gaze directions of the speaker and listener and a gaze target. For example, gaze interaction may consist of no interaction, mutual gaze, gaze avoidance, gaze tracking, and mutual interest.

Specifically, the gaze interaction may be classified according to the gaze processing of A and B when A and B are talking to each other. Referring to FIG. 6 , it is an example for describing a gaze interaction classification system (a gaze interaction criterion).

'No interaction' means that both A and B are looking at different places.

'Mutual gaze' refers to a case where A and B are looking at each other. This is the most representative interaction that occurs in a typical conversation scene.

'Gaze avoidance' refers to a case in which at least one person avoids the gaze while A and B gaze at each other.

'Gaze tracking' means that when A and B gaze at each other and at least one person gazes at an object, the other party (A is B, B is A) follows the gaze and gazes at the same object. .

'Mutual interest' refers to a case in which both A and B are gazing at the same object (eg, object).

The gaze interaction related analysis unit 360 may try various analyzes based on the collected gaze interaction relationship. The gaze interaction related analysis unit 360 may analyze a positive or negative relationship of a conversation from the extracted gaze interaction relationship. The gaze interaction-related analysis unit 360 is based on the collected information, such as the gaze direction of the characters, the gaze object, the gaze person, the gaze time, etc. You can analyze dialogue characters, main objects, etc.

The characters in the story-based video play an auxiliary role in expressing the character's disposition through intentional gaze processing in the dialogue scene. For example, a director and actor can intentionally turn their gaze downwards to non-verbally express the character's tendency to be passive and passive.

For example, when gaze data is continuously collected for each dialogue scene with respect to a specific person during one episode, the gaze interaction analysis system may infer the personality of the person based on the previously collected gaze-related data.

As another example, a conflicting dialogue in which conflicts between characters is seen may be different from the characteristics and structure of a dialogue conducted in a normal dialogue situation. For example, in a scene where A and B are talking, there are cases where the other person does not look at each other, or the other person who is facing only one side avoids the gaze. In this case, when no communication and gaze avoidance are detected multiple times based on the gaze interaction classification system, it can be inferred that a conflicting conversation is taking place between the speaker and the listener.

As another example, by tracking the gazes of main characters in dramas and movies, information on who they talk to often, what objects they pay attention to, and how long they stare at them for a long time may be acquired.

5 is a flowchart illustrating a gaze interaction analysis method according to an embodiment.

The gaze interaction analysis system may receive image information from a story-based image medium. The gaze interaction analysis system may detect a speaking action from the input image information and recognize a character who performs the speaking action as the speaker ( 510 ). The gaze interaction analysis system may restore depth information of the RGB image by the depth map generation model from the input image information ( 520 ). In detail, the gaze interaction analysis system may track the speaker's gaze information through a gaze tracking model using a depth map restored from the input image information and the speaker information recognized from the input image information.

The gaze interaction analysis system may recognize the listener by tracking the speaker's gaze information ( 530 ). The gaze interaction analysis system may recognize the listener from the tracked speaker's gaze information as the tracked speaker's gaze information is input to the gaze tracking model.

Referring to FIG. 7 , a learning model for recognizing a speaker and a listener will be described. A learning model for recognizing a speaker and a listener may include a depth map generation model and an eye tracking model.

There can be three paths in the network of the learning model, including the gaze (direction) path, the depth path, and the heat map path. The learning model may operate in an order in which a gaze path is first determined, a depth map path is determined, and then a heat map path is determined. The gaze path may include a gaze direction module for estimating a gaze angle and a gaze attention mask module for generating a gaze focus mask. A gaze attention mask can be generated that checks where the person is gazing in the gaze path and shows the part that needs attention in the entire image information. For example, when embedding a human head position, two fully connected layers may be used. The gaze direction may be determined by considering only the head image and the head position in the gaze path, and a gaze attention mask may be generated. For the depth path, a depth map may be extracted from the original image. The original image, the gaze attention mask, and the depth map generated in each module are connected, and the result is provided as an input to the heat map path, so that the final gaze point can be generated as a heat map.

In the learning model, three input data including an original image, a head image, and a head position may be input. The third party's original image and head image are adjusted to a size of 224x224, and the head center position coordinates (h _x , h _y ) can be normalized between 0 and 1. The output of the learning model may consist of gaze directions and heatmaps (p _x , p _y ). As shown in FIG. 7 , the features of the head image can be extracted using ResNet-50 pre-trained in ImgeNet as a backbone. At the same time, the head position coordinates can be entered into the fully connected layer to be included in the 256-dimensional feature. Next, the head feature map can be linked with the included head position through a fully connected layer for fusion. After that, the final gaze coordinates (p _x , p _y ) may be predicted through the last fully connected layer. After each layer, the ReLu activation function may be used. When the gaze direction module predicts the gaze coordinates (p _x , p _y ), it can generate three gaze attention masks radiating from the head center coordinates (h _x , h _y ) in a funnel-shaped cone towards the expected gaze point. have.

A heatmap may be generated from the heatmap path so that the fixed position gazed in the image can be predicted. First, a direction requiring specific attention in the entire image can be determined by using the gaze attention mask. Thereafter, the final gaze point may be estimated in consideration of the three-dimensional positional relationship between objects using the depth map. In this case, the convolutional layer of the heat map path may follow the structure of a Feature Pyramid Network (FPN) that strongly detects objects by integrating multi-scale functions. By concatenating the gaze attention mask, depth map, and the original RGB image, then input as FPN, a heatmap of size 56x56 can be generated as output. When training a heatmap path, the gradient may be updated to minimize cross-entropy loss between the predicted heatmap and the ground truth heatmap. In this case, the depth map is connected in front of the heat map path, and the FPN may adjust the final gaze point in consideration of the depth information.

The learning model can use different loss functions depending on the model's backbone network to learn the gaze direction and heatmap. As a loss function for training the gaze angle, the cosine similarity between the measured gaze direction and the predicted gaze direction may be used. The line from the head position H=(h _x , h _y ) to the ground truth gaze point G=(g _x , g _y ) is normalized to 1 and defined as L _HG , and the predicted point P=(p _x , p at the head position) The line up to _y ) can be normalized and defined as L _HP . After the cosine similarity of two lines is calculated, it can be subtracted from 1.

Binary cross entropy loss (BCE Loss) may be used as a heat map loss between the measured heat map and the predicted heat map.

The gaze interaction analysis system may classify the gaze interaction according to the recognized speaker and listener gaze information ( 540 ). The gaze interaction analysis system uses gaze information of the tracked object and gaze information of the tracked counterpart object to perform gaze interaction based on gaze interaction criteria including no interaction, mutual gaze, gaze avoidance, gaze tracking, and mutual interest. actions can be classified.

The gaze interaction analysis system may store the gaze interaction data in a database ( 550 ). The gaze interaction analysis system may store, in the database, gaze interaction data corresponding to the recognized speaker information, the tracked speaker gaze information tracking result, the tracked listener gaze information tracking result, and mutual gaze.

The gaze interaction analysis system may determine whether sufficient data is collected in the database. In this case, the gaze interaction analysis system may determine whether data has been sufficiently collected according to a preset criterion. For example, it may be determined whether data in the database is satisfied based on a preset capacity criterion. Alternatively, data may be collected for a preset period. The gaze interaction analysis system may terminate the process when it is determined that sufficient data has been collected in the database. In addition, the gaze interaction analysis system may perform step 510 again as it is determined that data is not sufficiently collected in the database.

Recognizing the gaze of the main characters in story-based video plays an important role in character profiling, analyzing social relationships, and analyzing implicit intentions of conversations. In spite of this importance, in the past, a large amount of money was consumed in collecting information because people had to collect gaze information one by one. On the other hand, according to the present invention, it is possible to automatically detect a dialogue scene in a story-based video medium, and to collect information about the speaker-listener's gaze and gaze interaction information.

In addition, the existing gaze tracking technology tracks the gaze in 2D or 3D spatial coordinates using an RGB-D image obtained from an infrared camera or a stereo camera. However, in the case of video media already filmed with a mono camera, such as a drama or movie, if the existing technology is applied as it is, the depth information is lost when the 3D reality is projected onto the 2D video object, so the accuracy of eye tracking is significantly reduced. had limitations. On the other hand, if the technology of the present invention is used, a deep learning model that automatically generates a depth map from RGB image frames of video media such as dramas and movies is used without additional hardware equipment, sensors, or camera equipment for drama and movie media. It can recognize the gaze with high accuracy. In other words, it is possible to automatically collect meaningful information such as who is watching what and who is talking with whom from a simple RGB image medium.

If the technology of the present invention is applied, a search service for gaze-related scenes can be devised based on the collected speaker-listener recognition, gaze information, and gaze interaction information, and the speaker and listener can be automatically recognized and converted into data in a drama dialogue scene. When you want, you can search for the conversation scene you want. Analyzing the gaze interaction relationship along with dialogue, facial expression, and behavior can be utilized for the analysis of drama stories and social relationships between characters, which can be applied to academic research and related commercial services.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (15)

In the gaze interaction analysis method performed by the gaze interaction analysis system,
receiving image information from a story-based image medium;
tracking gaze information of an object using a gaze tracking model using a depth map restored from the received image information and object information recognized from the received image information;
tracking gaze information of a counterpart object from gaze information of the tracked object as the gaze information of the tracked object is input to the gaze tracking model; and
Classifying gaze interaction information based on gaze information of the tracked object and gaze information of the tracked counterpart object
Gaze interaction analysis method comprising a. According to claim 1,
The object means a speaker among the characters in the story-based video medium,
The step of tracking the gaze information of the object,
receiving the image information into a depth map generation model, and generating a depth map from the image information using the depth map generation model
Gaze interaction analysis method comprising a. According to claim 1,
The object means a speaker among the characters in the story-based video medium,
The step of tracking the gaze information of the object,
receiving the image information into a speaker recognition model, and obtaining speaker recognition information from the image information by using the speaker recognition model
Gaze interaction analysis method comprising a. According to claim 1,
The step of tracking the gaze information of the object,
Storing the acquired speaker recognition information in a database
Gaze interaction analysis method comprising a. According to claim 1,
The step of tracking the gaze information of the object,
Storing the tracking result of the gaze information of the tracked object in a database
Gaze interaction analysis method comprising a. According to claim 1,
The counterpart object refers to a listener who is either staring at the speaker or speaking with the speaker among the characters in the story-based video medium,
The step of tracking the gaze information of the counterpart object comprises:
Storing the tracking result of the tracked listener's gaze information in a database
Gaze interaction analysis method comprising a. According to claim 1,
Classifying the gaze interaction information includes:
Classifying gaze interaction based on gaze interaction criteria including no interaction, mutual gaze, gaze avoidance, gaze tracking, and mutual interest using gaze information of the tracked object and gaze information of the tracked counterpart object step
Gaze interaction analysis method comprising a. 8. The method of claim 7,
Classifying the gaze interaction information includes:
Storing gaze interaction data corresponding to mutual gaze among the classified gaze interactions in a database
Gaze interaction analysis method comprising a. According to claim 1,
Classifying the gaze interaction information includes:
Gaze interaction through a search of a database including a result of the recognized object, a result of tracking gaze information of the tracked object, a tracking result of gaze information of the tracked counterpart object, and gaze interaction data corresponding to mutual gaze providing search information related to
Gaze interaction analysis method comprising a. 10. The method of claim 9,
Classifying the gaze interaction information includes:
Analyzing the character's disposition, conflict, dialogue discrimination, dialogue positive or negative relationship, main dialogue person in the story, and main object based on the interaction relationship between the speaker and the listener through the classified gaze interaction information
Gaze interaction analysis method comprising a. In the gaze interaction analysis system,
an input unit for receiving image information from a story-based image medium;
a speaker gaze tracking unit that tracks gaze information of an object using a gaze tracking model based on a depth map restored from the received image information and object information recognized from the received image information;
a listener recognition unit that tracks gaze information of a counterpart object from gaze information of the tracked object as the gaze information of the tracked object is input to the gaze tracking model; and
A gaze interaction classification unit for classifying gaze interaction information based on gaze information of the tracked object and gaze information of the tracked counterpart object
A gaze interaction analysis system comprising a. 12. The method of claim 11,
The speaker gaze tracking unit,
receiving the image information into a depth map generation model, and generating a depth map from the image information using the depth map generation model;
receiving the image information into a speaker recognition model, and obtaining speaker recognition information from the image information using the speaker recognition model;
Gaze interaction analysis system, characterized in that. 12. The method of claim 11,
The gaze interaction classification unit,
Classifying gaze interaction based on gaze interaction criteria including no interaction, mutual gaze, gaze avoidance, gaze tracking, and mutual interest using gaze information of the tracked object and gaze information of the tracked counterpart object
Gaze interaction analysis system, characterized in that. 12. The method of claim 11,
A storage unit for storing the result of the recognized object, the tracking result of the gaze information of the tracked object, the tracking result of the gaze information of the tracked counterpart object, and gaze interaction data corresponding to mutual gaze in a database
Gaze interaction analysis system further comprising. 14. The method of claim 13,
The gaze interaction classification unit,
Based on the interaction relationship between the speaker and the listener through the classified gaze interaction information, the character's disposition, conflict, dialogue discrimination, dialogue positive or negative relationship, main dialogue characters in the story, and main objects are analyzed.
Gaze interaction analysis system, characterized in that.

Images