KR102443797B1 - Apparatus and method for spatial image recommendation through eye tracking - Google Patents

Abstract

A device for recommending a spatial image through eye tracking and a method thereof are disclosed. According to the present invention, the device comprises a control module having a calculation processing function. The control module is able to search a spatial image corresponding to a search keyword from a terminal of a user in a spatial image table, transmit the spatial image as a source image to a terminal, collect an eye movement data of the user for the source image from the terminal displaying the source image, analyze the eye movement based on the collected eye movement data and the spatial image table, calculate the similarity between the source image and another spatial image of the spatial image table based on the results of analysis of eye movement, and provide a spatial image similar to the source image as a recommended spatial image among a large number of spatial images to the terminal based on the calculated similarity between the spatial images. The present is able to recommend a proper spatial image.

Description

Apparatus and method for spatial image recommendation through eye tracking}

The present invention relates to an apparatus and method for recommending a spatial image through eye tracking, and more particularly, to an apparatus and method for providing a recommended spatial image to a user through eye tracking.

When using services such as news, mail, video, SNS, and messenger in personal daily life, information consumption is increasing due to display-based (PC monitor, tablet, TV, mobile phone, etc.).

In the case of modern people, they stare at the display more than 50% of the time during the day, and in the case of cell phones, they are in close contact with the body for almost 24 hours.

There have been many technologies in the fields of advertising, marketing, and communication to identify the intention of a user by tracking the gaze. In recent years, electronic products such as computers, mobile phones, and TVs are equipped with front cameras by default, so simple gaze information can be collected by using them.

The matters described in the background art above are intended to help the understanding of the background of the invention, and may include matters that are not disclosed 　 prior art.

Prior art 1: Republic of Korea Patent Registration No. 10-1564042 (Spatial search characteristic information extraction method using spatial gaze characteristic) Prior art 2: Republic of Korea Patent Registration No. 10-1350526 (image manipulation based on tracked gaze movement)

The present invention has been proposed in view of the above-described conventional circumstances, and a spatial image through eye tracking that tracks the user's gaze to provide an appropriate recommended spatial image according to where the user's gaze mainly stays among the spatial images. An object of the present invention is to provide a recommended apparatus and method.

In order to achieve the above object, an apparatus for recommending a spatial image through eye tracking according to a preferred embodiment of the present invention includes: a storage module for storing a spatial image table including information on features of a plurality of spatial images; and a control module having an arithmetic processing function, wherein the control module searches the spatial image table for a spatial image corresponding to a search word received from a user's terminal based on a plurality of commands stored in the storage module. to transmit to the terminal as a source image, collect user's gaze movement data for the source image from the terminal displaying the source image, and gaze movement based on the collected gaze movement data and the spatial image table , calculates a similarity between the source image and another spatial image in the spatial image table based on the analysis result of the gaze movement, and calculates a similarity between the spatial images from among the plurality of spatial images based on the calculated similarity between the spatial images. A spatial image similar to the image is provided to the terminal as a recommended spatial image.

The gaze movement data includes time-based location information of the user's gaze with respect to the source image, and the control module analyzes the gaze movement on the source image based on the gaze movement data to determine the characteristics of the source image. A gaze time to which the user's gaze is directed may be calculated for each of the features, and a degree of similarity between spatial images may be calculated based on the gaze time of each of the features.

The features include the objects displayed on the spatial image and the atmosphere of the spatial image, and the control module is configured to determine a time for which the user's gaze continues to be located on each of the objects for a predetermined time or more to each of the objects. may be determined as a gaze time for the atmosphere, and a time in which the user's gaze is not located on any of the objects for a predetermined time or longer may be determined as the gaze time for the atmosphere.

The control module is configured to calculate an atmosphere weight of the source image by using the gaze time for the atmosphere, calculate a weight for each object of the source image by using the gaze time for each of the objects, and calculate the calculated atmosphere The similarity between the spatial images may be calculated based on the weight and the weight for each object.

The atmosphere weight may be calculated through Equation 1 below, and the weight for each object may be calculated through Equation 2 below.

(Equation 1)

(Equation 2)

(T1 _i means a time when the user's gaze is continuously located on the i-th object of the source image for a predetermined time or more, and T2 is a time when the user's gaze is not located on any of the objects for a predetermined time or more means)

The control module obtains the similarity between the atmosphere of the source image and the atmosphere of the other spatial image, obtains the similarity between the object of the source image and the object of the other spatial image, and uses the calculated similarity between atmospheres and the similarity between objects Thus, the similarity between the spatial images is obtained, and the atmosphere weight is applied to the similarity between the atmospheres, and the weight for each object is applied to the similarity between the objects to calculate the similarity between the source image and the other spatial image.

The control module may collect gaze movement data from the terminal when the user looks at the source image until the image recommendation menu is selected.

On the other hand, the spatial image recommendation method through eye tracking according to a preferred embodiment of the present invention is a method performed in a spatial image recommendation device through eye tracking, and a spatial image including information on features of each of a plurality of spatial images. Searching for a spatial image corresponding to the search word received from the user's terminal in the table and transmitting it to the terminal as a source image; collecting the user's gaze movement data with respect to the source image from the terminal displaying the source image; analyzing the gaze movement based on the collected gaze movement data and the spatial image table; calculating a similarity between the source image and another spatial image of the spatial image table based on the analysis result of the gaze movement; and providing a spatial image similar to the source image from among the plurality of spatial images as a recommended spatial image to the terminal based on the calculated similarity between the spatial images.

According to the present invention having such a configuration, it is possible to provide an appropriate recommended spatial image according to where the user's gaze mainly stays among the spatial images by tracking the user's gaze.

1 is a block diagram of a system employing a spatial image recommendation apparatus through eye tracking according to an embodiment of the present invention.
FIG. 2 is an internal configuration diagram of the server shown in FIG. 1 .
3 is a diagram illustrating an example of a spatial image (ie, a source image) provided according to a search word in an embodiment of the present invention.
4 and 5 are diagrams for explaining a spatial image table stored in the database shown in FIG. 1 .
6 is a diagram illustrating an example of gaze movement data collected by gaze tracking in an embodiment of the present invention.
7 is a diagram employed to explain a process of calculating a similarity between spatial images in an embodiment of the present invention.
8 is a flowchart for explaining a spatial image recommendation method through eye tracking according to an embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that this does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. In describing the present invention, in order to facilitate the overall understanding, the same reference numerals are used for the same components in the drawings, and duplicate descriptions of the same components are omitted.

1 is a block diagram of a system employing a spatial image recommendation apparatus through eye tracking according to an embodiment of the present invention.

The system of FIG. 1 may include a terminal 10 , a camera 20 , a server 30 , and a database 40 . The system of the present invention provides the terminal 10 with a spatial image (source image) corresponding to the search word from the terminal 10, then tracks and analyzes the user's gaze movement, and stores pre-stored images based on the gaze movement analysis result. A service may be performed by calculating the similarity between spatial images, and determining and providing a recommended spatial image similar to a spatial image corresponding to a search word based on the calculated similarity.

Each component of FIG. 1 is generally connected through a network. For example, as shown in FIG. 1 , the terminal 10 is connected to the server 30 through a network (not shown).

The above-described network may be a wireless communication type network or a wired communication type network.

For example, a wireless communication type network includes a wireless LAN (WLAN), a wireless broadband (Wibro), a wideband CDMA (WCDMA), an IEEE 802.16, a long term evolution (LTE), LTE-A (Long Term Evolution-Advanced), Bluetooth (Bluetooth), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, Near Field Communication (Near) Field Communication (NFC)), ultrasonic communication (Ultra Sound Communication (USC)), visible light communication (VLC), Wi-Fi (Wi-Fi), and the like may be included, but are not limited thereto.

On the other hand, wired communication type networks include wired LAN (Local Area Network), wired WAN (Wide Area Network), Power Line Communication (PLC), USB communication, Ethernet, serial communication, It may include, but is not limited to, optical/coaxial cable communication.

The terminal 10 is an electronic device used by a user, and a camera 20 is installed therein.

The terminal 10 may input a predetermined search word in order to receive a spatial image recommendation service through eye tracking.

The terminal 10 may transmit a predetermined search word by the user's key input to the server 30 through a network (not shown).

The terminal 10 may receive a spatial image corresponding to the search word. Here, the spatial image corresponding to the search word may be referred to as a source image for generating gaze movement data of the user.

Of course, if the user does not like the source image displayed on the screen of the terminal 10, he/she may request a replacement.

The terminal 10 may track the user's gaze movement in the source image in connection with the camera 20 and generate gaze movement data according to the tracking. The gaze movement data may include location information of the user's gaze with respect to the source image according to time. Of course, before tracking the user's gaze movement, a zero point adjustment operation of matching the user's gaze to a predetermined test image may be performed first. Here, the test image need not necessarily be a spatial image. A source image having the same size as the test image may be provided after performing a task of looking at the outer edge of the test image through a predetermined test image before eye movement tracking. Thereafter, the terminal 10 may track the user's gaze movement and generate gaze movement data according to the tracking.

The terminal 10 may transmit gaze movement data to the server 30 .

For example, when the terminal 10 receives and displays a space image (ie, a source image) corresponding to a search word, the user selects various objects (eg, a window, a table, a chair, a sofa, etc.) in the source image, respectively. You can stare at the time of , or scan the source image in its entirety. Accordingly, the gaze may stay on the first object for a predetermined time and then move to the second object, and the gaze may momentarily pass by an object.

As described above, the terminal 10 may generate information about the location of the user's gaze in the source image in real time.

The terminal 10 includes a processor (not shown) capable of generating eye movement data according to eye movement tracking.

Here, a method of tracking a user's gaze movement and generating gaze movement data can be fully understood by those skilled in the art, and thus a detailed description thereof will be omitted.

The terminal 10 may transmit an image recommendation request to the server 30 . The image recommendation request may be made by a user's selection of a corresponding menu (eg, an image recommendation request menu). The image recommendation request menu may be disposed at a position independent of an area where the source image is displayed. In this case, the image recommendation request may include gaze movement data. That is, it can be seen that the terminal 10 collects eye movement data from the point in time when the user looks at the source image until the image recommendation request menu is selected. Therefore, when transmitting the image recommendation request to the server 30 , the terminal 10 may transmit the gaze movement data collected before the image recommendation is selected.

Of course, among the gaze movement data, there may be gaze movement data in which the user's gaze exceeds the range of the source image. As such, it is preferable that the gaze movement data out of the range of the source image is excluded from transmission to the server 30 .

In the above description, it is assumed that the terminal 10 collects eye movement data from the time the user looks at the source image until the image recommendation request menu is selected. However, it may be done differently from this. For example, the terminal 10 transmits gaze movement data to the server 30 in real time until the image recommendation request menu is selected, and the server 30 receives the image recommendation request from the terminal 10 until The eye movement data may be collected from the terminal 10 . And, when the time comes to transmit the image recommendation request, the terminal 10 may stop transmitting the gaze movement data.

In an embodiment of the present invention, the subject collecting the eye movement data is not very important. From the standpoint of the server 30, it is only necessary to be able to analyze the gaze movement based on the gaze movement data at any time when the gaze movement data is received.

The terminal 10 may receive a recommended space image from the server 30 .

The terminal 10 may display the received recommended space image on the screen.

Accordingly, the terminal 10 may include an input module (not shown), a display module (not shown), and a communication module (not shown).

If necessary, the terminal 10 may further include a storage module (not shown) for storing gaze movement data.

The terminal 10 is a computing device having an arithmetic processing function, and may be a mobile device in which the camera 20 is installed or a fixed device. For example, the terminal 10 may mean a computer, a personal computer (PC), a smart phone, a navigation system, a laptop computer, a tablet computer, a wearable device, or a tablet, but embodiments of the present invention are not limited thereto.

Although only one terminal 10 is illustrated in FIG. 1 , it is better to understand that a plurality of terminals 10 exist in reality.

The above-described terminal 10 may be referred to as a user terminal or a customer terminal.

The camera 20 is an electronic device installed in the terminal 10 and may track the gaze movement of the user. That is, a spatial image (ie, a source image) corresponding to the search word will be displayed on the screen of the terminal 10 , and when the user moves his gaze while looking at the source image, the camera 20 may track the gaze movement.

For example, the camera 20 may be installed in the terminal 10 in a stationary form, or may be attached to or embedded in the terminal 10 in the form of a webcam.

The server 30 may be a spatial image recommendation apparatus through eye tracking according to an embodiment of the present invention. Of course, if necessary, the apparatus 30 for recommending a spatial image through eye tracking according to an embodiment of the present invention may be included in the server 30 .

The server 30 may provide a spatial image recommendation service through eye tracking to the user.

The server 30 may receive a search term from the terminal 10 .

As the server 30 receives the search word, the server 30 may search for a spatial image corresponding to the search word in the database 40 . Here, if there is one searched spatial image, the server 30 may transmit one searched spatial image to the terminal 10 as a source image. On the other hand, if there are several searched spatial images, the server 30 may randomly select one spatial image from among several searched spatial images as a source image and transmit it to the terminal 10 . The above-described source image is a spatial image provided to enable the terminal 10 to generate the user's gaze movement data.

Of course, if necessary, a plurality of source images provided to the terminal 10 may be provided.

The server 30 may receive or collect eye movement data according to eye movement tracking in the terminal 10 . For example, the server 30 may receive an image recommendation request from the terminal 10 . In this case, the image recommendation request may include gaze movement data. Accordingly, it can be seen that the server 30 receives gaze movement data as well as receiving the image recommendation request. As another example, it can be seen that the server 30 collects the gaze motion data by receiving and storing the gaze motion data from the terminal 10 in real time until the image recommendation request is received. After all, from the standpoint of the server 30, reception and collection are not technically very different.

The gaze movement data received or collected by the server 30 are those in which the user's gaze is within the range of the source image. This is because, in the terminal 10, gaze movement data in which the user's gaze exceeds the range of the source image is excluded from transmission.

Hereinafter, it is assumed that the server 30 collects gaze movement data in real time.

As the server 30 receives the image recommendation request from the terminal 10 , the server 30 may analyze the user's gaze movement based on the gaze movement data collected so far. As described above, since each gaze movement data includes time-based location information of the user's gaze with respect to the source image, the server 30 determines which feature of the source image the gaze gazes at in the source image based on the gaze motion data. can judge Here, the feature may include objects appearing on the spatial image (including the source image) and the atmosphere of the spatial image (including the source image). In addition, whether or not to gaze can be determined depending on whether the position is located for a predetermined time or longer.

That is, the server 30 may analyze (or interpret) the movement of the user's gaze on the source image based on the gaze movement data to calculate the gaze time at which the user's gaze is directed for each feature of the source image. In other words, the server 30 may calculate how long the gaze stayed on for each of the features in the source image. For example, if the object 1 has 5 focusing positions and the total time spent at the 5 focusing positions is totaled, the time spent by the gaze on the object 1 may be calculated.

On the other hand, the server 30 obtains the atmosphere weight (MW, Mood Weight) of the source image and the weights (OW ₁ ~ OW _n ) (Object Weight) of the source image based on the gaze movement analysis result, and then stores the results in the space in the database 40 . It can be reflected in the similarity between images (TSIM, Total Similarity). Here, the similarity between spatial images (TSIM) may be generated based on the similarity between atmospheres and objects between two spatial images to be compared. For example, there is an atmosphere and one or more objects for each spatial image. Therefore, assuming that the similarity between spatial image 1 and spatial image 2 is calculated, the server 30 may obtain a similarity (MSIM, Mood Similarity) between the atmosphere of spatial image 1 and the atmosphere of spatial image 2, and spatial image 1 The degree of similarity (OSIM, Object Similarity) between the object of and the object of spatial image 2 can be obtained. Here, the object of the spatial image 1 and the object of the spatial image 2 should be of the same type. The server 30 may obtain the atmosphere similarity (MSIM) between the two spatial images and the similarity between objects (OSIM) based on the information of the spatial image table in the database 40 . In particular, the server 30 may obtain an atmosphere similarity (MSIM) between two spatial images and a similarity between objects (OSIM) through a model trained in advance according to a deep learning algorithm. In addition, the server 30 may sum the calculated atmosphere similarity (MSIM) between the two spatial images and the similarity between objects (OSIM), and the sum result may be referred to as a similarity (TSIM) between the two spatial images.

In this way, the server 30 can obtain the similarity between all spatial images in the database 40 .

In addition, the server 30 may determine a recommended spatial image similar to the source image corresponding to the search word based on the calculated similarity between spatial images (TSIM). Then, the server 30 may transmit the determined recommended space image to the terminal 10 .

The database 40 stores a spatial image table including information on characteristics of each of a plurality of spatial images. Here, the features may include objects appearing on the spatial image and the atmosphere of the spatial image.

Although the database 40 is configured independently in FIG. 1 described above, it may be regarded as being included in the server 30 if necessary.

FIG. 2 is an internal configuration diagram of the server 30 shown in FIG. 1 , and FIG. 3 is a diagram illustrating an example of a spatial image (ie, a source image) provided according to a search word in an embodiment of the present invention, and FIG. 4 and FIG. 5 is a diagram for explaining a spatial image table stored in the database shown in FIG. 1 , FIG. 6 is a diagram illustrating an example of gaze movement data collected by gaze tracking in an embodiment of the present invention, and FIG. 7 is a diagram employed to explain a process of calculating the similarity between spatial images in an embodiment of the present invention.

The server 30 may include a communication module 32 , a storage module 34 , and a control module 36 . If necessary, the communication module 32, the storage module 34, and the control module 36 may also be referred to as a communication module unit, a storage module unit, and a control module unit.

The communication module 32 may exchange data with the terminal 10 . The communication module 32 may exchange data with the terminal 10 according to a wireless communication method or a wired communication method.

The storage module 34 may store data necessary for the operation of the server 30 . According to embodiments, the storage module 34 may store gaze movement data from the terminal 10 received by the communication module 32 .

Also, the storage module 34 may store the spatial image table read by the control module 36 from the database 40 .

If necessary, the storage module 34 may be a database, but is not limited thereto.

The storage module 34 may store a program composed of instructions for performing a series of operations performed by the server 30 .

The storage module 34 may include a non-volatile memory device or a volatile memory device. According to embodiments, the storage module 34 may be included in the control module 36, but is not limited thereto.

The control module 36 may control overall operations of the server 30 .

The control module 36 may receive a search word from the terminal 10 through the communication module 32 .

The control module 36 may search for a source image (refer to FIG. 3 ) corresponding to the corresponding search term in the spatial image table of the database 40 . More specifically, the control module 36 transmits the searched source image to the terminal 10, but may transmit one source image. For example, if there is one searched source image, the control module 36 may transmit one searched source image to the terminal 10 . On the other hand, if there are several searched source images, the control module 36 may randomly select one source image from among several searched source images and transmit it to the terminal 10 . The above-described source image is a spatial image for generating the user's gaze movement data in the terminal 10 . The source image is a spatial image corresponding to a search term among a plurality of spatial images in the database 40 .

Here, the source image may include various objects as illustrated in FIG. 3 . For example, a window (1), a wall (2), a table (3), a floor (4), etc. can be called objects.

In addition, as illustrated in FIG. 4 , the above-described spatial image table may include information (eg, a level value or an attribute value) of an atmosphere (mood) for each spatial image. Here, the atmosphere may mean a feeling for each spatial image, such as a warm atmosphere or a cold atmosphere.

In addition, as illustrated in FIG. 5 , the spatial image table may include information on the location, type, and property of each object for each spatial image. Here, the position may mean a position where the corresponding object is located in the spatial image. A type may mean a type of a corresponding object, such as a table or sofa. The properties may be the size, color, etc. of the corresponding object.

Accordingly, the spatial image table of FIG. 4 and the spatial image table of FIG. 5 may be collectively referred to as the spatial image table of the database 40 .

The control module 36 may collect gaze movement data (refer to FIG. 6 ) according to gaze movement tracking in the terminal 10 in real time. For example, with respect to the source image shown in FIG. 3 , location information of the gaze according to time may be the same as in FIG. 6 . If it is assumed that the user looks at the table 3 after looking at the window 1 and then the wall 2, 7 focused gazes are located on the window 1, and 5 focused gazes are located on the wall 2 It is located in several places, the table 3 may be positioned in 10 places of the focused gaze. In addition, the length of time the gaze stays may be different for each location of the focused gaze. Here, being focused can be understood as the user's gaze looking at the object at the corresponding location. As described above, the terminal 10 may transmit gaze movement data according to the gaze movement tracking in real time. Accordingly, the control module 36 may collect gaze movement data transmitted in real time.

The control module 36 may store the collected gaze movement data in the storage module 34 . In other words, the terminal 10 may generate and transmit gaze movement data in real time from the point in time when the user views the source image until the image recommendation menu is selected to the server 30 . Accordingly, the control module 36 may collect and store the gaze movement data in real time until the image recommendation request is received in the storage module 34 . Here, it is assumed that the image recommendation request is generated in the terminal 10 as the user selects the image recommendation request menu.

The control module 36 may analyze the user's gaze movement based on the gaze movement data stored in the storage module 34 upon receiving the image recommendation request from the terminal 10 .

In the case of gaze movement analysis, the control module 36 may determine which feature of the source image the user's gaze focuses on in the source image based on the collected gaze movement data and the spatial image table. In other words, the control module 36 analyzes (or may be referred to as an interpretation) the movement of the user's gaze on the source image based on the collected gaze movement data, so that the gaze of the user's gaze is directed to each of the features of the source image. time can be calculated. That is, the control module 36 may determine whether the user gazes at features (eg, objects or atmosphere) appearing in the source image based on the gaze movement data, and may calculate a gaze time during which the gaze continues.

When the user's gaze is continuously positioned on a specific object among the objects displayed on the source image, the user may be considered to be gazing at the specific object. On the other hand, when the user's gaze is not continuously positioned on any of the objects displayed on the source image, the user may be considered to be gazing at the atmosphere of the source image rather than gazing at a specific object.

According to embodiments, the control module 36 may determine whether the user's gaze is continuously located on each of the objects shown in the source image for a predetermined time or more through time series analysis of the gaze movement data. In this case, the predetermined time may be predetermined or may be changed. For example, the control module 36 refers to the spatial image table of FIG. 5 , and compares the position of the user's gaze with the positions of the objects on the source image so that the user's gaze is placed on each of the objects indicated in the source image for a certain period of time or longer. It can be determined whether or not it is continuously located. At this time, the control module 36 sets a predetermined area including each of the objects on the source image, and when the location of the user's gaze is included in the predetermined area, the user's gaze is located on the object shown in the source image It can be judged that

In this case, the control module 36 may reduce the size of the predetermined area in proportion to the time the user's gaze is located in the predetermined area. When the user gazes at a specific object, the user initially looks at a wide area centered on the specific object, and as time elapses, the user focuses on a narrower area centered on the specific object. Accordingly, if the size of the predetermined area for determining whether to gaze is reduced as the time at which the user's gaze is positioned elapses, it is possible to determine with high reliability whether the user's gaze gazes at a specific object.

The control module 36 may calculate a time T1 at which the user's gaze is continuously located on each of the objects for a predetermined period or more through analysis of the gaze movement data. In this case, the control module 36 may calculate a time T1 at which the user's gaze continues to be located for each object displayed on the source image for a predetermined period or more. In addition, the control module 36 may calculate a time T2 during which the user's gaze is not located on any of the objects displayed on the source image for a predetermined period of time or longer through analysis of the gaze movement data.

The control module 36 may determine a time T1 at which the user's gaze is continuously located on each of the objects for a predetermined time or more as the gaze time for each of the objects. Meanwhile, the control module 36 may determine a time T2 in which the user's gaze is not positioned on any of the objects for a predetermined time or longer as the gaze time for the atmosphere.

After analyzing the gaze movement in this way, the control module 36 calculates the mood weight (MW, Mood Weight) of the source image using the gaze time (ie, T2) for the atmosphere. In addition, the control module 36 calculates the weights OW ₁ to OW _n (Object Weight) for each object of the source image by using the gaze time (ie, T1 ) for each of the objects. Thereafter, the control module 36 may calculate a similarity (TSIM, Total Similarity) between spatial images based on the atmosphere weight (MW) of the source image and the weight (OW _i ) of each object calculated as in FIG. 7 .

Here, the atmosphere weight (MW) of the source image can be obtained through Equation 1 below, and the weight OW _i for each object of the source image can be obtained through Equation 2 below.

(Equation 1)

(Equation 2)

In Equations 1 and 2, T1 _i means the time the user's gaze is continuously located on the i-th object of the source image for a predetermined time or more, and T2 is the time when the user's gaze is located on any of the objects for a predetermined time. It means the time when it is not located continuously for more than one time.

The predetermined time may be appropriately set.

Here, since the time the gaze stays on the corresponding object is considered for the weight for each object, the weight of the object on which the gaze stays for the most time will be larger than the weight of other objects.

Reflecting the above-described weights (MW, OW _i ) to the degree of similarity (TSIM) between spatial images will be described in detail.

As illustrated in FIG. 7 , the control module 36 calculates a similarity (MSIM) between the atmosphere of spatial image 1 and the atmosphere of spatial image 2, which is a comparison target, and compares the object (eg, chair, window, etc.) of spatial image 1 and The degree of similarity (OSIM1, OSIM2) between objects (eg, chair, window, etc.) of spatial image 2 is calculated. Here, spatial image 1 is a spatial image of the spatial image table and is a source image corresponding to a search term, and spatial image 2 is a spatial image other than the source image among spatial images of the spatial image table.

Then, the control module 36 calculates the similarity between the spatial images (TSIM) by summing the calculated similarities between atmospheres (MSIM) and objects (OSIM1, OSIM2). At this time, the control module 36 applies an atmosphere weight (MW) to the calculated similarity between atmospheres (MSIM), and applies a weight (OW _i ) for each object to the calculated similarities between objects (OSIM1, OSIM2), so that the spatial images are separated. Find the similarity (TSIM). Here, application may mean summing or multiplying. The above-described similarity between spatial images (TSIM) means a similarity (TSIM) between a source image (eg, spatial image 1) and another spatial image.

That is, the control module 36 may calculate the similarity between the source image and other spatial images in the spatial image table. In this way, the similarity (TSIM) between the source image (eg, spatial image 1) and spatial image 2, the similarity (TSIM) between the source image (eg, spatial image 1) and spatial image 3, and the source image (eg, spatial image 1) and the similarity (TSIM) between the spatial image 4 and the like can be obtained. If the source image was spatial image 3 instead of spatial image 1, the similarity between the source image (spatial image 3) and spatial image 1 (TSIM), the similarity between the source image (spatial image 3) and spatial image 2 (TSIM), and the source The similarity (TSIM) between the image (spatial image 3) and the spatial image 4 may be obtained.

The number of spatial images listed above is not limited to four, but may be more.

When the similarity (TSIM) between the spatial images to which the weights (MW, OW _i ) is reflected is obtained, the control module 36 performs a recommendation space similar to the source image corresponding to the search word based on the calculated similarity (TSIM) between the spatial images. Images (eg, about 1 or 2) may be determined. In this case, the control module 36 may determine, among the spatial images in the spatial image table, a spatial image including an object similar to an object on which the gaze has stayed the longest among objects of the source image as the recommended spatial image.

The control module 36 may transmit the determined recommended space image to the terminal 10 through the communication module 32 .

The control module 36 may refer to a device capable of performing a series of operations or judgments for the operation of the server 30 . For example, the control module 36 may be a central processing unit (CPU), a micro controller unit (MCU), a graphic processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). , but is not limited thereto.

In addition, in the embodiment of the present invention, the control module 36 may include at least one processor, and the at least one processor may control the operations performed in the server 30 and provide a spatial image recommendation service through eye tracking. It is possible to control other components included in the server 30 so that the operation is performed. Accordingly, even if the control module 36 controls the spatial image recommendation service operation through eye tracking to be performed as an example, at least one processor included in the control module 36 performs the spatial image recommendation service through eye tracking. It will be obvious that the operation can be controlled to be performed.

8 is a flowchart for explaining a spatial image recommendation method through eye tracking according to an embodiment of the present invention.

First, the server 30 receives a search term from the terminal 10 (S10).

Upon receiving the search word, the server 30 searches for a spatial image corresponding to the search word in the database 40 and transmits it to the terminal 10 as a source image (S20).

Next, the terminal 10 tracks the user's gaze movement in the source image in association with the camera 20 , and generates and transmits gaze movement data in real time according to the tracking to the server 30 ( S30 ).

Accordingly, the server 30 collects the gaze movement data from the terminal 10 and stores it in the storage module 34 (S40).

Thereafter, the server 30 receives an image recommendation request from the terminal 10 (S50).

When the server 30 receives the image recommendation request from the terminal 10, the server 30 analyzes the user's gaze movement based on the gaze movement data collected so far (S60).

Then, the server 30 calculates a similarity (TSIM) between spatial images based on the gaze motion analysis result (S70). For example, after obtaining the atmospheric weight (MW) of the source image and the weights (OW ₁ to OW _n ) of the source image based on the gaze movement analysis result, the similarity (TSIM) between spatial images is calculated based on these.

The server 30 determines a recommended spatial image similar to the source image corresponding to the search term based on the calculated similarity between spatial images (TSIM) (S80).

Next, the server 30 transmits the determined recommended space image to the terminal 10 (S90).

In addition, the spatial image recommendation method through eye tracking of the present invention described above can be implemented as a computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. In addition, the computer-readable recording medium is distributed in a computer system connected through a network, so that the computer-readable code can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the method can be easily inferred by programmers in the art to which the present invention pertains.

The above description is merely illustrative of the technical spirit of the present invention, and various modifications and variations will be possible without departing from the essential characteristics of the present invention by those skilled in the art to which the present invention pertains. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

10: terminal 20: camera
30: server 32: communication module
34: storage module 36: control module
40: database

Claims (10)

a storage module for storing a spatial image table including information on characteristics of each of the plurality of spatial images; and
Including; a control module having an arithmetic processing function;
The control module, based on the plurality of commands stored in the storage module,
Searching for a spatial image corresponding to the search word received from the user's terminal in the spatial image table and transmitting it to the terminal as a source image,
Collecting the user's gaze movement data for the source image from the terminal displaying the source image,
Analyze the gaze movement based on the collected gaze movement data and the spatial image table,
calculating a similarity between the source image and another spatial image of the spatial image table based on the analysis result of the gaze movement;
providing a spatial image similar to the source image among the plurality of spatial images to the terminal as a recommended spatial image based on the calculated similarity between the spatial images;
The gaze movement data includes time-based location information of the user's gaze with respect to the source image, and the control module analyzes the gaze movement on the source image based on the gaze movement data to obtain features of the source image. calculating the gaze time to which the user's gaze is directed to each, and calculating the similarity between spatial images based on the gaze time of each feature,
The features include the objects displayed on the spatial image and the atmosphere of the spatial image, and the control module determines the time for which the user's gaze continues to be located on each of the objects for a predetermined time or longer for each of the objects. determining the gaze time, and determining the time when the user's gaze is not located on any of the objects for a certain period of time or longer as the gaze time for the atmosphere;
The control module is
The atmosphere weight of the source image is calculated using the gaze time for the atmosphere, the weight of each object of the source image is calculated using the gaze time of each of the objects, and the calculated atmosphere weight and the weight for each object are calculated. Calculate the similarity between the spatial images based on
The source image includes n objects, and the atmosphere weight is calculated through Equation 1 below,
The weight for each object is calculated through Equation 2 below,
Spatial image recommendation device through eye tracking.
(Equation 1)

(Equation 2)

(T1 _i means the time when the user's gaze is continuously located on the i(i≤n)-th object of the source image for a predetermined time or more, and T2 is the time when the user's gaze is continuously located on any of the objects for a predetermined time or more It means the time when it is not located.) delete delete delete delete The method of claim 1,
The control module is
The degree of similarity between the atmosphere of the source image and the atmosphere of the other spatial image is obtained, the degree of similarity between the object of the source image and the object of the other spatial image is obtained, and the degree of similarity between spatial images using the calculated similarity between atmospheres and the similarity between objects calculating the similarity between the source image and the other spatial image by applying the atmosphere weight to the degree of similarity between the atmospheres, and applying the weight for each object to the degree of similarity between the objects,
Spatial image recommendation device through eye tracking. The method of claim 1,
The control module is
Collecting gaze movement data from the terminal from the time the user looks at the source image until the image recommendation request menu is selected,
Spatial image recommendation device through eye tracking. As a method performed in a spatial image recommendation device through eye tracking,
retrieving a spatial image corresponding to a search word received from a user's terminal in a spatial image table including information on characteristics of each of a plurality of spatial images and transmitting it to the terminal as a source image;
collecting the user's gaze movement data with respect to the source image from the terminal displaying the source image;
analyzing the gaze movement based on the collected gaze movement data and the spatial image table;
calculating a similarity between the source image and another spatial image of the spatial image table based on the analysis result of the gaze movement; and
providing, as a recommended spatial image, a spatial image similar to the source image among the plurality of spatial images to the terminal based on the calculated similarity between the spatial images;
The gaze movement data includes time-based location information of the user's gaze with respect to the source image,
The characteristics include objects appearing on the spatial image and an atmosphere of the spatial image,
In the analyzing the gaze movement, the gaze movement on the source image is analyzed based on the gaze movement data to calculate a gaze time at which the user's gaze is directed to each of the features of the source image, but the user's gaze A time to be continuously located on each of the objects for a predetermined time or more is determined as a gaze time for each of the objects, and a time during which the user's gaze is not located on any of the objects for a certain period of time or longer is said Determined by the watch time for the atmosphere,
Calculating the similarity between the spatial images includes calculating the similarity between spatial images based on the gaze time of each of the features, calculating the atmosphere weight of the source image using the gaze time for the atmosphere, and calculating a weight for each object of the source image using the gaze time for each, and calculating a similarity between the spatial images based on the calculated atmosphere weight and weight for each object;
The source image includes n objects, and the atmosphere weight is calculated through Equation 1 below,
The weight for each object is calculated through Equation 2 below,
A spatial image recommendation method through eye tracking.
(Equation 1)

(Equation 2)

(T1 _i means the time when the user's gaze is continuously located on the i(i≤n)-th object of the source image for a predetermined time or more, and T2 is the time when the user's gaze is continuously located on any of the objects for a predetermined time or more It means the time when it is not located.) delete delete

Images