KR102581502B1 - Image reconmmendation system using olfactory information - Google Patents

Abstract

An image recommendation system utilizing olfactory information according to an embodiment of the present invention includes a database storing a plurality of images and a plurality of olfactory information, an olfactory analysis unit that analyzes a plurality of olfactory information to generate olfactory feature values, and analyzes a plurality of images. a visual analysis unit that generates a visual feature value, a similarity calculation unit that calculates the similarity between the feature value and the visual feature value, and a transmission unit that provides the client with at least one recommended image selected from a plurality of images based on the similarity; It includes a similarity calculation unit that calculates similarity according to the distance between olfactory feature values and visual feature values through contrastive learning, and performs clustering between olfactory feature values and visual feature values.

Description

Image recommendation system using olfactory information {IMAGE RECONMMENDATION SYSTEM USING OLFACTORY INFORMATION}

The present invention relates to a system for recommending highly relevant images using olfactory information.

The amount of data that can be collected is increasing, and various technologies are being proposed to analyze and process this data. In particular, in the case of image processing technology that analyzes image data containing visual information such as video frames or photos, derives features of the image to classify or locate objects in the image, or searches for similar images, deep neural networks are used. When using a network, it provides improved accuracy and efficiency than conventional image processing technology.

In particular, when a user wants to receive an image recommendation, a conventional image recommendation technology that provides a recommended image corresponding to the user's inquiry may be an example. Conventional image recommendation technology requires the user to input a query or query sentence in the form of text, or a photo, video, or voice in the form of an image.

However, when it is difficult for a user to construct a query using only text, images, or voice, for example, when entering a query about a mood or emotion felt through the sense of smell, conventional image recommendation technology does not allow the user to specify the mood or emotion through text or voice. Since the user must properly convert and input images and voices, there is a problem in that the recommendation accuracy is lowered during the conversion process.

In addition, the conventional image recommendation system selects recommended images using only the visual characteristics shown in the image, which makes it difficult to sufficiently reflect the person's five senses regarding the image.

Korean Patent Publication No. 10-2020-0024538 (2020.03.09.)

The present invention is intended to solve the problems of the prior art described above. In the process of searching for a recommended image, not only visual information but also olfactory information related to the image is used to improve image search ease and search efficiency. We would like to provide a recommendation system.

In addition, we would like to provide an image recommendation system utilizing olfactory information that can receive not only image information but also olfactory information from a client who wishes to receive image recommendations, and provides an image corresponding to the received olfactory information as a recommended image.

In addition, when providing images, we aim to provide an image recommendation system that utilizes olfactory information to improve search efficiency by providing not only images but also meta information related to the images.

However, the technical challenges that this embodiment aims to achieve are not limited to the technical challenges described above, and other technical challenges may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention includes a database storing a plurality of images and a plurality of olfactory information, an olfactory analysis unit that analyzes the plurality of olfactory information and generates olfactory characteristic values, A visual analysis unit that analyzes a plurality of images to generate a visual feature value, a similarity calculation unit that calculates the similarity between the olfactory feature value and the visual feature value, and at least one recommended image selected from the plurality of images based on the similarity. A transmission unit provided to the client; Including, the similarity calculation unit may calculate similarity according to the distance between the olfactory feature values and the visual feature values through contrastive learning, and may perform clustering between the olfactory feature values and the visual feature values.

The above-described means for solving the problem are merely illustrative and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description of the invention.

According to one of the means for solving the problem of the present invention described above, image recommendation that increases search ease and search efficiency by utilizing olfactory information that utilizes not only visual information but also olfactory information related to the image in the process of searching for a recommended image. system can be provided.

In addition, it is possible to receive not only image information but also olfactory information from a client who wishes to receive image recommendations, and an image recommendation system utilizing olfactory information that provides an image corresponding to the received olfactory information as a recommended image can be provided.

In addition, when providing an image, it is possible to provide an image recommendation system that utilizes olfactory information to improve search efficiency by providing not only the image but also meta information related to the image.

1 is a diagram illustrating the configuration of an image recommendation system utilizing olfactory information according to an embodiment of the present invention.
Figure 2 is a diagram for explaining the configuration of an olfactory sensor according to an embodiment of the present invention.
3 to 5 are diagrams illustrating an example of a screen for entering a search target into an image recommendation system according to an embodiment of the present invention.
FIG. 6 is a diagram illustrating an example of a screen on which a recommended image provided to a client is displayed according to an embodiment of the present invention.
7 to 9 are flowcharts illustrating an image recommendation method using olfactory information according to an embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" with another element in between. . In addition, when a part is said to "include" a certain component, this does not mean excluding other components unless specifically stated to the contrary, but may further include other components, and one or more other features. It should be understood that it does not exclude in advance the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

In this specification, 'part' includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more pieces of hardware, and two or more units may be realized using one piece of hardware.

In this specification, some of the operations or functions described as being performed by a terminal or device may instead be performed on a server connected to the terminal or device. Likewise, some of the operations or functions described as being performed by the server may also be performed on a terminal or device connected to the server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the attached drawings.

1 is a diagram illustrating the configuration of an image recommendation system utilizing olfactory information according to an embodiment of the present invention.

Referring to FIG. 1, an image recommendation system using olfactory information may include a client 100 and a recommendation server 200, and the recommendation server 200 includes a database 210, an olfactory analysis unit 220, and a visual It may include an analysis unit 230, a similarity calculation unit 240, a provision unit 250, and a reception unit 260.

The database 210 may store a plurality of image files and visual information about the composition, content, and metadata of each image file. Additionally, the database 210 may store olfactory information generated in the form of multidimensional feature data or time-series data by being sensed by an olfactory sensor, etc. and then converted into digital preferences.

In FIG. 1, the database 210 is shown as being located inside the recommendation server 200, but it may be located outside the recommendation server 200 depending on the designer's intention or need. In this case, the recommendation server 200 ) can receive data by connecting to an external database by wire or wirelessly.

Each component of the recommendation server 200 is generally connected through a network. Network refers to a connection structure that allows information exchange between nodes such as terminals and servers, including Local Area Network (LAN), Wide Area Network (WAN), and World Wide Area Network (WWW). Wide Web), wired and wireless data communication networks, telephone networks, wired and wireless television communication networks, etc. Examples of wireless data communication networks include 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth communication, infrared communication, and ultrasound. This includes, but is not limited to, communication, Visible Light Communication (VLC), LiFi, etc.

The recommendation server 200 may receive visual information and olfactory information from the outside or the internal database 210, and may use the olfactory analysis model of the olfactory analysis unit 220 or the visual analysis unit 230 based on these data. A visual analysis model can be trained. The olfactory analysis unit 220 may derive olfactory characteristic values from olfactory information using a learned olfactory analysis model, and the visual analysis unit 230 may derive visual characteristic values from visual information using a learned visual analysis model. can do.

Here, the olfactory information is data generated by an olfactory sensor, which will be described later, and may mean a digital signal generated by the olfactory sensor through at least one gas sensor from the inhaled gas, and the olfactory information is multidimensional data or continuous time series. It may be a series of time series data. In this specification, olfactory information refers to information previously stored in the database 210, and can be distinguished from query olfactory information, which will be described later. That is, the query olfactory information may refer to olfactory information received from the client.

Additionally, visual information may refer to data about one frame of an image or video. The types and locations of various objects displayed in the image can be derived by the visual analysis unit 230, which will be described later. In this specification, visual information refers to information previously stored in the database 210, and can be distinguished from the query image, which will be described later. In other words, the query image may refer to visual information received from the client.

More specifically, the olfactory analysis unit 220 may include an olfactory learning unit 222 that trains an olfactory analysis model to generate olfactory characteristic values by analyzing a plurality of olfactory information. Additionally, the olfactory analysis unit 220 may include an olfactory feature extraction unit 224 that extracts olfactory feature values from olfactory information input through a learned olfactory analysis model.

The olfactory learning unit 222 trains an olfactory analysis model for analysis through a multi-layer perceptron of deep learning when the olfactory information is multi-dimensional feature data, and when the olfactory information is time series feature data, the olfactory learning unit 222 trains the olfactory analysis model through deep learning. You can learn an olfactory analysis model using LSTM (Long Short Term Memory), 1D Convolution, or Transformer methods. In addition, the olfactory learning unit 222 may derive feature values through self-supervised learning of deep learning, which performs expression learning by reducing the dimensionality of data and converting it into a more meaningful form. Here, self-supervised learning is performed based on data without labeling information and may mean classifying the learning data on one's own.

The olfactory analysis model that has been trained through the olfactory learning unit 222 can then be used in the olfactory feature extraction unit 224. The olfactory feature extraction unit 224 may derive query olfactory feature values from the input query olfactory information.

The visual analysis unit 230 may include a visual learning unit 232 that trains a visual analysis model to generate visual feature values by analyzing a plurality of visual information. Additionally, the visual analysis unit 230 may include a visual feature extraction unit 234 that extracts visual feature values from visual information input through a learned visual analysis model.

The visual learning unit 232 can extract visual feature values from visual information using a method such as a convolutional neural network or vision transformer for visual information having data on pixel units of an image. there is. Additionally, the visual learning unit 232 may extract feature values using another pre-trained model or through self-supervised learning.

The visual analysis model that has been trained through the visual learning unit 232 can then be used in the visual feature extraction unit 234. The visual feature extractor 234 may derive query visual feature values from the input query image.

The similarity calculation unit 240 may calculate the similarity between the olfactory feature value generated by the olfactory analysis unit 220 and the visual feature value generated by the visual analysis unit 230. At this time, the similarity calculation unit may calculate similarity according to the distance between the olfactory feature values and the visual feature values through contrastive learning, and may perform clustering between the olfactory feature values and the visual feature values.

More specifically, contrastive learning may be a method of learning a representation that shortens the distance between similar samples and increases the distance between dissimilar samples. At this time, it is necessary to measure the distance between samples to determine whether they are similar, and samples that are less than a certain distance can be considered similar to each other, and samples that are farther than a certain distance can be considered dissimilar to each other.

In contrastive learning, the distance between samples can be calculated through various distance functions. Examples of distance functions include the Euclidean distance function, Manhattan distance, Chebychev distance, Cosine distance, Minkowski distance, Levenshtein distance, and Mahalanobis. (mahalanobis) distance, Jaccard distance, Hamming distance, etc. may be examples.

In contrastive learning, learning is performed by narrowing the distance between a sample and a sample in a positive pair relationship and increasing the distance to a sample in a negative pair relationship. You can.

The similarity calculation unit 240 may set positive and negative pairs through matching information between olfactory information and visual information that is input in advance. For example, the gases A, B, and C are given as olfactory information input in advance, and the images X, Y, and Z are given as visual information, where the X image is the image corresponding to the A gas, and the Y If the image is an image corresponding to B gas and the Z image is an image corresponding to C gas, the similarity calculation unit 240 determines that the distance between the olfactory feature value extracted from A gas and the visual feature value extracted from X image is close to each other. Then, the distance between the A gas and the Y image or Z image can be learned to increase.

The similarity analysis model created through this learning process can cluster the olfactory feature values of the olfactory information and the visual feature values of the corresponding visual information. Through clustering, at least one visual feature value corresponding to one olfactory feature value can be matched, and conversely, at least one olfactory feature value corresponding to one visual feature value can be matched.

In the above example, one image is similar to one gas, but the data clustered according to an embodiment of the present invention is not limited to this example, and one olfactory information or olfactory feature value is combined with multiple images or visual feature values. Clustering may occur, or one image or visual feature value may be clustered with multiple olfactory information or olfactory feature values.

When learning of the similarity analysis model is completed, the similarity calculation unit 240 can find a visual feature value that is included in the same cluster as the olfactory feature value and matches when the olfactory feature value is input, and has the corresponding visual feature value. Images can now be extracted. At this time, the extracted image may be provided to the client as a recommended image, and not only one but two or more images may be selected according to a preset value. When selecting one image, you can select the image with the highest similarity. That is, the similarity calculation unit 240 can select the image with the closest visual feature value within the same clustering. Additionally, when selecting two or more images, multiple images may be selected in order of highest similarity.

Conversely, when training of the similarity analysis model is completed, the similarity calculation unit 240 may find olfactory feature values that are included and matched in the same cluster as the visual feature values when visual feature values are input. In this case, it is possible to find matching visual feature values that are included in the same cluster for each of the matched olfactory feature values, and extract an image with the corresponding visual feature. At this time, the extracted image may be provided to the client as a recommended image, and not only one but two or more images may be selected according to a preset value. When selecting one image, you can select the image with the highest similarity. That is, the similarity calculation unit 240 may select the olfactory information with the olfactory feature value with the closest distance within the same clustering, and then select the image with the closest distance to the corresponding olfactory information. Additionally, when selecting two or more images, multiple images may be selected in order of highest similarity.

The providing unit 250 may provide the image selected by the similarity calculating unit 240 as a recommended image to the client 100. At this time, the providing unit 250 may provide not only the recommended image but also metadata associated with each recommended image.

Here, the metadata may include at least one of location information, time information, and similarity information of the recommended image. The location information of the recommended image may include information about the coordinates or location of the place where the image was taken. The time information of the recommended image may include information about the time when the image was taken. Similarity information may mean information that quantifies the similarity of how similar the recommended image is to the query olfactory information or query image input from the client 100.

The receiving unit 260 may receive the query image and/or the query olfactory information from the client 100 and provide the query to the olfactory analysis unit 220 and/or the visual analysis unit 230. The providing unit 250 and the receiving unit 260 are located inside the recommendation server 200, but may be connected to the client 100 by wire or wirelessly.

Figure 2 is a diagram for explaining the configuration of an olfactory sensor according to an embodiment of the present invention. The olfactory sensor 10 may include a chamber 11, a pump 12, a gas sensor 13, and a signal transducer 14. Chamber 11 refers to a closed space for obtaining gas from the outside. The olfactory sensor 10 controls the timing of external gas intake by controlling the valve (not shown) of the chamber 11, and the pump 12 can control the suction power of olfactory information. Additionally, the gas sensor 130 can sense olfactory information, and at least one sensor can be configured in multiple configurations to generate sensing data for each sensor. The sensed data can be converted into a digital signal by the signal converter 14 and generated as olfactory information.

3 to 5 are diagrams illustrating an example of a screen for entering a search target into an image recommendation system according to an embodiment of the present invention.

Figure 3 illustrates that the recommendation server receives a query image from a client, searches for olfactory information corresponding to the query image, searches for an image corresponding to the retrieved olfactory information, and displays it on the screen as a recommended image. At this time, the client can limit the displayed search results by entering the time or location of the section where the search results are displayed as a search condition.

Figure 4 illustrates that the recommendation server receives query olfactory information from a client (see Sensor_data.dat in Figure 4), searches for images corresponding to the query olfactory information, and displays the searched images on the screen as recommended images. At this time, the client can limit the displayed search results by entering the time or location of the section where the search results are displayed as a search condition.

Figure 5 illustrates that a recommendation server receives query olfactory information and a query image from a client and displays a recommendation image corresponding to the query olfactory information and query image. At this time, the recommendation server may search for an image corresponding to the query olfactory information and display the searched image as a recommended image on the screen. In addition, the recommendation server searches for olfactory information corresponding to the query image, searches for an image corresponding to the searched olfactory information, and displays it on the screen as a recommended image. If recommended images corresponding to the query olfactory information or query image overlap, they can be displayed at the top by giving them higher priority, and the arrangement of the displayed images can be changed according to the designer's intention.

At this time, the client can limit the displayed search results by entering the time or location of the section where the search results are displayed as a search condition.

FIG. 6 is a diagram illustrating an example of a screen on which a recommended image provided to a client is displayed according to an embodiment of the present invention.

In the process of providing a recommended image to a client, the recommendation server may also provide metadata corresponding to the recommended image so that it can be displayed on the client's screen. Here, the metadata may include at least one of location information, time information, and similarity information of the recommended image. The location information of the recommended image may include information about the coordinates or location of the place where the image was taken. The time information of the recommended image may include information about the time when the image was taken. Similarity information may refer to information that quantifies the similarity of how similar the recommended image is to the query olfactory information or query image input from the client.

In Figure 6, the recommendation server provides a recommended image to the client, and also provides similarity information (98%), location information (Seolleung-ro, Gangnam-gu, Seoul), and time information (October 10, 2022) as metadata for the recommended image. By providing it, it can be seen that the recommended image and meta data are displayed together.

7 to 9 are flowcharts illustrating an image recommendation method using olfactory information according to an embodiment of the present invention. The image recommendation method shown in FIGS. 7 to 9 includes steps processed in time series according to the embodiments shown in FIGS. 1 to 6. Therefore, even if the content is omitted below, the image recommendation system using olfactory information according to the embodiments shown in FIGS. 1 to 6 is also applied to the image recommendation method using olfactory information below. The image recommendation device below may correspond to the recommendation server 200 in the embodiments shown in FIGS. 1 to 6.

Referring to FIG. 7, the image recommendation method using the olfactory information of the image recommendation device includes receiving the query olfactory information (S100), analyzing the query olfactory information (S110), and searching for an image that matches the query olfactory information. It may include a step (S120) of providing a searched image as a recommended image (S130).

Referring to FIG. 8, the image recommendation method using olfactory information of an image recommendation device includes receiving a query image (S200), analyzing the query image (S210), and searching for olfactory information matching the query image. It may include (S220), analyzing olfactory information (S230), searching for an image matching the olfactory information (S240), and providing the searched image as a recommended image (S250).

Referring to FIG. 9, the image recommendation method using the olfactory information of the image recommendation device includes receiving the query image and the query olfactory information (S300), analyzing the query image and the query olfactory information (S310), query image and It may include searching for an image matching the query olfactory information (S320) and providing the searched image as a recommended image (S330).

In the above description, steps S100 to S130, steps S200 to S250, and steps S300 to S330 may be further divided into additional steps or combined into fewer steps, depending on the implementation example of the present invention. Additionally, some steps may be omitted or the order between steps may be switched as needed.

The image recommendation method using olfactory information described in FIGS. 7 to 9 may also be implemented in the form of a computer program stored on a computer-readable recording medium executed by a computer or a recording medium containing instructions executable by a computer. You can. Additionally, the image recommendation method using olfactory information described with reference to FIGS. 7 to 9 may also be implemented in the form of a computer program stored in a computer-readable recording medium that is executed by a computer.

Computer-readable recording media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Additionally, computer-readable recording media may include computer storage media. Computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

100: Client
200: Recommended server
210: database
220: Olfactory analysis unit
230: Visual analysis unit
240: Similarity calculation unit
250: Provider
260: Receiving unit

Claims (5)

A database storing a plurality of images and a plurality of olfactory information;
an olfactory analysis unit that analyzes the plurality of olfactory information and generates olfactory characteristic values;
a visual analysis unit that analyzes the plurality of images and generates visual feature values;
a similarity calculation unit that calculates a similarity between the olfactory feature value and the visual feature value; and
a transmission unit providing a client with at least one recommended image selected from the plurality of images based on the similarity; Including
The similarity calculation unit calculates the similarity according to the distance between the olfactory feature value and the visual feature value and the query olfactory feature value and the query visual feature value through contrastive learning, respectively, and calculates the olfactory feature value and the visual feature value and Performs clustering between query olfactory feature values and query visual feature values,
The olfactory analysis unit further derives the query olfactory feature value from the query olfactory information received from the client,
An image recommendation system utilizing olfactory information, wherein the visual analysis unit further derives the query visual feature value from the query image received from the client.
According to paragraph 1,
a receiving unit that receives the query olfactory information from a client; It further includes,
The similarity calculator selects at least one image having the query olfactory feature value and the clustered visual feature value,
An image recommendation system utilizing olfactory information, wherein the transmission unit provides at least one selected image as a recommended image to the client.
According to paragraph 1,
a receiving unit that receives the query image from a client; It further includes,
The similarity calculation unit derives at least one olfactory feature value clustered with the query visual feature value, and selects at least one image having the clustered olfactory feature value and each clustered visual feature value,
An image recommendation system utilizing olfactory information, wherein the transmission unit provides the selected image as a recommended image to the client.
According to paragraph 1,
a receiving unit that receives the query olfactory information and the query image from a client; It further includes,
The similarity calculator selects at least one image having the query olfactory feature value and the clustered visual feature value,
The similarity calculation unit derives at least one olfactory feature value clustered with the query visual feature value, and selects at least one image having the clustered olfactory feature value and each clustered visual feature value,
An image recommendation system utilizing olfactory information, wherein the transmission unit provides the selected image as a recommended image to the client.
According to paragraph 1,
The transmitting unit further provides metadata including at least one of location information, time information, and similarity information of the recommended image to the client.