WO2024101667A1

WO2024101667A1 - Method and system for searching for media message by using text extracted from media file

Info

Publication number: WO2024101667A1
Application number: PCT/KR2023/015308
Authority: WO
Inventors: 고원섭
Original assignee: 라인플러스 주식회사
Priority date: 2022-11-08
Filing date: 2023-10-05
Publication date: 2024-05-16

Abstract

A method and a system for searching for a media message by using text extracted from a media file are disclosed. The media message search method according to an embodiment may comprise the steps of: generating a particular session of an instant messaging service or participating in the particular session; extracting text from visual information included in a media message to be transmitted from a computer device through the particular session or a media message having been received by the computer device through the particular session; storing the extracted text in a local database of the computer device in association with the corresponding media message; providing a search interface in association with the instant messaging service; and searching the local database for text associated with a keyword input via the search interface and a media message associated with the text.

Description

Method and system for searching media messages using text extracted from media files

The description below relates to a method and system for searching media messages using text extracted from media files.

In an instant messaging service, a sequence of instant messages sent and received through a specific session may be displayed to participants in the session through a chat room corresponding to the session. At this time, as the number of instant messages increases, it may become difficult for participants to find specific messages previously sent and received within the chat room.

To help participants find specific messages, a function exists that searches for messages containing specific keywords and displays the location of the searched message in a sequence of instant messages within the chat room.

Provides a media message search method and system that extracts text by analyzing visual information contained in messages sent and received through a specific session of an instant messaging service, and searches for the visual information using the text of the extracted visual information. .

1. A media message retrieval method in a computer device including at least one processor, comprising: creating, by the at least one processor, a specific session of an instant messaging service or participating in the specific session; extracting, by the at least one processor, text from visual information included in a media message to be transmitted from the computer device through the specific session or a media message received from the computer device through the specific session; storing, by the at least one processor, the extracted text in association with a corresponding media message in a local database of the computer device; providing, by the at least one processor, a search interface in conjunction with the instant messaging service; and searching, by the at least one processor, a text associated with a keyword input through the search interface and a media message associated with the text in the local database.

According to one side, the step of extracting the text may be characterized by extracting text included in the visual information.

According to another aspect, the step of extracting the text may be characterized by extracting text for a category into which the visual information is classified.

According to another aspect, the step of extracting the text may be characterized by extracting text for a format in which the text included in the visual information is classified.

According to another aspect, the format may include at least one of a phone number format, an email address format, an account number format, a price format, and a business card format.

According to another aspect, the step of extracting the text includes extracting the text of the sentence unit included in the visual information, and the step of searching the media message includes the text of the sentence unit containing the keyword and the sentence unit It may be characterized by searching media messages associated with the text.

According to another aspect, the step of extracting the text may be characterized by extracting the text from the visual information based on vision-based machine learning.

According to another aspect, the step of extracting the text is based on an API (Application Programming Interface) call to a server providing a vision-based machine learning service from the visual information through a machine learning service provided by the server. It may be characterized by extracting text.

According to another aspect, the extracted text may not be stored on a server providing the instant messaging service.

According to another aspect, the visual information includes at least one of an image and a video, and the step of extracting the text includes using a thumbnail extracted from the image or video as an input to a machine learning module to extract the text. It can be characterized by extracting .

According to another aspect, the media message retrieval method may further include providing the retrieved media message.

According to another aspect, providing the searched media message includes providing a search result list for text and media messages searched in the local database using the keyword; and providing an instant message so that in the sequence of instant messages transmitted and received through the specific session, a portion corresponding to the position of the instant message corresponding to the search result selected from the search result list is displayed. there is.

According to another aspect, the step of providing the searched media message includes providing the searched media message by displaying a portion corresponding to the location of the searched media message in a sequence of instant messages transmitted and received through the specific session. You can do this.

According to another measurement, the media message retrieval method may further include registering the extracted text as a timeline tag for the visual information.

A computer program stored on a computer-readable recording medium is provided in conjunction with a computer device to execute the method on the computer device.

Provided is a computer-readable recording medium on which a program for executing the above method on a computer device is recorded.

At least one processor configured to execute instructions readable on a computer device, wherein the at least one processor creates or participates in a specific session of an instant messaging service, and through the specific session, Extract text from visual information included in a media message to be transmitted from a computer device or a media message received by the computer device through the specific session, and associate the extracted text with the corresponding media message to localize the computer device to the local media message. A computer that stores text in a database, provides a search interface in connection with the instant messaging service, and searches the local database for text associated with a keyword entered through the search interface and media messages associated with the text. Provides a device.

Text can be extracted by analyzing visual information contained in messages sent and received through a specific session of an instant messaging service, and the visual information can be searched using the text of the extracted visual information.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.

Figure 2 is a block diagram showing an example of a computer device according to an embodiment of the present invention.

Figure 3 is a flowchart showing an example of a media message search method according to an embodiment of the present invention.

Figure 4 is a flowchart showing another example of a media message search method according to an embodiment of the present invention.

5 to 9 are diagrams illustrating an example of providing a search interface according to an embodiment of the present invention.

Figure 10 is a flowchart showing an example of a search method for a media message containing visual information, according to an embodiment of the present invention.

Figure 11 is a diagram showing an actual example of visual information and an example of text that can be extracted from visual information, according to an embodiment of the present invention.

Figure 12 is a diagram illustrating an example of searching for a media message according to an embodiment of the present invention.

Figure 13 is a diagram illustrating another example of searching for a media message according to an embodiment of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

A media message retrieval system according to embodiments of the present invention may be implemented by at least one computer device. At this time, the computer program according to an embodiment of the present invention may be installed and driven in the computer device, and the computer device may perform the media message search method according to the embodiment of the present invention under the control of the driven computer program. there is. The above-described computer program can be combined with a computer device and stored in a computer-readable recording medium to cause the computer to execute the media message retrieval method.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. The network environment in FIG. 1 shows an example including a plurality of

electronic devices

110, 120, 130, and 140, a plurality of

servers

150 and 160, and a network 170. Figure 1 is an example for explaining the invention, and the number of electronic devices or servers is not limited as in Figure 1. In addition, the network environment in FIG. 1 only explains one example of environments applicable to the present embodiments, and the environment applicable to the present embodiments is not limited to the network environment in FIG. 1.

The plurality of

electronic devices

110, 120, 130, and 140 may be fixed terminals or mobile terminals implemented as computer devices. Examples of the plurality of

electronic devices

110, 120, 130, and 140 include smart phones, mobile phones, navigation devices, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), and PMPs (Portable Multimedia Players). ), tablet PC, etc. For example, in FIG. 1, the shape of a smartphone is shown as an example of the electronic device 110. However, in embodiments of the present invention, the electronic device 110 actually communicates with other devices through the network 170 using a wireless or wired communication method. It may refer to one of various physical computer devices capable of communicating with

electronic devices

120, 130, 140 and/or

servers

150, 160.

The communication method is not limited, and may include not only a communication method utilizing a communication network that the network 170 may include (for example, a mobile communication network, wired Internet, wireless Internet, and a broadcast network), but also short-range wireless communication between devices. For example, the network 170 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , may include one or more arbitrary networks such as the Internet. Additionally, the network 170 may include any one or more of network topologies including a bus network, star network, ring network, mesh network, star-bus network, tree or hierarchical network, etc. Not limited.

Each of the

servers

150 and 160 is a computer device or a plurality of computers that communicate with a plurality of

electronic devices

110, 120, 130, 140 and a network 170 to provide commands, codes, files, content, services, etc. It can be implemented with devices. For example, the server 150 provides services (e.g., instant messaging service, transaction (e.g., remittance) service, etc.) to a plurality of electronic devices (110, 120, 130, and 140) connected through the network 170. Payment services, virtual exchange services, risk monitoring services, gaming services, group call services (or voice conference services), messaging services, mail services, social network services, map services, translation services, financial services, search services and/or content provision. It may be a system that provides services, etc.).

Figure 2 is a block diagram showing an example of a computer device according to an embodiment of the present invention. Each of the plurality of

electronic devices

110, 120, 130, and 140 described above or each of the

servers

150 and 160 may be implemented by the computer device 200 shown in FIG. 2.

As shown in FIG. 2, this computer device 200 may include a memory 210, a processor 220, a communication interface 230, and an input/output interface 240. The memory 210 is a computer-readable recording medium and may include a non-permanent mass storage device such as random access memory (RAM), read only memory (ROM), and a disk drive. Here, non-perishable large-capacity recording devices such as ROM and disk drives may be included in the computer device 200 as a separate permanent storage device that is distinct from the memory 210. Additionally, an operating system and at least one program code may be stored in the memory 210. These software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210. Such separate computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD/CD-ROM drives, and memory cards. In another embodiment, software components may be loaded into the memory 210 through the communication interface 230 rather than a computer-readable recording medium. For example, software components may be loaded into memory 210 of computer device 200 based on computer programs installed by files received over network 170.

The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to the processor 220 by the memory 210 or the communication interface 230. For example, processor 220 may be configured to execute received instructions according to program code stored in a recording device such as memory 210.

The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (eg, the storage devices described above) through the network 170. For example, a request, command, data, file, etc. generated by the processor 220 of the computer device 200 according to a program code stored in a recording device such as memory 210 is transmitted to the network ( 170) and can be transmitted to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device 200 through the communication interface 230 of the computer device 200 via the network 170. Signals, commands, data, etc. received through the communication interface 230 may be transmitted to the processor 220 or memory 210, and files, etc. may be stored in a storage medium (as described above) that the computer device 200 may further include. It can be stored as a permanent storage device).

The input/output interface 240 may be a means for interfacing with the input/output device 250. For example, input devices may include devices such as a microphone, keyboard, or mouse, and output devices may include devices such as displays and speakers. As another example, the input/output interface 240 may be a means for interfacing with a device that integrates input and output functions, such as a touch screen. At least one of the input/output devices 250 may be configured as one device with the computer device 200. For example, like a smart phone, a touch screen, microphone, speaker, etc. may be included in the computer device 200.

Additionally, in other embodiments, computer device 200 may include fewer or more components than those of FIG. 2 . However, there is no need to clearly show most prior art components. For example, the computer device 200 may be implemented to include at least some of the input/output devices 250 described above, or may further include other components such as a transceiver, a database, etc.

Figure 3 is a flowchart showing an example of a media message search method according to an embodiment of the present invention. The media message search method according to this embodiment may be performed by the computer device 200 that implements a terminal that sends and/or receives instant messages by participating in a specific session using an instant messaging service. At this time, the processor 220 of the computer device 200 may be implemented to execute control instructions according to the code of an operating system included in the memory 210 or the code of at least one computer program. Here, the processor 220 causes the computer device 200 to perform steps 310 to 350 included in the method of FIG. 3 according to control instructions provided by code stored in the computer device 200. can be controlled.

In step 310, the computer device 200 may create a specific session of an instant messaging service or participate in a specific session. A session of an instant messaging service may correspond to a specific chat room. A sequence of instant messages sent and received through the session may be provided to participants of the session through a corresponding chat room. For example, the at least one computer program described above may be an application associated with an instant messaging service, and the computer device 200 is configured to provide instant messaging by at least one processor (e.g., processor 220) under the control of the application. You can access a server that provides messaging services to create a specific session or participate in a specific session that has already been created.

In step 320, the computer device 200 analyzes a media message to be transmitted through a specific session or a media message received through a specific session and extracts keywords of the media included in the media message to be transmitted or the received media message. can do. The computer device 200 may send instant messages through a corresponding session of an instant messaging service, and may also receive instant messages from other participants. At this time, an instant message containing only text can be classified as a text message, and an instant message containing media such as images, audio, and/or video can be classified as a media message. When the computer device 200 wishes to transmit a media message containing media, the computer device 200 may analyze the media message before transmitting the media message and extract keywords of the media message to be transmitted. As another example, when receiving a media message containing media, the computer device 200 may extract keywords of the media included in the received media message. The method for extracting keywords from media will be explained in more detail later.

Meanwhile, in one embodiment, the computer device 200 may extract keywords of the media by directly analyzing the media included in the received media message. In another embodiment, the computer device 200 may simply receive keywords extracted from the sender's terminal that transmitted the media message. For example, the sender's terminal can analyze the media message it wants to transmit and extract keywords for the media included in the media message. Afterwards, the sender's terminal can transmit the extracted media keyword along with the media message. The transmitted media message may be delivered to the computer device 200 through a server providing an instant messaging service, and the computer device 200 may use the keywords contained in the received media message as keywords of the media contained in the media message. It can be extracted as. At this time, the server delivering the media message may store the instant messages sent and received on the server for a certain period of time. In this case, when the server stores a media message, it can store the keyword of the media delivered with the media message in association with the media message. Keywords of media transmitted and stored to the server can be automatically registered as tags for timeline content related to the media at a later date.

In step 330, the computer device 200 may store the keywords of the extracted media in a local database of the computer device in association with the identifier of the media message including the corresponding media. These media keywords are media metadata and can be used to search for media messages containing specific media. Here, the local database may be implemented through the memory 210 of the computer device 200 or a persistent storage device described separately.

In step 340, the computer device 200 may provide a search interface for instant messages transmitted and received through a specific session. For example, the search interface may include a user interface for receiving a keyword from the user, but depending on the embodiment, it may also include a user interface for receiving a keyword or category from the user.

In step 350, the computer device 200 may provide a media message corresponding to an identifier searched in a local database using a keyword entered through a search interface.

In one embodiment, the computer device 200 may provide a media message corresponding to the searched identifier such that a portion corresponding to the location of the media message corresponding to the searched identifier is displayed in a sequence of instant messages transmitted and received through a specific session. . As a more specific example, the computer device 200 may sequentially provide instant messages corresponding to keywords entered through a search interface from a sequence of instant messages transmitted and received in the corresponding session. As already explained, instant messages may include text messages and media messages. In this case, the computer device 200 may sequentially provide a text message including a specific keyword and/or a media message with an identifier stored in the local device in association with the specific keyword according to a sequence of instant messages. Here, providing an instant message may include causing the position of the instant message in the sequence to be displayed in the chat room.

In another embodiment, the computer device 200 may first display a list of search results corresponding to the input keyword on the screen. The list of search results may include one or more search results, where each search result may include information about the corresponding instant message. For example, in the case of a text message, the search result may include an identifier of the sender of the text message, a profile image of the sender, and at least part of the corresponding text. As another example, for media messages, search results may include the sender's identifier, the sender's profile image, and information about the type of media the media message contains (images (photos), audio, video, etc.). there is. In this case, when a specific search result is selected by the user (for example, when a touch input is recognized in the area where the icon 520 is displayed by the user in a touch screen environment), the computer device 200 sends a sequence of instant messages. By displaying a part corresponding to the location of the instant message corresponding to the selected search result, an instant message corresponding to the search result can be provided to the user.

Accordingly, the computer device 200 can search for and provide not only text messages corresponding to keywords entered by the user through a search interface, but also media messages among instant messages transmitted and received through a specific session.

Meanwhile, the computer device 200 may utilize a machine learning model to extract keywords from media included in a media message.

For example, when the media includes an image, the computer device 200 extracts the category of the image as a keyword of the media including the image using a vision-based machine learning model learned to classify the category of the input image. You can.

As another example, when the media includes audio, the computer device 200 extracts the audio category as a keyword of the media including audio using a sound analysis-based machine learning model learned to classify the category of the input audio. can do.

As another example, media may include video. At this time, the video may include multiple thumbnail images and audio. In this case, the computer device 200 may extract the first category of thumbnail images of the video using a vision-based first machine learning model learned to classify the category of the input image, and select the category of the input audio. The second category of audio of the video can be extracted using a second machine learning model based on sound analysis learned to classify. Thereafter, the computer device 200 may determine at least one of the first category and the second category as a keyword for media including video. At this time, the first category may include a plurality of categories for a plurality of thumbnail images. For example, the computer device 200 may finally determine the same category with the largest number among the plurality of categories included in the first category as the first category. Afterwards, one or both of the first category and the second category can be determined as the keyword of the media. In this way, media keywords may include one or more keywords.

Meanwhile, multiple keywords may be extracted for one image. For example, in the case of 'Labrador Retriever', the keyword "Labrador" and the keyword "Retriever" may each be extracted as keywords for one image. In this way, keywords of media may include multiple keywords. In this case, a media message containing the corresponding media may be searched for any one of a plurality of keywords.

Meanwhile, each machine learning model can be trained to output a category that is higher than a preset reliability level for the input image or audio. Examples of these machine learning models include Multi-Layer Perceptron (MLP), Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Group Convolutional Neural Network (G-CNN), and R-CNN. One or more of various algorithms such as (Recurrent Convolutional Neural Network) can be used, and it is not limited to a specific algorithm. Additionally, each machine learning model can utilize one of the well-known models implemented to classify images or audio and output categories. For example, to classify images, the well-known Core Machine Learning (Core Machine Learning) framework and Vision framework can be used, and if the confidence level of the classification is more than 70%, the identifier of the classification is used as a keyword. It can be regarded as Additionally, the well-known sound analysis framework (SoundAnalysis Framework) can be used to classify audio. Similar to the classification of images, if the reliability of the classification is more than 70%, the identifier of the classification can be considered a keyword.

Figure 4 is a flowchart showing another example of a media message search method according to an embodiment of the present invention. The media message retrieval method according to this embodiment may be performed by the computer device 200 implementing a server providing an instant messaging service. At this time, the processor 220 of the computer device 200 may be implemented to execute control instructions according to the code of an operating system included in the memory 210 or the code of at least one computer program. Here, the processor 220 causes the computer device 200 to perform steps 410 to 440 included in the method of FIG. 4 according to control instructions provided by code stored in the computer device 200. can be controlled.

In step 410, the computer device 200 may receive a media message and a keyword of the media included in the media message extracted by the sender's terminal that transmitted the media message through a specific session of the instant messaging service. Here, the keyword extracted by the sender's terminal may be a keyword of the media extracted for the media message to be transmitted in step 320 of FIG. 3. When transmitting a media message, the sender's terminal may extract the keywords of the media included in the media message and then transmit the extracted keywords of the media to the server along with the media message.

In step 420, the computer device 200 may store keywords of the received media in association with the media message. In one example, computer device 200 may store keywords of received media in a local database of computer device 200 in association with an identifier of the received media message. Here, the local database may be implemented through the memory 210 of the computer device 200 or a persistent storage device described separately.

In step 430, the computer device 200 may transmit the received media message and the keyword of the media to the terminals of participants of a specific session. Through this, one media message can be searched on both the sender's terminal and the receiver's terminal of the media message.

In step 440, the computer device 200 may register a keyword of the received media as a tag of timeline content related to the media. The server may provide a timeline service for each member of the instant messaging service. At this time, when media included in a specific media message is registered in the timeline service as timeline content, the server can automatically register and utilize the keywords of the media extracted from the media message as tags of the timeline content.

The embodiment of FIG. 5 shows an example of a chat room screen 510 on which at least part of a sequence of instant messages transmitted and received through a specific session of an instant messaging service is displayed. Chat room screen 510 may include various user interfaces as well as at least a portion of a sequence of instant messages. At this time, the chat room screen 510 according to this embodiment may display an icon 520 for activating the search interface. When the icon 520 is selected by the user (for example, when a touch input is recognized in the area where the icon 520 is displayed by the user in a touch screen environment), the search interface may be activated.

The embodiment of FIG. 6 shows an example of a chat room screen 610 with the search interface activated. The chat room screen 610 shows an example of an input window interface 620 for displaying keywords input as a search interface and a keyboard interface for inputting keywords, such as the first dotted line box 630.

The embodiment of FIG. 7 shows an example of a chat room screen 710 in which the input keyword is displayed in the input window interface 620 as the keyword is input through the keyboard interface, and a list of search results corresponding to the input keyword is displayed. there is. At this time, the second dotted box 720 represents part of the list of search results displayed on the chat room screen 710. Meanwhile, the user can select one of the displayed search results to access the instant message corresponding to the selected search result. As shown in the chat room screen 710, the search results indicate that in addition to text messages, they may include media messages including photos, videos, and/or audio. Here, it can be easily understood that the media messages shown in the embodiment of FIG. 7 are media messages whose identifiers are linked to the input keyword “cat” and are stored in the local database of the terminal of the participant of the session. In other words, as the user inputs the keyword “cat,” the user's terminal can search for media messages through the identifier stored in association with the keyword “cat” in the local database. Additionally, the user's terminal can search for text messages containing the keyword “cat.” The searched media messages and text messages can be sorted according to the sequence of instant messages sent and received in the session to generate a search result list.

The embodiment of FIG. 8 shows an example of a chat room screen 810 in which an instant message corresponding to the second search result is provided as the user selects the second search result in the chat room screen 710. At this time, the chat room screen 810 displays a portion corresponding to the location of media message 3 820 among the sequence of instant messages transmitted and received through the session. In addition, in the embodiment of FIG. 8, the third dotted box 830 serves as a first user interface for displaying previous search results, and the fourth dotted line box 840 represents a second user interface for displaying next search results. It is showing. Additionally, in the embodiment of FIG. 8, the fifth dotted box 850 indicates that the currently displayed search result is the second search result out of 10 total search results.

The embodiment of FIG. 9 shows an example of a chat room screen 910 that displays the next search result as the user selects the second user interface displayed through the fourth dotted line box 840 in the chat room screen 810. As described in the embodiment of FIG. 7, the next search result may correspond to a text message (text message 16 (920)) containing the text “The cat is pretty”, and the user's terminal may display instant messages sent and received in the session. The portion corresponding to the location of the text message (text message 16 (920)) in the sequence can be displayed on the chat room screen (910).

Below, specific embodiments of extracting text from visual information included in a media message and using it to search for a media message will be described.

Figure 10 is a flowchart showing an example of a search method for a media message containing visual information, according to an embodiment of the present invention. The media message search method according to this embodiment may be performed by the computer device 200 that implements a terminal that sends and/or receives instant messages by participating in a specific session using an instant messaging service. At this time, the processor 220 of the computer device 200 may be implemented to execute control instructions according to the code of an operating system included in the memory 210 or the code of at least one computer program. Here, the processor 220 causes the computer device 200 to perform steps 1010 to 1070 included in the method of FIG. 10 according to control instructions provided by code stored in the computer device 200. can be controlled.

In step 1010, the computer device 200 may create a specific session of an instant messaging service or participate in a specific session. As already described, a session of an instant messaging service may correspond to a specific chat room. A sequence of instant messages sent and received through the session may be provided to participants of the session through a corresponding chat room. For example, the at least one computer program described above may be an application associated with an instant messaging service, and the computer device 200 is configured to provide instant messaging under the control of the application by at least one processor (e.g., processor 220). You can access a server that provides messaging services to create a specific session or participate in a specific session that has already been created.

In step 1020, the computer device 200 extracts text from a media message desired to be transmitted from the computer device 200 through a specific session or visual information included in a media message received to the computer device 200 through a specific session. can do.

As already explained, vision-based machine learning can be used to extract text. The computer device 200 may directly include a machine learning model, but may receive a machine learning service provided by the server based on an API (Application Programming Interface) call to the server providing the vision-based machine learning service. there is. In this case, the computer device 200 uses a thumbnail extracted from an image and/or video as visual information as an input to a machine learning module (a machine learning module included in the computer device 200 or included in the server). You can extract text.

Meanwhile, as an example, the computer device 200 may extract text included in visual information. As a more specific example, the computer device 200 may recognize text in a photo or video and extract the recognized text. At this time, the computer device 200 can extract text in sentence units, but is not limited to this.

As another example, the computer device 200 may extract text for a category into which visual information is classified. As a more specific example, if the category of a photo in which many people exist is classified as “person,” the computer device 200 may extract the category “person” as text.

As another example, the computer device 200 may extract text for a format in which text included in visual information is classified. At this time, the format may include at least one of a phone number format, an email address format, an account number format, a price format, and a business card format. As a more specific example, if a phone number such as “010-1234-5678” is included in the photo, the computer device 200 extracts text for formatting from the photo, such as “Phone Number” rather than “010-1234-5678”. It can be extracted.

Additionally, the computer device 200 may extract text according to two or more of the above-described examples. For example, if a phone number such as “010-1234-5678” is included in the photo, the computer device 200 may display the text “010-1234-5678” and the text “010-1234-5678” included in the photo. You can also extract and utilize all of the text “Phone Number” for the format. As another example, from a photo containing the text "ABCDE" and many people, both the text "ABCDE" in the photo and the text "person" for the category can be extracted and utilized. In other words, multiple text sets may be extracted from one media message.

In step 1030, the computer device 200 may store the extracted text in a local database of the computer device 200 in association with the corresponding media message. To protect personal information, extracted text may not be stored on servers that provide instant messaging services. However, as in the previously described embodiments, the server providing the instant messaging service may temporarily store the text on the server to provide it to other users for matching with keywords.

In step 1040, the computer device 200 may provide a search interface in connection with an instant messaging service. As already described, the search interface may include, for example, a user interface for receiving a keyword from the user, but depending on the embodiment, it may also include a user interface for receiving a keyword or category from the user.

In step 1050, the computer device 200 may search a local database for text associated with a keyword entered through a search interface and media messages associated with the text. For example, in a local database, the text and the identifier of the media message including the visual information from which the text was extracted may be stored in association with each other. In this case, the computer device 200 searches the local database for text containing the keyword entered through the search interface and then extracts the identifier of the media message associated with the text from the local database to search for text and media messages. You can.

In step 1060, the computer device 200 may provide the retrieved media message. For example, the computer device 200 may provide a search result list for text and media messages searched in a local database using keywords. At this time, when a specific search result is selected from the search result list, the computer device 200 selects a portion corresponding to the location of the instant message corresponding to the search result selected from the search result list in the sequence of instant messages transmitted and received through the specific session. You can provide an instant message to be displayed. As another example, the computer device 200 may provide the searched media message by displaying a portion corresponding to the location of the searched media message in a sequence of instant messages transmitted and received through a specific session without providing a search result list.

In step 1070, the computer device 200 may register the extracted text as a timeline tag for visual information. Visual information in media messages sent and received through a session of an instant messaging service may be uploaded as a post on a timeline associated with a specific user account. In this case, the computer device 200 may automatically register text previously extracted from visual information as a timeline tag for visual information uploaded through a post on the timeline.

Figure 11 is a diagram showing an actual example of visual information and an example of text that can be extracted from visual information, according to an embodiment of the present invention. FIG. 11 shows an example of a photo 1100 prepared as an example of visual information and texts extracted from the photo 1100. The photo 1100 can be input into a vision-based machine learning module, and the machine learning module can classify the category of the photo 1100 as “person.” In this case, the text “person” for the category may be extracted as the first text for the visual information. In addition, the machine learning module can extract the texts included in the photo 1100, “#CLEANUPCOVID”, “Be A Good Neighbor”, and “Pick Up After”, as the second text, third text, and fourth text, respectively. there is. In this case, the first to fourth texts may be stored in the local database of the computer device 200 in connection with the photo 1100 and/or the identifier of the photo 1100, which is visual information. If the text included in the visual information is text with a specific format, such as a phone number or account number, text such as “phone number” or “account number” for that format may be extracted.

Figure 12 is a diagram illustrating an example of searching for a media message according to an embodiment of the present invention. 12 shows not only a search result 1220 for a text message including the keyword "Up" as the keyword "Up" is input through the input window interface 1210 as a search interface for displaying the input keyword, but also the keyword "Up". This shows an example in which a search result list including search results 1230 for media messages containing text including “Up” is displayed.

Figure 13 is a diagram illustrating another example of searching for a media message according to an embodiment of the present invention. 13 shows a search result 1320 for a media message containing text including the keyword "Perso" as the keyword "Perso" is entered through the input window interface 1310 as a search interface for displaying the input keyword. This shows an example in which a list of search results containing the text is displayed.

At this time, when the user selects a specific search result from the search result list shown in FIG. 12 or 13, a media message corresponding to the selected search result may be provided to the user. A media message may be provided to a user by displaying a portion corresponding to the location of the media message among a sequence of instant messages transmitted and received through a specific session, but the present invention is not limited to this. The media message of the selected search result may be provided to the user by simply displaying the corresponding media message separately.

In this way, users can search for media messages containing visual information not only through the text containing the visual information, but also through the format of the text and/or the category (or topic, genre, etc.) of the visual information.

As such, according to embodiments of the present invention, text is extracted by analyzing visual information included in a message transmitted and received through a specific session of an instant messaging service, and the text of the extracted visual information is used to search for the visual information. You can.

The system or device described above may be implemented with hardware components or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), etc. , may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied in . Software may be distributed over networked computer systems and thus stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. The medium may continuously store a computer-executable program, or may temporarily store it for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various other software, or servers. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments and equivalents of the claims also fall within the scope of the following claims.

Claims

A computer program coupled to a computer device and stored on a computer-readable recording medium for executing a media message retrieval method on the computer device, comprising:

The media message search method is:

creating or participating in a specific session of an instant messaging service;

extracting text from visual information included in a media message to be transmitted from the computer device through the specific session or a media message received by the computer device through the specific session;

storing the extracted text in a local database of the computer device in association with a corresponding media message;

providing a search interface in conjunction with the instant messaging service; and

Searching the local database for a text associated with a keyword entered through the search interface and a media message associated with the text.

containing

A computer program characterized by .
According to paragraph 1,

The step of extracting the text is,

A computer program, characterized in that extracting text included in the visual information.
According to paragraph 1,

The step of extracting the text is,

A computer program characterized in that it extracts text for a category into which the visual information is classified.
According to paragraph 1,

The step of extracting the text is,

A computer program, characterized in that extracting text in a format in which the text included in the visual information is classified.
According to clause 4,

A computer program characterized in that the format includes at least one of a phone number format, an email address format, an account number format, a price format, and a business card format.
According to paragraph 1,

The step of extracting the text is,

Extract sentence-level text included in the visual information,

The step of retrieving the media message is,

Searching for text in sentence units containing the keyword and media messages linked to the text in sentence unit.

A computer program characterized by .
According to paragraph 1,

The step of extracting the text is,

A computer program characterized by extracting text from the visual information based on vision-based machine learning.
According to paragraph 1,

The step of extracting the text is,

A computer program characterized in that it extracts text from the visual information through a machine learning service provided by the server based on an API (Application Programming Interface) call to a server that provides a vision-based machine learning service.
According to paragraph 1,

A computer program, wherein the extracted text is not stored on a server providing the instant messaging service.
According to paragraph 1,

The visual information includes at least one of an image and a video,

The step of extracting the text is,

Extracting the text by using the thumbnail extracted from the image or video as input to the machine learning module.

A computer program characterized by .
According to paragraph 1,

The media message search method is,

Providing the retrieved media message

A computer program further comprising:
According to clause 11,

The step of providing the retrieved media message is,

providing a search result list for text and media messages searched in the local database using the keywords; and

Providing an instant message so that in the sequence of instant messages transmitted and received through the specific session, a portion corresponding to the position of the instant message corresponding to the search result selected from the search result list is displayed.

A computer program comprising:
According to clause 11,

The step of providing the retrieved media message is,

A computer program, characterized in that it provides the searched media message by displaying a portion corresponding to the location of the searched media message in the sequence of instant messages transmitted and received through the specific session.
According to paragraph 1,

The media message search method is,

Registering the extracted text as a timeline tag for the visual information

A computer program further comprising:
A method for retrieving a media message in a computer device including at least one processor, comprising:

creating, by the at least one processor, a specific session of an instant messaging service or joining the specific session;

extracting, by the at least one processor, text from visual information included in a media message to be transmitted from the computer device through the specific session or a media message received from the computer device through the specific session;

storing, by the at least one processor, the extracted text in association with a corresponding media message in a local database of the computer device;

providing, by the at least one processor, a search interface in conjunction with the instant messaging service; and

Searching, by the at least one processor, a text associated with a keyword entered through the search interface and a media message associated with the text in the local database.

Method for retrieving media messages containing .
According to clause 15,

The step of extracting the text is,

A media message search method comprising extracting text included in the visual information.
According to clause 15,

The step of extracting the text is,

A media message search method characterized by extracting text for a category into which the visual information is classified.
According to clause 15,

The step of extracting the text is,

A media message search method, characterized in that extracting text in a format in which the text included in the visual information is classified.
According to clause 15,

The step of extracting the text is,

Extract sentence-level text included in the visual information,

The step of retrieving the media message is,

Searching for text in sentence units containing the keyword and media messages linked to the text in sentence unit.

A media message search method featuring:
A computer-readable recording medium recording a computer program for executing the method of any one of claims 15 to 19 on a computer device.