KR20220079433A - Method for retrieving a screenshot image based on tag information and apparatus performing the same - Google Patents

Abstract

A user terminal according to an embodiment of the present application includes a storage unit for storing a plurality of screenshot images, wherein at least one or more tag information is mapped to each of the plurality of screenshot images; a display unit configured to display at least one screenshot image among the plurality of screenshot images; an input unit receiving a user input; a communication unit for communicating with an external server; and a control unit configured to determine at least one or more screenshot images to be displayed on the display unit among the plurality of screenshot images based on the user input, wherein the control unit includes, when the user input is input through the input unit, After determining matching tag information corresponding to the user input among the tag information stored in the storage unit, a screenshot image to which the matching tag information is mapped is controlled to be output through the display unit, wherein the control unit reads characters After extracting text information from the screenshot image through a module, the text information may be transmitted to the server, and the server may extract the tag information from the text information and transmit it to the user terminal.

Description

Method for retrieving a screenshot image based on tag information and apparatus performing the same}

The present invention relates to a method for classifying and searching for a screenshot image stored in a terminal based on text or image information extracted from the screenshot image.

Through the terminal, a user captures an image and stores it in the terminal, and, if necessary, captures various contents obtained through the terminal as a screenshot and stores the captured image in the terminal. On the other hand, both an image captured by the user (hereinafter referred to as a captured image) and a screenshot image captured and stored for the purpose of obtaining information are all stored in a single content storage.

Over time, a myriad of shot videos and screenshot images will be stored in a single content repository, and when a user wishes to search for a specific screenshot image according to their needs, the user can search the content repository to find the screenshot image. You have to search all the images stored in .

In this way, in order to find a screenshot image that a user wants, tens or even hundreds of images must be directly searched for, so this search process is time-consuming and cumbersome.

One object of the present invention is to provide a method of extracting tag information from a screenshot image.

An object of the present invention is to provide a method of searching for a screenshot image based on tag information mapped to the screenshot image.

One object of the present invention relates to a method of providing a screenshot image search result based on tag information mapped to a screenshot image.

The problem to be solved by the present invention is not limited to the above-mentioned problems, and the problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings. .

The user terminal disclosed in the present application includes a storage unit for storing a plurality of screenshot images, wherein at least one or more tag information is mapped to each of the plurality of screenshot images; a display unit configured to display at least one screenshot image among the plurality of screenshot images; an input unit receiving a user input; a communication unit for communicating with an external server; and a control unit configured to determine at least one or more screenshot images to be displayed on the display unit among the plurality of screenshot images based on the user input, wherein the control unit includes, when the user input is input through the input unit, After determining matching tag information corresponding to the user input among the tag information stored in the storage unit, a screenshot image to which the matching tag information is mapped is controlled to be output through the display unit, wherein the control unit reads characters After extracting text information from the screenshot image through a module, the text information may be transmitted to the server, and the server may extract the tag information from the text information and transmit it to the user terminal.

The solutions to the problems of the present invention are not limited to the above-described solutions, and solutions not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings. will be able

According to the embodiment of the present application, since tag information is mapped to the screenshot image captured and stored by the user, the user can quickly select a necessary screenshot image from among numerous screenshot images stored in the terminal based on the mapped tag information. and search accurately.

Effects of the present invention are not limited to the above-described effects, and effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention pertains from the present specification and accompanying drawings.

1 is a diagram for explaining a method of extracting tag information from a screenshot image according to an embodiment.
2 is a diagram for explaining the configuration of a user terminal.
3 is a diagram illustrating various images stored in a storage unit provided in a user terminal by way of example.
4 is a diagram for illustratively explaining a screenshot image.
5 is a diagram for explaining an overall process of a method of extracting tag information from a screenshot image according to an embodiment.
6 is a diagram for explaining the classification of a screenshot image into a text area or a non-text area through a neural network model.
7 and 8 are diagrams for explaining a method of extracting text from a text area.
9 to 11 are diagrams for exemplarily explaining text information extracted from a text area.
12 is a diagram for explaining a method for extracting image information from a non-text area.
13 is a diagram for explaining the entire process of extracting a keyword from text information in a server.
14 is a diagram for explaining a method of acquiring a keyword from text information through a second neural network model according to an embodiment.
15 is a diagram for exemplarily explaining keywords and target information extracted from a screenshot image.
16 is a view for explaining a screen displayed through an output unit of a user terminal by way of example.
17 is a diagram for explaining a method of selecting a recommendation tag and providing it to a user.
18 is a diagram for explaining a method of searching for a screenshot image based on tag information and a method of providing a search result to a user.

The above-described objects, features and advantages of the present application will become more apparent from the following detailed description in conjunction with the accompanying drawings. However, since the present application may have various changes and may have various embodiments, specific embodiments will be exemplified in the drawings and described in detail below.

Throughout the specification, like reference numerals refer to like elements in principle. In addition, components having the same function within the scope of the same idea shown in the drawings of each embodiment will be described using the same reference numerals, and overlapping descriptions thereof will be omitted.

If it is determined that a detailed description of a known function or configuration related to the present application may unnecessarily obscure the gist of the present application, the detailed description thereof will be omitted. In addition, numbers (eg, first, second, etc.) used in the description process of the present specification are only identification symbols for distinguishing one component from other components.

In addition, the suffixes "module" and "part" for the components used in the following embodiments are given or mixed in consideration of only the ease of writing the specification, and do not have distinct meanings or roles by themselves.

In the following examples, the singular expression includes the plural expression unless the context clearly dictates otherwise.

In the following embodiments, terms such as include or have means that the features or components described in the specification are present, and the possibility that one or more other features or components will be added is not excluded in advance.

In the drawings, the size of the components may be exaggerated or reduced for convenience of description. For example, the size and thickness of each component shown in the drawings are arbitrarily indicated for convenience of description, and the present invention is not necessarily limited to the illustrated bar.

In cases where certain embodiments are otherwise implementable, the order of specific processes may be performed different from the order in which they are described. For example, two processes described in succession may be performed substantially simultaneously, or may be performed in an order opposite to the order described.

In the following embodiments, when components are connected, it includes not only cases in which components are directly connected but also cases in which components are interposed between components and connected indirectly.

For example, in the present specification, when it is said that components and the like are electrically connected, it includes not only the case where the components are directly electrically connected, but also the case where the components are interposed in the middle and electrically connected indirectly.

According to one embodiment, there is provided a system for extracting tag information from a screenshot image, comprising: a user terminal in which at least one screenshot image is stored; and a server for transmitting and receiving information about the screenshot image with the user terminal, wherein the screenshot image includes at least one text, and the user terminal includes text in which the text exists in the screenshot image. After obtaining an area and a non-text area in which the text does not exist, extracting text information from the text area through a character reading module, mapping the text information to an identification number assigned to the screenshot image, and then the server After receiving the text information from the user terminal, the server extracts a keyword from the text information, maps the tag information derived based on the keyword to the identification number, and transmits it to the user terminal can

The screenshot image may include at least one text area, wherein the text area includes a first text area having a first property and a second text area having a second property to an N-th text area having an N-th property. and the first to Nth properties may be determined by at least one of a text size and a font.

The text area includes a first text area, an area in which text having a first size within an error range, and a second text area, an area in which text having a second size within an error range, among texts in the screenshot image. may include

The first text area is an area of text adjacent within a predetermined range among texts having the first size in the screenshot image, and the second text area is the second size text in the screenshot image. It may be a text area that is adjacent within a certain range among texts having .

The text area may include a first text area, which is an area in which text having a first font, among texts in the screenshot image, and a second text area, which is an area in which text having a second font exists.

The first text area is an area of text adjacent within a certain range among texts having the first font in the screenshot image, and the second text area is the second font in the screenshot image. It may be a text area that is adjacent within a certain range among texts having .

The text information includes first text information and second text information, and the user terminal extracts the first text information from the first text area through the character reading module, and the character from the second text area After extracting the second text information through a reading module, the first text information and the second text information are extracted, wherein the first text area and the second text area can be independently input to the character reading module have.

The user terminal obtains the text area from the screenshot image by using a pre-trained neural network model, wherein the pre-trained neural network model divides the text area into the first text area and the second text area to the Nth text area. It can be learned to distinguish text areas.

The keywords extracted from the text information may be plural, and the keywords may be words, numbers, sentences, or a combination thereof representing the screenshot image among texts included in the text information.

The server extracts a plurality of keywords from the text information, wherein importance is reflected in the plurality of keywords, and the importance is a word, number, sentence or a word representing the screenshot image included in the text information. It may be determined based on a probability of representing the screenshot image among combinations.

The server may extract the keyword from the text information through a pre-trained neural network model, and the pre-trained neural network model may be trained to obtain the keyword based on the text information.

After the user terminal extracts the object information from the non-text area, the object information is mapped to the screenshot image and transmitted to the server, and the server derives the tag information based on the object information and the keyword Thereafter, the derived tag information may be mapped to the screenshot image and then transmitted to the user terminal.

According to an embodiment, in a system for providing tag information extracted from a screenshot image to a user, a user terminal in which a plurality of screenshot images are stored - the screenshot image is a first screenshot image and a second screenshot image Contains images -; and a server for transmitting and receiving information about the screenshot image with the user terminal, wherein the screenshot image includes at least one text, and the user terminal comprises: a character reading module from the first screenshot image After extracting the first text information through and, the server extracts first tag information based on the first text information, extracts second tag information based on the second text information, and then extracts the extracted first tag information and the second tag information. The second tag information may be transmitted to the user terminal, and the user terminal may determine at least one of the first tag information and the second tag information as a recommendation tag according to a predetermined criterion and provide it to the user.

wherein the user terminal receives the first tag information from the server at a first time and receives the second tag information at a second time, and the user terminal determines the first tag information as the recommendation tag; The first time point may be earlier than the second time point.

The user terminal may receive a plurality of tag information extracted from the plurality of screenshot images from the server, and the user terminal may determine the recommendation tag based on a frequency at which the plurality of tag information is received from the server. .

The user terminal may determine, as the recommendation tag, tag information with the highest frequency received from the server among the plurality of tag information, and provide it to the user.

The user terminal acquires a first text area in which text exists in the first screenshot image, extracts the first text information from the first text area through the character reading module, and the second screen acquire a second text area in which text exists in the shot image, and extract the second text information from the second text area through the character reading module, wherein the first text area and the second text area are It can be independently input to the character reading module.

The first text area is an area in which text exists in the first screenshot image, and includes a text area having a first property and a text area having a second property to a text zero having an Nth property, The first to Nth properties may be determined based on at least one of a size and a font of the text.

The second text area is an area in which text exists in the second screenshot image, and includes a text area having a first property and a text area having a second property to a text zero having an Nth property, The first to Nth properties may be determined based on at least one of a size and a font of the text.

The first text area is an area of text adjacent within a certain range among texts in the first screenshot image, and the second text area is text adjacent within a certain range of texts in the second screenshot image. may be the area of

After receiving the first text information from the user terminal, the server extracts a first keyword from the first text information, extracts the first tag information based on the first keyword, and the user terminal After receiving the second text information from , a second keyword may be extracted from the second text information, and the second tag information may be extracted based on the second keyword.

The first keyword is at least one word, number, sentence, or a combination thereof representing the first screenshot image among texts included in the first text information, and the second keyword is included in the second text information. Among the included text, it may be at least one word, number, sentence, or a combination thereof representing the second screenshot image.

The first tag information may include a first representative image determined based on the first keyword, and the second tag information may include a second representative image determined based on the second keyword, When providing the recommendation tag to the user, the user terminal may provide the user with at least one of the first representative image and the second representative image.

Importance is reflected in the first keyword, and the importance is the first screenshot image among words, numbers, sentences, or combinations thereof representing the first screenshot image among texts included in the first text information. is determined based on a probability of representing Among these combinations, it may be determined based on a probability of representing the second screenshot image.

According to an embodiment, in a user terminal, a storage unit for storing a plurality of screenshot images, wherein at least one or more tag information is mapped to each of the plurality of screenshot images; a display unit configured to display at least one screenshot image among the plurality of screenshot images; an input unit receiving a user input; a communication unit for communicating with an external server; and a control unit configured to determine at least one or more screenshot images to be displayed on the display unit among the plurality of screenshot images based on the user input, wherein the control unit includes, when the user input is input through the input unit, After determining matching tag information corresponding to the user input among the tag information stored in the storage unit, a screenshot image to which the matching tag information is mapped is controlled to be output through the display unit, wherein the control unit reads characters After extracting text information from the screenshot image through a module, the text information may be transmitted to the server, and the server may extract the tag information from the text information and transmit it to the user terminal.

The user input includes a first user input and a second user input, and the input unit receives the first user input at a first time point and inputs the second user input at a second time point different from the first time point. received, the control unit may determine the matching tag information in consideration of a time point at which the first user input and the second user input are input to the input unit.

The control unit determines the matching tag information based on the first user input and the second user input, and determines the matching tag information by giving importance to the first user input, wherein the first time point is earlier than the second time point can

The matching tag information may be tag information corresponding to at least one of the first user input and the second user input among the plurality of tag information.

The matching tag information may be tag information corresponding to the second user input from among the plurality of tag information corresponding to the first user input.

The control unit may control the screen shot image to which the matching tag information is mapped to be output through the display unit, and control so that the screenshot image is displayed on the display unit according to a time sequence stored in the storage unit.

The control unit controls the screen shot image to which the matching tag information is mapped to be output through the display unit, and controls the matching tag information to be displayed on the display unit in order of increasing probability values corresponding to the user input. can

One whole process

Hereinafter, a method of extracting tag information from a screenshot image according to an embodiment will be described.

1 is a diagram for explaining a method of extracting tag information from a screenshot image according to an embodiment, and FIG. 2 is a diagram for explaining the configuration of a user terminal.

Referring to FIG. 1 , a method of extracting tag information from a screenshot image according to an embodiment may be performed through a user terminal 1000 and a server 2000 .

Referring to FIG. 2, the user terminal 1000 includes a control unit 100, an image capturing unit 200, a storage unit 300, a user input unit 400, an output unit 500, a power supply unit 600, and a communication unit ( 700) may be included. In this case, the user terminal 1000 may include a portable information and communication device, for example, a smart phone or a tablet.

The image capturing unit 200 is a digital camera and may include an image sensor and an image processing unit. An image sensor is a device that converts an optical image into an electrical signal, and may be configured as a chip in which a plurality of photodiodes are integrated. For example, the image sensor may include a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or the like. Meanwhile, the image processing unit may generate image information by image processing the captured result.

The storage unit 300 is a storage means for storing data that can be read by a microprocessor, and includes a hard disk drive (HDD), a solid state disk (SSD), a silicon disk drive (SDD), a ROM, a RAM, CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices may be included.

More specifically, data received by the user terminal 1000 may be stored in the storage unit 300 . For example, the storage unit 300 may store an image directly captured by the user through the image capturing unit 200 , and a screenshot image obtained by a user capturing information obtained online and output on the output unit 500 . can be stored.

The user input unit 400 receives a user input for the user terminal 1000 . The received input may be transmitted to the controller 100 . According to an embodiment, the user input unit 400 may receive a user input through a touch display. Also, the user input unit 400 may refer to a user interface screen on which a command is input from a user.

The output unit 500 outputs various types of information according to a control command of the control unit 100 . According to an embodiment, the output unit 500 may output information through the display panel. More specifically, the output unit 500 may output information related to the user's hair loss state through the display panel. However, the output unit 500 is not limited to a display panel, and may include various means for outputting information, such as a speaker.

The power supply unit 600 includes a battery, and the battery may be embedded in the user terminal 1000 or may be detachably provided from the outside. The power supply unit 600 may supply power required by each component of the user terminal 1000 .

The communication unit 700 may include a wireless communication module and/or a wired communication module. Here, the wireless communication module may include a Wi-Fi communication module, a cellular communication module, and the like.

The controller 100 may include at least one processor. In this case, each processor may execute a predetermined operation by executing at least one instruction stored in the memory. Specifically, the controller 100 may control the overall operation of components included in the user terminal 1000 . In other words, the user terminal 1000 may be controlled or operated by the controller 100 .

According to an embodiment, the user may perform a web search through the user terminal 1000 and obtain necessary information through the search contents. If necessary, the user may store the acquired information in the terminal, and in this case, the screen shot function provided in the user terminal 1000 may be utilized.

Meanwhile, the storage unit 300 may include a plurality of images (hereinafter, referred to as captured images) stored by the user directly photographing an object in addition to the screenshot image captured by the user. At this time, the storage unit 300 may store the captured image and the screenshot image accumulated.

3 is a diagram illustrating various images stored in a storage unit provided in a user terminal by way of example. Referring to FIG. 3 , for example, a screenshot image captured by a user and a photographed image may be stored in a single storage unit 300 . As another example, the screenshot image and the captured image may be stored after being classified in different locations of the storage unit 300 .

In this case, according to an embodiment, only a screenshot image may be separated from among a plurality of images stored in the storage unit 300 , or only a desired screenshot image may be searched from among a plurality of screenshot images stored in the storage unit 300 . can do.

Hereinafter, a method of extracting tag information from a screenshot image as a method for searching for a desired screenshot image among a plurality of screenshot images stored in the storage unit 300 will be described.

2 How to extract tag information from screenshot image

2.1 screenshot image description

4 is a diagram for explaining a screenshot image by way of example. A screenshot image will be described by way of example with reference to FIG. 4 .

According to an embodiment, a plurality of screenshot images may be stored in the storage unit 300 . In this case, the screenshot image may mean an image captured by the user of the screen displayed on the output unit 500 of the user terminal 1000 .

The screenshot image may be an image in which a part of the web surfing screen (eg, shopping mall, news, etc.) is captured or an image in which a part of the execution screen (eg, gifticon image, etc.) of an application stored in the terminal is captured. have.

The captured screenshot image includes various information, and the information may be provided in the form of an image or text. For example, referring to FIG. 4 , the screenshot image may include a plurality of image areas NTA1 and NTA2 and may include a plurality of text areas TA1 to TA8 .

In this case, the text included in the plurality of text areas TA1 to TA8 may be configured with the same font or may be configured with the same size. Alternatively, texts included in the plurality of text areas TA1 to TA8 may be configured in different fonts or have different sizes. Alternatively, texts included in the plurality of text areas TA1 to TA8 may have the same font but different sizes. Alternatively, texts included in the plurality of text areas TA1 to TA8 may be configured in different fonts but have the same size.

The method of extracting tag information from a screenshot image according to an embodiment may extract tag information based on a plurality of text areas TA1 to TA8 included in the screenshot image, which will be described in detail below. .

2.2 The whole process performed on the terminal

5 is a diagram for explaining an overall process of a method of extracting tag information from a screenshot image according to an embodiment.

Referring to FIG. 5 , the method for extracting tag information from a screenshot image according to an embodiment includes the steps of obtaining a screenshot image (S1100), analyzing the screenshot image (S1200), and converting the screenshot image into a plurality of Separating into regions (S1300), extracting text from a text region among a plurality of regions (S1400), mapping the extracted text with a screenshot image and storing it (S1500), sending the extracted text to a server It may include a step (S1600).

The step of obtaining a screenshot image (S1100) includes at least one screenshot image or two or more screenshot images among a plurality of screenshot images captured by the user's action and stored in the storage unit 300 as described above. It may include the step of obtaining.

In addition, the step of acquiring the screenshot image ( S1100 ) may include acquiring the screenshot image from the outside. For example, the screenshot image acquired in the screenshot image acquisition step S1100 may be an image in which a screenshot image captured by an external device is shared by others.

In the step of analyzing the screenshot image ( S1200 ), the analysis of the acquired screenshot image may be performed. In addition, analyzing the screenshot image ( S1200 ) may include performing image pre-processing so that the obtained screenshot image is suitable for analysis. For example, through the screenshot image analysis step ( S1200 ), image preprocessing may be performed so that the obtained screenshot image has a resolution, sharpness, brightness, saturation, etc. suitable for image analysis.

Through the step of dividing the screenshot image into a plurality of regions ( S1300 ), the screenshot image may be divided into a plurality of regions. Separating the screenshot image into a plurality of regions ( S1300 ) may include dividing at least some regions of the obtained screenshot image into a text region or a non-text region.

6 is a diagram for explaining the classification of a screenshot image into a text area or a non-text area through a neural network model. Referring to FIG. 6 , the obtained screenshot image SI may be divided into a text area TA or a non-text area NTA through the first neural network model NN1 .

The text area TA may mean an area including text within the acquired screenshot image, and the text area TA may include a plurality of text areas, for example, a first text area, a second text area, and the like. can

In this case, the text area TA may include a first text area and a second text area divided in consideration of the size and/or font of texts included in the screenshot image.

For example, the text area TA may include a first text area, a second text area, and the like, which are divided based on the size (or size) of the text. For example, the text area TA includes a first text area including texts having a first size among texts included in the screenshot image, and a second text area including texts having a second size. can do.

As another example, the text area TA may include a first text area, a second text area, and the like, which are divided based on a font (typeface) of the text. For example, the text area TA includes a first text area including texts having a first font among texts included in the screenshot image, and a second text area including texts having a second font. can do.

As another example, the text area TA may include a first text area, a second text area, etc. provided based on the size and font of the text. For example, the text area TA has a first size among texts included in the screenshot image and includes a first text area including texts corresponding to the first font and a second size corresponding to the second font. A second text area including texts may be included.

In other words, the text area TA may include a first text area having a first property and a second text area having a second property to an N-th text area having an N-th property. In this case, the first to Nth properties may be determined based on at least one of a text size and a font.

As a result, the obtained screenshot image SI may be divided into a first text area having a first property, a second text area having a second property, an N-th text area having an N-th property, and a non-text area (NTA). can

The obtained screenshot image SI may be divided into a first text area, a second text area to an N-th text area, and a non-text area through a pre-trained neural network model, but is not limited thereto.

The non-text area (NTA) may mean an area composed of a non-text portion of content included in the screenshot image. For example, the non-text area NTA may mean an area including photos of objects or people included in the screenshot image, logos, trademarks, product photos, and the like.

In this case, the non-text area NTA may be formed in plurality to correspond to the number of non-text parts of the content included in the screenshot image. That is, the non-text area NTA may include a first non-text area, a second non-text area, and the like.

Referring to FIG. 6 , according to an embodiment, the controller 100 divides a screenshot image SI into a text area TA or a non-text area NTA through the first neural network model NN1 trained in advance. can

The first neural network model NN1 may be trained to receive a screenshot image and obtain a text area TA and/or a non-text area NTA from the screenshot image. The first neural network model NN1 may be trained to obtain a text area TA and a non-text area NTA based on the screenshot image.

The first neural network model NN1 may be trained using training data including a screenshot image and labeling data. In this case, the labeling data may include a first labeling value corresponding to the text area TA. Also, the labeling data may include a second labeling value corresponding to the non-text area NTA.

Specifically, the first neural network model NN1 may obtain an output value after receiving a screenshot image. Thereafter, the first neural network model NN1 may be trained by a method of updating the first neural network model NN1 based on an error value calculated in consideration of the difference between the output value and the labeling data. In this case, the output value may include a first output value corresponding to the first labeling value and a second output value corresponding to the second labeling value.

The controller 100 may acquire a plurality of text areas TA by using the first neural network model NN1 . The controller 100 may obtain the first text area and the second text area by using the first neural network model NN1 .

Exemplarily, the controller 100 may obtain a text area (eg, a first text area) having a first size among texts included in the screenshot image by using the first neural network model NN1, A text area (eg, a second text area) having 2 sizes may be obtained.

In this case, the first text area may mean a text area having a first size among texts located within a predetermined range within the screenshot image, and the second text area may be located within a predetermined range within the screenshot image. It may mean a text area having a second size among texts. More specifically, the first text area may mean an area of adjacent texts within a predetermined range among texts having the first size in the screenshot image. Similarly, the second text area may mean an area of adjacent texts within a predetermined range among texts having the second size in the screenshot image.

As another example, the controller 100 may obtain a text area (eg, a first text area) having a first font from among texts included in the screenshot image by using the first neural network model NN1 , and the second A text area having a font (eg, a second text area) may be acquired.

In this case, the first text area may mean a text area having a first font among texts located within a predetermined range in the screenshot image, and the second text area may be located within a predetermined range within the screenshot image. It may mean a text area having the second font among texts. More specifically, the first text area may mean an area of adjacent texts within a predetermined range among texts having the first font in the screenshot image. Similarly, the second text area may mean an area of adjacent texts within a predetermined range among texts having the second font in the screenshot image.

Meanwhile, among texts included in the screenshot image, a text area having a first size or a text area having a first font may be plural. For example, there may be a plurality of text areas having a first size in the screenshot image, and the plurality of text areas may exist at different positions in the screenshot image. Similarly, there may be a plurality of text areas having the first font in the screenshot image, and the plurality of font areas may exist in different positions within the screenshot image.

Referring to FIG. 4 , the controller 100 uses a first neural network model NN1 to have a first text area TA1 having a first size among texts included in a screenshot image SI and a second size. A second text area TA2 and a third text area TA3 having a third size may be obtained. In this case, the text area having the first size may be a plurality (eg, the third text area TA3 , the fifth text area TA5 , and the sixth text area TA6 ), and the text area having the second size may be plural. There may be a plurality of text areas (eg, the seventh text area TA7 and the eighth text area TA8).

Similarly, the controller 100 uses the first neural network model NN1 to use a first text area TA1 having a first font among texts included in the screenshot image SI, and a fourth text area having a second font. (TA4), a third text area TA3 having a third font may be obtained. In this case, there may be a plurality of text areas having the third font (eg, the third text area TA3 , the fifth text area TA5 , and the sixth text area TA6 ).

Meanwhile, the text area may mean an area in which text is included in the acquired screenshot image. For example, the text area may mean a text-related image area in a screenshot image. In the screenshot image described in this specification, an area including text is defined as a text area, but it may also be defined as a text-related image area.

After the step (S1300) of dividing the above-described screenshot image into a plurality of areas is performed, the step of extracting text from the text area among the plurality of areas (S1400) may be performed.

Text information included in the text area may be extracted through the step of extracting the text ( S1400 ), and the step of extracting the text ( S1400 ) may be performed on the user terminal 1000 . That is, the controller 100 of the user terminal 1000 may obtain a plurality of text areas from the screenshot image and extract text information from the plurality of text areas.

7 and 8 are diagrams for explaining a method of extracting text from a text area.

Referring to FIG. 7A , the controller 100 may extract text information by analyzing the screenshot image. The controller 100 may extract text information from the screenshot image through the character reading module TM. In other words, the controller 100 may output text information as output data by using the entire screenshot image as input data of the character reading module TM.

As shown in (a) of FIG. 7 , when the entire screenshot image is used as input data of the character reading module TM, the character recognition rate is low, and thus the accuracy of extracted text information may be deteriorated. In addition, when the entire screen shot image is used as input data of the character reading module TM, the data processing speed of the character reading module TM is required because a large amount of data processing is required.

In contrast, referring to FIG. 7B , the controller 100 may extract text information by analyzing a partial region of the screenshot image. The controller 100 may extract text information by analyzing at least some of the plurality of text areas TA included in the screenshot image through the character reading module TM. In other words, the controller 100 may output text information as output data by using at least some of the plurality of text areas TA included in the screenshot image as input data of the character reading module TM.

When at least a portion of a plurality of text areas included in the screenshot image is used as input data of the character reading module TM as shown in FIG. . In addition, when at least a portion of a plurality of text areas included in the screenshot image is used as input data of the character reading module TM, the data processing speed of the character reading module TM may be improved.

The character reading module TM may include a character recognition algorithm (program or function), and obtain a character image based on image data using the character recognition algorithm (program or function), and convert the acquired character image into pixels Characters can be recognized by analyzing units. In addition, the character reading module TM may include an optical character reading algorithm (program or function), and using the optical character reading algorithm (program or function) to obtain data as a result of reading an image for a scanned character can do. In addition, the character reading module TM may be a lightweight algorithm as a character reading algorithm.

Referring to FIG. 8 , the controller 100 may extract a plurality of text information from each of a plurality of text areas through the character reading module TM.

The controller 100 may extract the first text information from the first text area TA1 through the character reading module TM. The controller 100 may extract the second text information from the second text area TA2 through the character reading module TM. That is, each of the plurality of text areas may be individually input to the text reading module TM, and the text reading module TM may extract respective text information from each text area.

For example, the controller 100 may extract text information from a text area having a first size among texts included in the screenshot image through the character reading module TM. As another example, the controller 100 may output text information from a text area having a first font among texts included in the screenshot image through the character reading module TM.

9 to 11 are diagrams for exemplarily explaining text information extracted from a text area. 9 to 11 , text information may be extracted from a screenshot image captured by a user.

For example, referring to FIG. 9 , the user may capture the SNS screen displayed through the output unit 500 of the user terminal 1000 as shown in FIG. By acquiring an image, text information as shown in FIG. 9(b) may be extracted.

More specifically, as shown in FIG. 9A , the controller 100 controls a plurality of text areas (eg, a first text area TA1 , a second text area TA2 , and a third text area from the obtained screenshot image). area TA3) may be obtained. Thereafter, the controller 100 may extract text information as shown in FIG. 9B from at least one of the plurality of acquired text areas by using the character reading module TM.

As another example, referring to FIG. 10 , the user may capture the messenger screen displayed through the output unit 500 of the user terminal 1000 as shown in FIG. It is possible to extract text information as shown in (b) of FIG.

More specifically, as shown in (a) of FIG. 10 , the controller 100 may obtain a plurality of text areas (eg, the first text areas TA1 to TA6) from the obtained screenshot image. can Thereafter, the controller 100 may extract text information as shown in FIG. 10B from at least one of the plurality of acquired text areas by using the character reading module TM.

As another example, referring to FIG. 11 , as shown in FIG. 11A , the user may capture a news screen displayed through the output unit 500 of the user terminal 1000 , and the controller 100 may control the captured image It is possible to extract text information as shown in (b) of FIG. 11 by obtaining .

More specifically, as shown in FIG. 11A , the controller 100 may obtain a plurality of text areas (eg, first text areas TA1 to ninth text areas TA9) from the obtained screenshot image. can Thereafter, the controller 100 may extract text information as shown in FIG. 11B from at least one of the plurality of acquired text areas by using the character reading module TM.

12 is a diagram for explaining a method for extracting image information IN from a non-text area NTA. Referring to FIG. 12 , the controller 100 may extract image information IN from a plurality of non-text areas NTA obtained from a screenshot image through the image reading module IM.

Although not shown in FIG. 5 , after the step of dividing the screenshot image into a plurality of areas ( S1300 ) is performed, the step of extracting object information from the non-text area among the plurality of areas may be performed.

The step of extracting object information from the non-text area may be performed through a pre-trained neural network model. The pre-trained neural network model may be a model trained to obtain object information based on an image.

The object information may be stored by being mapped to a screenshot image, and a user may classify and/or search for the screenshot image based on the object information mapped to and stored on the screenshot image. In addition, the user may search for a desired screenshot image by using the stored object information and/or keywords mapped to the screenshot image. For example, the user may search for a desired screenshot image by inputting the content related to the stored object information mapped to the screenshot image as a search term.

Meanwhile, the step of extracting the object information from the non-text area may be performed in the user terminal 1000, but is not limited thereto. The step of extracting object information from the non-text area may be performed in the server S.

For example, the control unit 100 of the user terminal 1000 may transmit a non-text area (NTA) obtained from the screenshot image to the server (S), and the server (S) receives the received non-text area (NTA). Object information (TI) may be obtained from an image reading module (IM) or a neural network model.

As another example, the control unit 100 of the user terminal 1000 may transmit a screenshot image to the server S, and the server S may use an image reading module IM from the received screenshot image or an object through a neural network model. Information TI may be obtained.

The control unit 100 may map the extracted text information with the screenshot image and then store the extracted text information with the screenshot image ( S1500 ), and may map the extracted text information with the screenshot image, and then store it in the storage unit 300 .

After extracting text information from the screenshot image, the controller 100 may map the extracted text information to the screenshot image. Here, the mapping of the extracted text information to the screenshot image may mean, for example, to display identification so that the control unit 100 or the server S can recognize that the text information is extracted from the screenshot image. have.

The controller 100 may transmit the extracted text to the server through the step of transmitting the extracted text to the server ( S1600 ). The controller 100 may transmit the text information to the server in order to extract a keyword from the extracted text information.

2.3 The whole process performed on the server

13 is a diagram for explaining the entire process of extracting a keyword from text information in a server. Referring to FIG. 13 , the method for extracting a keyword from text information in the server includes the steps of receiving text information from a user terminal (S2100), analyzing text information to obtain a keyword (S2200), and taking a screenshot of the obtained keyword It may include a step of storing the image after mapping (S2300), and transmitting the obtained keyword to the user terminal (S2400).

The server S may receive text information from the user terminal 1000 through the step S2100 of receiving the text information from the user terminal. Here, the text information may be information about text extracted from the screenshot image by the controller 100 of the user terminal 1000 as described above.

The server S may analyze the text information obtained through the user terminal 1000 through a step S2200 of analyzing the text information to obtain a keyword, and obtain a keyword based thereon.

14 is a diagram for explaining a method of acquiring a keyword from text information through a second neural network model according to an embodiment.

Referring to FIG. 14 , the server S may obtain a keyword from text information through the second neural network model NN2. The second neural network model NN2 may be trained to obtain keywords from text information. The second neural network model NN2 may be trained to obtain keywords based on text information.

Although not shown in the drawing, the server S may acquire important keywords from text information using a keyword extraction algorithm. In this case, the keyword extraction algorithm may include various known algorithms defined to extract key words or sentences from a plurality of words or sentence structures.

The keyword may mean a key word, number, sentence, etc. from various texts (eg, words, numbers, sentences, etc.) included in text information.

The keyword may mean at least one or more key words, numbers, sentences, etc. that can represent a screenshot image from various texts (eg, words, numbers, sentences, etc.) included in text information.

There may be a plurality of keywords obtained from text information. That is, since there may be a plurality of words or sentences that may represent the screenshot image, a plurality of keywords obtained from the text information may also be present.

When the server S acquires a plurality of keywords from the text information, the server S may acquire a plurality of keywords whose importance is reflected from the text information. That is, each of the plurality of keywords obtained from the server S may have their respective importance levels reflected, and in this case, the importance reflected on the plurality of keywords may be the same or different.

For example, the server S may obtain two keywords from the text information, and the same importance may be reflected in both obtained keywords.

As another example, the server S may obtain two keywords from the text information, and one of the obtained keywords may reflect a higher importance than the other. In this case, a keyword reflecting a relatively high importance may be related to the screenshot image with a higher probability. Alternatively, the keyword to which the relatively high importance is reflected may be a word or sentence more representative of the screenshot image.

The keyword may be any one of various texts included in the text information, but is not limited thereto. The keyword may be a word, number, or sentence not included in the text information. In other words, the keyword may be a new word, number, sentence, or a combination thereof extracted based on various texts included in text information.

The server S may map the acquired keyword to the screenshot image and store the screenshot image to which the keyword is mapped through the step (S2300) of mapping the acquired keyword to the screenshot image and then storing it. The mapping of the obtained keyword to the screenshot image may mean, for example, to display identification so that the controller 100 or the server S can recognize that the keyword represents the screenshot image.

The server S may transmit the acquired keyword to the user terminal 1000 through the step S2400 of transmitting the acquired keyword to the user terminal. The server S may transmit the acquired keyword to the user terminal 1000 so that a screen shot image search and classification can be performed on the user terminal 1000 based on the acquired keyword.

15 is a diagram for exemplarily explaining keywords and target information extracted from a screenshot image.

Referring to FIG. 15 , various types of text or non-text may be included in a screenshot image SI captured and stored by a user's motion. In this case, at least one or more keywords and/or object information may be acquired from the screenshot image through the user terminal 1000 and/or the server S.

For example, referring to FIG. 15 , the first keyword may be obtained from the first text area TA1 included in the screenshot image SI, and the second keyword may be obtained from the second text area TA2. , a third keyword may be obtained from the third text area TA3 , a fourth keyword may be obtained from the fourth text area TA4 , and a fifth keyword may be obtained from the fifth text area TA2 . , a sixth keyword may be obtained from the sixth text area TA6 , a seventh keyword may be obtained from the seventh text area TA7 , and an eighth keyword may be obtained from the eighth text area TA8 . can be obtained.

In this case, importance may be reflected in the plurality of keywords (eg, first to eighth keywords) obtained as described above. For example, when the keyword that can best represent the screenshot image SI is the first keyword, the highest importance may be reflected in the first keyword.

Also, referring to FIG. 15 , first object information may be obtained from the first non-text area NTA1 and second object information may be obtained from the second non-text area NTA2 .

At least one or more keywords and/or object information obtained from the screenshot image SI may be defined as tag information, and the tag information may be mapped to the screenshot image SI and stored in the user terminal 1000 . . The user can search for a desired screenshot image by using the tag information. A method of retrieving a screenshot image stored in the user terminal 1000 using the acquired at least one or more tag information will be described later.

3 How to select and provide recommended keywords

The controller 100 may acquire any one of a plurality of screenshot images stored in the user terminal 1000 based on at least one piece of tag information obtained from the screenshot image SI.

16 is a view for explaining a screen displayed through an output unit of a user terminal by way of example. Referring to FIG. 16 , the output unit 500 of the user terminal 1000 may display a search unit SE capable of obtaining a user's input, a recommendation tag display unit REC, and a search result display unit RES. .

The control unit 100 may select at least one of a plurality of screenshot images stored in the storage unit 500 based on a user input obtained through the search unit SE and provide it to the user.

The controller 100 may display at least one or more tag information obtained from the screenshot image SI through the recommendation tag display unit REC of the output unit 500 and provide it to the user. Specifically, the controller 100 may display tag information that meets a predetermined criterion through the output unit 500 and recommend it to the user. In this case, the tag information recommended to the user may be a recommendation tag.

According to an embodiment, after extracting the first text information from the first screenshot image through the character reading module, the user terminal 1000 may transmit the first text information to the server, and the server may transmit the first text information After extracting the first tag information based on , it may be transmitted to the user terminal 1000 . Similarly, after extracting the second text information from the second screenshot image through the character reading module, the user terminal 1000 may transmit the second text information to the server, and the server may send the second text information to the server based on the second text information. 2 After the tag information is extracted, it may be transmitted to the user terminal 1000 .

In this case, the user terminal 1000 may determine at least one of the first tag information and the second tag information as a recommendation tag according to a predetermined criterion and provide it to the user.

The user terminal 1000 may receive the first tag information from the server at a first time and receive the second tag information at a second time. In this case, the user terminal 1000 may determine the first tag information as the recommendation tag, and the first time point may be earlier than the second time point.

The user terminal 1000 may receive a plurality of tag information extracted from a plurality of screenshot images from the server. In this case, the user terminal 1000 may determine the recommendation tag based on the frequency at which a plurality of pieces of tag information are received from the server. For example, the user terminal 1000 may determine, as a recommendation tag, tag information with the highest frequency received from the server among a plurality of tag information and provide it to the user.

As a result, after obtaining a keyword from the text information received from the user terminal, the server may extract tag information based on the obtained keyword. The keyword may be at least one word, number, sentence, or a combination thereof representing a screenshot image among texts included in the text information. Here, the user terminal 1000 may extract tag information from the keyword, and may display the recommendation tag through the recommendation tag display unit REC by using it.

According to an embodiment, a recommendation tag composed of characters may be displayed on the recommendation tag display unit REC as shown in (a) of FIG. 16 . That is, the recommendation tag extracted by the above-described method may be provided in the form of text.

According to another embodiment, as shown in (b) of FIG. 16 , a recommendation tag composed of text and a representative image may be displayed on the recommendation tag display unit REC. Also, although not shown in the drawing, a recommendation tag composed of a representative image may be displayed on the recommendation tag display unit REC. In this case, the representative image displayed on the recommendation tag display unit REC may be an image determined based on the keyword. For example, the representative image is determined based on the keyword, and may be an image that can represent the type or information of the screenshot image well.

17 is a diagram for explaining a method of selecting a recommendation tag and providing it to a user. Referring to FIG. 17 , the method of selecting a recommendation tag and providing it to the user includes the steps of obtaining a screenshot image (S3100), analyzing the screenshot image (S3200), and extracting text information from the screenshot image ( S3300), obtaining a plurality of keywords from text information (S3400), selecting a recommended tag from among the plurality of keywords (S3500), and providing the recommended tag to the user (S3600) may include.

The controller 100 may acquire a screenshot image captured and stored by the user through the step of acquiring the screenshot image ( S3100 ). The control unit 100 captures by the user's action through the step (S3100) of obtaining a screenshot image and stores at least one screenshot image or two or more screenshot images among a plurality of screenshot images stored in the storage unit 300 image can be obtained. In addition to this, the control unit 100 may obtain a screenshot image from the outside of the user terminal 1000, and in this regard, the description is duplicated because it has been described above through the step (S1100) of obtaining the screenshot image of FIG. 5 . is omitted.

The controller 100 may analyze the screenshot image obtained through the step of analyzing the screenshot image ( S3200 ). The controller 100 may perform a step of pre-processing the image so that the obtained screenshot image is suitable for analysis. The step of analyzing the screenshot image ( S3200 ) is the same as or corresponding to the step of analyzing the screenshot image of FIG. 5 ( S1200 ), so a redundant description will be omitted.

The controller 100 may extract text information by analyzing the screenshot image obtained through the step of extracting text information from the screenshot image ( S3300 ). More specifically, the controller 100 may divide the screenshot image into a text area and/or a non-text area, and extract text information from the text area through a character reading module. In this regard, since the steps of dividing the screenshot image of FIG. 5 into a plurality of areas ( S1300 ) and extracting text from the text area among the plurality of areas ( S1400 ) have been described above, the overlapping description will be omitted.

The controller 100 may transmit the obtained text information to the server S. The server S may obtain a plurality of keywords from the received text information. The server S may acquire a plurality of keywords from text information through a pre-trained neural network model or a known algorithm. .

The controller 100 may select a recommendation tag to be provided to the user from among the acquired keywords through the step of selecting a recommendation tag from among the plurality of keywords ( S3500 ).

The recommendation tag may be selected based on a predetermined criterion among tag information stored in the storage unit 400 .

The control unit 100 may classify the tag information stored in the storage unit 400 according to time order and provide it to the user as a recommended tag. For example, the control unit 100 may provide the tag information stored in the storage unit 400 to the user in the order of recently acquired and stored tags. That is, the recommendation tag provided to the user by the controller 100 may be tag information extracted from the most recently captured and stored screenshot image.

The control unit 100 may classify the tag information stored in the storage unit 400 according to the frequency and provide it to the user as a recommended tag. More specifically, since a plurality of tag information is stored in the storage unit 400 , in this case, there may be repeatedly stored tag information. In this case, the control unit 100 may provide the user with the order of the most stored tags among the tag information stored in the storage unit 400 . That is, the recommendation tag provided to the user by the controller 100 may be a tag stored repeatedly the most.

The controller 100 may classify the tag information stored in the storage 400 according to the user's activity log and provide it to the user as a recommended tag. After obtaining information about the user's activity log in the user terminal 1000 , the controller 100 may provide suitable tag information to the user based thereon.

The control unit 100 may classify the tag information stored in the storage unit 400 by type and provide it to the user as a recommended tag. More specifically, the plurality of screenshot images stored in the storage 400 may be classified by type based on tag information. For example, a screenshot image to which tag information 'webpage' is mapped among a plurality of screenshot images stored in the storage unit 400 may be classified as a first type, and a screen to which tag information 'gifticon' is mapped. The shot image may be classified into a second type, a screenshot image to which tag information of 'shopping mall' is mapped may be classified as a third type, and a screenshot image to which tag information of 'SNS' is mapped may be classified as a fourth type can be classified as

As such, the screenshot image may be classified into a plurality of types based on the mapped tag information, and the controller 100 provides tag information based on classifying the screenshot image into a plurality of types as a recommended tag to the user. can do. That is, the recommendation tag provided by the controller 100 to the user may be a 'web page', a 'gifticon', a 'shopping mall', or the like.

The control unit 100 may display the recommendation tag selected through the step of providing the recommendation tag to the user ( S3600 ) on the output unit 500 and provide it to the user.

There may be various methods in which the controller 100 provides the recommendation tag to the user through the output unit 500 . For example, the controller 100 may provide only text related to the recommendation tag to the user through the output unit 500 . As another example, the controller 100 may provide, through the output unit 500 , at least a partial region of the screenshot image to which the recommendation tag is mapped together with the text related to the recommendation tag to the user. When the recommendation tag is provided to the user in the form of an image together with text, the user may more intuitively understand information about the recommendation tag.

In this case, the shape of the image provided to the user together with the recommendation tag may relate to at least a partial area of the screenshot image. For example, the shape of the image may relate to any one of a text area and/or a non-text area among the screenshot images. Alternatively, the form of the image provided to the user together with the recommendation tag may be an image processed based on the recommendation tag.

4 How to search for and provide screenshot images based on tag information

The user terminal 1000 includes a storage unit 300 for storing a plurality of screenshot images, an output unit 500 for displaying at least one screenshot image among the plurality of screenshot images, and an input unit for receiving a user input ( 400), a communication unit 700 for communicating with an external server, and a control unit 100 for determining at least one screenshot image to be displayed on the output unit among the plurality of screenshot images based on the user input can

Hereinafter, a method of retrieving at least one of a plurality of screenshot images stored in the storage unit 300 based on tag information will be described with reference to the drawings.

18 is a diagram for explaining a method of searching for a screenshot image based on tag information and a method of providing a search result to a user.

Referring to FIG. 18 , the output unit 500 includes a search unit SE that allows a user to input a search word for a screen shot image search, and a related search unit SE that displays keywords related to a user input input to the search unit. '), and a search result unit RES for displaying a screen shot image searched for in response to a user input.

A plurality of screenshot images may be stored in the storage unit 300 , and at least one piece of tag information may be mapped to each of the plurality of screenshot images. In this case, the tag information is extracted from the screenshot image, and since this has been described above, a redundant description will be omitted.

When the user input is input through the input unit 400 , the controller 100 may determine matching tag information corresponding to the user input from among the tag information stored in the storage unit 300 .

For example, the controller 100 may determine the same tag information as the user input from among the tag information stored in the storage 300 or tag information showing a degree of similarity greater than or equal to a predetermined standard as the matching tag information.

According to an embodiment, as shown in (a) of FIG. 18 , there may be one user input input through the search unit SE. That is, there is one user input obtained through the user input unit 400, and the control unit 100 determines matching tag information based on the input user input, and then at least one or more screenshots to which the matching tag information is mapped. The image may be output to the search result unit RES.

According to another embodiment, as shown in (b) of FIG. 18 , there may be two or more user inputs obtained through the user input unit 400 . The user input obtained through the user input unit 400 may include a first user input and a second user input. In this case, the first user input is input at a first time point, and the second user input is input at a second time point, but the first time point and the second time point may be different.

The controller 100 may determine matching tag information based on at least one of a first user input and a second user input. For example, the controller 100 may determine tag information corresponding to at least one of the first user input and the second user input as the matching tag information.

The first time point may be earlier than the second time point. In this case, the control unit 100 determines the matching tag information in consideration of the time points at which the first user input and the second user input are input to the input unit 400 . can

For example, the control unit 100 may determine the matching tag information based on a first user input and a second user input, and determine the matching tag information by giving importance to the first user input. More specifically, the controller 100 may determine, as the matching tag information, tag information having a higher probability of corresponding to the first user input from among the plurality of tag information corresponding to the first user input and the second user input. .

As another example, the controller 100 may determine tag information corresponding to the second user input from among the plurality of tag information corresponding to the first user input as matching tag information. More specifically, the controller 100 may determine the first tag information corresponding to the first user input from among the plurality of tag information. Thereafter, the controller 100 may determine second tag information corresponding to a second user input among the first tag information as matching tag information.

The control unit 100 may control the screen shot image to which the matching tag information is mapped to be output through the output unit, and control so that the screenshot image is displayed on the output unit according to the time sequence stored in the storage unit. .

For example, captured time information may be input to the screenshot image stored in the storage unit 300 . In this case, the controller 100 may control the plurality of screenshot images to which the matching tag information is mapped to be output to the output unit 500 after arranging them according to the chronological order stored in the storage unit 300 .

More specifically, the control unit 100 may control the output to the output unit 500 after sorting in the order most recently stored in the storage unit 300 among a plurality of screenshot images to which the matching tag information is mapped. .

In addition, the control unit 100 controls the screen shot image to which the matching tag information is mapped to be output through the output unit, and the matching tag information is output to the output unit in the order of the highest probability value corresponding to the user input. You can control the display.

For example, the matching tag information is determined based on a user input, and more specifically, tag information corresponding to a user input among a plurality of tag information is determined as the matching tag information.

In this case, the matching tag information may be determined based on a probability value corresponding to a user input among a plurality of tag information. For example, tag information in which a probability value corresponding to a user input is greater than or equal to a predetermined criterion among a plurality of tag information may be determined as matching tag information.

Features, structures, effects, etc. described in the above embodiments are included in at least one embodiment of the present invention, and are not necessarily limited to only one embodiment. Furthermore, features, structures, effects, etc. illustrated in each embodiment can be combined or modified for other embodiments by those of ordinary skill in the art to which the embodiments belong. Accordingly, the contents related to such combinations and modifications should be interpreted as being included in the scope of the present invention.

In addition, although the embodiment has been mainly described in the above, this is only an example and does not limit the present invention, and those of ordinary skill in the art to which the present invention pertains to the above in the range that does not depart from the essential characteristics of the present embodiment. It will be appreciated that various modifications and applications not illustrated are possible. That is, each component specifically shown in the embodiment can be implemented by modification. And differences related to such modifications and applications should be construed as being included in the scope of the present invention defined in the appended claims.

Claims (7)

In the user terminal,
a storage unit for storing a plurality of screenshot images, wherein at least one or more tag information is mapped to each of the plurality of screenshot images;
a display unit configured to display at least one screenshot image among the plurality of screenshot images;
an input unit receiving a user input;
a communication unit for communicating with an external server; and
A control unit configured to determine at least one screenshot image to be displayed on the display unit from among the plurality of screenshot images based on the user input;
When the user input is input through the input unit, the control unit determines matching tag information corresponding to the user input from among the tag information stored in the storage unit, and then a screenshot image to which the matching tag information is mapped. is controlled to be output through the display unit,
The control unit extracts text information from the screenshot image through a text reading module and then transmits the text information to the server, wherein the server extracts the tag information from the text information and transmits it to the user terminal,
user terminal.
According to claim 1,
The user input includes a first user input and a second user input,
The input unit receives the first user input at a first time point and receives the second user input at a second time point different from the first time point,
The control unit determines the matching tag information in consideration of a time point at which the first user input and the second user input are input to the input unit,
user terminal.
3. The method of claim 2,
The control unit determines the matching tag information based on the first user input and the second user input, and determines the first user input with importance;
The first time point is a time earlier than the second time point,
user terminal.
3. The method of claim 2,
The matching tag information is tag information corresponding to at least one of the first user input and the second user input among the plurality of tag information,
user terminal.
5. The method of claim 4,
The matching tag information is tag information corresponding to the second user input among tag information corresponding to the first user input among the plurality of tag information;
user terminal.
According to claim 1,
The control unit controls the screen shot image to which the matching tag information is mapped to be output through the display unit, and controls the screenshot image to be displayed on the display unit according to the time sequence stored in the storage unit,
user terminal.
According to claim 1,
The control unit controls the screen shot image to which the matching tag information is mapped to be output through the display unit, and controls the matching tag information to be displayed on the display unit in order of increasing probability values corresponding to the user input ,
user terminal.

Images