KR102048638B1 - Method and system for recognizing content - Google Patents

Abstract

The present invention relates to a method for recognizing content, which is capable of increasing a content recognition rate. According to the present invention, the method for recognizing content comprises the following steps: receiving a first image including at least a part of first content; extracting first text information from the first image; extracting one or more partial images of a first set from the first image; calculating a text similarity score between first and second images based on the first text information and second text information associated with a second image; calculating an image similarity score based on the one or more partial images of the first set and one or more partial images of a second set associated with the second image; and calculating a total similarity score based on the text and image similarity scores; and determining that at least a part of the first content is included in the second image when the total similarity score is equal to or greater than a preset first threshold value.

Description

Content Recognition Method and System {METHOD AND SYSTEM FOR RECOGNIZING CONTENT}

The present disclosure relates to a method for recognizing content and a system therefor. More particularly, the present disclosure relates to a method and system for recognizing content that can improve content recognition rate by using a text search and an image search together.

Recently, as information and communication technologies such as big data and AI are developed, various content materials can be easily accessed. Accordingly, there is a need for a search system for recognizing content and searching for the same or similar content.

Conventional image retrieval methods include a text-based retrieval method for recognizing text included in content material and searching based on extracted text information, and an image feature-based retrieval method using image features included in content material. The text-based retrieval method is the simplest and easiest way to recognize and search content materials, but the recognition rate may be degraded due to various distortions, and accurate search is difficult when the content includes many elements other than text.

The image feature-based search method is a method of searching for similar images based on image features such as color and shape. In the case of a picture image, a relatively accurate search is possible, but in the case of an image including text, there is a problem that the search accuracy is low.
[Preceding technical literature]
1. Korea Registered Patent Publication No. 10-0670003 (2007.01.10)
2. Korean Patent Publication No. 10-2016-0057125 (2016.05.23)
3. Korean Unexamined Patent Publication No. 10-2018-0039596 (2018.04.18)
4. US Patent Publication US 8,041,139 (October 18, 2011)

Embodiments disclosed herein relate to a method and a system for increasing content recognition rate by recognizing content received through a user terminal using a text search and an image search together.

A method for recognizing content according to an embodiment of the present disclosure may include receiving a first image including at least a portion of first content, extracting first text information from the first image, and extracting the first text information from the first image. Extracting at least one partial image of the first set, calculating a text similarity score of the first image and the second image based on the first text information and the second text information associated with the second image, the first set of Calculating an image similarity score based on the one or more partial images and a second set of one or more partial images associated with the second image, calculating a composite similarity score based on the text similarity score and the image similarity score and the composite similarity score Is greater than or equal to a preset first threshold, determine that at least a portion of the first content is included in the second image. It can include.

In a non-transitory computer readable storage medium having stored thereon instructions for recognizing content according to an embodiment of the present disclosure, when executed by the processor, the instructions cause the processor to generate a first image including at least a portion of the first content. Receive, extract first text information from the first image, extract the first set of one or more partial images from the first image, and based on the first text information and the second text information associated with the second image Calculate a text similarity score of the image and the second image, calculate an image similarity score based on the at least one partial image of the first set and the at least one partial image associated with the second image, and calculate the text similarity score and the image Comprehensive similarity score of the first image and the second image is calculated based on the similarity score, and the comprehensive similarity point If the number is greater than or equal to a preset first threshold, it may cause the determination that at least a portion of the first content is included in the second image.

According to an embodiment of the present disclosure, a content recognizing system includes a communication unit configured to receive a first image including at least a portion of first content, an OCR system configured to extract first text information from the first image, and a first image. An object recognition system configured to extract a first set of one or more partial images, a text search configured to calculate a text similarity score of the first image and the second image based on the first text information and the second text information associated with the second image A system, an image retrieval system configured to calculate an image similarity score based on the first set of one or more partial images and the second set of one or more partial images associated with the second image, and a composite based on the text similarity score and the image similarity score Include a search system configured to calculate a similarity score, System can, if the comprehensive degree of similarity score greater than a preset first threshold value, at least a portion of the first content is determined to be included in the second image.

A method for recognizing content in accordance with an embodiment of the present disclosure includes receiving first text information associated with a first image and data for a first set of one or more partial images, wherein the first image is selected from the first content. Including at least a portion; calculating a text similarity score of the first image and the second image based on the first text information and the second text information associated with the second image; Calculating an image similarity score based on the second set of one or more partial images associated with the two images, calculating a composite similarity score based on the text similarity score and the image similarity score, and a first preset similarity score If at least a threshold, determining that at least a portion of the first content is included in the second image, wherein the image The similarity score includes one or more partial image similarity scores, the composite similarity score is calculated by applying weights to the text similarity score and the one or more partial image similarity scores, and the weights may be applied differently according to the object type in the partial image.

According to various embodiments of the present disclosure, by text search and image search for an image received through the user terminal, the text search and the image search are complementary to each other. Therefore, it is possible to overcome various distortions in content recognition, and to improve the recognition rate. In addition, even if the position of the text and the object in the image is changed by re-editing, the same content can be recognized.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

Embodiments of the present disclosure will be described with reference to the accompanying drawings, which are described below, wherein like reference numerals denote similar elements, but are not limited thereto.
1 is a diagram illustrating an environment in which a content recognition system recognizes content at the request of a user terminal according to an embodiment of the present disclosure.
2 is a block diagram illustrating a detailed configuration of a content recognizing system according to an embodiment of the present disclosure.
3 is a diagram illustrating an example of a first image captured by a user terminal according to an exemplary embodiment.
4 is a diagram illustrating an example of recognizing text in a first image according to an embodiment of the present disclosure.
FIG. 5 is a diagram illustrating a table that calculates scores of candidate words for a word "count" in the recognized text in FIG. 4.
6 is a diagram illustrating an example of correcting text recognized in a first image based on a dictionary DB and a language model.
FIG. 7 is a diagram illustrating an example of calculating a text similarity score by comparing first text information extracted from a first image with second text information associated with a second image stored in a content DB.
8 is a diagram illustrating an example of a first image in which a text line area is removed.
9 is a diagram illustrating an example of calculating an image similarity score according to an embodiment of the present disclosure.
10 to 12 are diagrams illustrating an example of calculating an image similarity score according to another exemplary embodiment of the present disclosure.
13 is a flowchart illustrating a content recognition method according to an embodiment of the present disclosure.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. However, in the following description, when there is a risk of unnecessarily obscuring the subject matter of the present disclosure, a detailed description of well-known functions and configurations will be omitted.

In the accompanying drawings, the same or corresponding components are given the same reference numerals. In addition, in the following description of the embodiments, it may be omitted to repeatedly describe the same or corresponding components. However, even if the description of the component is omitted, it is not intended that such component is not included in any embodiment.

In the present disclosure, “first image” may refer to an image photographed or input by a user through a user terminal. In the present disclosure, “second image” may refer to any image stored in a database. In the present disclosure, “content image” may refer to an image including at least a portion of content.

1 is a diagram illustrating an environment in which the content recognizing system 130 recognizes content at the request of the user terminals 110_1 to 110_n according to an embodiment of the present disclosure. According to an embodiment, one or more user terminals 110_1 to 110_n may communicate with the content recognition system 130 through the communication network 120. The communication network 120 may be a wired network such as Ethernet, a wired home network, a power line communication, a telephone line network, or an RS-serial communication, or a wireless LAN (WLAN), Bluetooth, and ZigBee, depending on an installation environment. It may be configured and selected in a variety of wireless networks, such as).

The user terminals 110_1 to 110_n may photograph content using a camera module (not shown), and may transmit data about the captured image to the content recognition system 130 through the communication network 120. Alternatively, the user terminals 110_1 to 110_n may transmit data about an internally stored image to the content recognizing system 130. In an embodiment, the user terminals 110_1 to 110_n may include a smart phone, a tablet PC, a notebook computer, a personal digital assistant (PDA), a mobile communication terminal, and the like, but are not limited thereto, and a camera module and / or a communication module. It may be any device having a.

The content recognizing system 130 is a server device for recognizing content in an image transmitted by the user terminals 110_1 to 110_n and may include a communication unit 140, a processor 150, and a database 160. The communicator 140 may receive image data from the user terminals 110_1 to 110_n through the communication network 120. The processor 150 may receive the image data received by the communicator 140 and process the corresponding image data.

In an embodiment, the processor 150 may include at least one of a central processing unit (CPU), a graphic processing unit (GPU), and a digital signal processor (DSP) to perform an operation operation. The processor 150 extracts the text information and the partial image from the image data, performs text search, object recognition, and image search, and the like or similar to the content included in the received image data among the content images stored in the database 160. You can search for content. The process of the content recognizing system 130 to recognize content in the database 160 that is the same as or similar to the received content image will be described in detail with reference to FIG.

2 is a block diagram illustrating a detailed configuration of a content recognizing system 200 according to an embodiment of the present disclosure. As shown in FIG. 2, the content recognition system 200 may include a communication unit 140, a database 160, and a processor 150. Among the functions or components of the content recognizing system 200, components having the same reference numbers or names as described in FIG. 1 may be omitted in order to avoid repetition, and only changes or additional parts may be used. It can be explained.

The processor 150 may include an OCR system 210, an object recognition system 220, and a search system 230. In the case of search system 230, it may include a text search system 232 and an image search system 234. The database 160 may include a content DB 162, a dictionary DB 164, and a language model 160.

The communication unit 140 of the content recognition system 200 may communicate with an external device such as a user terminal through the communication network 120. According to an embodiment of the present disclosure, the content recognition system 200 may include the same or similar content by the communicator 140 receiving an image, and the processor 150 comparing the received image with content images stored in the database 160. Can be configured to retrieve the captured image. The image including the retrieved same or similar content may be transmitted to the user terminal through the communication network 120. In one embodiment, the content included in the image may be learning content such as a math problem, but is not limited thereto, and may be any content including text and non-text objects (pictures, figures, graphs, formulas, etc.). have.

In detail, the communication unit 140 may receive the first image from the user terminal through the communication network 120. Here, the first image may be an image captured by the user using the user terminal or an image stored in the user terminal, and the first image may include at least a part of the first content. The communicator 140 may provide the processor 150 with the first image received from the user terminal.

OCR system 210 may be configured to extract textual information within an image. In one embodiment, the OCR system 210 may extract the first text information from the first image transmitted by the user terminal. To this end, the OCR system 210 may detect one or more text line regions within the first image. At this time, in order to increase the text line area recognition rate and the text recognition rate, the OCR system 210 may perform image preprocessing such as binarization and contrast processing to remove the background and make the text portion sharp. The image similarity score and the partial similarity score which are made later may also be calculated based on the image after the image preprocessing is performed.

The OCR system 210 may then recognize the text in the detected text line region using optical character recognition (OCR) technology. At this time, in order to increase the text recognition rate, the text line area image may be resized to an optimal size. For example, the size of the text line area image may be resized such that the text height is the size of font 20.

The OCR system 210 may correct the recognized text based on the dictionary DB 164 and the language model 166. The dictionary DB 164 and the language model 166 may be data prepared to be suitable for the language and content field of the text to be recognized. For example, if the content to be recognized is Hangul mathematics learning content, the dictionary DB 164 may include words that frequently appear in the Hangul mathematics learning content, and the language model 166 learns based on the Hangul mathematics learning content. It may be a language model.

According to an embodiment, the OCR system 210 may correct the recognized text on a word-by-word basis. In this case, the OCR system 210 may determine one or more candidate words for each word based on the dictionary DB 164. For example, the OCR system 210 calculates an edit distance between the recognized word and each word stored in the dictionary DB 164 based on the edit distance algorithm, and the calculated edit distance is a preset threshold. Words within the range can be determined as candidate words. Here, the editing distance may represent the minimum number of operations required to make the recognized word the same as the word stored in the dictionary DB 164.

After selecting candidate words, the OCR system 210 may calculate a score for each candidate word based on the language model 166. In calculating the score, the OCR system 210 may calculate a score by referring to words positioned before the word and words positioned behind the word. For example, the language model 166 may be a Recurrent Neural Network (RNN) language model and may refer to 5 to 10 words before the word and 2 or 3 words after the word. The language model 166 is not limited to the RNN language model, and may be variously implemented without departing from the present disclosure.

The OCR system 210 may select the candidate word having the highest score as the final word when the score of the candidate word having the highest score is greater than or equal to a preset threshold. On the other hand, when the score of the candidate word having the highest score is less than the preset threshold, the OCR system 210 may delete the word and exclude the area containing the word from the text line area. The above-described process may be repeated for each word to extract text information in the first image. Through this process, misrecognized text is corrected, and a part which is incorrectly extracted as a text line area without text is included in the text line area.

The object recognition system 220 may extract one or more partial images of the first set from the first image received from the user terminal. Specifically, object recognition system 220 may remove a text line region from the first image and extract one or more sets of one or more partial images from the first image using a clustering technique. For example, a technique such as Density-Based Spatial Clustering of Applications with Noise (DBSCAN) may be used to extract one or more partial images based on a margin from a first image from which text lines have been removed. In addition, the object recognition system 220 may extract the entire first image and include it in the one or more partial images of the first set as a partial image.

The object recognition system 220 can then recognize the objects in the one or more partial images of the first set extracted from the first image. For example, the object recognition system 220 may recognize an object in the partial image using a deep learning based classification model. Here, the recognized object may include at least one of an equation, a graph, a figure, and a picture. The entire image extracted as the partial image may be tagged as the entire image without performing object recognition separately.

The search system 230 stores the first image and the content DB 162 transmitted from the user terminal based on the text similarity score calculated by the text search system 232 and the image similarity score calculated by the image search system 234. And calculate a comprehensive similarity between the second images stored in the. In one embodiment, the content DB 162 may include a plurality of content images, and may also store information (text information, information about a partial image, object information included in the partial image, etc.) associated with the content image. have.

The text search system 232 calculates a text similarity score of the first image and the second image based on the first text information extracted from the first image and the second text information associated with the second image stored in the content DB 162. can do. In one embodiment, the text search system 232 may calculate a text similarity score by comparing the entire text in the first text information and the second text information. In another embodiment, the text search system 232 may calculate a text similarity score by comparing the keywords in the first text information and the second text information.

Image retrieval system 234 calculates an image similarity score based on the first set of one or more partial images extracted from the first image and the second set of one or more partial images associated with the second image stored in content DB 162. can do. Here, the image similarity score may include one or more partial image similarity scores, and the image similarity score may be calculated as a weighted sum of the partial image similarity scores. In one embodiment, image retrieval system 234 may calculate partial image similarity scores between partial images having the same object type. In another embodiment, image retrieval system 234 may calculate partial image similarity scores between the partial images with the highest similarity. In addition, the entire image extracted as the partial image may be compared with each other to calculate the partial image similarity score.

The partial image similarity score may be calculated based on image features extracted from the two partial images. For example, image features may be extracted using Scale Invariant Feature Transform (SIFT), Speed-Up Robust Feature (SURF), Binary Robust Independent Elementary Features (BRIEF), and Oriented FAST and Rotated BRIEF (ORB) techniques. .

In some embodiments, search system 230 may apply a weight to the text similarity score and the one or more partial image similarity scores to calculate a composite similarity score. For example, the overall similarity score may be calculated by the following equation.

은 텍스트 유사도 점수를 나타내고,

는 부분 이미지 유사도 점수를 나타내고,

는 종합 유사도 점수를 나타내고,

는 텍스트 유사도 점수에 적용되는 가중치를 나타내고,

는 부분 이미지 유사도 점수에 적용되는 가중치를 나타내고, n은 부분 이미지 유사도 점수의 개수를 나타낸다. 일 실시예에서, 텍스트 유사도 점수에 적용되는 가중치

는 부분 이미지 유사도 점수에 적용되는 가중치

보다 클 수 있다.here,

Indicates a text similarity score,

Represents a partial image similarity score,

Represents a composite similarity score,

Represents the weight applied to the text similarity score,

Denotes a weight applied to the partial image similarity score, and n denotes the number of partial image similarity scores. In one embodiment, the weight applied to the text similarity score

Is a weight applied to the partial image similarity score

Can be greater than

는 부분 이미지 내의 객체 유형(예를 들어, 수식, 그래프, 도형, 그림, 전체 이미지)에 따라 상이하게 적용될 수 있다. 예를 들어, 부분 이미지 내의 객체가 그림인 경우, 부분 이미지 내의 객체가 도형인 경우보다 높은 가중치가 적용될 수 있고, 부분 이미지 내의 객체가 도형인 경우, 부분 이미지 내의 객체가 그래프인 경우보다 높은 가중치가 적용될 수 있다. 또한, 부분 이미지 내의 객체가 그래프인 경우, 부분 이미지 내의 객체가 수식인 경우보다 높은 가중치가 적용될 수 있고, 부분 이미지 내의 객체가 수식인 경우, 부분 이미지가 전체 이미지인 경우보다 높은 가중치가 적용될 수 있다.Also, the weight applied to the partial image similarity score

May be applied differently according to the object type (eg, formula, graph, figure, picture, whole image) in the partial image. For example, when the object in the partial image is a picture, a higher weight may be applied than when the object in the partial image is a shape, and when the object in the partial image is a shape, the weight is higher than when the object in the partial image is a graph. Can be applied. In addition, when the object in the partial image is a graph, a higher weight may be applied than when the object in the partial image is a formula, and when the object in the partial image is a formula, a higher weight may be applied than when the partial image is a full image. .

The search system 230 may determine a relationship (same content, similar content, dissimilar content, etc.) between the first image and the second image based on the calculated comprehensive similarity score. For example, when the comprehensive similarity score is equal to or greater than the first threshold value, the first image and the second image may be determined to include the same content. When the overall similarity score is equal to or greater than a second preset threshold and less than the first threshold, it may be determined that the first image and the second image include similar content. On the other hand, when the comprehensive similarity score is less than the second threshold, the first image and the second image may be determined to include dissimilar content.

The search system 230 may search for the same or similar content by comparing the first image transmitted from the user terminal with all images stored in the content DB 162 according to the above-described process. In an embodiment, when the same or similar content as the first image is found, the content recognizing system 200 may transmit an image including the corresponding content to the user terminal. In addition, one or more of associated text information, information about the partial image, and object information included in the partial image may be provided together with the image. Additionally or alternatively, the content recognition system 200 may provide the user terminal with information (grade information, difficulty information, subject information, etc.) associated with the retrieved same or similar content. In another embodiment, if the same content as the first image is not found, the content recognizing system 200 stores the first image along with the content DB (including related text information, information about the partial image, object information included in the partial image), and the like. 162).

In FIG. 2, the content recognition system 200 is illustrated as including the OCR system 210 and the object recognition system 220, but is not limited thereto. The user terminal may use the OCR system 210 and the object recognition system 220. It may be configured to include. In this case, the user terminal transmits the first text information extracted from the first image, the first set of one or more partial images extracted from the first image, and the information of the object included in each partial image to the content recognizing system 200. Can be.

As described above, the content recognition system 200 is configured to search the content DB 162 in consideration of all of the text similarity score, the partial image similarity score, and the object information in the partial image, thereby re-editing the text or the content within the content. Even if the position of the object is changed, it is possible to recognize the same content. In addition, since the text retrieval system 232 and the image retrieval system 234 are complementary to each other, it is possible to overcome various distortions (differences in editing, differences in shooting conditions, etc.) included in images captured by the user terminal, It is possible to improve the recognition rate. An example of the operation of the content recognizing system 200 will be described in detail with reference to FIGS. 3 to 9.

3 is a diagram illustrating an example of a first image 310 photographed by a user terminal according to an exemplary embodiment. As illustrated in FIG. 3, a user may photograph content using a user terminal (eg, a smart phone), store the first image 310, and transmit the content to the content recognition system through a communication network. Here, the first image 310 is an image of a math problem.

4 is a diagram illustrating an example of recognizing text in a first image 310 according to an embodiment of the present disclosure. The first image 310 includes math problem content. The OCR system can detect text line regions 410-460 in the first image 310. In an embodiment, the OCR system may perform image preprocessing such as binarization and contrast processing to remove the background and to sharpen the text portion before the text line region detection to increase the text line region recognition rate and the text recognition rate.

The OCR system can then recognize the text in the text line areas 410-460. As shown in Fig. 4, in the modifier portion, the text line area 460 is not properly recognized, so that the text may be misrecognized as "g") ..- ln (.r + l) '". In the case of a formula, the OCR system often outputs an incorrect text recognition result because it is difficult to properly recognize the text line area due to the integral, derivative, matrix, etc. Also, the "count" in the properly recognized text line areas 410 and 420 may be used. And text may not be recognized properly.

FIG. 5 is a diagram illustrating a table 500 in which scores of candidate words are calculated for the word “count” in the recognized text in FIG. 4. The OCR system may determine candidate words for "count" based on the dictionary DB. In one embodiment, the OCR system may determine candidate words by searching for words having an "count" among words in the dictionary DB and an edit distance within a preset threshold.

For example, as shown, words of "count", "count", "count", and "count" may be searched for words within an editing distance of 2 and may be determined as candidate words. In the case of the odd number, the editing distance may be 0, and in the case of the coefficient, "ㅔ" may be replaced with "ㅖ" and the editing distance may be 1 by the replacement operation. Similarly, the "number" can be edited distance 1 and house 2 by editing operation. Here, the insertion operation means an operation generated by adding a new character to a specific string, and the replacement operation means an operation generated by replacing a character included in the specific string with a new character. In addition, the delete operation refers to an operation generated by deleting a character included in a specific string, and the prefix operation refers to an operation generated by changing the order of adjacent characters included in a specific string.

After electing one or more candidate words, the OCR system may calculate a score for each candidate word based on the language model. In calculating the score, the OCR system may calculate a score by referring to words positioned before the word and words positioned behind the word. For example, the language model may be a Recurrent Neural Network (RNN) language model, and may refer to 5 to 10 words before the word and 2 to 3 words after the word.

Referring to FIG. 4, the first word of "colon" included in the first text information may be one "secondary term", and the second word may be "secondary function", "f (x)", "function", or the like. . In the case of "coefficient" of candidate words, the OCR system is shown based on the language model because it has a high correlation with the preceding word, the second term, and the following words, "secondary function", "f (x)", and "function". As described above, a score of 95 points can be calculated. On the contrary, the remaining candidate words are not highly related to the front word and the back word, so that relatively low scores of 45, 55, and 35 points can be given.

If the score of the candidate word having the highest score among the candidate words is equal to or greater than a preset threshold, the candidate word having the highest score may be selected as the final word. On the other hand, when the score of the candidate word having the highest score is less than the preset threshold, the word may be deleted from the recognized text, and the area containing the word may be excluded from the text line area. For example, when the preset threshold is 80 points, the "coefficient" may be selected as the final word because the "coefficient" having the highest score among candidate words for the "count" is 95 or more thresholds.

6 is a diagram illustrating an example of correcting text recognized in the first image 310 based on a dictionary DB and a language model. As described above, the recognized text may be corrected by repeating the correcting process of FIG. 5 for all words included in the recognized text in the first image 310. For example, as shown in FIG. 6, "counts" and "medium" may be corrected to "coefficient" and "segment" based on dictionary DB and language model. Further, in the case of "g") ..- ln (.r + l) '", the candidate word having the highest score is determined to be below a preset threshold, and is deleted from the recognized text, and the corresponding area 460 is deleted. The first text information may be extracted from the first image 310 through the process.

FIG. 7 is a diagram illustrating an example of calculating a text similarity score by comparing first text information extracted from the first image 310 with second text information associated with the second image 710 stored in the content DB. The second text information may be stored in the content DB in association with the second image 710. As shown, the first image 310 and the second image 710 contain the same mathematical problem content, but the overall layout is different because the width of the second image 710 is longer due to the difference in editing. Since the first image 310 and the second image 710 are identical in their overall layout but include the same text, the text search engine calculates a high text similarity score of the first image 310 and the second image 710. can do. For example, the text similarity score of the first image 310 and the second image 710 may be 95 points.

8 is a diagram illustrating an example of a first image 810 from which text line regions 410 to 450 are removed. The object recognition system may remove the text line areas 410 through 450 from the first image 310. As described above, the region 460 shown in FIG. 4 is excluded from the text line region and is not removed from the first image 310.

The object recognition system may extract the first set of one or more partial images from the first image 810 from which the text line regions 410-450 have been removed. As illustrated, the partial image 820 including the formula may be extracted from the first image 310 from which the text line regions 410 to 450 are removed. In addition, the first image 310 in which the text line area is not removed may also be extracted as a partial image. Accordingly, in the case of the first image 310, the partial image 820 including the modifier part and the first image 310 may be included in the first set of partial images.

The object recognition system can then recognize the object in the partial image 820. For example, an object in the partial image 820 may be recognized by an equation. In the case of the first image 310, the object recognition system may recognize the entire image.

9 is a diagram illustrating an example of calculating an image similarity score according to an embodiment of the present disclosure. The image retrieval system scores an image similarity based on the first set of partial images 310, 820 extracted from the first image 310 and the second set of partial images 710, 910 associated with the second image 710. Can be calculated. The second set of partial images 710, 910 associated with the second image 710 may be stored in the content DB in association with the second image 710.

In one embodiment, the image retrieval system may calculate partial image similarity scores by comparing partial images including objects of the same type (eg, equations, graphs, figures and drawings, and entire images). For example, the image retrieval system calculates a partial image similarity score of the first image 310 and the second image 710 corresponding to the entire image, and the partial image 820 and the partial image 910 including the equation object. The partial image similarity score of) can be calculated. In another embodiment, the image retrieval system may calculate partial image similarity scores by comparing partial images having high similarity. For example, the image retrieval system may calculate the partial image similarity score with the second image 710 having a high similarity by comparing the first image 310 with the second image 710 and the partial image 910.

As shown, the first partial image similarity score between the first image 310 and the second image 710 is arranged although the first image 310 and the second image 710 contain the same mathematical problem content. Since it is different, it can be calculated low. For example, the first partial image score may be 50 points. Since the partial image 820 and the partial image 910 include the same equation, the second partial image similarity score between the partial image 820 and the partial image 910 may be calculated to be high. For example, the second partial image similarity score may be 90 points.

In one embodiment, the image similarity score may be calculated through a weighted sum of each calculated partial image similarity score. The weight applied to the partial image similarity score may be applied differently according to the object type (formula, graph, figure, picture, whole image) in the partial image. For example, when the object type is a formula, a higher weight may be applied than when the object type is an entire image. Therefore, even if the content arrangement in the first image 310 and the second image 710 is different, the image similarity score may be calculated to be high.

Thereafter, the search system may calculate a comprehensive similarity score of the first image 310 and the second image 710 based on the text similarity score and the image similarity score. In one embodiment, the weight applied to the text similarity score may be greater than the weight applied to the partial image similarity score. For example, the comprehensive similarity score of the first image 310 and the second image 710 may be calculated by the following equation.

은 텍스트 유사도 점수(95점)를 나타내고,

는 제1 부분 이미지 유사도 점수(50점)를 나타내고,

는 제2 부분 이미지 유사도 점수(90점)를 나타내고,

는 종합 유사도 점수를 나타내고,

는 텍스트 유사도 점수에 적용되는 가중치(0.6)를 나타내고,

는 제1 부분 이미지 유사도 점수에 적용되는 가중치(0.3)를 나타내고,

는 제2 부분 이미지 유사도 점수에 적용되는 가중치(0.1)를 나타낼 수 있다. 이 경우, 종합 유사도 점수(

)는 89점으로 산출될 수 있다.here,

Indicates a text similarity score (95 points),

Represents a first partial image similarity score (50 points),

Represents a second partial image similarity score (90 points),

Represents a composite similarity score,

Represents the weight (0.6) applied to the text similarity score,

Represents a weight (0.3) applied to the first partial image similarity score,

May represent a weight (0.1) applied to the second partial image similarity score. In this case, the composite similarity score (

) Can be calculated as 89.

In one embodiment, the search system may determine a relationship (same content, similar content, dissimilar content, etc.) between the first image 310 and the second image 710 based on the calculated overall similarity score. For example, when the overall similarity score is equal to or greater than a first threshold value (80 points), the first image and the second image may be determined to include the same content. Since the total similarity score between the first image 310 and the second image 710 is calculated as 89 points, the search system may determine that the first image and the second image include the same content.

10 to 12 are diagrams illustrating an example of calculating an image similarity score according to another exemplary embodiment of the present disclosure. 10 is a diagram illustrating a first image 1010 and a second image 1020 according to another embodiment of the present disclosure. As shown, the first image 1010 and the second image 1020 contain the same mathematical problem content, but the arrangement of the text and the object are different from each other due to a difference in editing. The first text information is extracted from the first image 1010 in the same manner as described above, and the text is compared by comparing the similarity between the extracted first text information and the second text information associated with the second image 1020 stored in the content DB. Similarity scores can be calculated.

In addition, a first set of partial images may be extracted from the first image 1010 in the same manner as described above. As illustrated, the first image 1010, the first partial image 1112 including the equation, and the second partial image 1210 including the figure may be extracted. The image retrieval system compares the similarity between the first set of partial images 1010, 1112, 1210 and the second set of partial images 1020, 1122, 1220 stored in association with the second image 1020 in the content DB. An image similarity score and an image similarity score may be calculated. For example, a first partial image similarity score is calculated by comparing the first image 1010 and the second image 1020, which are the entire image, and the first partial image 1112 including the formula and the formula including the formula. Comparing the three partial images 1122, a second partial image similarity score is calculated, and a third partial image similarity is compared by comparing the second partial image 1210 including a figure and the fourth partial image 1220 including a figure. The score can be calculated.

The image similarity score may be calculated through a weighted sum of the first to third partial image similarity scores. The weight applied to the partial image similarity score may be applied differently according to the object type (formula, graph, figure, picture, whole image) in the partial image. For example, when the object type is a figure, a higher weight may be applied than when the object type is a formula, and when the object type is a formula, a higher weight may be applied than when the object type is an entire image. Thereafter, in the same manner as described above, a comprehensive similarity score may be calculated based on the text similarity score and the image similarity score, so that the first image 1010 and the second image 1020 may include the same content. .

13 is a flowchart illustrating a content recognizing method 1300 according to an embodiment of the present disclosure. The method 1300 for recognizing content may begin with receiving 1010 a first image that includes at least a portion of the first content. In an embodiment, the communication unit of the content recognizing system may receive the first image transmitted from the user terminal and provide the received first image to the processor.

Thereafter, in operation 1320, first text information may be extracted from the first image. For example, the OCR system may detect one or more text line regions in the first image to recognize text using Optical Character Recognition (OCR) technology. The recognized text may be corrected based on a dictionary DB and a language model.

After extracting the first text information, an operation 1330 of extracting the first set of one or more partial images from the first image may be performed. In one embodiment, the object recognition system may remove a text line area from the first image and extract a first set of one or more partial images from the first image using a clustering technique. The object recognition system may then recognize the object included in the extracted partial image, for example based on the deep learning based classification model. Here, the recognized object may include an equation, a graph, a figure, a picture, and an entire image.

In operation 1340, a text similarity score between the first image and the second image is calculated based on the first text information extracted from the first image and the second text information associated with the second image stored in the content DB. Can be. In one embodiment, the text search system may calculate a text similarity score by comparing the entire text in the first text information and the second text information. In another embodiment, the text search system may calculate a text similarity score by comparing the keywords in the first text information and the second text information.

Thereafter, calculating 1350 an image similarity score based on the first set of one or more partial images extracted from the first image and the second set of one or more partial images stored in association with the second image in the content DB. Can be performed. Here, the image similarity score may include one or more partial image similarity scores, and the image similarity score may be calculated as a weighted sum of the partial image similarity scores. In one embodiment, the image retrieval system may calculate partial image similarity scores between partial images having the same object type.

After calculating the text similarity score and the image similarity score, an operation of calculating a comprehensive similarity score of the first image and the second image based on the text similarity score and the image similarity score may be performed 1360. In some embodiments, the search system may calculate a composite similarity score by applying weights to the text similarity score and the one or more partial image similarity scores. For example, the overall similarity score may be calculated by the following equation.

은 텍스트 유사도 점수를 나타내고,

는 부분 이미지 유사도 점수를 나타내고,

는 종합 유사도 점수를 나타내고,

는 텍스트 유사도 점수에 적용되는 가중치를 나타내고,

는 부분 이미지 유사도 점수에 적용되는 가중치를 나타내고, n은 부분 이미지 유사도 점수의 개수를 나타낸다. 일 실시예에서, 텍스트 유사도 점수에 적용되는 가중치

는 부분 이미지 유사도 점수에 적용되는 가중치

보다 클 수 있다. 또한, 부분 이미지 유사도 점수에 적용되는 가중치

는 부분 이미지 내의 객체 유형(수식, 그래프, 도형, 그림, 전체 이미지)에 따라 상이하게 적용될 수 있다.here,

Indicates a text similarity score,

Represents a partial image similarity score,

Represents a composite similarity score,

Represents the weight applied to the text similarity score,

Denotes a weight applied to the partial image similarity score, and n denotes the number of partial image similarity scores. In one embodiment, the weight applied to the text similarity score

Is a weight applied to the partial image similarity score

Can be greater than Also, the weight applied to the partial image similarity score

May be applied differently according to the object type (formula, graph, figure, picture, whole image) in the partial image.

Finally, if the overall similarity score is equal to or greater than the first threshold value, step 1070 may be performed to determine that at least a portion of the first content is included in the second image. In some embodiments, the search system may determine a relationship (same content, similar content, dissimilar content, etc.) between the first image and the second image based on the calculated overall similarity score. For example, when the comprehensive similarity score is equal to or greater than the first threshold value, the first image and the second image may be determined to include the same content.

When the overall similarity score is equal to or greater than a second preset threshold and less than the first threshold, it may be determined that the first image and the second image include similar content. In addition, when the comprehensive similarity score is less than the second threshold, the first image and the second image may be determined to include dissimilar content. The search system may search for the same or similar content by comparing the first image transmitted from the user terminal with all images stored in the content DB. In addition, the first image transmitted from the user terminal, the first text information extracted from the first image, and the first set of partial images may be stored together in the content DB.

In FIG. 13, the extracting of the first text information 1320 and the calculating of the text similarity score 1340 are performed before the extracting the partial image 1330 and the calculating the image similarity score 1350. Although illustrated, the present invention is not limited thereto, and the extracting of the partial image 1330 and the calculating of the image similarity score 1350 may be performed first. Alternatively, extracting first text information 1320 and calculating text similarity score 1340 are in parallel with extracting partial images 1330 and calculating image similarity score 1350. It can proceed to.

The method for recognizing the above-described content may be embodied as computer readable code on a computer readable recording medium. Computer-readable recording media include all kinds of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, optical data storage devices, and the like. The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. In addition, functional programs, codes, and code segments for implementing the above embodiments can be easily inferred by programmers in the art to which the present invention belongs.

The techniques described herein may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques may include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs) Field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, It may be implemented within a computer, or a combination thereof.

Accordingly, various exemplary logic blocks, modules, and circuits described in connection with the disclosure herein may be used in general purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or It may be implemented or performed in any combination of those designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

In firmware and / or software implementations, the techniques are random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), PROM ( on computer readable media such as programmable read-only memory (EPROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, and the like. It may be implemented as stored instructions. The instructions may be executable by one or more processors, and may cause the processor (s) to perform certain aspects of the functionality described herein.

If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. Storage media may be any available media that can be accessed by a computer. By way of non-limiting example, such computer-readable media may be in the form of RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or desired program code in the form of instructions or data structures. Or any other medium that can be used for transport or storage to a computer and that can be accessed by a computer. Also, any connection is properly termed a computer readable medium.

For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, wireless, and microwave, the coaxial cable , Fiber optic cable, twisted pair, digital subscriber line, or wireless technologies such as infrared, wireless, and microwave are included within the definition of a medium. As used herein, disks and disks include CDs, laser disks, optical disks, digital versatile discs, floppy disks, and Blu-ray disks, where the disks are usually magnetic Data is reproduced optically, while discs are optically reproduced using a laser. Combinations of the above should also be included within the scope of computer-readable media.

The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other type of storage medium known in the art. An example storage medium may be coupled to the processor such that the processor can read information from or write information to the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may be present in the user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

The previous description of the disclosure is provided to enable a person skilled in the art to make or use the disclosure. Various modifications of the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to various modifications without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the examples described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Although example implementations may refer to utilizing aspects of the presently disclosed subject matter in the context of one or more standalone computer systems, the subject matter is not so limited, but rather in connection with any computing environment, such as a network or a distributed computing environment. It may also be implemented. Moreover, aspects of the presently disclosed subject matter may be implemented in or across a plurality of processing chips or devices, and storage may be similarly affected across a plurality of devices. Such devices may include PCs, network servers, and handheld devices.

Although the subject matter has been described in language specific to structural features and / or methodological acts, it will be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are described as example forms of implementing the claims.

Although the method described in this specification has been described with reference to specific embodiments, it is possible to embody it as computer readable code on a computer readable recording medium. Computer-readable recording media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, optical data storage devices, and the like. The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. And, the functional program, code and code segments for implementing the embodiments can be easily inferred by programmers in the art to which the present invention belongs.

Although the present disclosure has been described herein in connection with some embodiments, various modifications and changes may be made without departing from the scope of the present disclosure to those skilled in the art. Also, such modifications and variations are intended to fall within the scope of the claims appended hereto.

110_1 to 110_n: user terminal
120: communication network
130: content recognition system
140: communication unit
150: processor
160: database
162: content DB
164: dictionary DB
166: language model
210: OCR system
220: object recognition system
230: search system
232: text search system
234: image search system

Claims (12)

A method performed by a processor to recognize content,
Receiving a first image that includes at least a portion of the first content;
Extracting first text information from the first image;
Extracting a first set of one or more partial images from the first image, wherein the one or more partial images of the first set correspond to a partial region of the first image including an equation, graph, figure or picture Contains a partial image;
Calculating a text similarity score between the first image and the second image based on the first text information and second text information associated with the second image;
Calculating an image similarity score based on the one or more partial images of the first set and the one or more partial images of the second set associated with the second image, wherein the one or more partial images of the second set are formulas, graphs, figures Or a second partial image corresponding to a partial region of the second image including a picture, wherein the image similarity score is calculated based on the similarity between the first partial image and the second partial image;
Calculating a comprehensive similarity score based on the text similarity score and the image similarity score; And
Determining that at least a portion of the first content is included in the second image when the comprehensive similarity score is equal to or greater than a first threshold value preset;
Comprising a method performed by a processor to recognize content.
The method of claim 1,
Extracting the first text information may include:
Detecting one or more text line regions in the first image;
Recognizing text in the one or more text line areas; And
Correcting the recognized text based on a dictionary database and a language model.
Comprising a method performed by a processor to recognize content.
The method of claim 2,
Correcting the recognized text based on a dictionary database and a language model, for each word included in the recognized text,
Determining one or more candidate words based on the dictionary database;
Calculating a score for each of the one or more candidate words based on the language model; And
If the score of the candidate word having the highest score is greater than or equal to a preset threshold, the candidate word having the highest score is selected as the final word, and if the score of the candidate word having the highest score is less than the preset threshold, the word is included. To exclude the broken region from the text line region
Comprising a method performed by a processor to recognize content.
The method of claim 3,
Extracting the first set of one or more partial images from the first image,
Removing the text line area from the first image; And
Extracting the one or more partial images of the first set based on a clustering technique
Including,
And the at least one partial image of the first set further comprises a third partial image corresponding to the entire area of the first image.
The method of claim 4, wherein
Recognizing an object in the one or more partial images of the first set,
And the object comprises at least one of a formula, graph, figure, and figure, performed by a processor to recognize content.
The method of claim 5,
The image similarity score is calculated by comparing partial images including objects of the same type and comparing the third partial image with a fourth partial image corresponding to the entire area of the second image. The method performed by.
The method of claim 5,
The image similarity score comprises one or more partial image similarity scores,
The composite similarity score is calculated by applying weights to each of the text similarity score and the one or more partial image similarity scores,
Wherein the weighting is applied differently to each of the one or more partial image similarity scores according to the type of object in the partial image.
The method of claim 7, wherein
The comprehensive similarity score is calculated by the following formula,

here,

Indicates a text similarity score,

Represents a partial image similarity score,

Represents a composite similarity score,

Represents the weight applied to the text similarity score,

Denotes a weight applied to the partial image similarity score, n denotes the number of partial image similarity scores,
The weight applied to the text similarity score is greater than the weight applied to the partial image similarity score,
If the object in the partial image is a picture, a greater weight is applied than if the object in the partial image is a shape,
If the object in the partial image is a shape, a greater weight is applied than if the object in the partial image is a graph,
If the object in the partial image is a graph, a greater weight is applied than if the object in the partial image is a formula,
If the object in the partial image is a formula, a greater weight is applied than if the partial image is a full image.
The method of claim 1,
Determining that the first content is similar to second content included in the second image when the comprehensive similarity score is greater than or equal to a second predetermined threshold and less than the first threshold. The method performed by the processor to recognize.
A non-transitory computer readable storage medium having stored thereon instructions for recognizing content, wherein the instructions, when executed by a processor, cause the processor to:
Receive a first image that includes at least a portion of the first content,
Extracting first text information from the first image,
Extract a first set of one or more partial images from the first image, wherein the one or more partial images of the first set correspond to a partial region of the first image that includes an equation, graph, figure or picture Contains image-,
Calculate a text similarity score between the first image and the second image based on the first text information and second text information associated with the second image,
Calculate an image similarity score based on the one or more partial images of the first set and the one or more partial images of the second set associated with the second image, wherein the one or more partial images of the second set are formulas, graphs, figures or A second partial image corresponding to a partial region of the second image including a picture, wherein the image similarity score is calculated based on the similarity between the first partial image and the second partial image;
Calculating a comprehensive similarity score between the first image and the second image based on the text similarity score and the image similarity score,
If the comprehensive similarity score is equal to or greater than a first threshold value, it is determined that at least a part of the first content is included in the second image.
Computer readable storage media.
As a content recognition system,
A communication unit configured to receive a first image including at least a portion of the first content;
An OCR system configured to extract first text information from the first image;
An object recognition system configured to extract a first set of one or more partial images from the first image, wherein the one or more partial images of the first set correspond to a partial region of the first image including an equation, graph, figure or picture A first partial image to be included;
A text retrieval system configured to calculate a text similarity score of the first image and the second image based on the first text information and second text information associated with the second image;
An image retrieval system configured to calculate an image similarity score based on the one or more partial images of the first set and the one or more partial images of the second set associated with the second image, wherein the one or more partial images of the second set are formulas; And a second partial image corresponding to a partial region of the second image including a graph, a figure, or a picture, wherein the image similarity score is calculated based on the similarity between the first partial image and the second partial image. ; And
A search system configured to calculate a comprehensive similarity score based on the text similarity score and the image similarity score
Including,
And the search system determines that at least a portion of the first content is included in the second image when the comprehensive similarity score is equal to or greater than a first predetermined threshold.
A method performed by a processor to recognize content,
Receiving first text information associated with a first image and data for a first set of one or more partial images, the first image comprising at least a portion of first content, wherein the one or more partial images of the first set Includes a first partial image corresponding to a partial region of the first image including an equation, graph, figure or picture;
Calculating a text similarity score between the first image and the second image based on the first text information and second text information associated with the second image;
Calculating an image similarity score based on the one or more partial images of the first set and the one or more partial images of the second set associated with the second image, wherein the one or more partial images of the second set are formulas, graphs, figures Or a second partial image corresponding to a partial region of the second image including a picture, wherein the image similarity score is calculated based on the similarity between the first partial image and the second partial image;
Calculating a comprehensive similarity score based on the text similarity score and the image similarity score; And
Determining that at least a portion of the first content is included in the second image when the comprehensive similarity score is equal to or greater than a first threshold value preset;
Including,
The image similarity score includes one or more partial image similarity scores, wherein the one or more partial image similarity scores are calculated by comparing partial images including objects of the same type,
The composite similarity score is calculated by applying weights to each of the text similarity score and the one or more partial image similarity scores,
Wherein the weighting is applied differently to each of the one or more partial image similarity scores according to the type of object in the partial image.

Images