KR102142238B1 - Method of extracting text information such as abbreviation, handwriting, atypical word and sentence included in a predetermined image and automatically translating the extraction result into a predetermined language - Google Patents

Abstract

The present invention relates to a method of extracting text information such as abbreviation, handwriting, atypical words and sentences included in a predetermined image and automatically translating the extraction result into a predetermined language. Disclosed is a new method of extracting the text information included in the image using various data sets and then automatically translating the same into a specific language. The technical idea of the present invention relates to a method of extracting the text information such as abbreviations, handwriting, atypical words and sentences included in the predetermined image and then automatically translating the extraction results into a predetermined language. In the present invention, when a part in which the predetermined language is displayed is photographed through a terminal in daily life and transmitted to a server, the server automatically translates in real time into a language set by the user and provides the set language to the terminal. Therefore, if you take pictures of menus, signboards, etc. while traveling abroad using the terminal equipped with an app proposed in the present invention, there will be an advantage of translating the pictures into a native language in real time and providing the same.

Description

Method of extracting text information such as abbreviation, handwriting, atypical word and sentence included in a predetermined image and automatically translating the extraction result into a predetermined language}

The present invention relates to a method for automatically extracting text information such as abbreviations, handwriting, unstructured words, and sentences included in a predetermined image, and then automatically translating the extracted result into a predetermined language. Text information included in an image using various data sets A new method to automatically translate to a specific language after extracting

In general, in the development of existing IT technologies, data sets required for various fields were built and utilized, but there is a hassle of manually updating, and manual updates incur constant labor and update costs, as well as artificial use of data sets There is no doubt that the accuracy, precision, and delay of the time used to derive the results of the learning of intelligence must be questioned.

For example, assuming that a program for matching a puppy's name in a puppy image is developed, the existing technique stores a number of puppy-related images in a server's DB and matches the puppy image to be compared with the images (data set) to match the image similarity. The method of deriving the name of the dog with the highest dog is used.

However, the problem with this method is that when a new puppy (hybrid) comes out, the images of the new puppy should be newly stored on the server.

In addition, when the puppy image not stored in the data set matches the data set of the server, a result value is not displayed or an incorrect result value is derived.

In order to solve such a problem, a technology and a system capable of continuously updating the data set by artificializing the data set itself and a translation system utilizing the system are required.

1. Patent Application No.: 10-2013-0142744, Title of the invention: "Method for recognizing characters by adapting the captured image and information processing device for performing the method" 2. Patent application number: 10-2012-0104762, Title of invention: "Natural video multilingual character recognition and translation system and method using mobile camera"

The present invention is to propose a new method for extracting text information, which is text information, from a predetermined image transmitted to a server through a terminal, and then automatically translating the text information into a desired language.

The first embodiment of a method of automatically extracting text information such as abbreviations, handwriting, atypical words, and sentences included in a predetermined image, which is a technical idea of the present invention, and automatically translating the extracted result into a predetermined language

Transmitting an image captured by the camera of the terminal or an image stored in the terminal to the server;

Scanning the image from the server;

Converting the scanned image into an image of a predetermined resolution set by the server;

Extracting color information for each pixel constituting the image;

Collecting blob information which is one or more shape information composed of the same color;

Detecting text line information of the blob information and determining a first language (Korean, English, Chinese, etc.) corresponding to the blob information;

Normalizing the size and height of the blob information;

Extracting text information corresponding to the blob information;

Selecting a word having the highest similarity to the text information among words in a first language (Korean if it is determined to be Korean in the above) stored in a language database (eg, a second data set described in the detailed description);

Translating the selected word into a second language (eg, English) different from the first language (Korean);

And transmitting the second language to the terminal.

In the first embodiment of the present invention

Detecting text line information of the blob information and determining a first language corresponding to the blob information includes:

Acquiring contour information of each of the blob information;

Stochastically determining whether or not the contour information of a certain language is similar to each other by combining each of the contour information and each of the contour information;

And determining in the first language, which is determined to have the highest similarity according to the similarity determination.

The second embodiment of a method of automatically extracting text information such as abbreviations, handwriting, atypical words, and sentences included in a predetermined image, which is a technical idea of the present invention, and automatically translating the extracted result into a predetermined language

A first data set that manages a predetermined search term and image and periodically accesses a plurality of sites to collect and store first information matching each of the predetermined search term and image,

A second data set that periodically stores and updates the first information transmitted from the first data set,

Extracting text information such as abbreviations, handwriting, atypical words, and sentences included in a given image using a server having a third data set that extracts and manages the first text information contained in the given image transmitted through the terminal. Then, as a method of automatically translating the extraction result into a predetermined language,

Transmitting an image captured by the camera of the terminal or an image stored in the terminal to the server;

Scanning the image from the server;

Converting the scanned image into an image of a predetermined resolution set by the server;

Extracting color information for each pixel constituting the image;

Collecting blob information which is one or more shape information composed of the same color;

Detecting text line information of the blob information and determining a first language corresponding to the blob information;

Normalizing the size and height of the blob information;

Extracting text information corresponding to the blob information;

Selecting a word having the highest similarity to the text information from words of the first language stored in the second data set;

Translating the selected word into a second language different from the first language;

And transmitting the second language to the terminal.

In the second embodiment of the present invention,

Detecting text line information of the blob information and determining a first language corresponding to the blob information includes:

Acquiring contour information of each of the blob information;

Stochastically determining whether or not the contour information of a certain language is similar to each other by combining each of the contour information and each of the contour information;

And determining in the first language, which is determined to have the highest similarity according to the similarity determination.

In the second embodiment of the present invention

When the text information is an abbreviation or a new word that does not exist in the second data set, the first data set retrieves information about the abbreviation or the new word from the plurality of sites and stores the information in the second data set. Characterized by learning.

When extracting text information such as abbreviations, handwriting, unstructured words, and sentences included in a predetermined image proposed by the present invention and then performing an automatic translation of the extracted results into a predetermined language, the following effects can be expected.

In the present invention, when a portion in which a predetermined language is displayed through a terminal in everyday life is photographed and transmitted to a server, the server automatically translates the language in a language set by the user and provides it to the terminal.

Therefore, when taking a picture of a menu, signboard, etc. while traveling abroad using a terminal equipped with the app proposed in the present invention, there will be an advantage that it can be provided in translation in real time in the native language.

In addition, when the TTS function is provided, any foreign language may be translated into a native language desired by the user and then simply listened through the speaker of the user terminal.

1 is a diagram conceptually showing a functional block diagram of a server for implementing the method proposed in the present invention.
2 is a view for explaining an example of a method for obtaining predetermined data information in the image.
3 and 4 are diagrams illustrating a process of determining what kind of language the blob information is by acquiring a data line from blob information of a predetermined image.

Hereinafter, a method of extracting text information such as abbreviations, handwriting, unstructured words, and sentences included in a predetermined image proposed by the present invention with reference to the drawings and then automatically translating the extracted result into a predetermined language will be described.

1 is a diagram conceptually showing a functional block diagram of a server for implementing the method proposed in the present invention.

1, the server of the present invention includes a first data set, a second data set, and a third data set. For reference, the first to third data sets in the present invention are for functional description, and when performing the functions of the present invention, they belong to the protection scope of the present invention regardless of the name or the number of data sets. Should be interpreted. For reference, the data set of the present invention collectively refers to a configuration in which data used for learning AI is collected and a predetermined algorithm for managing and operating the data is installed.

In FIG. 1, the first data set manages and stores predetermined search terms and images, and periodically accesses a plurality of sites (various portals such as Daum, Naver, and Google, various SNS such as blogs, Twitter, and Facebook). It performs a function of collecting and storing information (hereinafter, first information) that matches each of the predetermined search terms and images.

For example, a predetermined search term stored in the first data set includes languages for each country (English, Chinese, Japanese, Korean, etc.; country-specific languages include new words, terms, etc. of the country. , Korean abbreviations, and comprehensive words including abbreviated words for a specific age group), and certain images include all kinds of images (animals, plants, mountains, sea, fruits, buildings, etc.) . For reference, the predetermined search word and image stored and managed in the first data set may be newly updated at any time by an administrator.

The server of the present invention automatically accesses various sites such as Naver, Daum, Google, Yahoo, and various social media sites such as Twitter, Facebook, and various cafes by utilizing predetermined search terms and images stored in the first data set. Alternatively, a function for requesting a search operation for information matching an image is performed periodically or by a manager's setting. Information collected by such a search operation is referred to as first information in the present invention.

Next, the server of the present invention updates the second data set while automatically storing the first information collected by the search operation in the second data set.

Accordingly, it can be seen that various search results can be automatically stored in the second data set over time without administrator intervention. That is, according to the implementation of the present invention, the second data set may be continuously and automatically updated by an artificial intelligence method such as deep learning, and improves accuracy and precision of text and images collected and stored through continuous updating. I can expect.

The third data set of the present invention is linked to a terminal (a terminal that can be connected to the server of the present invention through an Internet network, such as a computer, a smartphone, a tablet, etc.) on which a predetermined app is installed. In particular, the third data set of the present invention includes at least one or more text information (hereinafter referred to as convenience of description) included in an image transmitted from the terminal (including an image captured in real time by the camera of the terminal or an image previously stored in the terminal). For the first text information).

For example, the third data set extracts and collects text information from the image transmitted through the terminal, and the server of the present invention performs a function of selecting the word with the highest similarity by matching these text information with the second data set. do.

In the present invention, a method of extracting text from a predetermined image transmitted from a terminal is processed as follows.

1. The user transmits the image captured by the terminal's camera or the image stored in the terminal to the server. The server then scans the image.

The scanned image is converted into an image of a predetermined resolution set by the server in the server.

This is to standardize the image transmitted from each terminal.

2. The server of the present invention extracts color information for each of all pixels (all pixels) constituting a standardized image (hereinafter referred to as an image).

3. Next, blob information that is shape information composed of the same color is collected. Here, the blob information will be at least one.

Here, the shape information, that is, the blob information means the outline information of pixels composed of the same color. The contour information includes information about a predetermined contour line

For example, in the third data set of the present invention, at least one predetermined shape line is extracted by interconnecting pixels having the same color.

For example, referring to FIGS. 3 and 4, if the image transmitted to the server is “no”, the outline information of the blob information having the same color is, for example, “o”, “,”, “b”, “ah”, Can include "no"

4. Next, the server (or the third data set) of the present invention uses a text line detector to detect whether the blob information corresponds to a type of text line.

Here, the text line detector includes information on the line information (hereinafter, text line) of alphabets, vowels, consonants, and words of various languages as well as symbols and numbers. Therefore, in the server of the present invention, it is detected whether the outline information of each blob information and the combination of each blob information collected in the same color corresponds to a text line in a language of the text line detector. Through this, it is determined which language vowel, consonant, alphabet, or word is the blob information.

For example, with respect to "o", "발명", "b", "ah", "an", etc., the text line detector of the server of the present invention can be determined to be most similar to Hangul among languages stored in the third data set.

By determining the similarity as 70% in Korean, 1% in Chinese, and 29% in English, for example, it is possible to set a priority for which country the language is.

For reference, even when it is written atypically, such as handwriting or italics, a text line derived by mutual combination of each blob information and adjacent blob information is analyzed to determine the language having the highest similarity (for example, for the word goodbye) ㅇ, ㅏ, ㄴ, ㄴ, ㅕ, ㅇ, ah, an, nyeo, nyeo, ning, etc. to derive various combinations)

Summarizing this step by step, detecting the text line information of the blob information and determining a language corresponding to the blob information includes:

Acquiring contour information of each of the blob information;

Stochastically determining whether or not the contour information of a certain language is similar to each other by combining each of the contour information and each of the contour information;

It is composed of the step of determining in the language determined to have the highest similarity according to the similarity determination.

5. Next, the server of the present invention standardizes the size and height of the text line using a pitch detector.

This is to uniformize the standard size set by the server of the present invention because the size of the text line is very different.

6. Next, text information corresponding to the blob information is extracted.

That is, text information corresponding to a specific language corresponding to blob information is derived through a text line detector and a pitch detector.

7. Next, the server of the present invention determines whether the text information thus derived is similar to a word stored in a language database, for example, a second data set.

8. When the word with the highest similarity (for example, COFFEE) is selected, the server of the present invention is set in the user terminal, translates into a language (for example, Korean), and transmits it to the user terminal.

Therefore, the user can check the text information translated in his/her native language through the terminal in real time by simply photographing a signboard, menu board, book, etc. containing a foreign language.

Hereinafter, a method of extracting text information from a specific image will be described in more detail.

As illustrated in FIG. 2, it is determined whether the language of a country is similar by analyzing a feature of a line of text extracted from an image in which “Hello Handwriting” is written.

That is, blob information (shape information) composed of the same color is derived for all pixel regions of the image of FIG. 2.

After extracting the text line from the contour information of the blob information derived in this way, the text information is normalized using a pitch detector, and then the word with the highest similarity compared to various language data stored in the language database (for example, the second data set). To extract.

For example, when it is determined that the text line detected in the third data set is “hello,” the similarity is determined as 70% of Korean, 1% of Chinese, and 29% of English, thereby setting the priority for the language of the country. Can.

Next, when the probability corresponding to Korean is the highest in the priority, the priority of judgment is compared to Korean stored in the second data set in which the newly updated information is accumulated and stored, for example, 90% of well-being, 7% of anxiety, and darkness You can make the same judgment as 3%.

In this case, in the present invention, the text extracted from the image is judged as "hi", which is the word with the highest probability.

When the method according to the present invention is used, there is an advantage in that it is possible to grasp the language type and meaning of text written in handwriting, handwriting, and bad writing, which are difficult to recognize with conventional techniques.

That is, as described above, the present invention derives a text line from contour information of blob information composed of the same color in an image, and after first determining which language the text line corresponds to, the corresponding language stored in the language database Since it goes through the process of determining which words are similar to, it has the advantage of being able to determine which language corresponds to not only the normalized text but also the unstructured text information (eg, handwriting, design writing, etc.).

Next, the present invention provides a translation module that automatically translates the final selected text into another foreign language.

Therefore, when using the method provided by the present invention, text written in a specific image may be easily translated into other foreign languages, and such technology can be easily used by anyone using a terminal in which an app that implements the present invention is installed. .

That is, when the terminal user installs the main function app provided by the present invention and sets his/her native language, the terminal user, for example, takes a predetermined image containing a foreign language and transmits it to the server of the present invention. In the above, a foreign language text included in a predetermined image may be translated and provided in a native language desired by the user.

Therefore, when installing the app to which the method proposed in the present invention is applied to a terminal, for example, there is an advantage in that the inconvenience of travel caused by not knowing a foreign language when traveling abroad can be solved.

On the other hand, if the translation result provided by the present invention is strange, the terminal user can select the corresponding part and transmit it to the server of the present invention, and in this case, the server of the present invention can newly translate the relevant part and transmit it to the terminal.

The reason for this phenomenon is that in the case of an image composed of handwriting, an error may occur in the determination of the language or word corresponding to the handwriting.

In this case, in the present invention, in performing the above-described process, when selecting a type of language and selecting a word, the next probability may be retranslated into the selected language and word to provide the terminal.

On the other hand, when a new abbreviation (shortened) or a new word is detected in an image in the fast information age, there may be a case where the abbreviation or new word is not stored in the second data set.

In this case, when an image in which these abbreviations or new words are described is input to the server, accurate translation results may not be provided in performing the translation method according to the present invention.

In order to solve this problem, the server of the present invention may instruct a new or abbreviated word that is not stored in the second data set to find and update the meaning of the new or abbreviated word through the first data set. The data set can be upgraded to the second data set by automatically accessing a portal or various online sites to acquire the meaning of the new or abbreviated word.

That is, in the case of the present invention, if the text information derived from the image is an abbreviated word or a new word that does not exist in the second data set, the first data set sends information about the abbreviated word or the new word to the plurality of sites. Search and store it in the second data set for training.

Therefore, according to the present invention, even if a newly created new or abbreviated word is detected, there is an advantage that a proper translation result can be provided because learning about the abbreviated or new word is performed in a short time.

The present invention described so far is a method for automatically extracting text information such as abbreviations, handwriting, unstructured words, and sentences included in a predetermined image and automatically translating the extracted results into a predetermined language differs from existing technologies in the following points. have.

1. Recognizing the subject with the camera is the same as Naver's Pagogo or Samsung's Bixby, but due to the construction of an artificial intelligence data set and system, there is no need to upgrade a separate data set, thereby significantly reducing manpower and maintenance costs.

2. Unlike the service of the above companies, the user wants after properly recognizing various texts that are difficult to recognize with existing technologies, such as hand-written text, handwriting, and shedding, by using the data set and artificial intelligence module, not the DB holding method. The result can be translated into a foreign language.

3. By using the CPU and GPU at the same time in the parallel module, the AI module can outperform other services under the assumption that it uses the same hardware resources than the services provided by the above companies, and can have a superior performance and processing speed.

4. If the user is not satisfied with the translation result after checking the translation result, the re-translation function continuously provides the translation result with a similarity and priority from the first, and the last translation when the user no longer uses the re-translation function. The accuracy of the translation can be improved over time because the results are added with similarity and priority.

Claims (5)

Transmitting an image captured by the camera of the terminal or an image stored in the terminal to the server;
Scanning the image from the server;
Converting the scanned image into an image of a predetermined resolution set by the server;
Extracting color information for each pixel constituting the image;
Collecting blob information which is one or more shape information composed of the same color;
Detecting text line information of the blob information and determining a first language corresponding to the blob information ;
Normalizing the size and height of the blob information;
Extracting text information corresponding to the blob information;
Selecting a word having the highest similarity to the text information from words stored in the language database;
Translating the selected word into a second language different from the first language;
And transmitting the second language to the terminal.
Detecting text line information of the blob information and determining a first language corresponding to the blob information includes:
Acquiring contour information of each of the blob information;
Probabilistically determining whether or not the contour information of a certain language is similar by combining each of the contour information and each of the contour information;
And determining the first language determined to have the highest similarity according to the similarity determination , extracting text information such as abbreviations, handwriting, atypical words, and sentences included in a predetermined image and extracting the extracted text information. How to automatically translate results into a given language delete A first data set that manages a predetermined search term and image, periodically accesses a plurality of sites, and collects and stores first information matching each of the predetermined search term and image,
A second data set that periodically stores and updates the first information transmitted from the first data set,
Extracting text information such as abbreviations, handwriting, atypical words, and sentences included in a given image using a server having a third data set that extracts and manages the first text information contained in the given image transmitted through the terminal. Then, as a method of automatically translating the extraction result into a predetermined language,
Transmitting an image captured by the camera of the terminal or an image stored in the terminal to the server;
Scanning the image from the server;
Converting the scanned image into an image of a predetermined resolution set by the server;
Extracting color information for each pixel constituting the image;
Collecting blob information which is one or more shape information composed of the same color;
Detecting text line information of the blob information and determining a first language corresponding to the blob information;
Normalizing the size and height of the blob information;
Extracting text information corresponding to the blob information;
Selecting a word having the highest similarity to the text information from words of the first language stored in the second data set;
Translating the selected word into a second language different from the first language;
And transmitting the second language to the terminal.
Detecting text line information of the blob information and determining a first language corresponding to the blob information includes:
Acquiring contour information of each of the blob information;
Stochastically determining whether or not the contour information of a certain language is similar to each other by combining each of the contour information and each of the contour information;
And extracting text information such as abbreviations, handwriting, atypical words, and sentences included in a predetermined image, and determining the first language, which is determined to have the highest similarity according to the similarity determination. How to automatically translate results into a given language delete The method of claim 3
When the text information is an abbreviation or a new word that does not exist in the second data set, the first data set retrieves information about the abbreviation or the new word from the plurality of sites and stores the information in the second data set. A method of extracting text information, such as abbreviations, handwriting, unstructured words, and sentences, included in a predetermined image, characterized by learning, and automatically translating the extracted results into a predetermined language

Images