KR101377601B1 - System and method for providing recognition and translation of multiple language in natural scene image using mobile camera - Google Patents

Abstract

The present invention relates to a system and a method for recognizing and translating multiple language text from natural images using a mobile camera. The system for recognizing and translating multiple language text from natural images includes: a common text recognition/translation engine which extracts, recognizes, and translates text from an image acquired by a mobile camera; and a management unit which manages a recognition/translation database used for recognizing and translating the extracted text, builds a database specialized for each application where text recognition and translation is performed including data about combinations among alphabets and words, among alphabets, and among words and data about wrong-recognition patterns, and provides the database to the common text recognition/translation engine. The common text recognition/translation engine provides a framework applicable commonly to applications where text recognition and translation is performed and recognizes and translates text using a database specialized for each application which is provided by the management unit. According to the present invention, by providing the common engine framework and the management unit, even if recognition targets vary with applications, it is possible to recognize various kinds of text such as natural images without separately developing an engine. Also, text from a natural image can be successfully extracted even if being damaged due to various factors, and the recognition rate is not deteriorated even if the fonts of text are diverse. [Reference numerals] (100) Mail send system; (110) Send list generation unit; (112) Natural image pre-processing unit; (114) Text extraction unit; (116) Text verification unit; (120) Multiple language text recognition unit; (122) Character string recognition unit; (124) Image recognition unit; (130) Multiple language translation unit; (150) Management tool; (155) Database management tool by language; (160) Text recognition performance tuning tool; (165) Text recognition learning tool; (170) Test bed by language; (175) Distribution management tool; (180) Commercial software; (185) Software for a smartphone; (190) Software for a terminal; (195) Package software; (197) Software for a server service

Description

System and method for providing recognition and translation of multiple language in natural scene image using mobile camera

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technology for recognizing text information present in an image, and in particular, a mobile camera that extracts, recognizes, and translates specific text information present in a natural scene image acquired using a camera. The present invention relates to a multilingual character recognition and translation method and system using natural image.

Conventional text recognition technology has been concentrated on image generation and document processing through a scanner. However, in recent years, in which the mobility of the user has been increased due to the development of the mobile terminal environment, there is an urgent need for combining information acquisition and service through image acquisition through a mobile camera and character recognition of a natural scene image.

In addition, the conventional character recognition engine has some degree of character recognition performance in a specific application, but when the recognition target is different, the recognition performance is inferior, and there is inconvenience and difficulty in developing a separate engine suitable for the recognition target.

The problem to be solved by the present invention is that the character recognition performance in a specific application in the conventional character recognition engine, but the recognition performance is reduced when the recognition target is different, but it was necessary to develop a separate engine suitable for the recognition target, but it recognizes like natural video Even if the targets are diverse and the targets of recognition are different according to the application, it is possible to recognize various characters such as natural images without developing the engine separately, and automatically extracts and recognizes the characters included in the natural images acquired from the camera. The present invention provides a multilingual character recognition and translation system for natural video using a mobile camera.

Another problem to be solved by the present invention is that even if the recognition target is different, such as a natural image, and the recognition target is different depending on the application, it is possible to recognize a variety of characters, such as a natural image without developing an engine, and acquired from the camera The present invention provides a method for recognizing and translating natural image multilingual characters using a mobile camera, which automatically extracts and recognizes characters included in a natural image and translates them into other languages.

According to an aspect of the present invention, there is provided a natural image multilingual character recognition and translation system using a mobile camera, including: a common character recognition / translation engine for extracting, recognizing, and translating characters existing in an image acquired by a mobile camera; And manages a recognition / translation DB used for recognition and translation of the extracted characters, and includes information on the letter and word, letter combination, word combination information specialized according to an application in which the character recognition and translation is performed. A management tool for constructing a database of misrecognition pattern information as a database and providing the common character recognition / translation engine, wherein the common character recognition / translation engine is commonly used for an application in which the character recognition and translation is performed. It provides a possible framework, and recognizes and translates characters using a database specialized for each application provided by the management tool.

The common character recognition / translation engine may include: a natural image character region extraction unit for extracting a character by searching for an area in which a character string exists in a natural image photographed by a mobile camera; A multilingual character recognition unit for recognizing a character string of the extracted character area; And a multilingual translation unit for translating the recognized character strings into various languages by searching a multilingual database.

The natural image character region extraction unit may include a natural image preprocessing unit converting an input image file into a form capable of image processing and correcting distortion included in the image; A character extracting unit for selecting a character region candidate group in the input image, separating a background image and a character string of the character region among the character region candidates, and separating the extracted character string into respective characters; And a character verification unit for verifying characters by extracting characters for each language by applying a language model including at least word combination information, word combination information, and misrecognition pattern information.

The multi-language character recognition unit includes a string recognition unit for recognizing character strings in the form of printed characters, and the character string recognition unit divides and divides a split point into predicted character division points in a string image, and then combines and recognizes each segment to obtain an optimal combination. It is characterized by recognizing a string by searching. The multilingual text recognition unit may further include an image recognition unit recognizing a character or a logo including a graphic element, and the image recognition unit extracts a corner point and a junction point representing characteristics of an image and then surrounds the surrounding information. Feature extractor to vectorize using; And a feature matching unit that matches the vectorized image feature information with information of an image matching DB to find the most similar image information.

The multilingual translation unit may include a general term translation database in which a general word or a combination of words is matched with a plurality of words or combinations of words; And a specialized term translation database in which a specialized word or word combination is matched one-to-one, and when the recognized character string is the general word or combination of words, a plurality of words by referring to the general term translation data database. Or a combination of words, and when the recognized character string is the specialized word or a combination of words, the character string is translated into a single word or a combination of words by referring to the specialized terminology translation data database.

The management tool additionally provides a performance tuning function and a learning function for generating a character recognition model DB, and manages a recognition / translation DB used to recognize and translate the extracted character, and registers, changes, A language-specific DB management tool that provides a deletion and inquiry function; Character recognition performance tuning tool providing performance tuning; And a character recognition learning tool that provides a learning function for generating a character recognition model DB.

The recognition / translation DB may include: a font DB configured to classify and store piece information used for recognizer learning by font type; An image matching DB consisting of original image information and feature extraction information; Language consisting of word information consisting of words, frequency of occurrence of words and combination information between words, word information consisting of words, word occurrence frequency and combination information between words, and misrecognition pattern information consisting of words that are easy to be mistaken and patterns of word misrecognition Model DB; A character recognition model DB for managing a list of character recognition models generated by the character recognition engine learning tool; And a translation DB having a form of a word of the reference language, a foreign language word matching the reference language, and a foreign language word table matching the reference language table.

The character recognition learning tool includes: a feature extraction unit for each class that generates a learned character recognition model by receiving the character information of the font DB, extracts the image characteristic using the boundary information of each character, and expresses it as a feature vector; A dimension reduction unit for each class for reconstructing the feature vector by extracting only an important element from the dimension of the feature vector in order to reduce the memory and the amount of computation used for learning and recognition; And a learning unit generating a reference vector representing a class by using a vector of classes.

The character recognition performance tuning tool selects DB and engine tuning information necessary for a specialized field by using the DB information received from the DB management tool for each language and a test sample for a specialized field, and selects the selected DB and engine tuning information. It characterized in that the tuning of the engine using.

Natural image multi-language character recognition and translation system using a mobile camera according to the present invention includes a test bed for each language for receiving the optimal tuning DB information and the optimal tuning engine information from the character recognition performance tuning tool to reconfigure and test the engine; And a distribution management tool that receives the optimal tuning DB information and the optimal tuning engine information from the character recognition performance tuning tool, generates a library suitable for the service environment by constructing an engine and a DB in a usable form, and outputs a module for distribution. It further comprises.

Natural image multi-language character recognition and translation method using a mobile camera according to the present invention for achieving the above technical problem is a combination of words and words, combinations between words, words and combinations that are specialized according to the application (s) that character recognition and translation is performed Constructing information and misrecognition pattern information into a database; Extracting a character string image existing in an image acquired by a mobile camera by a character recognition / translation engine providing a framework commonly applied to the application field; Analyzing, recognizing, and outputting the extracted character string image in text form by using the database by the character recognition / translation engine; And translating, by the character recognition / translation engine, the text-type characters into another language using a predetermined database.

The extracting of the string image may include converting an input image file into a form capable of image processing and correcting distortion included in the image; Selecting a character region candidate group in the input image, separating a background image and a character string of the character region among the character region candidates, and separating the extracted character string into respective characters; And verifying a character by extracting characters for each language by applying a language model including at least word combination information, word combination information, and misrecognition pattern information.

The character string analysis and recognition may be performed by subdividing and dividing a split point into character segment prediction points in a character string image, and then searching for an optimal combination by combining and recognizing each segment.

The character string image recognition and the recognized character string translation may include a font DB for classifying font information used for learning a recognizer according to font type; An image matching DB consisting of original image information and feature extraction information; Language consisting of word information consisting of words, frequency of occurrence of words and combination information between words, word information consisting of words, word occurrence frequency and combination information between words, and misrecognition pattern information consisting of words that are easy to be mistaken and patterns of word misrecognition Model DB; A character recognition model DB for managing a list of character recognition models generated by the character recognition engine learning tool; And a translation DB having a form of a word of the reference language, a foreign language word matching the reference language, and a foreign language word table matching the reference language table.

The present invention provides a recording medium readable by a processor that records a program executed by the processor.

In the conventional character recognition engine, the character recognition performance is shown to some extent in a specific application, but when the recognition target is different, the recognition performance is deteriorated and a separate engine suitable for the recognition target has to be developed, but the natural image multilingual character recognition using the mobile camera according to the present invention And a translation system and method, by providing an engine common framework and a management tool including at least learning and performance tuning functions, the recognition targets are varied, such as natural video, and even if the recognition targets vary depending on the application. By providing management tools suitable for recognition objects to the common framework of the engine, it is possible to recognize various characters such as natural video without developing the engine separately.

And according to the natural image multilingual character recognition and translation system and method using a mobile camera according to the present invention, it is possible to extract, recognize, and translate characters existing in the natural image acquired by the camera. Character recognition of the natural image is often difficult to extract and recognize the character because the image is often damaged by various factors, according to the present invention can extract the character of the natural image damaged by the various factors normally and the font of the character The variety does not reduce the recognition rate.

1 is a block diagram showing an embodiment of a natural image multilingual character recognition system using a mobile camera according to the present invention.
2 shows a configuration of an image recognition unit.
3 shows the configuration of the recognition / translation DB.
Figure 4 shows the configuration of the character recognition learning tool.
Figure 5 shows a block diagram of a more specific embodiment of the configuration of the present invention.
6 is a block diagram showing the configuration of the engine common framework.
7 is a block diagram showing the configuration of the natural video character extraction engine.
8 illustrates examples of problems occurring in natural video when camera-based video is taken.
9 is a diagram illustrating an algorithm for removing reflection effects.
10 illustrates an example in which image merging is required.
11 illustrates a matching method of a panoramic image.
12 illustrates an image in which image merging is completed.
FIG. 13 illustrates a boundary detection method in a natural image with shadow.
14 illustrates a process of extracting a character candidate region applying MRF.
15 shows an example of perspective effect detection.
16 illustrates an example in which character merging determination is difficult.
17 shows an example of character merging after application of a language model.
18 is a block diagram illustrating a string recognizer configuration.
19 shows an example of a hyperdivision recognition process.
20 illustrates an example of a collection Korean font.
21 shows an example of a collection target latin font.
22 shows an example of post-processing of a recognition result of a corrupted image.
23 shows the configuration of image matching.
24 shows an example of feature point extraction in image matching.
25 shows an example of image matching.
26 illustrates candidate selection through clustering.
FIG. 27 shows an example of the configuration of a unidirectional 1: N translation table as an example of a Korean and English general terminology translation DB, and an example of a bidirectional 1: 1 translation table as an example of a Korean and Latin specialized term translation DB.
Fig. 28 shows the construction of the management tool.
29 shows a configuration diagram of the DB for each language.
30 shows an example of font information and font DB management.
Fig. 31 shows a font DB construction method.
32 shows a flowchart of a language model DB construction.
33 shows an image matching DB construction flowchart.
34 shows a translation DB management configuration diagram.
35 is a block diagram of a character recognition engine learning tool.
36 exemplarily illustrates a feature extraction process.
37 shows a flowchart of the character recognition performance tuning tool.
38 shows the configuration of a test bed for each language.
39 shows a configuration diagram of a distribution management tool.
40 shows a commercial software configuration.
41 is a flowchart illustrating an embodiment of a natural image multilingual character recognition and translation method using a mobile camera according to the present invention.
FIG. 42 illustrates the string image extraction process of FIG. 41 in more detail.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The embodiments described in the present specification and the configurations shown in the drawings are merely preferred embodiments of the present invention and are not intended to represent all of the technical ideas of the present invention and therefore various equivalents And variations are possible.

First, terms used in the specification and drawings of the present invention will be defined. 'Natural video' is a video taken by the camera of general texts on various objects such as signs, signs, and atypical documents.The characters included in the video taken by the camera are free of font, size, and arrangement, and the sunlight Alternatively, a lot of distortion may be generated by the surrounding environment such as lighting.

The present invention provides an engine for automatically extracting and recognizing characters included in a natural image obtained from a camera, for example, Korean and Latin languages, a language model and software management tool for optimizing the engine performance, and a service-specific term translation database. DB).

1 is a block diagram showing an embodiment of a natural image multilingual character recognition system using a mobile camera according to the present invention, the natural image multilingual character recognition system using the mobile camera is a common character recognition / translation engine 100 and It comprises a management tool 150. In addition, a commercial software module 180 including a custom module and an app may be added based on the common character recognition / translation engine 100.

The common character recognition / translation engine 100 extracts, recognizes, and translates the characters existing in the image acquired by the mobile camera, and extracts the natural image character region extractor 110, the multilingual character recognition unit 120, and the multilingual translation unit 130. )

The management tool 150 manages a recognition / translation DB used for recognition and translation of the extracted characters, and combines letters, words, and letters that are specialized according to an application in which the character recognition and translation is performed. And constructing a database of word combination information and misrecognition pattern information into a database and providing the common character recognition / translation engine. At this time, the common character recognition / translation engine 100 provides a framework that can be commonly used for an application in which the character recognition and translation is performed, and is provided from the management tool 150. Characters are recognized and translated using a database specialized for each application.

The natural image character region extraction unit 110 searches for a region in which a character string exists in the natural image photographed by the mobile camera, and extracts the character. The natural image preprocessing unit 112, the character extraction unit 114, and the character verification unit ( 116).

The natural image preprocessing unit 112 converts the input image file into a form capable of image processing and corrects distortion included in the image.

The character extracting unit 114 selects a character region candidate group from the input image, separates the background image and the character string of the character region among the character region candidates, and separates the extracted character string into respective characters.

The character verification unit 116 verifies a character by extracting characters for each language by applying a language model including at least word combination information, word combination information, and misrecognition pattern information.

The multilingual character recognition unit 120 recognizes the extracted character string of the text area, and includes a string recognition unit 122 and an image recognition unit 124.

The character string recognition unit 122 recognizes a character string in the form of a printed character. More specifically, the character string is divided into subdivided points into character predicted points in a character string image, and then each segment is combined and recognized to search for an optimal combination. Recognize.

The image recognizing unit 124 recognizes a letter or logo including graphic elements, and includes a feature extracting unit 200 and a feature matching unit 250 as shown in FIG. 2.

The feature extractor 200 extracts a corner point and a junction point representing the characteristics of the image and then vectorizes the image using surrounding information. The feature matching unit 250 searches for the most similar image information by matching the vectorized image feature information and the information of the image matching DB.

The multilingual translation unit 130 translates the recognized character strings into various languages, and a general term translation database in which a general word or a combination of words is arranged as a plurality of words or a combination of words and a specialized word or a combination of words is 1 A specialized term translation database that is matched one-to-one, and if the recognized character string is the general word or a combination of words, the term is translated into a plurality of words or a combination of words with reference to the general term translation data database. When the recognized character string is the specialized word or combination of words, the recognized character string is translated into one word or combination of words by referring to the specialized terminology translation data database.

The management tool 150 manages the recognition / translation DB used for recognition and translation of the extracted characters, and additionally provides a performance tuning function and a learning function for generating a character recognition model DB. The management tool 155, the character recognition performance tuning tool 160, and the character recognition learning tool 165 may be made to further include a language-specific test bed 170 and a distribution management tool 175.

The language-specific DB management tool 155 manages the recognition / translation DB used to recognize and translate the extracted characters, and provides registration, change, deletion and inquiry functions necessary for the management. The character recognition performance tuning tool 160 tunes the character recognition performance, and more specifically, the DB required for the specialized field using the DB information received from the language-specific DB management tool 155 and the test sample for the specialized field. Engine tuning information is selected, and the engine is tuned using the selected DB and engine tuning information.

Character recognition learning tool 165 provides a learning function for generating a character recognition model DB.

3 illustrates the configuration of the recognition / translation DB 300, the font DB 310, the image matching DB 320, the language model DB 330, the character recognition model DB 340, and the translation DB 350. It is preferable to have a.

The font DB 310 stores the piece information used for learning the recognizer by font type. The image matching DB 320 consists of original image information and feature extraction information. The language model DB 330 includes word information consisting of words, frequency of occurrence of words and combination information between words, word information consisting of words, frequency of occurrence of words and combinations between words, and misrecognition patterns of words and words that are easy to be mistaken. Contains misrecognition pattern information made. The character recognition model DB 340 manages a list of the character recognition models generated by the character recognition engine learning tool.

The translation DB 350 has a form of a word of a reference language, a foreign language word matching with the reference language, and a foreign language word table matching with the reference language table.

Character recognition learning tool 165 receives the character information of the font DB to generate a learned character recognition model, as shown in Figure 4 feature extraction unit 400, dimensional reduction unit 420 for each class and learning It is preferred to have a portion 440.

The feature extractor 400 for each class extracts an image characteristic by using boundary information of each piece and expresses the feature as a feature vector. The class reduction unit 420 for each class reconstructs the feature vector by extracting only an important element in the dimension of the feature vector in order to reduce the memory and the amount of computation used for learning and recognition. The learner 440 generates a reference vector representing a class by using a vector of classes.

The language-specific test bed 170 receives the optimal tuning DB information and the optimal tuning engine information from the character recognition performance tuning tool and reconfigures and tests the engine.

The distribution management tool 175 receives the optimal tuning DB information and the optimal tuning engine information from the character recognition performance tuning tool 160 and generates and distributes a library suitable for an environment to be configured and serviced by using an engine and a DB in a usable form. Output the module.

Hereinafter, more specific embodiments of the above-described configuration of the present invention will be described.

5 is a block diagram showing the configuration of a more specific embodiment of the above-described configuration of the present invention, which includes an engine common framework 500 and a management tool 550, and is commercially available based on the engine common framework 500. Software 580 is additionally provided.

The natural video multilingual character recognition and translation method and system according to the present invention construct an engine common framework in order to construct a multilingual language recognizer, develop a part that is commonly processed in all languages, and distinguish a part specialized in each language. to provide.

6 is a block diagram illustrating a configuration of the engine common framework 500. The engine common framework 500 corresponds to the common character recognition / translation engine 100 of FIG. 1, the natural image character extraction engine 600 corresponding to the natural image character region extraction unit 110 of FIG. 1, The multilingual character recognition engine 640 corresponding to the multilingual character recognition unit 120 of FIG. 1, and the multilingual translation engine 680 corresponding to the multilingual translation unit 130 of FIG. 1.

The natural image character extraction engine 600 searches for an area in which a character string exists in the natural image acquired by the user.

The multilingual recognition engine 640 is a module that recognizes the searched string region and improves the recognition success rate through the steps.

The multilingual translation engine 680 is a module for translating recognized texts into various languages. The multilingual translation engine 680 is a general term translation database in which a general word or a combination of words is arranged in a plurality of words or combinations of words and a specialized word or word. A specialized term translation database having a combination of one-to-one matching, and when the recognized character string is the general word or a combination of words, the common term translation data database is translated into a plurality of words or a combination of words with reference to the general term translation data database. When the recognized character string is the specialized word or combination of words, the recognized character string is translated into a single word or combination of words with reference to the specialized term translation data database.

First, the natural image character extraction engine 600 shown in FIG. 6 will be described. Conventional image processing engines can recognize images with a small number of functions because they have fewer conditions that can be distorted than natural images based on a scanner, but they are insufficient to process various distorted phenomena of natural images.

7 is a block diagram illustrating a configuration of the natural video character extraction engine 600. The natural image character extraction engine 600 removes the distorted effects of the image input in the preprocessing process, finds the region of the characters included in the image, and creates an optimal string image to increase the recognition rate in the recognition engine.

The natural image character extraction engine 600 includes the entire process of providing a string image to a recognition engine from an image obtained from a camera or a file, and includes a natural image preprocessing engine 700, a character extraction engine 740, and a character verification engine. 780 consists of three modules.

The natural image preprocessing engine 700 corresponds to the natural image preprocessing unit 112 of FIG. 1, converts the input image file into a form capable of image processing, and corrects various distorted effects included in the image. . In addition, the merge process is performed according to whether the input image is overlapped.

The character extraction engine 740 corresponds to the character extracting unit 114 of FIG. 1, and selects a candidate group having a high probability of being a text area in the input image, and uses the image processing information to determine the accuracy of selecting a text area candidate group. Increase Among the text area candidates, the background image and the text of the actual text area are distinguished by a probabilistic method, and the extracted text is divided into the respective letters.

If you do not know the language of the character string during the character extraction process, the character extraction engine cannot determine whether the characters extracted are divided according to the language. The character verification engine improves the accuracy of character extraction by applying language model.

A natural image preprocessing engine 700 will be described in more detail with reference to FIG. 7.

As illustrated in FIG. 7, the natural image preprocessing engine 700 includes an image codec engine 705, an image optimization engine 710, and an image merging engine 715.

Referring to the image codec engine 705, an 8M pixel camera has an image size of about 3264 x 2448 when taken at the maximum resolution. When decompressing such a video in a mobile device, a large amount of memory and a large amount of execution time are required, so the image codec engine 705 can reduce decoding time in a lightweight codec and a mobile environment to efficiently use memory space in a mobile environment. It provides a JPEG codec module that decodes a reduced image according to a codec and parameters.

Referring to the image optimization engine 710, natural image processing is difficult to accurately recognize the object by the complex distortion of the image. Problems that occur in natural video when taking a camera-based image include rotation & distortion, blurring, transparency, reflection, occlusion, and shadow, as shown in FIG. 8.

The image optimization engine 710 is an image enhancement processing engine for extracting and recognizing an optimal recognition object from a natural image, and corrects and recognizes an image having a distortion such as rotation, skew, and perspective. In addition, when the input image is made of reflectance illumination in consideration of the change in illumination, the letter region is effectively extracted by mathematically modeling and removing the illumination effect.

9 is a diagram illustrating an algorithm for removing a reflection effect. As shown in FIG. 9, a text area is detected by regenerating a color of an actual object through a preprocessing process of an image in which a lighting effect is generated due to a shadow. To this end, the image optimization engine includes an algorithm for removing blurring effect, an algorithm for removing transparency effect, an algorithm for removing occlusion effect, an algorithm for removing shadow effect, and a binarization algorithm suitable for natural image.

Referring to the image merging engine 715, the image merging engine 715 may perform image recognition by matching images photographed by dividing a severely curved image or an object into multiple images into a single image on a plane. . The image merge engine 715 includes an algorithm for detecting overlapping regions between images and a rotation estimation and correction algorithm of the images. 10 illustrates an example in which image merging is required, and FIG. 11 illustrates a matching method of a panoramic image. 12 illustrates an image in which image merging is completed.

Referring to FIG. 7, after the natural image is preprocessed by the natural image preprocessing engine 700, a character is extracted through the character extraction engine 740. The character extraction engine 740 will be described.

Referring to the character extraction engine 740, a character region is detected by searching for a region in which a character may exist in a natural image, and the image conditions are applied to a DB to apply image processing information by using the MRF (Markov). Random field (SLR) engine divides the background and text area from the natural image and extracts each character from the text area.The text area detection engine (745), MRF engine (755), and connected component (CC) extraction engine (760) ).

The most researched field in the recognition problem of natural image is the field of detection of character domain. This means that the technical level of character extraction is at an early stage and an improved character area detection engine 745 is needed.

The text area detection engine 745 generally determines the text area based on the distribution of the boundary portion of the image by using the feature that the distribution of the boundary line per unit area is high in the natural image. By using the artificial neural network based recognizer to determine whether the character region can be determined candidates of the character region. In addition, the text area detection engine 745 clusters the text area candidates determined by the boundary analysis through connection element analysis and morphological operations, and analyzes the text area by analyzing the size, position, and shape of the area in the clustered candidates. Extract. FIG. 13 illustrates a boundary detection method in a natural image with shadow.

In order to improve the detection rate of the text area, it is preferable to use an image processing information DB such as color information and shape information acquired under various conditions of the image, and have higher character area candidate detection rate in the natural image by using the registered information. Can be. In addition, by applying the image processing information (750), that is, by making the conditions of the image DB to use the character area detection, the character area detection time is reduced.

In general, images taken with a digital camera have a great variety, and thus the position, size, and color of the characters cannot be known in advance, so it is difficult to divide the characters included in the natural image from the images. The engine 755 extracts characters from the background, and replaces the problem of character segmentation in the image with a problem of label determination, and determines the label of each pixel according to the highest probability for a given image. The MRF engine 755 extracts all character candidates and then extracts portions of actual characters using relationship information between characters. Through this, it is possible to extract a character candidate with higher reliability. 14 illustrates a process of extracting a character candidate region applying MRF.

The connected component (CC) extraction engine 760 splits the characters in units of characters in the extracted string. To this end, the extracted string is labeled with the CC which is the minimum unit of the character area using CC extraction, and each CC is reconstructed into one character through a merge process. Then, the concatenated lines of the reconstructed characters are found and composed of intact strings. In order to remove the perspective effect, the baseline of the strings is searched to determine whether the perspective is applied. 15 shows an example of perspective effect detection.

Referring to FIG. 7, after preprocessing the natural image in the natural image preprocessing engine 700, the character is extracted through the character extraction engine 740, and the language model is applied through the character verification engine 780 to verify the extracted character. do. The character verification engine 780 will be described.

Because strings are merged by CC extraction, they are given only as strings and do not have basic information for recognition. If the extracted string is composed of two lines, it is difficult to distinguish whether it is a Korean initial consonant + neutral, an English two lines, or a mixed language. Therefore, a character verification engine 780 applied to a language model is required to increase the completion of character merging. FIG. 16 illustrates an example of difficult character determination, and FIG. 17 illustrates an example of character merging after application of a language model.

As shown in FIG. 17, the character verification engine recognizes the languages of the merged characters by applying a language model before composing the character strings after merging the characters and merges them with a merge method suitable for the corresponding language, or is composed of heterogeneous languages. Verify.

Meanwhile, referring to FIG. 6, when a character is extracted through the natural video character extraction engine 600, the multilingual character is recognized through the multilingual character engine 640. The multilingual character recognition engine 640 will be described.

The multilingual character recognition engine 640 analyzes and recognizes a string image that is a result of the natural image character extraction step, outputs a textual result, and includes a string recognizer 645 and an image recognizer 650. The string recognizer 645 recognizes a character string in the form of general print characters, and the image recognizer 645 recognizes a character or a logo including many graphic elements. .

The string recognizer 645 and the image recognizer 650 may be used together or only one part may be used depending on the service type. In most situations, the string recognizer 645 performs the recognition only, and the image recognizer 650 recognizes the string. Recognizes logos that are difficult to recognize.

FIG. 18 is a block diagram illustrating the configuration of the string recognizer 645, and the string recognizer 645 will be described.

The string recognizer 645 adopts a hyperdivision-based character recognition method, and the hyperdivision-based character recognition divides a split point into an expected point of character division in a string image, divides the split point, and then combines and recognizes each segment for an optimal combination. In this case, the two characters are in contact with each other to prevent the problem that normal recognition is impossible when two characters are included in one segment. The character string recognition function includes over-dividing, over-divided segment combination and recognition, post-processing using misrecognition pattern information, and post-processing using combination information between words.

19 shows an example of an overdivision recognition process, in which there is an overdivision process for horizontal writing and an overdivision process for vertical writing. The overdivided segment combinations and recognition in the horizontal division and the division division division division processing are improved using the language model.

Recognizers include Korean recognizers, Latin recognizers, Chinese recognizers, and Japanese recognizers.

Korean character string recognition engine is based on a common character recognition framework, and can be created by collecting Korean fonts, learning words, and developing and tuning a character string recognizer.

Since the recognizer is developed through learning and tuning using the collected font information, it is difficult to expect the accuracy of the recognition for the font not used for learning. Therefore, font information is the most important element in the recognizer and various font information is needed.

Korean fonts use the font information possessed by Myeongjo, Gothic, and Palace, and expand the range of fonts that can be recognized by collecting fonts with graphic elements. Since the collected font information is huge, the font DB management tool, which is a dedicated management tool, manages the collected fonts. 20 illustrates an example of a collection Korean font.

Korean word recognizer learns collected letters and recognizes Korean words, adjusts learning parameters and repeats learning and testing. The learning parameter refers to a parameter used in the learning process, that is, a parameter used for learning using feature extraction and extracted features, and a difference occurs in the performance of the recognizer according to the parameter value. Therefore, in order to build a high-performance learner, it is necessary to find suitable parameter values and repeat the training and testing. The word learning is received using font information collected from the font DB management tool and proceeds using the character recognition learning tool.

The Korean string recognizer includes a segment combination routine, a language model usage routine, and a segment combination post-processing routine suitable for Korean string recognition based on an engine common framework. Korean segment combination routine and combination post-processing routine can improve combination performance using Hangul's phonemic property information. Also, language model using routine can improve processing accuracy by using partitioned form of Hangul's phoneme unit. Using the acquired character image sample, the characteristics of Korean are analyzed and tuned in order to recognize them appropriately for Korean characters.

The Latin string recognition engine, like the Korean recognition engine, is based on the common framework of the engine and can be created through font collection, single word learning, and string recognizer development and tuning. Latin letters consist of the basic English alphabet and additional Latin letters used in each country of Europe. Additional Latin letters include routines with a diacritic in the form of dots above or below the base English alphabet and differing from other languages. 21 shows an example of a collection target latin font.

In order to recognize the Latin characters, the Latin character learning is performed by repeatedly performing the learning and the test by adjusting the learning parameters. The Latin string recognizer includes segment combination routines, language model usage routines, and segment combination post-processing routines suitable for Latin string recognition based on a common framework, and are tuned to fit Latin characters using actual character image samples.

Describe the Chinese recognizer. First collect Chinese fonts. Chinese fonts have a lot of cursive unique fonts compared to other language fonts. Therefore, the Chinese fonts are collected in the form of unusual fonts in addition to the standard fonts, and the Chinese word recognizer is developed.

Chinese and Japanese have a problem that it takes a long time to recognize due to the characteristics of recognition characters much more than Korean and Latin. Therefore, the clustering concept is introduced into the word recognizer to reduce the recognition time. Clustering is a method of forming clusters grouped into similar classes and comparing only classes in similar clusters in order to prevent comparison of all classes.

In order to generate the Chinese string recognizer, it includes the segment combination routine, language model usage routine, and segment combination post-processing routine suitable for Chinese string recognition based on the common framework. .

Explain the Japanese recognizer. First, Japanese fonts are collected and a Japanese word recognizer is generated. Like Chinese, the clustering concept is introduced to reduce the recognition time and to perform the learning and testing repeatedly to create the optimal word recognizer.

To create Japanese string recognizer, we modify segment combination routine, language model usage routine, segment combination post-processing routine suitable for Japanese string recognition based on common framework, and perform appropriate tuning for Japanese characters using actual character image samples. .

Referring to FIG. 18, the string recognizer 645 recognizes a multilingual language after the hyperdivision process, and performs a recognition post-process. Recognition post-processing is performed by performing post-processing using misrecognition pattern information, and then performing post-processing using combination information between words.

The post-processing using the misrecognition pattern information and post-processing using the combination information between words improves the recognition performance when a normal string recognition process cannot obtain a normal recognition result by inputting a corrupted string image.

Post-processing using the misrecognition pattern information corrects misrecognition of each recognized word using misrecognition pattern information and word frequency information of the language model, and additionally uses contextual information and character size. In addition, the post-processing using the word-to-word combination information corrects an incorrect word-to-word combination using the word-to-word combination information of the language model. FIG. 22 illustrates an example of post-processing of a recognition result post-processing on a corrupted image. When a corrupted character string image is input, after performing division and recognition on segment combination, postprocessing using a misrecognition pattern and then using post-word combination information. The processing indicates that the misperception is corrected. At this time, the misrecognition pattern information and words are corrected using the combination information using language model information.

Meanwhile, referring to FIG. 6, when a character is extracted through the natural image character extraction engine 600, the multilingual character is recognized by the multilingual character engine 640, and image matching is performed at this time. The image matching will be described.

FIG. 23 illustrates the configuration of image matching. Image matching is used for recognition of logos, imaged characters, and the like. The feature extraction extracts and vectorizes a main part of an image, and searches an image matching DB with vectorized feature information. Image matching is performed through feature matching to match features of an image.

The feature extraction is performed through vectorization using the surrounding information and the like for the main part representing the characteristics of the image well. Elements that represent image characteristics well include corner point and junction point, and these elements are extracted. Peripheral information of each element is vectorized using the distribution of pixel values, and the feature used for image matching should be able to handle the size change, rotation, and partial occlusion of the image. It is preferable. Typical algorithms include SIFT and SURF. The scale-invariant feature transform (SIFT) generates robust features in size, rotation transformation, Affine transformation of 3D projection, light intensity, and is an algorithm that forms the basis of camera-based object recognition and image matching technology. The Speeded Up Robust Features (SURF) is a high speed version of SIFT, which is fast and accurate, and is suitable for mounting a mobile terminal. 24 shows an example of feature point extraction in image matching.

The feature matching matches the image feature information with the information in the image matching DB and finds the most similar image information. 25 shows an example of image matching. The most similar image information is found by comparing each feature of the image matching DB registered in advance and feature information extracted from the input image. To compare the features of general character recognition, we use a lot of Discriminative Methods represented by neural networks. This methodology is only suitable for small classes. Trademark recognition cannot use this methodology because it has a large number of classes to be recognized, and using a center-based method suitable for many classes can yield high performance. In image matching, thousands of feature points are extracted and used per image. When all these dimensions are used, the computation speed is excessive and the recognition speed is greatly reduced. The performance is improved by applying techniques such as dimensionality reduction and matching candidate reduction (cluster selection through clustering), which can optimize computations while keeping the recognition rate almost the same. 26 illustrates candidate selection through clustering.

Meanwhile, referring to FIG. 6, when a character is extracted through the natural image character extraction engine 600, the multilingual character is recognized through the multilingual character engine 640, and the character recognized through the multilingual translation engine 680 is converted into another character. Translate into multiple languages. The multilingual translation engine 680 will be described.

Multilingual translations include unidirectional 1: N translation and bidirectional 1: 1 translation. Unidirectional 1: N is used when there is more than one translated meaning, and when one word is translated into several meanings like general terminology. The bidirectional 1: 1 method is used when there is only one translated meaning. In a specialized field, one word can be translated into one meaning. The multilingual translation engine uses table indexing and sorting techniques to reduce processing time. FIG. 27 shows an example of the configuration of a unidirectional 1: N translation table as an example of a Korean and English general terminology translation DB, and an example of a bidirectional 1: 1 translation table as an example of a Korean and Latin specialized term translation DB.

On the other hand, referring to Figure 5, a more specific embodiment of the configuration of the present invention, including the engine common framework 500 and the management tool 550, based on the engine common framework 500 commercial software ( 580 is additionally provided. The management tool 550 corresponds to the management tool 150 of FIG. 1, and the management tool 550 will be described.

Natural image character recognition and translation system and method according to the present invention is distributed through the engine tuning, testing and verification to perform the recognition and translation using the DB necessary for recognition and to generate the final commercial SW module.

The process from the sub-engine configuration to the commercial SW module generation needs to be carried out efficiently. Therefore, dedicated management tools are needed to effectively share relevant data at all stages and manage each data systematically.

28 is a block diagram of the management tool 550. The management tool 550 manages an image matching DB, a language model DB, a character recognition model DB, a translation DB, and a performance tuning function used in a recognition and translation process. It needs the ability to test and deploy the tuned engine and the learning function to create the character recognition model DB.

The management tool 550 supports a language-specific DB management tool (2800), character recognition engine learning tool (2820), character recognition performance tuning tool (2840), language-specific test bed (2860), deployment management tool (2880). The recognition engine requires a language-specific DB management tool (2800), character recognition engine learning tool (2820), character recognition performance tuning tool (2840).

29 is a view showing the configuration of the language-specific DB, the language-specific DB management tool manages the font DB, character recognition model DB, language model DB, image matching DB, translation DB, registration, change, deletion, inquiry required for management It consists of functions.

Referring to the font DB management, the font DB is data stored by dividing the piece information used for the recognizer learning by font type, and the piece information is composed of each piece image and a label of each image. 30 shows an example of font information and font DB management, and classifies and manages font information using language type, font name, font size, and font type. The font information is collected in two forms, real and virtual. Fig. 31 shows a method of constructing a font DB, which illustrates adding real letter samples and adding virtual word samples. Add real word sample captures real objects, splits them, labels each one, verifies and corrects them, and generates a file of information. Virtual sample add adds font type and font size suitable for specific field and executes automated program to generate single information file.

In addition, the language-specific DB management tool (2800) manages the character recognition model DB, specifically manages the list of the character recognition model generated by the character recognition engine learning tool.

The language-specific DB management tool 2800 manages a language model DB, and the language model DB is composed of word information, word information, and misrecognition pattern information. The word information is composed and managed in the form of words, word appearance frequency, combination information between words, the piece information is organized and managed in the form of single letter, single occurrence frequency, combination information between words, and misrecognition pattern information is a word that is easy to generate misrecognition For the information storage means for storing the pattern and correct the misrecognition from the pattern information in the recognition process, the misrecognition pattern information is configured and managed in the form of words, misrecognition patterns of words. 32 is a flowchart illustrating a language model DB construction, in which word information is added to construct a word information DB, word information is added to construct a word information DB, and misrecognition pattern information is added to construct a misrecognition pattern information DB. .

The language-specific DB management tool 2800 also manages an image matching DB, and the image matching DB is composed of original image information (image buffer, meaning, category) and feature extraction information (image feature information and image meaning (text)). do. 33 is a flowchart illustrating an image matching DB construction. The original image DB is added by adding original image information, and the image feature DB is constructed by adding feature extraction information.

Language-specific DB management tool (2800) also manages the translation DB, the translation information is defined in the form of words of the reference language, foreign language matching the reference language, translation DB is a foreign language word table matching the reference language table Manage configuration. The translation DB is composed of general term translation DB using unidirectional 1: N method and specialized term translation DB using bidirectional 1: 1 method according to translation processing method. 34 is a diagram illustrating a translation DB management configuration, and includes a general term translation DB and a specialized term translation DB.

Management tool 550 according to the present invention is a language-specific DB management tool (2800), character recognition engine learning tool (2820), character recognition performance tuning tool (2840), language-specific test bed (2860), distribution management tool (2880) ), The character recognition engine learning tool 2820 will be described.

The character recognition engine learning tool 2820 generates a learned character recognition model by receiving the character information of the font DB. 35 shows a configuration diagram of the character recognition engine learning tool 2820. .

Character recognition engine learning consists of feature extraction by class, feature dimension reduction by class, and learning phase. The feature extraction for each class is a process of extracting image properties for each character and quantifying the extracted information quantitatively. The feature uses boundary information of a character that is commonly used, and the quantized result is expressed in a vector form. 36 exemplarily illustrates a feature extraction process, inputting a single image, extracting feature information, and vectorizing a feature.

Dimension reduction by class is a process that reconstructs the feature vector by extracting only the important elements from the dimension of the feature vector extracted in the previous step in order to reduce the memory and the amount of computation used for learning and recognition. The commonly used Fisher's Discriminant Analysis (FDA) Use the method.

Learning creates a reference vector representing a class using a vector of each class.

Management tool 550 according to the present invention is a language management DB management tool (2800), character recognition engine learning tool (2820), character recognition performance tuning tool (2840), language-specific test bed (2860), distribution management tool distribution management A tool is included, and the character recognition performance tuning tool 2840 will be described.

37 is a flowchart illustrating a character recognition performance tuning tool 2840. The character recognition performance tuning tool 2840 is a tool necessary for efficiently performing tuning of an engine suitable for a specialized field, and is transmitted from a language-specific DB management tool. DB and engine tuning information necessary for specialized field can be selected by using received DB information and sample for test in specialized field. The tuning information of the engine can be generated by testing images acquired directly from a specialized field or virtually generated. By manually setting each setting for tuning and visually checking the processed image or report results, the optimum setting value is found.

The use of image preprocessing filter and recognition tuning factor are set to eliminate unnecessary processing and increase processing accuracy by using only methods suitable for specialized fields. Image tuning elements include filters to remove blurring effects, filters to remove transparency effects, filters to remove reflection effects, filters to remove occlusion effects, filters to remove shadow effects, binarization algorithms suitable for natural video, and italics , Kerning distance setting, etc.

Management tool 550 according to the present invention is a language-specific DB management tool (2800), character recognition engine learning tool (2820), character recognition performance tuning tool (2840), language-specific test bed (2860), distribution management tool (2880) It includes a), the language-specific test bed 2860 and the distribution management tool (2880) will be described.

38 illustrates a configuration of a test bed 2860 for each language, and the language-specific test bed 2860 receives an optimal tuning DB information and an optimal tuning engine information from a character recognition performance tuning tool, and configures an actual engine and a DB. The system performs a test by including a test sample, ground truth, performance measurement, and a reporting tool.

The test sample includes image data, and acquires and builds image data of a real environment, and the sample correct answer information manually writes a normal recognition text result, a text area, and the like used for performance measurement on the constructed test sample. To build.

Performance measurement and reporting tools include string extraction rates, string recognition rates, and translation success rates, and are organized in batches to facilitate testing of large numbers of test samples.

FIG. 39 illustrates a configuration diagram of a distribution management tool 2880. The distribution management tool receives an optimal tuning DB information and an optimal tuning engine information from a character recognition performance tuning tool, and configures an engine and a DB in a form that can be actually used. Create a library appropriate for your environment. In addition to the library creation function, it is possible to manage the distribution information and the version information of the distributed library.

On the other hand, Figure 5 is a block diagram showing the configuration of a more specific embodiment of the configuration of the present invention described above, including the engine common framework 500 and the management tool 550, based on the engine common framework 500 Commercial software 580 is additionally provided. The commercial software 580 will be briefly described.

In order to meet the demands required by demand agencies such as portal companies and mobile telecommunication companies, the company provides customized modules and apps based on the technology of processing natural images acquired through mobile cameras and extracting text areas. 40 shows the commercial software configuration, wherein the provided module includes a smartphone module such as a signage recognition system, a terminal mounting module such as a multilingual translation service, and a server & package module such as a real estate information service.

The signboard recognition system acquires natural images including signboards and extracts specific data of related signboards. The extracted data performs text recognition through a lightweight engine and a lightweight DB, and displays the shop information searched based on the recognized results. Gives or serves the user the results of nearby store information.

The multilingual translation service extracts specific data of the menu board by acquiring a natural image such as a menu board, and the extracted data performs character recognition on a string in the menu board through a recognition engine and a specialized recognition DB, and retrieves the recognized result. Based on this, the multilingual translation module is used to translate the translation into multiple languages (Korean, Latin, Chinese, Japanese) and serve the translation results to the user.

The real estate information service is stored in the server by acquiring the natural image of the new address sign, extracting the characters in the new address sign area from the image and recognizing the extracted character, and then attaching additional information to the recognized new address information and transmitting it to the server. The module consists of a function of obtaining the corresponding real estate information, which is configured to provide real estate information services and various application services. In case of general user who checks simple information, it is composed of package to handle image acquisition, text extraction and recognition, and server interworking process on the user's terminal. In this case, only image acquisition and server interworking are processed by the terminal, and when the image is uploaded to the server, the character extraction and recognition process is performed on the server, and then only the relevant information is transmitted to the terminal through DB search.

Meanwhile, FIG. 41 is a flowchart illustrating an embodiment of a natural language multilingual character recognition and translation method using a mobile camera according to the present invention.

Referring to FIG. 41, a natural language multilingual character recognition and translation method using a mobile camera will be described. First, it is necessary to build a database of words and words, word-to-word combinations, word-to-word combination information, and misrecognition pattern information specialized according to an application in which character recognition and translation are performed. In the state in which the database is constructed, a character recognition / translation engine providing a framework commonly applied to the application field extracts a string image existing in an image acquired by the mobile camera (step S4100).

Then, the character recognition / translation engine analyzes and recognizes the extracted string image and outputs it in text form. (S4120) The string image analysis and recognition subdivides a split point into predicted character division points in the string image. After segmentation, it is desirable to search for the best combination by combining and recognizing each segment. Then, the text-type characters are translated into another language (step S4140).

FIG. 42 illustrates the string image extraction in more detail. Referring to FIG. 42, the string image extraction first converts an input image file into a form capable of image processing and corrects distortion included in the image. In operation S4200, a character area candidate group is selected in the input image (step S4220), a background image of the character area candidates and a character string are separated, and the extracted character string is separated into respective characters (step S4240). Then, the characters are verified by extracting characters by language by applying a language model DB including words, frequency of occurrence of words and word combinations between words, word occurrences of letters, word occurrences of letters, combination information between letters, and misrecognition pattern information. (Step S4260)

Here, the character string image recognition and the recognized character string translation may include a font DB for classifying and storing the piece information used for recognizer learning by font type, an image matching DB including original image information and feature extraction information, word and word occurrence frequency, and words. A language model DB comprising word information consisting of inter-combination information, word information consisting of single words, frequency of occurrence and inter-combination information, misrecognition pattern information consisting of words and misrecognition patterns of words that are easy to be mistaken, and the character recognition engine At least one of a character recognition model DB that manages a list of character recognition models generated by the learning tool, and a translation DB consisting of a word of a reference language and a foreign language word table having a form of a foreign language word matching the reference language and matching with the reference language table. Is done using.

The present invention can be embodied as code that can be read by a computer (including all devices having an information processing function) in a computer-readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored. Examples of computer-readable recording devices include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

100: common character recognition / translation engine 110: natural image character region extraction unit
112: natural image preprocessing unit 114: character extraction unit
116: character verification unit 120; Multilingual Character Recognition
122: string recognition unit 124: image recognition unit
130: multilingual translation unit 150: management tool
155: language management DB management tool 160: character recognition performance tuning tool
165: character recognition learning tool 170: language-specific test bed
175: deployment management tool 180; Commercial software
185: smartphone software 190: terminal mounting software
195 Package Software 199 Server Service Software
200: feature extraction unit 250: feature matching unit
300: recognition / translation database 310: font DB
320: Image matching DB 330: Language model DB
340: character recognition model DB 350: translation DB
400: feature extraction unit for each class 420: dimension reduction unit for each class
440: learning unit

Claims (15)

A common character recognition / translation engine for extracting, recognizing and translating characters existing in an image acquired by a mobile camera; And
Manages the recognition / translation DB used for the recognition and translation of the extracted character, and the character and word, the combination between the characters, combination information between words and misrecognition that is specialized according to the application (s) where the character recognition and translation is performed A management tool for constructing a pattern information as a database and providing the common character recognition / translation engine;
The common character recognition / translation engine provides a framework commonly available for an application in which the character recognition and translation is performed, and uses a database specialized for each application field provided from the management tool. Natural image multilingual character recognition system using a mobile camera, characterized in that the character is recognized and translated. The method of claim 1, wherein the common character recognition / translation engine
A natural image character region extraction unit for extracting characters by searching for an area in which a character string exists in a natural image photographed by a mobile camera;
A multilingual character recognition unit for recognizing a character string of the extracted character area; And
Natural image multi-lingual character recognition system using a mobile camera, characterized in that it comprises a multi-language translation unit to search the multi-lingual database to translate the recognized string. The method of claim 2, wherein the natural image character region extraction unit
A natural image preprocessing unit converting the input image file into a form capable of image processing and correcting distortion included in the image;
A character extracting unit for selecting a character region candidate group in the input image, separating a background image and a character string of the character region among the character region candidates, and separating the extracted character string into respective characters; And
Natural image multilingual character recognition using a mobile camera, characterized in that it comprises a character verification unit for verifying the character by extracting the characters by applying a language model including at least the combination information between words, the combination information between words and misrecognition pattern information system. The method of claim 2, wherein the multilingual character recognition unit
A string recognition unit for recognizing character strings in the form of printed characters,
The string recognition unit
Natural image multi-lingual character recognition system using a mobile camera, characterized in that the segmentation point is divided into a character point predicted point in the character string image, and then each segment is combined and recognized to search for an optimal combination. The method of claim 4, wherein the multilingual character recognition unit
Further comprising an image recognition unit for recognizing a character or a logo containing a graphic element,
The image recognition unit
A feature extractor for extracting corner points and junction points representing characteristics of an image and then vectorizing them using surrounding information; And
Natural image multilingual character recognition system using a mobile camera, characterized in that it comprises a feature matching unit to find the most similar image information by matching the information of the vectorized corner point (junction point) and junction point (junction point) and the image matching DB . The method of claim 2, wherein the multilingual translation unit
A general term translation database in which a general word or combination of words is matched with a plurality of words or combinations of words; And
A specialized term translation database in which specialized words or word combinations are matched one-to-one,
When the recognized character string is the general word or a combination of words, the generalized word translation data database is translated into a plurality of words or a combination of words,
When the recognized character string is the specialized word or a combination of words, the natural image multilingual character recognition system using a mobile camera, characterized in that the translation to a single word or a combination of words with reference to the specialized term translation data database. The method of claim 1, wherein the management tool
It provides additional performance tuning function and learning function for creating character recognition model DB.
A language-specific DB management tool that manages a recognition / translation DB used to recognize and translate the extracted characters and provides a registration, change, deletion, and inquiry function necessary for the management;
Character recognition performance tuning tool providing performance tuning; And
Natural image multilingual character recognition system using a mobile camera, characterized in that it includes a character recognition learning tool that provides a learning function for generating a character recognition model DB. The method of claim 7, wherein the recognition / translation DB:
A font DB for storing the piece information used for learning the recognizer by font type;
An image matching DB consisting of original image information and feature extraction information;
Language consisting of word information consisting of words, frequency of occurrence of words and combination information between words, word information consisting of words, word occurrence frequency and combination information between words, and misrecognition pattern information consisting of words that are easy to be mistaken and patterns of word misrecognition Model DB;
A character recognition model DB for managing a list of character recognition models generated by the character recognition learning tool; And
A natural image multilingual character recognition system using a mobile camera, characterized in that it comprises at least one of a word of the reference language, a foreign language word matching the reference language and a translation DB consisting of a foreign language word table matching the reference language table. The method of claim 7, wherein the character recognition learning tool
Generates the learned character recognition model by inputting the font information of the font DB,
A feature extraction unit for each class, which extracts an image characteristic by using boundary information of each piece and expresses it as a feature vector;
A dimension reduction unit for each class for reconstructing the feature vector by extracting only an important element from the dimension of the feature vector in order to reduce the memory and the amount of computation used for learning and recognition; And
Natural image multi-lingual character recognition system using a mobile camera, characterized in that it comprises a learning unit for generating a reference vector representing a class by using the vector of each class. The method of claim 7, wherein the character recognition performance tuning tool
DB and engine tuning information required for a specialized field are selected using DB information received from the DB management tool for each language and a specialized field test, and engine tuning is performed using the selected DB and engine tuning information. Natural video multilingual character recognition system using a mobile camera, characterized in that. 8. The method of claim 7,
A language-specific test bed configured to test by reconfiguring the engine by receiving optimal tuning DB information and optimal tuning engine information from the character recognition performance tuning tool; And
It further includes a distribution management tool that receives the optimal tuning DB information and the optimal tuning engine information from the character recognition performance tuning tool, generates an environment library to be serviced by constructing an available engine and DB and outputs a module for distribution. Natural video multilingual character recognition system using a mobile camera, characterized in that. Constructing a database of words and words, word-to-word combinations, word-to-word combination information, and misrecognition pattern information specialized according to an application in which character recognition and translation are performed;
Extracting a character string image existing in an image acquired by a mobile camera by a character recognition / translation engine providing a framework commonly applied to the application field;
Analyzing, recognizing, and outputting the extracted character string image in text form by using the database by the character recognition / translation engine; And
The character recognition / translation engine natural character multilingual character recognition method using a mobile camera comprising the step of translating the text-type characters to another language using a predetermined database. The method of claim 12, wherein the character string image extraction
Converting the input image file into a form capable of image processing and correcting distortion included in the image;
Selecting a character region candidate group in the input image, separating a background image and a character string of the character region among the character region candidates, and separating the extracted character string into respective characters; And
Applying a language model including at least combination information between words, combination information between words, and misrecognition pattern information to verify a character by extracting characters for each language, wherein the natural image multilingual character recognition method using a mobile camera is provided. . The method of claim 12, wherein the string image analysis and recognition is performed.
A natural image multilingual character recognition method using a mobile camera, characterized by dividing a segmentation point into a character classification prediction point in a string image, and then searching for an optimal combination by combining and recognizing each segment. The method of claim 12, wherein the string image recognition and the recognized string translation are performed.
A font DB for storing the piece information used for learning the recognizer by font type;
An image matching DB consisting of original image information and feature extraction information;
Language consisting of word information consisting of words, frequency of occurrence of words and combination information between words, word information consisting of words, word occurrence frequency and combination information between words, and misrecognition pattern information consisting of words that are easy to be mistaken and patterns of word misrecognition Model DB;
A character recognition model DB for managing a list of character recognition models generated by the character recognition learning tool; And
Natural image multi-lingual character recognition method using a mobile camera, characterized in that using at least one of the translation language consisting of a word of the reference language, a foreign language word matching the reference language, and a foreign language word table matching the reference language table .

Images