JP2005135210A - Portable device with character recognition function - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To allow the input means of a portable terminal with a camera to automatically detect the subject of reading for character recognition from the surrounding scenes and recognize characters. <P>SOLUTION: A character string to be recognized is automatically set according to a user-designated keyword. A process for extracting the character string and/or recognizing characters is carried out on images continuously inputted from an image input means, and if recognition has been successful, it is reported to the user by means of voice or vibration. This makes it possible to automatically detect the character string in scenes and report it to the operator, which has been difficult with conventional devices. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention belongs to a technical field relating to input means in a portable terminal having a camera.

  Conventionally, devices such as a digital still camera, a camera-equipped mobile phone, and a camera-equipped PDA that can digitize images stored with a camera and store them as files have been put into practical use. Furthermore, attempts have been made to use these devices as image input means in character recognition. For example, H. Fujisawa, H. Sako, Y. Okada, and SW. Lee, "Information Capturing Camera and Developmental Issues," Proc. Int. Conf. Document Analysis and Recognition, ICDAR'99, Bangalore, India, Sep. 20 -22, 1999, pp. 205-208 (Non-Patent Document 1) describes that foreign language signs and signboard characters are read, translated, and searched by a portable device with a camera.

  In general, character recognition involves (1) detection of character line area (character line extraction), (2) detection of each character area from the character line area (character extraction), and (3) any character extracted from each character. (4) Interpretation of character identification results (language processing) is required. However, the processing is not necessarily performed in these order. For example, Hiroshi Murase, “Lettering and Recognizing Characters from Handwritten Character Strings Using Linguistic Information,” Science Review (D), vol.J69-D, no.9, pp.765-772 (non-patent document 2 In), after character extraction based on various hypotheses, the character extraction method is determined by the character identification result and language processing.

  Conventional character recognition technology mainly targets characters written on paper documents, and images are input by an image scanner. On the other hand, when reading a signboard or a sign using a camera as an input means, it is necessary to enhance the character line extraction and character extraction among the above functions. This is because it becomes difficult to realize element functions necessary for conventional character line extraction and character extraction, such as character and background separation and character layout analysis. For example, when inputting an image with a camera, unlike the case of inputting an image with a scanner, it is difficult to separate the characters and the background because the lighting conditions cannot be known in advance. Also, in paper documents, there are document-specific restrictions on the arrangement of character lines, but there are no such restrictions on signs and signs in images taken with cameras, so character lines are based on a priori knowledge of character arrangement. It becomes difficult to cut out.

H. Fujisawa, H. Sako, Y. Okada, and SW. Lee, "Information Capturing Camera and Developmental Issues," Proc. Int. Conf. Document Analysis and Recognition, ICDAR'99, Bangalore, India, Sep. 20-22 , 1999, pp. 205-208

Hiroshi Murase, “Character extraction and recognition from handwritten strings using linguistic information,” IEICE (D), vol.J69-D, no.9, pp.765-772 R.M.K.Sinha, B. Prasada, G.F.Houle, M. Sabourin, “Hybrid Contextual Text Recognition with String Matching,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 15, No. 9, December 1993 A.K. Jain, B. Yu, “Automatic Text Location in Images and Video Frames,” Pattern Recognition, Vol. 31, No. 12, pp. 2055-2076, 1998 C.-L. Liu, M. Koga and H. Fujisawa, "Lexicon-driven Segmentation and Recognition of Handwritten Character Strings for Japanese Address Reading," IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 24, No. 11, Nov . 2002, pp. 425-1437

  The portable device of the present invention uses a camera as an image input means for character recognition, and reads signboards / signs on which specific characters are written on a street or the like. In the conventional technology, an operator searches for a signboard / signs to be read by himself / herself and points a camera at the sign / signs to be recognized. In contrast, the apparatus of the present invention has a function of photographing a wide range using a wide-angle camera and automatically detecting a reading target. This eliminates the need for the operator to search for the reading object and point the camera at the user, greatly improving convenience.

  In order to realize such functions, the present invention intends to solve the following problems.

  First, it takes time to detect and recognize characters. Since the portable device of the present invention has a larger image area to be processed than the conventional device, the number of detected character lines increases, and the processing time for character extraction, character identification, and language processing increases. Due to the increase in processing time, the time that an operator waits to obtain necessary information is extended, and the convenience of the device is reduced. Therefore, reduction of processing time is an important issue.

  Second, while there are many unnecessary character strings in the image, the operator does not necessarily know exactly what characters are written on the signboard / sign that is being searched for. When an image is taken in the city, many characters appear in the image. However, what is of interest to the operator is often part of them. Here, unnecessary characters can be eliminated by limiting the recognition target to character strings designated in advance by the operator. However, there are often many strings that mean one thing. For example, there are various character strings such as “bathroom”, “toilet”, “rest room”, and “lavatory”. The operator cannot know in advance which of these various character strings appears in the image. Also, it takes a lot of time for the operator to specify all of these. As described above, it is important to eliminate unnecessary characters and to make various character strings to be recognized at the same time, which is a difficult problem to solve.

  Third, the captured image does not necessarily include a character string of interest to the operator. As described above, it is an object of the present invention to enable a mobile device to search for characters to be read from an image on behalf of an operator. In order to realize this purpose, it is necessary that the operator does not have to be aware of whether or not a character string of interest is captured when an image is taken. As a self-explanatory solution to this problem, there is a means in which after the image is captured and characters in the image are recognized, the portable device presents to the operator whether or not there is a character string of interest to the operator. However, when this means is used, the operator needs to repeat photographing and checking the presence / absence of a character string many times, and the convenience of the apparatus is deteriorated. As described above, it is an important problem to solve the problem that the character string of interest to the operator is not always present in the captured image without reducing the convenience of the apparatus.

  In order to solve the first problem, the present invention provides means for an operator to designate a set of character strings to be recognized so that the character strings to be recognized can be narrowed down in advance. Furthermore, a means for narrowing down character types for character identification according to a set of designated character strings to be recognized is provided. In general, the processing time for character identification increases according to the number of character types to be identified. Therefore, if the character strings to be recognized can be limited in advance, the processing time can be reduced by narrowing down the character types to be identified accordingly. Therefore, in the portable device of the present invention, the operator can designate a set of character strings of interest, and automatically narrow down the character types used for character identification according to the designated character string set. In addition, a recognition target character string can be easily specified by storing a recognition target character string group in advance and allowing the operator to select it as necessary. Furthermore, it will be even easier by providing means for the mobile device to automatically perform according to the situation.

  In order to solve the second problem, means for automatically acquiring a set of character strings having the same meaning is provided. For example, when the operator inputs the character string “toilet” or selects it from the menu, the portable device refers to information on the synonym stored in advance and acquires a set of synonyms for “toilet”. Furthermore, using a general language knowledge, synonyms are further changed as necessary, and a set of character strings to be recognized is determined. Here, changing synonyms with general linguistic knowledge means, for example, in the case of Japanese, from “Depository” to “Depositary”, “Depositary”, “Depositary”, etc. To create a set of strings. In the case of English, a set of character strings such as “rest room”, “rest room”, “Rest Room”, and “REST ROOM” is generated corresponding to the various notation of uppercase and lowercase letters.

  In addition, in order to solve the third problem, the mobile device automatically repeats imaging and character recognition, and when a character string of interest to the operator is recognized from the image, this is indicated. Is transmitted to the operator by voice or vibration, and the location of the recognized character string is displayed as an image. This eliminates the need for the operator to be aware of whether or not a character string of interest is shown in each image. Furthermore, if a character string of interest is found, it can be quickly known, and the character string can be easily found visually and the details can be confirmed without using a camera.

  It is possible to automatically detect the character string in the landscape and notify the operator, which was difficult in the past.

  FIG. 1 is a data flow diagram showing the flow of processing from image input to image file storage in the first embodiment of the present invention. A range indicated by a frame 101 in the figure is processing performed inside the mobile device.

  In this embodiment, after inputting an image by a camera or the like (109), the character string is recognized (110) from the image by referring to the recognition target character string dictionary 108. When the recognition is successful, that is, when a character string stored in the recognition target character string dictionary 108 is detected from the image, the operator is notified 111 by voice or vibration and the recognition result is displayed. (112).

  The recognition character string dictionary 108 stores a set of character strings to be recognized. The character string to be stored is determined by the following three methods.

(1) The operator inputs a word of interest, that is, a keyword to the portable device (102). In the portable device, the synonym dictionary 104 stored in the portable device or the external synonym dictionary 114 stored outside the portable device is referred to, and a set of synonyms of the input keyword is obtained (consent The word expansion process 103) is further changed by general language knowledge, and these are stored in the recognition target character string dictionary 108. The synonym dictionary 104 and the external synonym dictionary 114 store a relationship between a word and a set of synonyms thereof. The external synonym dictionary 114 is referred to via communication means described later. In addition, as a change due to general language knowledge, the following processing is performed.
・ Capitalize the first letter of English words.
・ Capitalize all English letters.

  The synonym expansion function eliminates the need for the operator to input a large number of synonyms into the portable device one by one, greatly improving convenience. Furthermore, when the recognition target is a foreign language, the keyword may be specified in the native language, and the corresponding synonym may be the foreign language.

  (2) The portable device refers to the recognition menu information 106 and displays the keyword to the operator in the form of a menu. The operator selects an item of interest from the menu (105). In accordance with the selection result, the portable device stores a set of recognition target character strings stored in the menu information 106 in the recognition target character string dictionary 108. By providing a means for selecting a keyword of interest from the menu, the operator does not need to input many synonyms into the portable device one by one, and the convenience of the portable device is greatly improved.

  (3) The portable device sends a signal requesting acquisition of the recognition reference character string to an external computer via the communication means. Upon receiving this, the computer transmits a set of recognition target character strings to the portable device. The mobile device stores the received set of recognition target character strings in the recognition target character string dictionary 108. The recognition target character string acquisition signal may be transmitted when the operator performs a specific operation. Moreover, you may make it transmit at a fixed time. Further, the convenience of the mobile device is further improved by changing the recognition target character string transmitted by the external computer in accordance with the location, time, operator, etc. of the mobile device.

  In this embodiment, for example, RMK Sinha, B. Prasada, GF Houle, M. Sabourin, “Hybrid Contextual Text Recognition with String Matching,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. , No. 9, December 1993 (Non-Patent Document 3). FIG. 2 shows an example of the data flow of character string recognition processing. First, in 201, a character line is cut out from the input image. For example, AK Jain, B. Yu, “Automatic Text Location in Images and Video Frames,” Pattern Recognition, Vol. 31, No. 12, pp. 2055-2076, 1998 (Non-patent Document 4) Such a method is used. Next, in 202, individual characters are cut out from the character line. When a plurality of character lines are cut out, all of them are subjected to subsequent processing. Next, what character each individual character extracted in step 203 is identified. At this time, the character type limitation 204 limits the identification character type to only the character types included in the recognized character string dictionary 108 and refers to the character identification dictionary 206. The character identification dictionary 206 stores information on the shape of each character. Finally, in step 205, the character identification result is interpreted as a character string. At this time, the recognized character string dictionary 108 is referred to. The output of the character string recognition process is the presence / absence of a character string to be recognized and the recognized character string.

  The processing time for character identification is roughly proportional to the number of character types to be identified. For this reason, the processing time can be significantly reduced by limiting the identification target to the character types that can appear in the character string to be recognized. This effect is particularly noticeable in languages with many character types such as Kanji and Korean.

  In this embodiment, character extraction, character recognition, and post-processing are sequentially performed. However, C.-L. Liu, M. Koga and H. Fujisawa, "Lexicon-driven Segmentation and Recognition of Handwritten Character Strings for Japanese Address Reading, "IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 24, No. 11, Nov. 2002, pp. 425-1437 (Non-Patent Document 5). May be. In this case, the character type is limited dynamically during the recognition process, so that higher speed and higher accuracy can be realized. As another embodiment, after character recognition is performed without using language information, a word in the recognized character string dictionary 108 may be collated with a character recognition result using a normal text matching algorithm. .

  FIG. 3 shows a hardware configuration in the embodiment of the present invention. An image is picked up by an optical device 302 including a lens, a diaphragm, etc., and then converted into an electric signal by a photoelectric conversion element 303 such as a CCD element. Further, the obtained electrical signal is converted into a digital signal by an analog / digital converter 304, and further subjected to processing such as color space conversion and filter processing by a signal processing element 305 such as a DSP. The result is transferred to the RAM 309. The recognition target character string dictionary 108 is also stored in the RAM 309. The arithmetic unit 307 refers to the processing procedure stored in the ROM 308 and data such as a character identification dictionary, and executes the character string recognition processing 110 using the image stored in the RAM 309 as an input. The input device 312 is used when inputting a keyword. The display device 306 is used to check an image at the time of shooting and display a character string recognition result. The sound generation device 313 and the vibration device 314 are used for the character string presence notification 111. The communication device 315 is used for receiving a recognition target character string and accessing the external synonym dictionary 113.

  FIG. 4 shows the appearance of the apparatus in the embodiment of the present invention. The lens unit 402 of the optical device 202 is disposed on the front surface of the housing 401. Use a lens with a wide angle of view sufficient to photograph signs and signs around the operator. In the upper part, a shutter 403 for instructing the image input 109 as a part of the input device 312, a power switch 404, and a speaker opening 410 which is a part of the sound generator 313 are arranged. The antenna 407 of the communication device 315 is disposed on the side surface. On the back side, a cancel button 405 and a cursor key 406, which are a part of the input device 312 are arranged. Further, a display surface 409 of the display device 306 is disposed on the back surface. The cursor key 406 sends a signal indicating a direction to the input device 312 when the upper, lower, left and right ends are pressed, and sends another signal to the input device 312 when the center is pressed. In this embodiment, it is preferable that the portable device is stored in an operator's breast pocket or the like so that the scenery can be photographed by taking out only the lens outside except during operation.

  FIG. 5 shows the state transition of the portable device in the present embodiment. First, when the power switch 404 is pressed, a transition is made from the start state 501 to the standby state 502. When the keyword input operation is activated in the standby state 502, the state transitions to the keyword input state 503. When the input is completed, the state returns to the standby state 502 again. In the keyword input state 503, for example, a virtual keyboard appears on the display surface 409, and this is operated with the cursor key 406 to input a character string. In addition, when the menu selection operation is started in the standby state 502, the state transitions to the menu selection state 504. When the menu selection is completed, the state returns to the standby state 502 again. Moreover, by starting the recognition character string acquisition in the standby state 502, the recognition character string acquisition state 505 is entered. When the acquisition of the recognition target character string is completed, the process returns to the standby state 502 again. In the recognized character string acquisition state 505, operations on the portable device are temporarily not accepted, and during that time, the portable device communicates with an external computer to acquire a character string to be recognized. Pressing the power switch 404 in the standby state 502 returns to the start state 501. When the recognition is activated in the standby state 502, the state transitions to the recognition state 506. In the recognition state 507, as will be described later, the portable device repeats the image input 109 and the character string recognition 110 until the recognition target character string is recognized from the image. In this state, the operator takes out only the camera lens and carries the portable device in a pocket or the like. If the character string is recognized, the state transitions to the character string presence notification state 507. In the character string presence notification state 507, the mobile device emits voice or vibration in order to call the operator's attention. When the operator presses the cancel button 405, the voice or vibration is stopped and the state transitions to the character string recognition result display state 508. In the character string recognition result display state 508, the character string recognition result is displayed on the display surface 409 so as to overlap the image. Here, when the cancel button 405 is pressed, the state transitions to the standby state 502 again.

  FIG. 6 shows the display surface 409 in the standby state 502. A menu 601 is displayed on the display surface 409. The selection item in the menu is changed by the operator pressing the upper and lower portions of the cursor key, and the center of the cursor key 406 is pressed when confirming. If neither is selected, the cancel button 405 is pressed. When transitioning to the keyword input state 503, the operator selects “keyword input” 603 and presses the center of the cursor key 406. In order to change to the menu selection state 504, the “keyword selection” 602 is selected and confirmed. In order to shift to the recognized character string acquisition state 505, “Keyword acquisition” 604 is selected and confirmed. In order to change to the recognition 506 state, “recognition” 605 is selected and confirmed.

  FIG. 7 shows the display screen 409 in the menu selection state 504. Here, keywords such as “toilet” and “information center” are displayed on the window 701, and a check box 702 is arranged to the left of each keyword. A keyword can be selected by operating a check box. Here, a keyword displayed on the menu and a synonym group corresponding to the keyword are stored in the recognition menu information 106 in advance, and they are used as a recognition target character string. For example, when “toilet” is selected here, character strings such as “rest room”, “Restroom”, and “toilet” are recognized.

  FIG. 8 shows an example of the input image. In this example, there are two character strings “Restroom” 801 and “Exit” 802 in the image.

  FIG. 9 shows a processing procedure from the recognition state 506 to the character string recognition result display state 508 in the state transition diagram of FIG. First, in step 901, character types to be identified by character identification are limited according to a set of recognition target character strings. Next, in a loop 902, image input 903 and character string recognition 904 are repeated. If the target character string is recognized from the image, the repetition is terminated at step 905. Next, in step 906, the operator is notified that the character string has been recognized by voice or vibration. In step 907, the character string recognition result is displayed.

  FIG. 10 shows the state of the display surface 409 in the recognition result display state 509. Here, the position of the recognized character string 810 is surrounded by a quadrilateral to clarify the position. The operator can easily find the character string recognized from the surroundings by quickly confirming the recognition result on the screen before moving or changing the direction after the notification of the recognition. In addition, the character string of the recognition result, the keyword that is the basis of the character string, and the recognized time are displayed on the window 1001. As a result, even when the operator is not cheerful about the synonym (for example, when the recognition target is a foreign language), it can be easily confirmed which of the specified keywords the recognized character string corresponds to.

  FIG. 11 shows a storage format in the synonym dictionary 104 and the external synonym dictionary 113. Here, synonym information is stored in a table in which the keyword 1101 and the synonym group 1102 are one record.

  FIG. 12 shows a storage format in the recognition menu information 106. Here, the menu information is stored in a table in which one record is a true / false value 1201 indicating whether to be included in the recognition target, a keyword 1202 displayed on the menu, and a character string group 1203 to be recognized.

The data flow figure which shows the flow of the process from an image input to an image file output. The data flow figure which shows the flow of a character string recognition process. The block diagram of hardware. FIG. The figure showing the state transition of operation of an apparatus. The figure showing the state of the display surface in a standby state. The figure showing the state of the display surface in a menu selection state. The schematic diagram of an input image. The figure showing the process sequence from a recognition state to a recognition result display state. Display surface when displaying the character recognition result. The figure showing the memory format of a synonym dictionary. The figure showing the storage format of recognition menu information.

Explanation of symbols

101 ··· Processing from image input to recognition result display, 102 ··· Keyword input, 103 ··· Synonym expansion, 104 ··· Synonym dictionary, 105 ··· Menu selection, 106 ··· Character string to be recognized Request: 107 ... Character string to be recognized, 108 ... Character string to be recognized, 109 ... Image input, 110 ... Character string recognition, 111 ... Character string presence notification, 112 ... Character Column recognition result display, 113 ... external synonym dictionary, 201 ... character line extraction, 202 ... character extraction, 203 ... character identification, 204 ... character type limitation, 205 ... post-processing, 206 ... Character identification dictionary, 301 ... Portable device, 302 ... Optical device, 303 ... Photoelectric conversion element, 304 ... Analog-to-digital converter, 305 ... Signal processing element, 306 ...・ Display device 307... Arithmetic unit, 308... ROM, 309... RAM, 310... I / O device, 311... Memory card, 312. ... Vibration device, 315 ... Communication device, 401 ... Case, 402 ... Lens, 403 ... Shutter, 404 ... Power switch, 405 ... Cancel button, 406 ... Cursor key, 407 ... antenna, 409 ... display surface, 410 ... speaker opening, 501 ... start state, 502 ... standby state, 503 ... keyword input state, 504 ... Menu selection state, 505 ... recognized character string acquisition state, 506 ... recognition state, 507 ... character string presence notification state, 508 ... character string recognition result display state, 601 ... menu 602 ... Keyword selection, 603 ... Keyword input, 604 ... Keyword acquisition, 605 ... Recognition, 701 ... Keyword specification menu, 702 ... Check box, 801 ... In the image Character string “Restroom”, 802... Character string “Exit” in the image, 901... Step of limiting the character type, 902... Loop, 903. Step, 905 ... Repeat end step, 904 ... Notification step, 905 ... Recognition result display step, 1001 ... Character string recognition result, 1101 ... Field for storing synonym dictionary keywords, 1102・ ・ ・ Field for storing synonym group of synonym dictionary, 1201... Field that contains the keywords that appear in the menu in the 202 ... recognition menu information, the field that stores the recognition target character string group in 1203 ... recognition menu information.

Claims (6)

Image capturing means for photoelectrically converting an image into a digital signal, recognition target character string storage means for storing a set of character strings to be recognized, and character strings stored in the recognition target character string storage means are recognized from the image. Means to
A portable device characterized in that a set of character strings to be recognized is automatically generated from a keyword representing a recognition target character string group and stored in a recognition target character string storage means.   2. The portable device according to claim 1, wherein the means for generating a character string to be recognized is to acquire a set of synonyms of keywords. Image capturing means for photoelectrically converting an image into a digital signal, recognition target character string storage means for storing a set of character strings to be recognized, and character strings stored in the recognition target character string storage means are recognized from the image. Means to
And a recognition menu information storage means for storing a plurality of pairs of a keyword representing the recognition target character string group and a recognition target character string set, and a means for selecting a keyword in the recognition menu information storage means. A portable device that copies a recognition target character string in a recognition menu information storage unit to a recognition target character string storage unit according to a result. Image capturing means for photoelectrically converting an image into a digital signal, recognition target character string storage means for storing a set of character strings to be recognized, and character strings stored in the recognition target character string storage means are recognized from the image. Means to
Furthermore, a portable device having means for communicating with an external computer, wherein a recognition target character string group is acquired from the outside and stored in a recognition target character string storage means. Image capturing means for photoelectrically converting an image into a digital signal, recognition target character string storage means for storing a set of character strings to be recognized, and character strings stored in the recognition target character string storage means are recognized from the image. Means to
A portable device that automatically repeats imaging and character string recognition and generates voice or vibration when the character string recognition is successful. Image capturing means for photoelectrically converting an image into a digital signal, recognition target character string storage means for storing a set of character strings to be recognized, and character strings stored in the recognition target character string storage means are recognized from the image. Means to
A portable device that automatically repeats imaging and character string recognition, and displays a captured image and a recognition result when the character string recognition is successful.

Images