JP2005038431A - Portable terminal and communication system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To photograph an object on which an arbitrary character row is written and to obtain information based on the character row by using a portable terminal incorporating an image pickup element. <P>SOLUTION: A portable terminal enables an operator to easily select an object character row by displaying a character row extraction process at a display part of the terminal as desired and by rendering the image to be inputted to recognition processing with a high resolution, and raises the character recognition precision by realizing high resolution. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  In the future, mobile terminals with built-in image sensors will become more popular due to higher performance of CPUs built into mobile terminals and mobile phones and lower prices for CCDs. Using a mobile terminal with a built-in image sensor, it is possible to extract relevant information from the network by collecting a landscape image that includes a character string focused by the user and recognizing the character string. In order to select a target character string from a landscape image, an instruction method that uses a limited user interface on a portable terminal and is easy for an operator to understand intuitively is required.

  In addition, it is difficult to store a device capable of collecting high-resolution images in the mobile terminal due to its structure. Therefore, character recognition is performed using low-resolution images collected by the camera of the mobile terminal. However, resolution and recognition accuracy are in a trade-off relationship, and when high recognition accuracy is required, it is necessary to process the input image.

  Information retrieval using character recognition results from landscape images is described in JP-A-2000-331006 and JP-A-10-254622. Japanese Patent Laid-Open No. 2000-23012 describes a translator using a portable terminal, and a line segment for applying the target character line to the GUI in the present invention is displayed on the image display unit. However, the line segments are fixed, and the operator needs to adjust the position and size of the character line only for use as a guide when the operator collects a landscape image. Further, there is no invention related to character recognition for an image collected by a low-resolution imaging system in a portable terminal.

JP2000-331006

JP 10-254622 A JP2000-23012

  Japanese Patent Laid-Open No. 2000-23012 describes a means of incorporating a CCD camera in a mobile phone or a mobile terminal, capturing a landscape image, performing character recognition, and installing a translation function, a search function, and the like. However, there is no means for the operator to easily select the target character line from the landscape image. Therefore, the present invention presents to the operator the character line extraction result from the input image at any time, and makes it possible to easily select any target character line.

  In addition, it is difficult to design an image sensor incorporated in a portable terminal or the like so as to collect a high-resolution image from the viewpoint of cost and installation volume. When recognition processing is performed using a low-resolution image, the recognition accuracy is adversely affected and the operator's purpose is hindered. Therefore, in the present invention, after extracting the character line position from the collected low-resolution image, the character line partial image is increased in resolution by performing enlargement processing on the partial image at the character line position. Improve accuracy.

  Among the inventions disclosed in the present application, outlines of typical ones are as follows. A portable terminal having means for extracting a character line description area in an acquired image, wherein the character line description area is continuously extracted while changing the character line description area until an input is made to the confirmation means, and the extracted character line description area Is always displayed on the display means, the character line description area to be extracted is determined, the means for recognizing the character line described in the determined character line description area, and the recognition result is transferred via the network. A portable terminal having means and means for receiving content based on the recognition result.

  Furthermore, the portable terminal which has the refinement means of the acquired image, the said character line description area | region is pixel-complemented, the said pixel-complemented partial image is smoothed, and also binarization is performed.

  A communication system comprising the mobile terminal and a computer connected to the mobile terminal via a network, wherein the recognition result or the character line extraction area acquired by the form terminal is transmitted to the computer via the network. And a means for performing an information search on the network based on the recognition result or the information of the character line extraction area transferred by the computer and transferring the search result to the portable terminal. Communications system.

  When the present invention is used, it is possible to capture an object in which an arbitrary character line is described using a portable terminal incorporating an image sensor and obtain information based on the character line. Further, when the operator selects an arbitrary character line, the operation of the portable terminal can be facilitated. Even when an image sensor with a low resolution is used, it is possible to improve recognition accuracy by generating a pseudo high resolution image and performing character recognition processing.

  FIG. 1 shows a portable terminal used in the present invention and an object on which a character line serving as input data is described. 101 is one form of a portable terminal, 102 has an image display unit, and a camera using an image sensor is stored on the opposite surface. 103 is an operation lever, and 104 is a button. The image display unit is a touch panel so that an operator can directly instruct. Using this mobile terminal, the URL address or character line image written on the object as in 105 is sampled, the character line description area is extracted, the character line is recognized, and the data is transmitted using the wireless communication line like 106. Using a computer connected to the network as in 107, the target content on the network is selected, transferred, and displayed. Bidirectional communication is performed by referring to the displayed content and giving an instruction again.

  2 and 3 show image display examples of the present invention. FIG. 2 shows an example in which character recognition processing is performed on a URL character line written on an imaged object, and a homepage corresponding to the URL is received and displayed from the recognition result. First, an object collected by the image sensor is displayed on the image display unit 201, and a marker is displayed as 202 in the image display unit. The marker can be moved by an instruction from the operator. The character line rectangle extraction process is continuously performed on the URL character line described at the position closest to the marker, and the extraction result rectangle is updated at any time as indicated by 203, and the display is repeated. As a result, the operator can easily select the recognition target character line by moving the terminal or moving the marker. When the rectangle correctly surrounds the target character line, the user issues a confirmation instruction. After that, the character line is confirmed and character recognition is performed for the character line in the rectangle, and the recognition result is displayed in a pop-up as shown at 204. If this recognition result is valid, the operator gives a confirmation instruction, and as a result, content corresponding to the URL of the recognition result is transferred from the computer on the network to the portable terminal as in 205. Thereafter, the operator refers to the contents of the display unit, and bidirectional communication is possible by a button operation or a button operation of the display unit such as 206.

  FIG. 3 shows an example in which character recognition processing is performed on a character line written on an imaged object, content on the network is searched based on the recognition result, and the search result is displayed. As in the case of FIG. 2, when the image captured at 301 is displayed, the nearest character line is extracted by the marker 302, and the user confirms the character line rectangle displayed continuously as 303, the recognition process is performed. The recognition result is displayed in a pop-up like 304. If this recognition result is valid, the operator gives a confirmation instruction. Thereafter, the recognition result is input to a search engine on the network, and the obtained search result is received by the portable terminal and displayed as 305. The operator can receive new content by referring to the displayed search result and by operating a button or touching the display unit.

  Fig. 4 shows the hardware configuration in the mobile terminal. In the portable terminal, there are a CPU (401) that performs control, image processing, character recognition, a memory (402), a display unit (403), an image sensor (404) for collecting input data, and a communication unit used for wireless communication ( 405), a speaker (406), a microphone (407), a power supply unit (408), a button (409), and a touch panel (410) that is incorporated in a display unit are connected by a bus.

  FIG. 5 shows an overall processing flow in the present invention. First, in the portable terminal, the target object is imaged by the CCD using the image sensor at 501 and the captured image is displayed to the operator on the display unit at 501. Thereafter, as shown in FIG. 2 and FIG. 3, character line extraction processing is performed at 502 with the nearest character from the marker as the nucleus. The extraction result obtained as a result is displayed as a rectangle in the form of a rectangle superimposed on the display unit at 503, and the process returns to 501 and repeats until the operator gives a confirmation instruction at 504.

  After the operator issues a confirmation instruction, image processing is performed using the partial image within the displayed rectangle and input to the recognition processing. The low resolution input image collected by the image sensor is subjected to high definition processing by 505, and the high definition partial image is input to the 506 character line recognition. Thereafter, the character recognition result is presented to the operator at 507 and an instruction is given. If the operator determines that the result is a valid recognition result, a confirmation instruction is given and transferred to a computer on the network by wireless communication. If the operator determines that the recognition result is not valid, the image is collected again from the image sensor at 501 and the process is performed again.

  The character line sent to the computer is received at 509, and at 510, it is determined whether the character line is a URL address or another character line. If the character line is a URL address, the home page content of the URL address is collected on the Internet at 511 and sent to the terminal at 512. Next, in 513, the character line is determined to be an E-mail address. If it is an E-mail address, a mail transmission program in which the address is inserted in the destination field is transmitted to the terminal in 514. This mail transmission program can be realized by using an application described in JAVA (registered trademark) or the like. Also, the mail transmission program once transmitted can be stored in the portable terminal and used again. If the character line is neither a URL address nor an e-mail address, it is input to a content search engine on the Internet at 515 and a search is performed using the character line as a keyword. The search result is transmitted to the terminal at 516 and presented to the operator.

  On the mobile terminal side, the content is received at 517, and the operator refers to the content at 518, and selects the desired content or sends an e-mail by a touch instruction or button operation on the display unit, thereby achieving the purpose.

  In addition, there are other possible data transfer procedures and processing sharing between the mobile terminal and the computer on the network. For example, after selecting a target character line description area on a portable terminal, only a partial image is transferred to a computer on the network, and image processing and character recognition are performed on the computer. As this transfer means, means for attaching a partial image to an e-mail and transmitting it can be considered. Based on a plurality of candidates obtained as a result of the character recognition process, content on the network is searched in the order of candidates. Searching using multiple candidates, if valid content is found, the content is sent to the terminal and displayed. Also, if no valid content is found even after searching using all candidates, the character recognition process is repeated with the parameters changed again.

  If the above processing procedure is used, the operator can refer to the desired content only by a character description area confirmation instruction. However, even if the character recognition result is incorrect, if content exists, content different from the operator's intention may be displayed.

  In addition, there may be a service for charging when transmitting / receiving data between the mobile terminal and the computer as described above. This service is provided to the operator when a company that manages computers connected to the network receives a character string as a recognition result or a partial image to be recognized from a mobile terminal and sends the content to the mobile terminal. The service fee is charged. The operator can receive the content after confirming the recognition result of the target character string, thereby enabling a stable service.

  FIG. 6 shows a conceptual diagram of the character line extraction process. First, 601 is an image collected by the camera of the portable terminal, and 602 is an object in which the target character line is described. 603 in the middle of the screen is a marker, and a process of extracting a character line written in the nearest vicinity from the position of this marker is performed. The position of the marker can be moved by the operator by touch panel or button operation. Next, 604 is an image that binarizes the input image, generates connected components that are black pixel blocks, and displays the circumscribed rectangle of each connected component. In order to increase processing speed, character line extraction processing is performed using circumscribed rectangle position information of connected components. Next, the circumscribed rectangle of the connected component at the nearest position of the marker is selected and is represented by a thick rectangle such as 605. This connected component is used as the “core” of the initial state of character line extraction. The connected component that is the “nucleus” is not displayed to the operator. Reference numeral 606 represents the progress of character line extraction, and a character line rectangle is formed by integrating neighboring connected components in the horizontal direction based on the “core” indicated by 605. Next, as a result of integration in the horizontal direction, 607 displays a character line rectangle composed of connected components.

  Reference numeral 608 denotes a display of a character line rectangle extraction result viewed from the operator, and a dotted rectangle is the character line extraction result. If the operator refers to this rectangle and the target character line correctly surrounds the rectangle, the confirmation instruction is given with the button, otherwise the terminal is moved or the marker is moved, so that the character line extraction result rectangle can be fluctuate.

  FIG. 7 shows a process flow of the character line extraction process. In the character line extraction process, first, an image is collected from the image sensor with 701, the input image is binarized with 702, and then noise is removed with 703. Thereafter, in 704, a connected component is generated from the binary image, and in 705, the nearest connected component from the marker is selected as the initial “core”. Next, in 706, a certain short-distance connected component is integrated from the connected component that becomes the “core” to form a character line rectangle, and rectangular coordinates are output. This rectangle is dynamically displayed on the display unit 707, and the process returns to 701 image collection until the user issues a confirmation instruction in 708, and the process is repeated. When the user issues a confirmation instruction, rectangular coordinates are input to the high-definition line extraction process.

  FIG. 8 shows a conceptual diagram of high-definition row image generation. Before inputting a character line image to recognition processing, pixel interpolation is performed on the target partial image to generate a high-resolution partial image, aiming to improve recognition accuracy. Reference numeral 801 denotes a display unit at the time when the operator gives an instruction to confirm the character line rectangle. Based on this character line rectangle position information, 802 images are obtained by cutting through partial images from the grayscale image obtained at the time of image collection. A pixel complementation process is performed on the grayscale partial image to obtain a high resolution image 803. When the resolution is increased by pixel interpolation in this way, a staircase shape appears in the diagonal line segment. Therefore, when the image is smoothed, noise is reduced as in 804, and the outline of the diagonal line segment is also smoothed. Next, when binarization processing is performed, a high-definition character pattern image having no noise and a clear outline as in the image 805 is generated. By inputting this high-definition row image to the recognition process, a highly accurate recognition result can be obtained even with a low-resolution imaging system.

  FIG. 9 shows a processing flow of high-definition row image generation. First, based on the coordinates of the character line rectangle determined by the character line extraction process, a partial image is cut out from the grayscale image obtained at the time of image collection in 901. The partial image is enlarged at 902 by pixel interpolation. Next, the image enlarged at 903 is smoothed, and binarized at 904. As a result, a high-definition binary image is generated.

  FIG. 10 shows a conceptual diagram of the character line recognition process. Reference numeral 1001 denotes an input character line image, and 1002 is a network expression that generates a combination of character pattern candidates for each connected component of the character line image. 1003 is a pattern in which only “h” is cut out, and 1004 is a pattern in which “ht” is combined. When a correct character pattern is selected from these combinations and a route search is performed on the network, the recognition result of the character line is obtained. The result of inputting this network pattern to the character recognition module is 1005. The 1003 “h” pattern gives a recognition result of “h” like 1006, and the recognition result for the pattern combining two characters “100” of “1004” becomes “n” like 1007. I understand that. In this way, the similarity between the character recognition results for all patterns and the template used for character recognition is recorded on the network. 1008 is a result of route search based on the similarity between the word knowledge and the recognition result for the network to which the record of the recognition result is added. In this example, the words that occur frequently as URLs (http,: //, www, co.jp, com, etc.) are route-constrained based on word knowledge, and other words are routed using the similarity of the recognition results as an index. To do. The word knowledge can be used by obtaining URL addresses from DNS (Domain Name Server), or by using geographical location information to limit existing words when signboards are used as objects. 1009 character lines are obtained by connecting the characters on the determined route, and this is the recognition result for the character line image.

  FIG. 11 shows a process flow of character line recognition. First, a combination of character pattern candidates is generated in 1101 for the input character line image to construct a network. Next, at 1102, character recognition is performed on each character pattern candidate, and the recognition result and similarity corresponding to each candidate are added as information. Next, route restriction on the network is performed using word knowledge stored in advance in 1103. Next, in 1104, the optimal route on the network is determined using the word knowledge constraint and the similarity of the character recognition result. As a result, the recognition result of the character line is obtained.

  By using the above means, the operator can select a character line of interest and obtain information content from the character recognition result.

It is a conceptual diagram of the information collection using the portable terminal with a recognition function. It is a figure of the display screen of the portable terminal using this invention. It is a figure of the display screen of the portable terminal using this invention. It is a hardware block diagram of a portable terminal. It is a figure of the processing flow of the computer in a portable terminal and a network. It is a conceptual diagram of character line extraction. It is a processing flow of character line extraction. It is a conceptual diagram of high resolution image generation. It is a processing flow of high-resolution image generation. It is a conceptual diagram of character line recognition. It is a processing flow of character line recognition.

Explanation of symbols

201: Screen display unit 202 ... Marker 203 ... Character line extraction rectangle 204 ... Character line recognition result 205 ... Screen display unit 206 ... Operation key

Claims (7)

An information search method for searching for information obtained from a mobile terminal via a network,
A first step of acquiring an image by the image acquisition means of the mobile terminal;
A second step of determining a character line description area in the acquired image;
A third step of recognizing a character line described in the determined character line description area;
A fourth step of determining whether the recognition result is a character string indicating address information or another character string;
A fifth step of searching for information in the network based on the character string when the determination result is another character string and receiving the search result at the terminal; Information search method characterized by The information search method according to claim 1, further comprising a step of displaying the recognition result and accepting a confirmed input by a user after the third step. A communication system comprising a mobile terminal and a computer connected to the mobile terminal via a network,
The mobile device
Image acquisition means and display means for the acquired image;
Means for determining a character line description area in the acquired image;
Means for recognizing character lines in the determined area;
Means for transferring the recognition result to the computer via the network,
The above calculator
Means for determining whether the acquired recognition result is a character string indicating address information or another character string;
Means for starting a search program based on the character string, searching for information in the network, and transmitting the search result to the terminal if the determination result is another character string; A communication system comprising: 4. The communication system according to claim 3, wherein the portable terminal has means for displaying the character line recognition result and receiving a confirmed input by a user. 4. The communication system according to claim 3, further comprising means for charging when the mobile terminal receives the search result transferred from the computer. A server device capable of communicating with a mobile terminal via a network,
A means for receiving a character recognition result based on the image acquired by the image acquisition means of the portable terminal, and a partial image including a character line in the image acquired by the image acquisition means of the portable terminal, At least one of means for performing character recognition based on,
Means for determining whether the recognition result is a character string indicating address information or another character string;
Means for starting a search program based on the character string, searching for information in the network, and transmitting the search result to the terminal if the determination result is another character string; A server device comprising: 7. The server device according to claim 6, further comprising means for charging when the search result is transmitted to the mobile terminal.

Images