JP2010103694A - Camera with translation function, and method of displaying text - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a camera with a translation function capable of quickly displaying prescribed text, and a method of displaying the text. <P>SOLUTION: The camera with a translation function includes a camera for picking up a frame image, a character extracting means for extracting characters reflected on the frame image, a character recognizing means for recognizing the extracted characters, a translating means for translating text composed of the recognized characters, a displaying means for displaying the translated text in the frame image, and a frame image analyzing means for comparing consecutively imaged frame images, calculating a difference in a position and a difference in zooming of the same object shown on the both frame images, and analyzing a difference between imaging conditions. The displaying means specifies a display position in the frame image of the translated text displayed in a frame image that is just previously imaged from the difference between the imaging conditions of the both frame images analyzed by the frame image analyzing means. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a technique for translating characters in an image into another language for display.

  When driving or walking in a foreign country, the characters on the signs and signs are often confusing and sometimes confusing. Patent Document 1 discloses a vehicle navigation system that translates characters written on signs and other objects provided along the street and informs the driver of the translated characters in order to avoid such confusion when driving a car. It is disclosed. This vehicle navigation system captures characters on signs and other objects with a camera, extracts characters from the captured images, recognizes them, and translates them into a predetermined language. Then, the translated characters are displayed together with the captured image on the display device to assist the driving of the automobile.

JP 2003-323893 A

  The vehicle navigation system disclosed in Patent Document 1 performs character extraction from a captured image, translation, and setting of the display position of the translated character for each frame that is continuously captured. For this reason, the load on each device constituting the vehicle navigation system is heavy, and it takes time to display the translated characters.

  The present invention has been made to solve the above-described problems, and an object thereof is to provide a camera with a translation function capable of quickly displaying a predetermined text and a text display method.

  In order to achieve the above object, a camera with a translation function according to the present invention includes an image capturing unit that continuously captures a plurality of frame images, and a character extraction unit that extracts characters from the frame images captured by the image capturing unit. A character recognition means for recognizing the character extracted by the character extraction means; a translation means for translating a text composed of the characters recognized by the character recognition means into text of a predetermined language; Display means for combining and displaying the text translated by the means; and frame image analysis means for obtaining a difference in imaging condition between the frame image captured by the image capturing means and the frame image captured immediately before the frame image, The display means translates the difference between the imaging conditions of both frame images analyzed by the frame image analysis means and the frame image captured immediately before And a position of the text was to identify the display position of the text translated in the frame image captured immediately thereafter, characterized by combining the text and the translated frame image captured immediately after the.

  The character recognizing unit further specifies a size of a display area of a text captured in the frame image captured by the image capturing unit, and the display unit is configured to determine an imaging condition of both frame images analyzed by the frame image analyzing unit. From the difference and the dimension of the display area specified in the frame image captured immediately before, the dimension of the display area of the translated text in the frame image captured immediately after that may be determined.

  The display means may display the translated text in a display area determined by the display means by adjusting a display size of the text translated by the translation means.

  Further, the frame image analysis unit detects a characteristic pattern common to the frame image captured by the image capturing unit and the frame image captured immediately before the frame image, and the difference in display position of the pattern detected in both the frame images From this, the difference between the imaging conditions in both frame images may be obtained.

  In order to achieve the above object, a text display method according to the present invention includes an imaging step of continuously capturing a plurality of frame images, and a display step of continuously displaying the frame images to which a predetermined text is added. And the frame image captured in the imaging step and the frame image captured immediately before it are calculated to calculate the difference in position and zoom of the same subject imaged in both frame images. A frame image analysis step for analyzing a difference between image capturing conditions, and displaying the difference between the image capturing conditions of both frame images analyzed in the frame image analyzing step and the frame image captured immediately before in the display step From the translated text position, the display position of the translated text in the frame image taken immediately after that is specified, and the translated text Characterized in that it added to the frame image of the captured strike immediately after the.

  Further, a character extraction step for extracting characters from the frame image captured in the imaging step, a character recognition step for recognizing characters extracted in the character extraction step, and a text composed of the characters recognized in the character recognition step A translation step of translating into a predetermined language, and the text displayed in the display step may be a text translated in the translation step.

  Further, in the character recognition step, the size of the display area of the text shown in the frame image captured in the imaging step is specified, and in the display step, the imaging conditions of both frame images analyzed in the frame image analysis step The size of the display area of the translated text in the frame image captured immediately after that may be determined from the difference between the two and the size of the display area specified in the frame image captured immediately before.

  In the display step, the translated text may be displayed in the display area determined in the display step by adjusting the display size of the text translated in the translation step.

  According to the present invention, it is possible to quickly display a predetermined text.

  A camera with a translation function and a text display method according to embodiments of the present invention will be described below with reference to the drawings. As shown in FIG. 1, the camera with a translation function 1 according to the embodiment of the present invention includes an apparatus main body 10, a camera 20, an input unit 30, and a display unit 40.

  The apparatus body 10 includes a text extraction unit 11, a text reading unit 12, a translation processing unit 13, a captured image analysis unit 14, a display processing unit 15, and a database 16. With this configuration, the apparatus body 10 extracts and translates characters from the frame image captured by the camera 20, sets the display position and display dimensions of the translated characters, and displays them on the display unit 40. Hereinafter, each part which comprises the apparatus main body 10 is demonstrated in detail.

  The text extraction unit 11 stores fonts of characters (including numbers, symbols, codes, and the like) in advance, and extracts characters that appear in a frame image captured by the camera 20 by a technique such as pattern matching.

  The text reading unit 12 recognizes each extracted character as character information using a character recognition technique. Here, the character recognition technique refers to, for example, searching for a match between the characteristics of characters constituting the extracted text and the characteristics of the characters stored in the database 16 (or a template of the characters themselves), and storing them in the matched database 16. The technology to recognize as a letter.

  The translation processing unit 13 translates text composed of characters recognized by the text reading unit 12 into a predetermined language. For example, the translation processing unit 13 translates the text into a predetermined language by searching the database 16 storing a large number of Japanese words and words in a plurality of languages corresponding to the words.

  The captured image analysis unit 14 analyzes a difference in imaging conditions between different frame images continuously captured by the camera 20 by a pattern matching method. Here, the difference in imaging conditions refers to a difference in composition and zoom between different frame images.

  The analysis contents by the captured image analysis unit 14 will be described in detail. First, the captured image analysis unit 14 captures a frame image to be subjected to translation processing (hereinafter referred to as a target frame image) and a frame image immediately before the target frame image. An analysis based on a so-called pattern matching method for finding a pattern common to a frame image (hereinafter referred to as a previous frame image) is executed. This pattern matching method detects, for example, a characteristic pattern in the previous frame image (for example, a region where the saturation changes significantly), and then matches the pattern detected in the previous frame image in the target frame image. It is an image analysis technique to find out.

  As shown in FIG. 2, the captured image analysis unit 14 detects characteristic patterns a1 (4, 3) and b1 (0, 0) in the previous frame image 41, and matches the pattern a1 in the target frame image 42. Assume that a1 ′ (6, 6) and a pattern b1 ′ (−2, 0) matching the pattern b1 are detected. Then, the captured image analysis unit 14 calculates the distance “5” between the pattern a1 and the pattern b1 and the distance “10” between the pattern a1 ′ and the pattern b1 ′ from the coordinate values of each pattern. Subsequently, the captured image analysis unit 14 compares the calculated distances “5” and “10”, and the target frame image 42 is captured by zooming in twice from the state of capturing the previous frame image 41. Recognize it as an image. Note that the X and Y coordinates shown in the figure have the imaging centers of the previous frame image 41 and the target frame image 42 as the origin.

  In addition, the captured image analysis unit 14 calculates the difference between the subject positions that are commonly projected in the previous frame image 41 and the target frame image 42. To explain using the image shown in FIG. 2, first, the captured image analysis unit 14 eliminates the zoom difference between the previous frame image 41 and the target frame image 42, and the pattern a <b> 1 of the previous frame image 41 and Each of the coordinate values of the pattern b1 is doubled. Next, the captured image analysis unit 14 compares the multiplied coordinate values with the coordinate values of the pattern a1 ′ and the pattern b1 ′ of the target frame image 42. That is, the pattern a1 (8, 6) and the pattern b1 (0, 0), which are coordinates obtained by doubling the coordinate value of each pattern of the previous frame image 41, and the pattern a1 ′ (6, 6) of the target frame image 42 and Comparing the pattern b1 ′ (−2, 0), the X coordinates of the pattern a1 ′ and the pattern b1 ′ are 2 smaller than the X coordinates of the pattern a1 and the pattern b1, respectively. Accordingly, the captured image analysis unit 14 recognizes that the subject displayed in the target frame image 42 is at a position moved −2 in the X direction with respect to the same subject displayed in the previous frame image 41.

  In addition, the captured image analysis unit 14 displays a new subject in the target frame image 42 based on the difference in the same subject position and the difference in zoom between the previous frame image 41 and the target frame image 42 obtained above. Find the area.

The display processing unit 15 sets the display position and display dimensions of the translated text, and displays the translated text together with the target frame image 42 on the display unit 40.
Specifically, the display processing unit 15 specifies the position of the extracted text in the frame image and its display area, and rewrites the extracted text into the translated text. At this time, the size of the translated text is appropriately changed or a line feed or the like is performed so that the translated text fits in the extracted text display area.

  For example, the camera 20 captures several to several tens of frame images per second, and transmits the frame image to the apparatus main body 10 for each imaging. The camera 20 preferably has a high shutter speed and a deep depth of field in order to facilitate the extraction and recognition of the captured text.

  The input unit 30 includes various switches for the operator to perform a desired operation of the camera 1 with a translation function. For example, the input unit 30 includes an imaging switch for capturing an image with the camera 20, a language switching switch for selecting a translated language, and the like.

  The display unit 40 is a liquid crystal display mounted on the camera 1 with a translation function, for example, and displays frame images continuously captured by the camera 20 in real time. Moreover, the display part 40 rewrites the text before translation into the text after translation, and displays it.

  Next, the translated word display processing of the text captured by the camera 1 with a translation function according to the present invention will be described with reference to the flowcharts shown in FIGS. In order to simplify the description, it is assumed that the captured frame image always includes text.

  First, the camera 20 captures a frame image and transmits the captured frame image to the apparatus main body 10 (step S110).

  Next, the text extraction unit 11 extracts the text shown in the captured frame image (step S120). The method for extracting the text from the frame image is as described above.

  Subsequently, the text reading unit 12 checks whether or not the characteristics of the characters constituting the text extracted in step S120 match the characteristics of the characters stored in the database 16 (step S130).

  In step S130, when the characteristics of the characters constituting the text match the characteristics of the characters stored in the database 16 (step S130: Yes), the text reading unit 12 changes the characters constituting the text to the database whose characteristics match. It is recognized as a character stored in 16 (step S140). Further, the text reading unit 12 obtains the size of the display area of the text formed by the characters recognized in this way.

  Next, the translation processing unit 13 translates text composed of characters recognized by the text reading unit 12 into a predetermined language (step S150). For example, the translation processing unit 13 translates the text into a predetermined language by searching the database 16 storing a large number of Japanese words and words in a plurality of languages corresponding to the words.

  Next, the display processing unit 15 sets the display position and display dimensions of the translated text, and displays the translated text together with the frame image on the display unit 40 (step S160). As described above, the display processing unit 15 displays the translated text on the display unit 40 so as to fit in the display area of the text before translation.

  In step S130, if the characteristics of the characters constituting the extracted text do not match the characteristics of the characters stored in the database 16 (step S130: No), the display processing unit 15 directly displays the captured frame image on the display unit 40. Displayed (step S180). Then, the step proceeds to step S170.

  Note that the steps up to the above show the translated word display processing steps when the first frame image is captured by the camera 20. By the way, as described above, the camera 20 captures several to several tens of frame images per second. Therefore, such continuous shooting of frame images and translated word display processing are performed in step S170. Hereinafter, step S170 will be described with reference to the flowchart shown in FIG.

  First, as in step S110, the camera 20 captures a frame image and transmits the captured frame image to the apparatus main body (step S171).

  Next, the captured image analysis unit 14 executes a pattern matching method to detect a characteristic pattern common to the previous frame image 41 (imaged in step S110) and the target frame image 42 (imaged in step S171). Then, the captured image analysis unit 14 analyzes the same subject position difference and zoom difference between the previous frame image 41 and the target frame image 42 (step S172). Based on the analysis result, the captured image analysis unit 14 obtains a newly captured area in the target frame image 42. Note that these specific analysis methods are as described above.

  Next, the text extraction unit 11 executes text extraction processing in the newly imaged area obtained in step S172 (step S173).

  Subsequently, the text extraction unit 11 determines whether a new text is extracted in step S173 (step S174).

  In step S174, if the text extraction unit 11 does not extract a new text (step S174: No), the display processing unit 15 determines that the previous frame image from the difference in the imaging conditions of the consecutive different frame images obtained in step S172. The display position and text size when the text already displayed in 41 is displayed on the target frame image 42 are set. Then, the display processing unit 15 displays the translated text together with the target frame image 42 on the display unit 40 (step S175). Note that when the text already displayed in the previous frame image 41 is outside the target frame image 42, the display processing unit 15 does not display this text.

Here, the process of step S175 will be described with an example. For example, assume that the previous frame image 41 and the target frame image 42 have the relationship shown in FIG. Further, it is assumed that the text display position of the previous frame image 41 is b1 (0, 0), and the text display area 43 has a size of 2 × 4 on the XY coordinates.
First, the display processing unit 15 enlarges the text display area 43 of the previous frame image 41 to double the size in order to eliminate the zoom difference between the previous frame image 41 and the target frame image 42. Then, the size 4 × 8 on the XY coordinates of the display area 43 ′ in the target frame image 42 is set. Next, the display processing unit 15 doubles the coordinate of the text display position b1 (0,0) and subtracts the X coordinate by 2 to display the text b1 ′ (− 2,0) in the target frame image 42. Position. Subsequently, the display processing unit 15 displays the enlarged text at the display position b1 ′ (−2.0) of the target frame image 42 with the text displayed in the previous frame image 41 being twice as large.

  Subsequently, it is determined whether or not to capture a frame image again according to the operator's input or the like (step S176).

  In step S176, when a frame image is not taken (step S176: No), the continuous translation word display process is terminated, and the translation word display process shown in FIG. 3 is terminated.

  In step S176, when further frame images are to be captured (step S176: Yes), the process returns to step S171, the frame images are captured again, and the above-described steps are repeated.

  When the text extraction unit 11 extracts a new text in step S174 (step S174: Yes), the text reading unit 12 determines that the characteristics of the characters constituting the newly extracted text are the database, as in step S130. It is checked whether or not it matches the character feature stored in 16 (step S177).

In step S177, when the characteristics of the characters constituting the text match the characteristics of the characters stored in the database 16 (step S177: Yes), the text reading unit 12 changes the characters constituting the text to the database whose characteristics match. It is recognized as a character stored in 16. Then, the translation processing unit 13 translates the text composed of the recognized characters into a predetermined language (step S178), and advances the step to step S175.
Note that the text newly translated in step S178 is displayed on the display unit 40 in step S175, with the display processing unit 15 setting the display position and display dimensions in the target frame image 42.

  In step S177, if the character features constituting the text do not match the character features stored in the database 16 (step S177: No), the process proceeds to step S175. In this way, the text that has not been translated is displayed on the display unit 40 in a state of being captured by the display processing unit 15 in step S175.

  Next, schematic diagrams of frame images taken by the camera with a translation function 1 according to the present invention are shown in FIGS. 5 and 6, and how the translated text is displayed on the frame images will be described. In addition, each figure (a)-(c) arranges the frame image in order of imaging.

  As shown in FIG. 5A, the frame image 50 shows buildings 52 built around the road 51 and along both sides thereof. The building 52 is provided with a signboard 53 on which a Korean text 55 is written. Further, FIGS. 5B and 5C show the frame images 50 captured while zooming up while moving the imaging center to the left side in the figure from the state in which the frame images 50 shown in FIG. . As a matter of course, the position of the text 55 projected on the frame image 50 changes each time the image is taken, and the size is also projected larger.

  When the operator of the camera 1 with a translation function turns on the translation function into Japanese at the time of capturing a frame image, the text 55 shown in FIG. 5 is changed to the translated text 56 as shown in FIG. It is converted and displayed. For example, a case where the Hangul text 55 written in the display area 58 shown on the right side of FIG. 5A is rewritten to the translated text 56 “Sollon Tan” is taken as an example. Although the translated text 56 “Sollon Tan” has a larger number of characters than the other translated text 56 shown in FIG. 6, the display area 58 is not so large. Therefore, as shown in FIG. 6, the translated text 56 “Sollon Tan” is displayed smaller than the other translated text 56.

  In addition, as shown in FIGS. 6B and 6C, even when a difference in the position of the same subject and a difference in zoom appear in the continuously captured frame images 50, the change in these imaging conditions occurs. Based on the above, the display position and size of the translated text can be determined and displayed.

  As described above, the camera with a translation function and the text display method according to the present invention determine the display position of the text displayed in the previous frame image in the target frame image from the difference in imaging conditions obtained in the continuously captured frame images. Identify. For this reason, once extracted, recognized, and translated text from the frame image, it is possible to continue displaying the text appropriately on the display unit simply by setting the display position and display size in the captured frame image. Therefore, it is possible to quickly display the translated text, and it is possible to suppress the visual jitter that the timing between the display of the frame image and the display of the translated text is shifted.

  In addition, the camera with a translation function according to the present invention specifies the size of the display area where the text captured in the frame image is displayed, and specifies the previous frame image from the difference in imaging conditions obtained in the continuously captured frame images. The size of the target frame image in the display area is determined. Then, the camera with a translation function displays the translated text so as to be within the display area. Therefore, it becomes clear where the translated text is the translation of the marked character, and the translated text can be prevented from being displayed in an overlapping manner.

The present invention is not limited to the above-described embodiments, and various modifications and applications are possible.
For example, in the above description, the distance between two characteristic points is calculated in order to obtain the zoom difference between the previous frame image and the target frame image. For example, a characteristic pattern area common to the previous frame image and the target frame image is calculated. May be obtained, and the zoom difference may be obtained by comparing the sizes of the patterns.

  Moreover, although the case where the translated text was displayed was described in the said embodiment, as shown in FIG. 7, you may make it write together the pronunciation text 57 in Japanese with the Korean text. Also in this case, by obtaining a difference in imaging conditions between consecutively captured frame images, the display position and display size of the pronunciation text 57 can be set appropriately, and the above-described visual jitter can be suppressed. it can.

The block diagram of the camera with a translation function which concerns on embodiment of this invention. It is explanatory drawing of the frame image which attached | subjected XY coordinate, (a) is explanatory drawing of a previous frame image, (b) is explanatory drawing of an object frame image. The flowchart which showed the translation word display process. The flowchart which showed the continuous translation word display process. Schematic which arranged the picked-up image of the camera with a translation function in order of image pick-up by (a)-(c). Schematic which arranged the picked-up image which displayed the translated text in order of picking-up in (a)-(c). Schematic which arranged the picked-up image of the camera with a translation function of other embodiment in order of image pick-up (a)-(c).

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Camera with a translation function 10 Apparatus main body 11 Text extraction part 12 Text reading part 13 Translation processing part 14 Captured image analysis part 15 Display processing part 16 Database 20 Camera 30 Input part 40 Display part 41 Previous frame image 42 Target frame image 43 Display area

Claims (8)

Image capturing means for continuously capturing a plurality of frame images;
Character extraction means for extracting characters from a frame image captured by the image capturing means;
Character recognition means for recognizing the character extracted by the character extraction means;
Translation means for translating a text composed of characters recognized by the character recognition means into text in a predetermined language;
Display means for combining and displaying the text translated by the translation means on the frame image;
Frame image analysis means for obtaining a difference in imaging conditions between a frame image captured by the image capturing means and a frame image captured immediately before the frame image;
The display means translates the frame image captured immediately after the difference between the imaging conditions of the two frame images analyzed by the frame image analysis means and the position of the translated text displayed in the frame image captured immediately before. A camera with a translation function, characterized in that the display position of the translated text is specified, and the translated text is synthesized with a frame image captured immediately after the text. The character recognizing unit further specifies a size of a display area of a text captured in a frame image captured by the image capturing unit;
The display means translates the frame image captured immediately after the difference between the imaging conditions of both frame images analyzed by the frame image analysis means and the size of the display area specified in the frame image captured immediately before. The camera with a translation function according to claim 1, wherein a dimension of a text display area is determined.   3. The translation function according to claim 2, wherein the display unit adjusts a display size of the text translated by the translation unit, and displays the translated text in a display area determined by the display unit. camera.   The frame image analysis unit detects a characteristic pattern common to the frame image captured by the image capturing unit and the frame image captured immediately before the frame image, and from the difference in display position of the pattern detected in both the frame images, The camera with a translation function according to claim 1, wherein a difference in imaging conditions between both frame images is obtained. An imaging step of continuously capturing a plurality of frame images;
A display step of continuously displaying the frame image with a predetermined text added thereto;
Comparing the frame image captured in the imaging step with the frame image captured immediately before it, calculating the difference in position and zoom of the same subject imaged in both frame images, A frame image analysis step for analyzing a difference in imaging conditions,
In the display step, the translation in the frame image captured immediately after the difference between the imaging conditions of the both frame images analyzed in the frame image analysis step and the position of the translated text displayed in the frame image captured immediately before A method for displaying a text, wherein the display position of the text is specified and the translated text is added to the frame image captured immediately after the text. A character extraction step of extracting characters from the frame image captured in the imaging step;
A character recognition step for recognizing the character extracted in the character extraction step;
A translation step of translating the text composed of the characters recognized in the character recognition step into a predetermined language,
The text display method according to claim 5, wherein the text displayed in the display step is a text translated in the translation step. In the character recognition step, further specify the size of the display area of the text shown in the frame image captured in the imaging step,
In the display step, the difference between the imaging conditions of both frame images analyzed in the frame image analysis step and the size of the display area specified in the frame image captured immediately before are translated in the frame image captured immediately thereafter. The text display method according to claim 6, wherein the size of the text display area is determined.   8. The text according to claim 7, wherein the display step further includes adjusting the display size of the text translated in the translation step and displaying the translated text in the display area determined in the display step. How to display.

Images