JP2012075044A - Device, method and program for image processing - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device for processing the part in which a user is interested out of an image including characters.SOLUTION: An image processing device in an embodiment includes a capture part for capturing the image of a subject, a detection part for detecting the character region of the image, an estimation part for estimating the character recognition accuracy for the character region and a display part for superimposing and displaying incomplete region information on the image according to the character recognition accuracy of the character region.

Description

  Embodiments described herein relate generally to an image processing apparatus for capturing an image including characters, for example, a captured image such as a magazine or a hanging advertisement as a memo with an image capturing apparatus such as a camera.

  2. Description of the Related Art In recent years, digital cameras and mobile phones with camera functions have been improved in performance, miniaturized, and popularized, so that it has become easy to shoot, and subjects including characters such as documents have been taken as simple memos. When many such photographed images (that is, bitmap images) are photographed and accumulated, it becomes difficult to extract a desired bitmap image. Therefore, character recognition of a character image included in the bitmap image is performed in advance, This character recognition result is used for search.

  Then, the character string data is recognized from the character image, and the search accuracy is improved by using the importance of the character string data calculated using the display mode such as the character image size and character color. ing.

  In general, not all characters included in a bitmap image are necessary for a search, and a portion that is noticed when a user takes a picture is important. It is considered that it is contrary to the user's intention that the characters included in the part not focused on when the user takes a picture are used for the search. That is, when a certain search query character string is given, it is desirable that an image that exists in a location where the query character string is noticed at the time of photographing is output to the top of the search result.

  However, in the case where the importance is used, the portion noticed by the photographer is not necessarily a portion having a strong visual impression. For example, there are multiple articles in newspapers, and there is often no big difference in the display mode of character images of each article, but users often pay attention to only a part of the articles that are included, It cannot cope with such a case.

JP 2007-058819 A

  Provided is an image processing apparatus that processes a portion of interest from an image including characters.

  An image processing apparatus according to an embodiment includes a capturing unit that captures an image of a subject, a detection unit that detects a character region of the image, an estimation unit that estimates character recognition accuracy for the character region, and the characters in the character region A display unit that superimposes and displays on the image according to recognition accuracy is provided.

1 is a schematic configuration diagram of an image capturing device according to an embodiment. The flowchart of the whole image imaging | photography concerning this embodiment. The figure which shows an example of the layout analysis result of the character area of the bitmap image which concerns on this embodiment. The figure which shows an example of the layout analysis result of the character area of the bitmap image which concerns on this embodiment. The figure which shows the example of expression of the character area information in the case of the layout analysis result of FIG. The flowchart of the character recognition precision estimation part 130. FIG. The figure which shows an example of the expression format of character recognition accuracy information. The figure which shows an example of the incomplete area | region where the character area has been cut | disconnected. The flowchart of the incomplete area | region estimation part 140. FIG. The figure which shows an example of the incomplete area | region in the lower right of the display screen of the superimposition display part. The figure which shows an example of the expression format of imperfect area information. The figure which shows the 1st example which superimposed and displayed the estimation result of recognition accuracy on the capture image. The figure which shows the character recognition accuracy information about each area | region of FIG. The figure which shows the 2nd example which superimposed and displayed the estimation result of recognition accuracy on the capture image. The figure which shows an example of incomplete area | region information. 10 is a flowchart of area designation by the layout designation unit 160. The figure which shows an example of the display screen in the selection area | region addition mode of the layout designation | designated part 160. FIG. The figure which shows an example of the display screen in the new area addition mode of the layout designation | designated part 160. FIG. The figure which shows the example of the screen state of the superimposition display part 150 at the time of imaging | photography. The figure which shows the example of the screen state of the superimposition display part 150 at the time of imaging | photography.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a schematic configuration diagram of an image capturing apparatus 100 according to the present embodiment. The image capturing apparatus 100 includes an image capturing unit 110 that captures a bitmap image of a subject, a layout detection unit 120 that detects a character region of the captured bitmap image, and a bitmap image of the subject for each detected character region. The character recognition accuracy estimation unit 130 that estimates the character recognition accuracy of the included characters, and whether each detected character region is an incomplete region that does not fall within the capture range of the image capture unit 110 The incomplete area estimation unit 140 for estimating whether or not the character recognition accuracy information for each character area obtained from the character recognition accuracy estimation unit 130 and the incomplete area information obtained from the incomplete area estimation unit 140 A superimposed display unit 150 that superimposes and displays on the bitmap image, and each character area detected by the layout detection unit 120, A layout designation unit 160 having a UI (user interface) capable of selecting a fixed area, a character recognition accuracy obtained from the character recognition accuracy estimation unit 130, and a designation area selected by the layout designation unit 160 are input, and in response to this input For example, when the estimated value of the character recognition accuracy exceeds a predetermined threshold value, the photographing determination unit 170 that outputs a shutter signal for releasing the shutter, the shutter unit 180 that releases the shutter according to the shutter signal, and the character recognition An index weighting unit 190 that determines the weighting of the search index by using the character recognition accuracy information obtained from the accuracy estimation unit 130 and the incomplete region information obtained from the incomplete region estimation unit 140 as inputs, and the search index and subject bit An image DB (database) 200 for storing map images is provided.

  FIG. 2 is a flowchart of the entire image shooting according to this embodiment. As shown in the figure, iterative processing (S210 to S280) is performed from image capture to shooting branch, character area information by the layout detection unit 120, character recognition accuracy information by the character recognition accuracy estimation unit 130, and incomplete region. The incomplete area information by the estimation unit 140 is output each time and used for processing.

  In this embodiment, the process starts from an iterative process. In this repetitive processing part, first, the image capturing unit 110 captures an image (S210), the layout detecting unit 120 estimates the character region of the captured image, and outputs character region information (S220). Next, in-character area information extraction processing is performed (S230). Among them, the output of the character recognition accuracy information estimated by the character recognition accuracy estimation unit 130 and the incomplete region information estimated by the incomplete region estimation unit 140 are output. Next, the character area information, the character recognition accuracy information, and the incomplete area information are input, and the superimposed display unit 150 displays the character recognition accuracy and whether or not the area is incomplete (S240). Next, the designated area is received by the layout designating section 160 (S250), and the character recognition accuracy information, the incomplete area information, and the designated area are input, and the photographing determining section 170 determines whether to perform automatic photographing (S260). If automatic shooting is performed depending on whether or not automatic shooting is to be performed, the repetitive process is skipped. Also, when there is a photographing operation by the user, the part of the repetitive processing is exited (Yes in S270). After exiting the iterative process, the character recognition accuracy information and the incomplete area information are input, and the index weighting unit 190 generates a search index (S290), and the captured image and the search index are stored in the image DB (S300). .

  Hereinafter, each functional block will be described in detail.

  The image capturing unit 110 has a function of capturing a bitmap image of a subject. At this stage, the subject is not limited, and any image can be captured. In the present embodiment, it is assumed that an image is captured as digital data through a lens like a digital camera, but the present invention is not necessarily limited thereto. For example, an image may be captured from each frame of a moving image, or a part or the whole of an image displayed on a display device that can operate display content connected to the image capturing unit 110 may be captured and captured.

  The layout detection unit 120 analyzes and acquires a character region existing in the bitmap image captured by the image capture unit 110. FIG. 3 is an example of the layout analysis result of the character area of the bitmap image according to the present embodiment. In this example, a group of characters having the same character size and the same font is acquired as a rectangle (region 1 to region 6). Note that the layout analysis result is not necessarily limited in this way, and may be acquired in units of areas of articles including characters, for example, as shown in FIG. In this example, a region having the same size and the same design is obtained as a rectangle. In addition, since there is no limitation on the captured image in the image capturing unit 110, an image that does not include a character image is also input. However, when such an image is input, it is output that there is no character area. .

  FIG. 5 is a representation example of character area information in the case of the layout analysis result of FIG. In this example, the vertices of the polygon are held side by side in the counterclockwise direction. In this way, by expressing each vertex of the polygon, it is possible to deal with an area having a complicated shape as well as a quadrangle.

  The character recognition accuracy estimation unit 130 calculates a predicted value of recognition accuracy when character recognition (OCR) is performed on the character image in each character region obtained by the layout detection unit 120. As an example of calculating the predicted value, there is a method of calculating from the average or minimum value of the accuracy value of the recognition result of each character output by the existing OCR. Although the OCR may actually be performed inside the device, it may be difficult to execute the OCR when the device is mounted in an environment where hardware resources are limited. In this case, only the predicted value of the character recognition accuracy when performing OCR may be calculated.

  FIG. 6 is an example of a flowchart for predicting character recognition accuracy without performing OCR. In this example, the accuracy prediction value is lowered by a predetermined constant when there is distortion, deflection, reflection, blurring or blurring at the time of photographing, which are factors that reduce character recognition accuracy in the character image (S630 to S680), and finally A predicted accuracy value is calculated (S690).

  FIG. 7 shows an example of the expression format of the output character recognition accuracy information. When predicting character recognition accuracy without performing OCR, a simple format as shown in FIG.

  In this way, the character recognition accuracy is predicted at the time of preview before photographing, and is superimposed as feedback information on the preview screen. For example, when the predicted value of the character recognition accuracy is equal to or lower than a predetermined threshold, the user is notified that the predicted value of the character recognition accuracy is low by performing a shading process or the like on the area.

  As shown in FIG. 8, the incomplete area estimation unit 140 determines whether each character area obtained by the layout detection unit 120 is an incomplete area in which the entire character area does not fall within the capture range. And output the estimation result.

  There are several possible methods for estimating whether or not the region is an incomplete region. For example, when there are many unknown words near the end of the imageable range, it is possible to estimate whether the area is an incomplete area by setting the incomplete area. In order to determine whether the word is an unknown word, there is a method in which a known word list is given in advance and whether or not a word exists in the list is checked. Alternatively, there is a method of examining by performing morphological analysis processing.

  FIG. 9 is an example of a flowchart for checking by performing a morphological analysis process. First, an image of the character area is acquired (S910), OCR is performed on the acquired character image (S920), morphological analysis is performed on the recognition result obtained by OCR (S930), and the line in contact with the imageable range frame. A morpheme existing before and after the folded portion or across the folded portion is acquired (S940). First, the number n_all of morphemes at the above-described line wrapping location is acquired (S950), and the number of morphemes n_unk that is an unknown word is acquired (S960). When n_unk / n_all is larger than the predetermined threshold t (Yes in S970), it is determined as an incomplete region (S980), and when it is equal to or smaller than the predetermined threshold t, it is determined as a complete region (S990).

  FIG. 10 is an example of a part of a bitmap image including an incomplete region in which the right frame of the character region is cut from the entire region. As a result of the morphological analysis, each morpheme such as “DV-R”, “Blue Coder”, “I”, “Building Ray Disc”, “Zalink”, “Blu-ray Disc” indicated in this figure is obtained at the end of the recordable area. These are unknown words. In the same figure, there are 12 lines of text, but since there are unknown words in the folded part of 6 lines, the ratio of the unknown word existing in the part in contact with the shootable range frame is 0.5. This value is compared with a predetermined threshold value, and when it is equal to or greater than the threshold value, it is determined as an incomplete region.

  As another method to estimate whether the area is incomplete, especially when OCR cannot be performed due to restrictions such as hardware environment, simply, if the character area is in contact with the shootable range frame, It may be an incomplete area.

  FIG. 11 shows an example of an expression format of incomplete area information. As shown in the figure, in the present embodiment, whether each area is an incomplete area is expressed in binary.

  The shutter unit 180 receives a shooting execution instruction as input and executes shooting. In the present embodiment, it is assumed that the moment of shooting is instructed as in a digital camera, but the present invention is not necessarily limited to such a configuration. For example, when capturing a certain frame from a moving image, an instruction for the frame to be captured is input and the capture is executed.

  The superimposed display unit 150 receives layout information, character recognition accuracy information, and incomplete area information, and superimposes and displays a frame line surrounding the character area on the captured image according to the layout information. Display areas darkly. What is necessary is just to display darkly, when a character recognition precision is below a predetermined threshold value. The percentage of the luminance of the part to be lowered may be specified in advance or may be set by the user. Alternatively, it may be determined to be directly proportional to the predicted value of character recognition accuracy in the character recognition accuracy information received from the character recognition accuracy estimation unit 130.

  FIG. 12 shows an example in which the recognition accuracy estimation result is superimposed on the captured image. The captured image is a bitmap image of a magazine, and a part of the character area has a reflection part.

  FIG. 13 shows character recognition accuracy information for each region in FIG. In this example, since the character recognition accuracy of the area including the reflection portion is low and the predetermined threshold value 0.6 is not reached, the area (areas 1 to 4) is displayed darkly. In addition, the area where the character image does not exist can be displayed darkly or brightly, and it is possible to make the part with good character recognition accuracy stand out by displaying it darkly, or character recognition by displaying it brightly. A portion with poor accuracy may be conspicuous or may be set by the user.

  In this embodiment, an area with low character recognition accuracy is displayed darkly, but any superposition display method may be used as long as the character recognition accuracy is good or bad. For example, the color of the frame surrounding the character area may be changed according to the character recognition accuracy, the predicted value of the character recognition accuracy may be directly displayed, or a specific recognition result may be displayed. .

  FIG. 14 is another example in which the recognition accuracy estimation result is superimposed on the captured image. In this example, the captured image is the same as that of FIG. 12 except that the captured image is a bitmap image of a magazine. However, an incomplete region (regions 1, 2, and 5) of the character region that is not entirely within the frame that can be photographed. Exists. Whether the character area is an incomplete area is received from the incomplete area estimation unit 140. This example is a result of receiving and displaying the incomplete area information shown in FIG. The lower right area (area 5) in FIG. 14 has no reflection and has good character recognition accuracy, but is dark because it is an incomplete area.

  The layout designating unit 160 superimposes and displays a region where a character image exists according to the layout information on a display screen provided with a touch panel, and selects at least one region of interest for which character recognition is desired at least accurately. One or more regions may be selected.

  FIG. 16 is a flowchart until one or more areas are designated by the layout designation unit 160. As shown in the figure, there are three modes until a region is determined, and each is a selection region addition mode (S1640) for selecting a region of interest from already displayed region candidates, which are not displayed as candidate regions. A new area addition mode (S1660) for designating an area and a candidate area deletion mode (S1680) for deleting from the added candidate area. Switching between modes may be performed from a button connected to the device, or a switching menu may be displayed on the touch panel screen to accept the switching. When the selected area is confirmed, the new area is output to the layout detection unit 120, and the selected area and the new area are output to the imaging determination unit 170. Regarding the confirmation of the designated area, it may be operated from a button connected to the device, or a confirmation menu may be displayed on the touch panel screen to accept the confirmation.

  The flow of the area designation process by the layout designation 160 shown in FIG. 16 will be described. First, the character region is superimposed and displayed on the captured image with a green frame (indicated by a dotted line in FIG. 17 for convenience) (S1610). Next, it is determined whether or not the current mode is the candidate region deletion mode (S1620). If it is not applicable, it is determined whether or not it is a new region mode (S1630). In the selection area addition mode, first, an area selected by the touch panel is added as a selection area to a candidate (S1640), and then the border color of the area is red, which is a predetermined color (in FIG. 17, for convenience, a thick solid line). Display) (S1650). In the new area designation mode, the area enclosed by the touch panel is added as a new area as a candidate (S1660), and at that time, the frame line of the area is additionally superimposed and displayed (S1670). In the candidate area deletion mode, when a region to be deleted is selected from the candidates already selected by the touch panel, the area is deleted from the candidate (S1680), and the frame line that has been superimposed is returned to the original state. (S1690).

  FIG. 17 is an example of a display screen in the selection area addition mode of the layout designation unit 160. In accordance with the layout information, it is superimposed on the captured image, and the character area is surrounded and displayed in a superimposed manner. One or more designations can be received by causing the user to touch an area of interest on the touch panel. Note that the device does not need to use a touch panel. For example, it is possible to determine which region to select by sequentially pressing buttons connected to the device. In the candidate area deletion mode, the candidate area is deleted by performing the same operation.

  FIG. 18 is an example of a display screen in the new area addition mode of the layout designation unit 160. In the present embodiment, as shown in the figure, the user is allowed to specify an area to be specified with a finger on the touch panel by an operation of surrounding with the finger. The surrounding operation is an operation of tracing with a finger on the touch panel. If the traced line is closed, the closed area is set as the input area.If the traced line is not closed, the start point and end point of the traced path are connected by a straight line, and the closed area is input. To do. Analyzes whether a character image exists in the rectangular area that includes the input area, and if found, adds the found area to the candidate, and if not found, displays on the screen that no character exists. To notify.

  The shooting determination unit 170 determines whether to perform shooting by looking at the character recognition accuracy and incomplete area information for the specified area received from the layout specifying unit 160. If it is determined that shooting is to be performed, the shooting determination unit 170 outputs a shutter signal to the shutter unit 180. To automatically shoot. In the present embodiment, it is determined that shooting is performed when the character recognition accuracy of the designated area is good and the area is not an incomplete area. Note that this automatic shooting does not necessarily require all of the above-mentioned conditions, and automatic shooting may be performed if at least the character recognition accuracy of the designated area is good or the designated area is not an incomplete area.

  In this way, at the time of preview before shooting, the user is allowed to specify a location where character recognition is desired with high accuracy, and waits until the predicted value of the character recognition accuracy at the location exceeds a predetermined threshold value. Automatically captures when the threshold is exceeded.

  The automatic shooting function may be disabled, the user may be notified that the character recognition accuracy of the designated area is good, or the designated area is not an incomplete area, and the user may press the shutter.

  The index weighting unit 190 receives character recognition accuracy and incomplete area information, and determines an index weight from these pieces of information. The device is a device for considering that the region photographed so as to improve the recognition accuracy at the time of photographing is the region noticed by the user who performed the photographing and placing the region at the top of the search result. In this embodiment, the character recognition accuracy and the incomplete area information are received, but the specified area information obtained by the layout specifying unit 160 may also be received and reflected in the search result.

  19 and 20 are examples of screen states of the superimposed display unit 150 at the time of shooting. Since the character strings “ABC” and “XYZ” exist in the subject, they are registered in the search index.

  In the example of FIG. 19, since the character string “ABC” itself can be recognized at the time of registration, there is a reflection in the area where the character string “ABC” exists, and the character recognition accuracy is low. The weight of the search index “ABC” is reduced, and the weight of the search index “XYZ” is left as it is.

  In the example of FIG. 20, the character recognition accuracy is good in both of the two character areas in which the character string “ABC” or “XYZ” exists. However, in the character area where the character string "XYZ" exists, the right side of the character area is cut off in the shootable range, so the search index "ABC" for this shot image is left as it is, and the search index "XYZ" The weight of is reduced. Accordingly, when “XYZ” is given to the search query, the captured image of FIG. 19 comes to the top of the search result, and when “ABC” is given to the search query, FIG. The captured image comes to the top of the search results.

  In this way, an image in which the search query character string exists at the location designated by the user at the time of shooting is output to the top of search results. Since it is considered that the use of characters included in a part that is not focused on when the user takes a picture is contrary to the user's intention, the predicted value of the character recognition accuracy is set to a predetermined threshold at the time of shooting. The index weighting of the characters that existed in the part that was not satisfied is set low when generating the search query. As a result, the image searched for by the location photographed by the user is output at the top of the search result.

  In the present embodiment, when a character string exists in the region of interest, the search result is arranged at the top. However, the search result is not necessarily limited to this. For example, the character string of the region not attracting attention is An index may not be created and may not be output as a search result.

  Note that the present embodiment is not limited to the above-described embodiment as it is, and can be embodied by modifying the components without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 100 Image photographing device 110 Image capture part 120 Layout detection part 130 Character recognition accuracy estimation part 140 Incomplete area estimation part 150 Superimposition display part 160 Layout designation part 170 Shooting judgment part 180 Shutter part 190 Index weighting part 200 Image DB

Claims (7)

A capture unit for capturing an image of a subject;
A detection unit for detecting a character region of the image;
An estimation unit for estimating character recognition accuracy for the character region;
An image processing apparatus comprising: a display unit that superimposes and displays on the image according to the character recognition accuracy of the character region.   The image processing apparatus according to claim 1, wherein the display unit displays a dark area of the character area where the character recognition accuracy is low. The character region further includes an incomplete region estimation unit that estimates whether the entire character region is an incomplete region that is not within the capture range of the capture unit,
The image processing apparatus according to claim 1, wherein the display unit superimposes and displays the incomplete area on the image.   The image processing apparatus according to claim 1, further comprising a determination unit that determines whether to perform shooting based on the character recognition accuracy.   The image processing apparatus according to claim 3, further comprising a determination unit that determines whether to perform imaging based on whether the region is an incomplete region. Capturing a subject image;
Detecting a character region of the image;
Estimating the character recognition accuracy of the character region;
An image processing method comprising: superimposing and displaying the character recognition accuracy of the character region on the image. Computer
Means for capturing an image of a subject;
Means for detecting a character region of the image;
Means for estimating the character recognition accuracy of the character region;
An image processing program for causing the character recognition accuracy of the character region to function as means for superimposing and displaying on the image.

Images