JP6598402B1 - Receipt and other form image automatic acquisition / reading method, program, and portable terminal device - Google Patents

Abstract

A method, a program, and a portable terminal device for easily obtaining a form image such as a receipt so as to be suitable for a character reading process. A method of acquiring a form image obtained by photographing a form so as to be suitable for character reading using a portable terminal device having a photographing means, wherein a plurality of preview images are captured during photographing of the preview image by the photographing means. A step of determining an edge that forms a form from each of the steps, a step of comparing each determined edge and determining whether or not the edge is stable, and when the edge is determined to be stable, A step of determining whether or not a predetermined area on the inside is in focus, and a step of acquiring a form image for character reading when it is determined that the focus is in focus. [Selection] Figure 13

Description

  The present invention relates to a method for acquiring a form image for reading a character using a photographing unit of a portable terminal device. In particular, the present invention relates to a method, a program, and a portable terminal device for easily acquiring a receipt image so as to be suitable for a receipt reading process having an irregular size.

  Various applications are known in which a receipt is photographed using a mobile terminal device such as a smartphone, and a result obtained by reading characters from a receipt image is used.

  When taking a receipt by an application user, for example, the receipt is placed on a table and fixed, the camera of the mobile terminal device is activated, the receipt is placed in the viewfinder, and a shutter button is pressed to prevent camera shake and blurring. This is done by pressing. The OCR (Optical Character Recognition) processing result of the photographed receipt image can be used for various applications.

  In the OCR processing, it is preferable that the number of pixels constituting one character included in the image is equal to or greater than a certain value (for example, 24 × 24 dots or more). Characters may not be read if the number of pixels constituting one character is small.

  When the entire receipt is photographed in a finder such as a smartphone, for example, if the receipt has an aspect ratio corresponding to the screen of the smartphone, it can be photographed using the screen widely, so the number of pixels constituting one character increases. Receipts generally become longer and longer as the number of purchased items increases, and the number of pixels per character decreases when an elongated receipt is placed on the screen. For this reason, with a small image size (for example, 2MP, 3MP, etc.), it is substantially difficult to recognize characters on a long receipt.

  The inventor of the present invention investigated the relationship between the number of lines printed on a receipt, the image size, and the OCR misread rate. When the number of printed lines was less than about 50 lines (short receipt), the misreading rate was low from 0 percent to around 10 to 20 percent regardless of whether the image size was 3MP (megapixel), 8MP, or 12MP. When the number of rows of receipts was about 50 or more (long receipt), the misreading rate was high at around 30 percent with 3MP, and exceeded 50 percent with about 60 or more rows. On the other hand, in 8MP and 12MP, even if the number of receipt lines is about 50 or more, the misreading rate is as low as about 10%, and the misreading rate tends to exceed 30%. About 90 lines or more. In recent years, mobile terminal devices such as smartphones have a tendency to support so-called 4K (3840 × 2160, 8MP) moving images, and thus it can be said that it has become easier to read characters on a long receipt successfully.

  However, generally, the operation of pressing the shutter button tends to cause camera shake, and if the image is out of focus, characters may not be read by OCR. Holding the terminal with one hand so as not to shake, and pressing the shutter button with the other hand, both hands are blocked. In that case, for example, it is not possible to adjust the position with a free hand so that the receipt does not become a shadow under the lighting, hold the long receipt so that it does not curl, lift it in the air or support it with a wall and shoot it Can not. If the image cannot be shot so as to be suitable for the OCR process, the application eventually returns an error, and the user is requested to capture again. Such an error is likely to be repeated, and there is a problem that the user gives up shooting and leaves the application when the error continues.

  2. Description of the Related Art Conventionally, as an image processing apparatus capable of accurately recognizing desired information from a recognition target (subject) or an image captured in a state where the camera is likely to shake, a recognition target existing in an image captured by a camera Has been proposed (Japanese Patent Laid-Open No. 2010-218061), which includes a recognition target detection unit for detecting the image and a recognition target state determination unit for determining whether or not the image area of the recognition target is in a recognizable state. Patent Document 1).

  In order to allow users of multi-function mobile devices to apply for various campaigns smoothly and quickly, when taking a long receipt, instructing the divisional shooting in the length direction of the long receipt, It has been proposed to make it impossible to release the shutter of the multifunctional portable terminal when the receipt of the multifunctional portable terminal is not focused on the receipt (Japanese Patent Laid-Open No. 2006-57676). .

  When performing OCR using a camera that requires adjustment of shooting size and focus, the keyword part is determined in real time during continuous shooting as an information processing device, program, and control method that eliminates the trouble of re-shooting many times. In addition, by presenting the candidates in the vicinity in a selectable format, it is proposed that the user can easily select and capture only the necessary text and eliminate the trouble of repeating the shooting that occurred at the time of still image shooting. (Japanese Patent Laid-Open No. 2017-1117027: Patent Document 3).

  As a method of automatically scraping a part of an image from a paper image obtained by photographing a paper surface, among a plurality of partition edges related to the paper image, upper and lower horizontal edges extending in the horizontal direction and left and right vertical extending in the vertical direction A section of a paper image is detected based on a set of edges composed of edges, a map area is determined based on a set of edges corresponding to each section, and a plurality of pixels constituting the map area are associated with a map number. It has been proposed that a map image is generated and a paper image is divided into a scraping area and a non-scraping area (Japanese Patent Laid-Open No. 2018-97551: Patent Document 4).

JP 2010-218061 A JP, 2006-57676, A JP 2017-1117027 A JP-A-2018-97551

  The prior art uses a guide frame or the like for a form to be photographed to determine the focus state and whether or not it can be recognized. For this reason, when taking an image with an indeterminate overall size in a finder like a receipt, the guide cannot be set in advance, and it cannot be determined whether or not it can be recognized. It was cumbersome and impractical to divide and photograph long receipts. Further, it is necessary to perform OCR processing of keywords during continuous shooting, and it has not been possible to determine in advance whether or not it is suitable for OCR before performing time-consuming OCR.

  In view of the above, an object of the present invention is to provide a method for acquiring a form image so as to be suitable for character reading processing before performing character reading processing such as OCR. In particular, a method, a program for automatically acquiring an appropriate image without requiring the user to press a shutter button or determine a focus state when shooting an unsized receipt without dividing it, And an apparatus. It is another object of the present invention to provide a method, a program, and an apparatus that can automatically perform character recognition by simply holding a camera of a mobile terminal device such as a smartphone over a form such as a receipt so as to improve operability.

  One aspect of the present invention for solving the above-described problem is a method of acquiring a form image obtained by photographing a form so as to be suitable for character reading using a portable terminal device having a photographing unit, the preview by the photographing unit While capturing an image, the step of determining an edge related to a form from each of a plurality of preview images, the step of comparing each determined edge, determining whether the edge is stable, and the edge being stable Determining whether or not a predetermined area inside the edge is in focus when determined, and acquiring a form image for character reading when it is determined to be in focus ,including.

  If the image is simply acquired when the edge is stable, the image may not be in focus. If the image is acquired only when the image is in focus, for example, it has camera shake prevention and sports shooting functions and is moving. In a mobile terminal device that focuses on an object, the receipt of the object may not be shown. According to the present invention, since the image is acquired by determining the stability of the edge and determining the focus inside the edge, an image suitable for character recognition can be automatically acquired.

  The form includes a receipt. According to the present invention, even for an irregular form such as a receipt, a form image suitable for character reading can be acquired by determining an edge and determining a focus.

  The edges of the form consist of top and bottom edges and left and right edges. The top and bottom edges and left and right edges block the analysis image for analyzing the preview image in the horizontal and vertical directions, and detect a maximum of eight white background parts. To be determined. By doing in this way, an edge can be determined even if the background of a form such as a receipt is various.

  Whether or not the image is in focus is determined by embossing the grayscale image of the preview image and obtaining the number of pixels (histogram) for each luminance value in a predetermined area. In this way, the focus state can be easily determined without imposing a load on the CPU.

  The form image acquisition method according to the present invention further requests an autofocus (AF) operation of the photographing means when the photographing means is activated, when an edge is not determined, and / or when it is determined that the focus is not achieved. Includes steps. In the present invention, it is preferable that the focus timing is controlled in order to obtain an image suitable for character reading. It is considered that it is easier to acquire an image suitable for character reading by appropriately performing one-shot AF.

  When the edge is determined, it is preferable to display the determined edge on the screen of the mobile terminal device as a guide. By doing so, it is considered that the user can hold the camera over and the edge becomes more stable.

  The form image acquisition method according to the present invention further includes a step of reading a character by clipping a region to be read in the acquired form image based on an edge and / or a white background portion. In this way, it is possible to read a character with high accuracy by removing a background that is not a character reading target without setting a character reading range in the image in advance. It is automatically performed from image acquisition to character reading, and an application with high operability can be realized.

  Another aspect of the present invention is a program that causes a computer of a mobile terminal device to execute any of the methods described above.

  Another aspect of the present invention is a portable terminal device for acquiring a form image over a form, and a plurality of previews during photographing of a preview image by the photographing means and photographing means for photographing the form. When it is determined that the edge is determined to be stable, edge determination means for determining the edge of the form from each of the images, edge stability determination means for comparing each determined edge and determining whether the edge is stable In addition, a focus determination unit that determines whether or not a predetermined area inside the edge is in focus, and an image acquisition unit that acquires a form image when it is determined that the focus is in focus.

  According to the present invention, a preview image suitable for character reading is automatically obtained from a preview image such as 4K or 8K without requiring the user to press the shutter button or confirm the focus state. can do. Regardless of the length of the receipt, the user can acquire an image suitable for character reading such as OCR by simply holding the camera such as a smartphone with one hand so that it fits on the screen, improving the operability of the application be able to. The user does not leave the application, and the character reading result can be used in various applications.

FIG. 1 is a functional block diagram of a portable terminal device according to the present invention. FIG. 2 schematically illustrates image blocking. FIG. 3 schematically shows a preview image obtained by photographing a receipt. FIG. 4 schematically shows detection of a vertical blank by horizontal blocking according to the present invention. FIG. 5 schematically shows detection of a horizontal blank by vertical blocking according to the present invention. FIG. 6 schematically shows determination of the first edge and determination of the character reading target range according to the present invention. FIG. 7 schematically shows the relationship between a block, a blank, and a rectangle according to an embodiment of the present invention. FIG. 8 schematically shows a preview image obtained by photographing a receipt. FIG. 9 schematically shows detection of a horizontal space by horizontal blocking according to the present invention. FIG. 10 schematically shows detection of a vertical space by vertical blocking according to the present invention. FIG. 11 schematically shows determination of the second edge and determination of the character reading target range according to the present invention. FIG. 12 is a focus histogram according to one embodiment of the present invention. FIG. 13 is a flowchart of a form image automatic acquisition / reading process according to an embodiment of the present invention.

  Various features of the present invention will now be described with reference to the drawings, together with preferred embodiments not intended to limit the invention. The drawings are simplified and schematic for illustrative purposes.

  FIG. 1 schematically shows a configuration of a portable terminal device 100 including a form automatic reading application program according to the present invention. The mobile terminal device 100 may be a small computer device such as a smartphone with a built-in camera, a personal digital assistant (PDA), or a tablet PC. A known portable terminal device 100 is mounted with a CPU, RAM, ROM, hard disk, and the like, and can execute various languages by executing a programming language under the control of an appropriate operating system (OS).

  According to the present invention, the mobile terminal device 100 includes a photographing unit 1, an input / output unit 2, a control unit 3, a storage unit 4, an edge determination unit 5, an edge stability determination unit 6, a focus determination unit 7, an image acquisition unit 8, and a reading unit. 9 is provided.

  The photographing means 1 is composed of a digital camera mounted on the portable terminal device 100, and can photograph a form such as a receipt, which is a photographing object, and convert it into digital image data. When the photographing means 1 is activated, generally, the photographing of the preview image of 10 to 30 frames per second can be continuously performed. The photographing means 10 can execute an autofocus (AF) operation. The form preview image is generally a color image and may have various sizes.

  The input / output means 2 is composed of a known touch panel or the like. For example, the processing operation start command of the application program according to the present invention can be issued on the screen displayed on the touch panel. For example, when the icon on the screen is tapped, the camera is activated, and the user performs from the acquisition of the image to the reading of the character while holding the portable terminal device over the receipt. May be displayed to accept user input. Further, the input / output means 2 can continuously display the preview images with a predetermined resolution, and the form edges detected as described later can be displayed superimposed on the preview image.

  The control means 3 includes an electronic device, an electronic circuit, and / or a program for controlling each means so as to execute the form image automatic acquisition / reading process according to the present invention. The control unit 3 can control the imaging unit 1 to activate the imaging unit 1 and execute the one-shot AF operation at an appropriate timing. Control can be performed so that images continuously photographed by the photographing means 1 are accepted as images for analysis such as edge detection at an appropriate timing. As an appropriate timing, for example, the preview image is skipped until the one-shot focus operation ends, and the preview image is accepted for analysis when the focus operation ends. Further, it is possible to control the analysis result or the like to be appropriately displayed on the input / output means 2.

  The storage unit 4 is mainly composed of a memory such as a flash memory, a hard disk (HDD), a RAM, and a ROM. The storage unit 4 stores an application program related to the form image automatic acquisition / reading process according to the present invention, and can store an image being processed, an analysis result, a final character reading result, and the like.

  The edge determination means 5 includes an electronic device, an electronic circuit, and / or a program for detecting and determining the edge of the receipt included in the received preview image.

  Edge detection / determination in the present invention has the following advantages. A general OCR engine provides a tool for setting a range for each format, where and where a form image is read. However, when a receipt image obtained by photographing an irregular receipt is acquired and character reading is performed, it is not possible to set a range in the image to be subjected to OCR processing in advance. If you try to process the entire form image with OCR, depending on the shape of the receipt, the proportion of the background in the image will increase, exceeding the number of characters that can be deciphered by processing the background as characters in OCR, etc. . By performing edge detection / determination as in the present invention, an unnecessary background can be removed from an image obtained by capturing an irregular receipt, and the image can be read successfully without setting a target range for reading in advance. Can do. Since it is possible to determine the focus of the inner portion of the edge, that is, the portion where the receipt is reflected, an image suitable for the OCR processing of the receipt can be acquired in advance.

  In addition, when a user photographs a receipt, the background of the receipt may be various, and may have a pattern (table grain or the like) or may be white as the receipt. According to the edge detection according to the present invention, the edge of a receipt can be detected corresponding to various backgrounds. Hereinafter, a form may be described as a receipt, but is not intended to be limited to a receipt. The present invention can be applied to various printed matters such as business cards, cards, tickets, receipts, hospital medical records, school prints printed in black and white, and black and white margins on the white background. .

  With reference to FIGS. 2 to 11, the edge determination of one embodiment according to the present invention will be described.

Initially, the edge determination unit 5, appropriately scaled preview image, trimmed, Two and value, and / or by running a filter, it is possible to generate an analysis image I _A for the detection of edges. To detect an edge, analyzing the image I _A is blocked. FIGS. 2 (a) represents the horizontal (width W) direction of the block of the analysis image _{I A,} FIG. 2 (b) shows the vertical (height H) direction of the block of analyzing the image _{I A.} Each block size is represented by Δw and Δh. Δw and Δh are preferably integer pixel numbers larger than one pixel. Exemplary Δw (Δh) is 2, 4, 6, 8, 10, 15 (pixel), and the like, but is not limited thereto.

  In the horizontal blocking, for example, when Δw = 10 pixels, all the 10 pixels in the horizontal direction in the block are white pixels and become white pixels, and one or more of the 10 horizontal pixels in the block Is a black pixel, all the pixels in the block are black pixels. Similarly, in the block formation in the vertical direction, for example, when Δh = 10 pixels, all the 10 pixels in the vertical direction in the block become white pixels and become white pixels, and 10 pixels in the vertical direction in the block When one or more of them are black pixels, all the pixels in the block are black pixels. By doing so, it becomes easy to detect a white background extending in the horizontal direction or the vertical direction. In the present specification, a continuous white pixel region in the same block may be referred to as a blank or a rectangle, and a series of blanks or rectangles connected between different blocks may be referred to as a space.

  Detection of the first edge of the receipt 200 according to one embodiment is described with reference to FIGS. FIG. 3 schematically shows a preview image I taken of the receipt 200 with a white background.

Figure 4 shows a blank of a detection in the analysis image I _A preview images I schematically. The analysis image I _A at appropriate filter, the black printing portion and the like in the receipt, as easier to distinguish from the surrounding white background, it may be extended than the actual printing portion (not shown). Regions for receipt 200 _A is detected in the analysis image _{I A.} The horizontal direction of the block of analyzing the image I _A (FIG. 4 (a)), the vertical direction of the blank is detected, a predetermined or longer among the detected blank (e.g., more than half of the height of the image) The blanks B1 to B11 that are included can be detected as edge candidates in the vertical direction (FIG. 4B).

Similarly with reference to FIG. 5, by the block of vertical analysis image I _A (FIG. 5 (a)), is detected horizontal blanking, the detected constant over the length of the blank (for example, Blanks B12 to B22 having half or more of the width of the image can be detected as horizontal edge candidates (FIG. 5B).

With reference to FIG. 6, the determination of the edge and the determination of the target range of the character reading process such as OCR are schematically shown. First, it is assumed that the receipt part (200 _A ) includes the center C of the image in the analysis image I _A , and the blank is assigned to any one of the upper side, the lower side, the left side, and the right side with respect to the position of the center C. The blanks sorted in order from the top, bottom, left, and right of the image are examined, and the four blanks closest to the image center C (B5, B6, B18, and B19 indicated by broken lines in the figure) have edges. It is detected as a blank to represent (FIG. 6 (a)). The rectangular four sides e1, e2, e3, and e4 defined by these blanks can be detected.

  Referring to FIG. 6B, the edges E1, E2, E3, and E4 related to the receipt 200 (shown by a broken line) are associated with the four sides e1, e2, e3, and e4 corresponding to the coordinate system of the original preview image I. E4 is determined. The X coordinate and Y coordinate of the four vertices of the rectangle defined by the edges E1, E2, E3, and E4 are stored as the vertex coordinates of the edge with the upper left corner of the image as the origin (0, 0). Used to determine edge stability. Further, the region R is determined as the OCR reading target range. An exemplary region R is a rectangular inner portion defined by edges E1, E2, E3, and E4. Region R corresponds to the character region of the receipt 200. When the preview image I is acquired as an image for character reading, for example, an image in which the outside of the region R is filled with black so as not to be an OCR processing target (gray colored portion in FIG. 6B). Can be generated (clipping). By doing so, even if the range of the receipt 200 is not set as the OCR reading range in advance, a necessary range can be set as an OCR processing target, and character recognition can be performed with high accuracy.

  Subsequently, detection of the second edge will be described.

  First, FIG. 7 schematically shows a relationship between a blank and a rectangle in one block horizontally formed with a block size Δw (= 6 pixels). Here, the blank has a size of Δw × 1 pixel (1 × Δh pixel in the case of blocking in the vertical direction), and the rectangle is made up of continuous blanks in the block. In the illustrated example, the rectangle r1 is composed of 8 blanks (6 × 8 pixels), and the rectangle r2 is composed of 4 blanks (6 × 4 pixels). These white background rectangles may store the upper right and lower left coordinates (raster data) in the storage means 4 respectively. A space is detected by examining the connection between different blocks of these rectangles. An exemplary space detection is described in US Pat. No. 6,057,056, but does not state that edge detection in the present invention is known.

  FIG. 8 schematically shows a preview image I obtained by photographing the receipt 200 with a patterned (not white) background.

Figure 9 shows the detection of space in the analysis image I _A preview images I schematically. In the analysis image I _A, by binarization and appropriate filter, the black portion of the background, and black printed portion in the receipt, as easier to distinguish from the white background of the periphery of the receipt, without prior extended can (shown ). A white rectangle for each block is detected by horizontal blocking (FIG. 9A). A series of rectangles having a certain length or more in the horizontal direction is detected by examining the connection between adjacent blocks of rectangles having a certain range of size (indicated by a solid line in FIG. 9B). Multiple rectangles). A space is determined based on a series of detected rectangles. In the example shown in the figure, a series of rectangles is determined as the spaces S1 and S2 as they are (same as gray colored portions). In general, in a series of rectangles, the form is inclined with respect to the vertical and horizontal sides of the image (skew), and in many cases, the position and size of the rectangle are not constant between blocks due to various print patterns. Therefore, the space may be determined to be a substantially rectangular or substantially parallelogram-shaped region in which the position and size of each of the series of rectangles are adjusted so as to be suitable for detection of the margin at the periphery of the receipt.

  Similarly, FIG. 10 schematically shows detection of a vertical space by vertical blocking (FIG. 10A). A rectangle having a certain range of size is detected for each block, and a series of rectangles having a certain length or more in the vertical direction is detected by checking the vertical connection of the rectangles of adjacent blocks (FIG. 10). (B) A plurality of rectangles represented by solid lines). As shown in the figure, the size of each of the series of rectangles varies between blocks, and the center position of the rectangle is not necessarily constant. In the example shown in the drawing, a range in which the position and size are common between a series of rectangles can be determined as the spaces S3 and S4 (same as gray colored portions). The space may not be determined as a common part of the series of rectangles, and may be determined as an area related to the series of rectangles so as to meet the purpose of detecting margins at the periphery of the receipt.

  FIG. 11 schematically illustrates receipt edge determination based on the detected space. Referring to FIG. 11A, the upper and lower edges of the receipt are detected based on the positional relationship from the center C of the image of the space (represented by a dotted line in the figure) detected by horizontal blocking. The The space above the image center and closest to the center C represents the upper edge of the receipt, and the space below the image center C and closest to the center C represents the lower edge of the receipt. Similarly, the left and right edges of the receipt are detected based on the positional relationship from the image center C of the space (represented by a dotted line in the figure) detected by blocking in the vertical direction. That is, the space that is to the left of the image center C and closest to the center C represents the left edge of the receipt, and the space that is to the right of the image center C and closest to the center represents the right edge of the receipt. By doing in this way, an appropriate space can be selected from the space where many can be detected depending on the background. For each of up to four detected spaces, e5, e6, e7, and e8, which can be the four sides of the rectangle, are determined. e5, e6, e7, and e8 may be a horizontal straight line and a vertical straight line equally dividing the area of each space. The determination of e5, e6, e7, and e8 is not limited to this, and may be a straight line closest to / far from the center C of the space. Also, e5, e6, e7, and e8 may be a horizontal straight line and a vertical straight line arbitrarily determined based on the space.

  Referring to FIG. 11 (b), by matching the straight lines e5, e6, e7, and e8 determined based on the space to the original image I so as to define a rectangle, the edge E5 for the receipt 200, E6, E7, and E8 are determined. Four vertex coordinates of a rectangle having the edges E5, E6, E7, and E8 as four sides are stored in the storage unit 4 as the vertex coordinates of the edge. Clipping can be done based on space. By filling the portions corresponding to the outside of the space (or each of the rectangles constituting the space) and the four corners of the image so that they are not subject to OCR processing (the gray colored portion in FIG. 11B), character reading A target region R ′ can be defined. In this way, even if the entire image I is subjected to OCR processing, the number of characters that can be read is not exceeded, and only the portion related to the receipt 200, that is, the portion corresponding to the character region of the receipt 200 is successfully read. can do.

  In view of the various backgrounds when the user captures a receipt, the present invention preferably combines the first detection and the second detection. That is, a maximum of four blanks representing edges are detected by the first detection, and a maximum of four spaces representing edges are detected by the second detection, and the edges of the receipt are determined based on these. For example, when an upper blank and an upper space are detected, one closer to the center C, one longer, or any one can be selected, or four edges can be determined by combining both. Depending on the background, only one of a blank or a space representing the edge is detected. Therefore, by detecting both, the edge can be appropriately determined in any background. If no edge is detected in both the first and second detections, the periphery of the image can be determined as the edge.

  The edge stability determination means 6 (FIG. 1) includes an electronic device, an electronic circuit, and / or a program for determining whether the edge determined as described above is stable.

  The edge stability determination means 6 compares, for example, the four vertex coordinates of the edge determined by the analysis of the first preview image and the four vertex coordinates of the edge determined by the analysis of the second preview image. To determine whether the edge is stable. For example, the minimum X coordinate values of each of the first four vertex coordinates and the second four vertex coordinates, the maximum X coordinate values, the minimum Y coordinate values, and the maximum Y coordinate value. By comparing each other, it can be determined that the edge is stable when the difference between them is not more than a certain value. By detecting the stability of the edge in this way, it is possible to detect a state in which the target receipt is determined in the finder and there is little camera shake.

  The edge stability determination unit 6 may compare two edges detected from the first preview image shot at time (t−1) and the second preview image shot at time t. The edge stability determination unit 6 may compare any N (≧ 2) times of edges to determine whether the edge is stable. When it is determined to be stable, N latest preview images (for example, time t) are used for the subsequent focus determination processing.

  As described above, the receipt edge is determined based on the detection of the white background. Even if the camera is not in focus, the edge is detected and determined. Even if an edge is simply detected, the image may not be in focus, and even if OCR processing is performed on such an image, characters may not be successfully read.

  When it is determined that the edge is stable, the focus determination unit 7 (FIG. 1) includes an electronic device, an electronic circuit, and an electronic device for determining whether or not the preview image reading target range where the edge is stable is in focus. / Or consists of programs.

  The focus determination unit 7 generates an embossed image obtained by embossing the grayscale image of the preview image. In embossing, “p = −s + p + 128” is calculated using the pixel (p) and the upper left pixel (s). If the difference in brightness between the pixels (p) and (s) is small, it approaches “128”, and if the difference in brightness is large, it becomes “128 ± n” according to the size (n). That is, embossing has the effect of raising the difference in lightness and the effect of making the same lightness an intermediate color. In the receipt image, the inventor of the present invention has many pixels close to “128” if the image is not in focus, “128 ± n” pixels increase if the image is in focus, and if the number of luminances, Focusing on the fact that it is not affected by the size (number of pixels), it was configured to easily perform focus determination using this.

  FIG. 12 is a focus histogram, where the vertical axis represents the number of pixels (0 to 300) and the horizontal axis represents the luminance (0 to 255). For example, the focus determination unit 7 is configured to detect an embossed image on a rectangle (for example, 300 to 400 × 300 to 400 pixels) with a certain ratio of the size of the edge rectangle inside the detected edge (rectangle). By examining the histogram, it can be determined whether or not the subject is in focus, and if it is not in focus, how much is out of focus (focus level). The position, shape, and size of a certain percentage of rectangles may be changed according to the characteristics of the form.

  Illustratively, at the focus level F1 (shown by a dotted line) in FIG. 12, the luminance is concentrated at 128 in the embossed image. For example, when the luminance range (difference) in which the number of pixels having the respective luminances in the direction larger and smaller than 128 is a certain number or more is less than 60, “the luminance is concentrated at 128”, that is, Is not suitable at all. " At the focus level F2 (indicated by a two-dot chain line), the focus level is still around 128, and it can be determined that “out of focus”. The focus level F3 (indicated by the alternate long and short dash line) has a slight difference in brightness, and it can be determined that “the focus is a little in focus / a little camera shake”. The focus level F4 (indicated by a broken line) may be determined as “in focus / motion”. At the focus level F5 (indicated by the solid line), it can be seen that the luminance is sufficiently dispersed and there is a large difference in brightness. For example, when the luminance range (difference) in which the number of pixels having the respective luminances in the direction larger and smaller than the luminance 128 is a certain number or more is 180 or more, “the luminance is sufficiently dispersed”, ie, It can be determined that “in focus”.

  The image acquisition means 8 (FIG. 1) is an electronic device for acquiring a preview image determined to have a stable edge and further determined to be in focus as a character reading image, It consists of electronic circuits and / or programs.

  Alternatively, when the image size of the preview image of the mobile terminal device 100 is not sufficient, the image acquisition unit 8 automatically performs the shutter operation when it is determined that the edge is stable and it is determined that the focus is in focus. It is possible to acquire an image photographed by performing as a character reading image. The image acquisition unit 8 can perform the above clipping process on the acquired image.

  The reading unit 9 (FIG. 1) includes an electronic device, an electronic circuit, and / or a program for performing processing related to character reading for an image acquired as a character reading image. The reading unit 9 may perform a known OCR process. For example, the brightness and highlight of the acquired image are adjusted and binarized into black and white, analyzed for layout, removed ruled lines, cut out characters, Characters can be read using a pattern dictionary, a font dictionary, or the like. The read result may be stored in the storage unit 4.

  With reference to FIG. 13, a flow for automatically acquiring and reading a form image such as a receipt according to an embodiment of the present invention will be described. When a predetermined application according to the present invention is started on the mobile terminal device 100 (START), the camera (imaging means 1) is activated and a one-shot autofocus (AF) operation is executed (S301). In the present invention, in order to control the focus of the camera and capture an image at an appropriate timing, a one-shot autofocus is executed instead of the continuous (continuous focus) mode.

  When the autofocus operation is finished, a preview image continuously taken is received and analyzed (S303). If the receipt is not shown or the edge is not determined due to severe camera shake, the subsequent preview image is analyzed. When the edge is determined, it is stored in the storage means 4 (S305). The determined edge may be displayed superimposed on the preview image. By doing so, the user is aware of the edge, so the edge is easily stabilized.

  When edge stabilization is performed based on detection of N edges, it is checked whether or not the previous ((N-1)) edges to be compared have been determined. Further, an edge is determined by analyzing the preview image. For example, when comparing two edges, it is checked whether the change in the position of the latest edge and the previous edge is small, and it is determined whether the edge is stable (S307). If the edge is not stable, the analysis of the preview image may be repeated from the beginning.

  When it is determined that the edge is stable, it is determined whether or not the predetermined range inside the edge of the preview image where the edge is stable is in focus (S309).

  If the image is not in focus, the camera is requested to perform a one-shot focus operation according to the focus level (F1 to F4) (S311). For example, when the edge is stable but the focus level is F1 or F2, it is considered that the user is in the middle of searching for a receipt with the camera after startup or the camera is moved up and down to fit the receipt on the screen. Assuming that the preparation is completed, the camera is requested to perform one-shot AF, and the processing after the preview image analysis is repeated. When the edge is stable and the focus level is F3 or F4, it is considered that there are many states in which the user has finely adjusted or the camera is slightly shaken, so the processing after the preview image analysis is repeated as it is. When the state of F3 or F4 is repeated several times, it is considered that the user is not conscious, so the camera may be requested to perform AF.

  An image that is determined to have a stable edge and is in focus is acquired as an image suitable for character reading and stored in the storage unit (S313). At this time, if the resolution of the preview image is small (2MP, 3MP, etc.), even if it is taken in as it is, it cannot be read with a long receipt image, so the shutter operation may be performed automatically. In this way, even if the model of the mobile terminal device has a preview image size that is not sufficient, the receipt is fixed and the image is automatically taken in focus and an image suitable for character reading is obtained. Can be acquired.

  For the acquired image, the background is appropriately removed and character reading processing is performed (S315). For example, a screen for confirming the reading result is displayed to the user, and the reading result is saved when the "Save" button is pressed. Good (S317). As described above, the image acquisition and the character reading are automatically performed only by the user holding the camera over the receipt. The character recognition result may be transmitted to the server via communication means (not shown) and further processed on the server side. Alternatively, the acquired image (or clipping image) may be transmitted to the server by communication means (not shown), and character recognition processing may be performed on the server side. In this way, an error returned from the server to the user Is reduced. When the character reading result is different from the user's intention or the like, a series of processes after the preview image analysis may be performed in response to pressing of the “redo” button.

  According to the present invention, even for an irregular receipt or a long receipt, the user can hold the camera of the mobile terminal device with one hand so as to fit in the preview screen, without having to confirm the focus or press the shutter button. The character reading result can be automatically obtained. With easy focus determination, an image suitable for OCR processing can be acquired without performing OCR processing, and characters can be read successfully without imposing a load on the CPU. According to the present invention, the operability of various applications that use the character reading result is improved, and the user satisfaction can be improved.

  Those skilled in the art will appreciate that many different modifications are possible without departing from the spirit and aspects of the invention. Accordingly, it goes without saying that the embodiments of the present invention are merely examples, and do not limit the scope of the present invention.

DESCRIPTION OF SYMBOLS 100 Portable terminal device 1 Imaging | photography means 2 Input / output means 3 Control means 4 Storage means 5 Edge determination means 6 Edge stability determination means 7 Focus determination means 8 Image acquisition means 9 Reading means

Claims (9)

A method of obtaining a form image obtained by photographing a form so as to be suitable for character reading using a portable terminal device having photographing means,
Determining an edge relating to the form from each of a plurality of preview images during photographing of the preview image by the photographing unit;
Comparing each determined edge to determine whether the edge is stable; and
Determining whether or not the predetermined area inside the edge is in focus when it is determined that the edge is stable;
Obtaining a form image for character reading when it is determined that the image is in focus.   The form image acquisition method according to claim 1, wherein the form is a receipt. The edge consists of upper and lower edges and left and right edges,
The upper and lower edges and left and right edges are determined by blocking an analysis image for analyzing the preview image in a horizontal direction and a vertical direction and detecting a maximum of eight white background portions. The form image acquisition method according to Item 1.   2. The focus determination according to claim 1, wherein whether or not the image is in focus is determined by embossing a grayscale image of the preview image and obtaining the number of pixels for each luminance value of the predetermined area. Form image acquisition method.   2. The method according to claim 1, further comprising: requesting an autofocus operation of the photographing unit when the photographing unit is activated, when the edge is not determined, and / or when it is determined that the focus is not achieved. Form image acquisition method.   2. The form image acquisition method according to claim 1, wherein when the edge is determined, the determined edge is displayed on the screen of the portable terminal device as a guide.   The form image acquisition method according to claim 3, further comprising a step of clipping a character reading target region in the acquired form image based on the edge and / or a white background portion and reading a character. .   The program which makes the computer of the said portable terminal device perform the method described in any one of Claims 1 thru | or 7. A portable terminal device for acquiring a form image over a form,
Photographing means for photographing the form;
An edge determination unit that determines an edge related to the form from each of a plurality of preview images during shooting of the preview image by the shooting unit;
Edge stability determination means for comparing each of the determined edges and determining whether the edge is stable;
A focus determination means for determining whether or not a predetermined area inside the edge is in focus when it is determined that the edge is stable;
An image acquisition unit that acquires a form image when it is determined that the image is in focus.