JP7292052B2 - Image processing device, guide image generation method and program - Google Patents

Description

The present invention relates to an image processing device, a guide image generation method, and a program.

Various application procedures by using image data obtained by photographing various certificates such as driver's licenses with terminal devices such as mobile phones and smartphones, and character information obtained by character recognition of the image data are increasingly being done electronically. For example, there is a mobile terminal device described in Japanese Patent Application Laid-Open No. 2002-200313 as a device that performs character recognition by photographing with a camera of a terminal device.

In the technique disclosed in Patent Document 1, for example, character recognition is performed on a plurality of document images obtained while a document to be read is continuously photographed, and character information such as characters and values contained in the document images is detected. do. Furthermore, in the technique disclosed in Patent Document 1, character information indicating a predetermined item name is detected from among the detected character information, and recorded corresponding to the item name, for example, Character information such as names and addresses that differ for each individual is read and acquired.

Japanese Patent No. 6364051

By the way, in the case of a form larger than a driver's license, it may not be possible to capture the entire form at once with a camera such as a smartphone in order to obtain an image with the resolution required for character recognition. . For such a form, if the technology disclosed in Japanese Patent Laid-Open No. 2002-200010 is used, all the character information recorded in the form can be acquired by dividing the form into a plurality of areas and continuously photographing the form. It becomes possible to

However, if the form is divided into a plurality of areas and continuously photographed, it becomes difficult to distinguish between areas where character information has been acquired and areas where character information has not been acquired. There is a problem.

SUMMARY OF THE INVENTION The present invention has been made in view of such circumstances, and its object is to provide an area in which character information can be obtained and a An object of the present invention is to provide an image processing device, a guide image generating method, and a program that can easily distinguish between unmarked areas while photographing.

In order to solve the above problem, the image processing apparatus of the present invention includes a frame image acquisition unit that acquires frame images obtained by partially continuously photographing one form, and character recognition of character images in the frame images. The character recognition unit, the characteristic information of the entire image of the form including the entire image of the form, and the characteristic information of the frame image from which the character information is obtained by the character recognition. a guide image generating unit for generating a guide image for shooting guide by combining area information indicating an area from which character information is obtained; and a display processing unit that displays in a second area different from the first area .

In the image processing apparatus of the present invention, the entire image of the form is a template image of the form.

In the image processing apparatus according to the present invention, the overall document image is an image obtained by capturing the overall image of the document to be captured when capturing the frame image.
In the image processing apparatus of the present invention, the guide image generation unit determines whether or not the area information is combined in all of the predetermined areas in the guide image.

A guide image generation method of the present invention includes a frame image acquisition process of acquiring frame images obtained by partially continuously photographing one form, a character recognition process of performing character recognition of character images in the frame images, and and the feature information of the frame image from which the character information is obtained by the character recognition. a guide image generating process for generating a guide image for shooting guide combining area information indicating and a display processing step of displaying in a different second area .

The program of the present invention comprises: frame image obtaining means for obtaining frame images obtained by partially continuously photographing one form; character recognition means for performing character recognition of character images in the frame images; and the feature information of the frame image from which the character information is obtained by the character recognition . guide image generating means for generating a guide image for shooting guide combining area information ; displaying the frame image in a first area of a screen; and displaying the guide image in a second area different from the first area on the screen This is a program for functioning as display processing means for displaying in the area of .

According to this invention, when one form is partially continuously photographed, it is possible to easily distinguish an area in which character information has been obtained from an area in which character information has not been obtained while photographing. .

1 is a schematic block diagram showing the configuration of a form data management system according to an embodiment of the present invention; FIG. It is a figure which shows an example of the template of the form in the same embodiment. It is a figure which shows an example of the form in the same embodiment. It is a figure which shows the external appearance of the terminal device in the same embodiment. It is an example (part 1) of a frame image captured in the same embodiment. It is a figure which shows the form definition table in the same embodiment. It is a figure which shows the form information table in the same embodiment. It is a figure which shows the form information total table in the same embodiment. It is a flowchart which shows the flow of the imaging|photography process of the form by the terminal device of the same embodiment. It is a flowchart which shows the flow of the guide image generation process by the terminal device of the same embodiment. It is a figure which shows an example of the process by the matching process part in the same embodiment. FIG. 10 is a diagram showing a frame image matched with the coordinate system of the template image in the same embodiment; FIG. 10 is a diagram showing an example (part 1) of a guide image in which shaded objects are combined in the same embodiment; FIG. 10 is an example (part 2) of a frame image captured in the same embodiment; FIG. FIG. 10 is a diagram showing an example (part 2) of a guide image in which shaded objects are combined in the same embodiment; It is a flowchart which shows the flow of the combined image production|generation process by the terminal device of the same embodiment. It is a figure which shows an example of the combined image in the same embodiment. FIG. 10 is a diagram showing an example of a guide image in which a shaded object is preliminarily combined with an area that does not need to be photographed in the same embodiment.

A form data management system S including the terminal device 1 according to the embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing the configuration of a form data management system S. As shown in FIG. The form data management system S includes a terminal device 1 , a form data management device 3 , a form information server device 4 and a communication network 5 . The communication network 5 connects the terminal device 1, the form data management device 3, and the form information server device 4 to each other.

The terminal device 1 is, for example, a device such as a smartphone with a built-in camera, and is capable of executing various application programs (hereinafter referred to as AP) stored in an internal storage area. Hereinafter, the configuration in which the terminal device 1 receives a user's operation and executes the form photographing AP stored in the internal storage area will be described.

The terminal device 1 includes a storage unit 10, an operation unit 11, an imaging unit 12, a display processing unit 13, a frame image acquisition unit 14, a character recognition unit 15, a recognition data management unit 16, a communication processing unit 17, a guide image generation unit 20, and a combined image generator 21 .

The form to be photographed by the terminal device 1 includes, for example, a title (○○○○ membership application form), item names (for example, date of application, phonetic, name, etc.), and ruled lines, as shown in FIG. Templates are pre-printed. When the information of characters and numbers is recorded in the form template shown in FIG. 2, the form shown in FIG. 3, for example, is obtained. The following description assumes that some information is always written in the entry fields corresponding to all the item names of the form, and that no item name is allowed to be left blank.

FIG. 4 is an external view of the terminal device 1. A housing of the terminal device 1 is provided with a screen 50 and operation buttons 55. As shown in FIG. The operation unit 11 is connected to, for example, operation buttons 55 and a touch pad arranged on the screen 50 . The operation unit 11 detects an operation of the operation button 55 and an operation of the touch pad, and outputs instruction information according to the detected operation. For example, the operation unit 11 detects an operation of the operation button 55 by the user, and outputs, for example, instruction information for instructing the start of imaging or instruction information for instructing the end of imaging according to the state of the imaging unit 12 at the time of detection. Output to the imaging unit 12 .

The imaging unit 12 is, for example, a CCD (Charge Coupled Device) camera, and when instruction information to start imaging is given from the operation unit 11, it continuously performs imaging at a frame rate of 10 to 30 frames per second. , frame image data is obtained for each frame rate cycle. For example, FIG. 5 is an example of a frame image 61 obtained by photographing a part of the form shown in FIG. Further, when the operation unit 11 gives instruction information to end the shooting, the shooting unit 12 ends the shooting.

The display processing unit 13 has a screen 50 and divides the display area of the screen 50 into two to generate a captured image display area 51 and a guide image display area 52 . The display processing unit 13 performs so-called preview display, in which images of the object to be photographed formed on the imaging device of the CCD camera of the photographing unit 12 are sequentially displayed on the screen in the photographed image display area 51 .

The display processing unit 13 displays the guide image 53 on the guide image display area 52 . The guide image 53 is an image for shooting guide, which is obtained by marking the area where the character information is obtained in the image of the template shown in FIG. It is an image. The display processing unit 13 also displays a guide enlargement button 54 in the guide image display area 52 .

The storage unit 10 includes a template image storage unit 101, a form definition information storage unit 102, a frame image storage unit 103, a template feature information storage unit 104, a form information storage unit 105, a guide image storage unit 106, and a combined image storage unit 107. Prepare. The storage unit 10 needs to secure a storage area when the application for photographing a form is activated, and release the storage area when the application ends. Therefore, the storage unit 10 is preferably a volatile storage area such as RAM (Random Access Memory). , a nonvolatile storage area such as a flash memory or a hard disk.

The template image storage unit 101 stores information of the template image of the form shown in FIG. The form definition information storage unit 102 stores, for example, a form definition table 1021 shown in FIG. The form definition table 1021 has items of "form identification information", "item name", "item number", and "definition information". The item of "form identification information" contains information that can uniquely identify the type of form, for example, form templates with different column formats and item names. Identification information is written.

Information on the item name included in the form corresponding to the form identification information is written in the item of "item name". An item number assigned in advance to each item name is written in the "item number" item. In the item of "definition information", the format of the information recorded corresponding to the item name (hereinafter referred to as item record information), for example, whether it is written horizontally, vertically, or in one line, is displayed. Information indicating whether it is a row is written. The definition information also includes information indicating the item name and the positional relationship of the item record information.

The frame image storage unit 103 stores frame image data acquired by the frame image acquisition unit 14 from the imaging unit 12 together with frame identification information associated with each frame image.

The template feature information storage unit 104 stores feature information including feature points and feature amounts calculated by the guide image generation unit 20 based on the template image stored in the template image storage unit 101 .

The form information storage unit 105 stores a form information table having the same data format generated for each frame image. For example, the form information table 1051-1 shown in FIG. 7 is a table generated corresponding to the frame image 61 shown in FIG. In the following description, any one table stored in the form information storage unit 105 is indicated as the form information table 1051 without a branch number.

The form information table 1051 has items of "frame identification information", "item number", and "acquired information". Frame identification information associated with the frame image corresponding to the table is written in “frame identification information”. An item number corresponding to the item name information acquired by the character recognition unit 15 is written in the item “item number”. Item recording information acquired by the character recognition unit 15 is written in the item of “acquired information”. When the character recognition unit 15 has acquired the item name but has not acquired the item record information corresponding to the item name, the item of "acquired information" corresponding to the item number of the item name is blank. becomes.

The guide image storage unit 106 stores guide images. The combined image storage unit 107 stores the combined image. Here, the combined image is, for example, an image obtained by superimposing a frame image, which has undergone image conversion by the combined image generation unit 21 so as to match the size of the template image and the positions of items and ruled lines, on the template image. be.

The frame image acquisition unit 14 acquires frame images generated and output by the imaging unit 12 . When the frame image is acquired, the frame image acquisition unit 14 generates new frame identification information, associates the acquired frame image with the generated frame identification information, and writes them to the frame image storage unit 103 . For example, the frame identification information is generated by the frame image acquisition unit 14 by incrementing the value from the lowest number, for example, "1", so that the chronological order of the frame images can be known.

The character recognition unit 15 extracts images of characters and numbers (character images) from the frame image by OCR (Optical Character Recognition) processing, performs character recognition, and obtains character information indicating characters and numbers. Further, the character recognition unit 15 acquires the coordinates of the circumscribing rectangle of each character or number during character recognition.

When character recognition processing is performed, the character recognition unit 15 detects all combinations of character information and coordinate information included in the frame image at once, and determines whether or not the character recognition processing has succeeded. When the character recognition unit 15 determines that the character recognition process has succeeded, it refers to the form definition table 1021 and detects character information included in a specific coordinate range from the acquired character information and coordinate information.

Here, the specific coordinate range is, for example, in the case of horizontal writing, the range specified by the coordinate information of each character included in a character string in which the characters are arranged in one line and the spacing between each character is constant. be. The character recognition unit 15 identifies the character or character string of the item name from the character information included in the detected specific coordinate range. The character recognition unit 15 separates the character information included in the specified coordinate range including the character or character string of the specified item name into the item name information and the corresponding item record information and acquires them. For example, in the case of the form shown in FIG. 3, the character recognition unit 15 separately obtains information on the item name of "name" and item record information of "Taro Tokkyo" for the item name of "name".

The character recognition unit 15 reads the item numbers corresponding to all the acquired item names from the form definition table 1021, and associates the read item numbers with the acquired item recording information. When the character recognition unit 15 completes acquiring the item name information and the item recording information in one frame image, it generates a new form information table 1051 in the form information storage unit 105, and stores the item in the generated form information table 1051. number, item recording information corresponding to the item number, and frame identification information are written.

Further, the character recognition unit 15 deletes the generated form information table 1051 when all the item record information corresponding to the item name has not been acquired in the generated form information table 1051 . Also, the character recognition unit 15 deletes the frame image corresponding to the deleted form information table 1051 from the frame image storage unit 103 .

The guide image generation unit 20 can acquire the character information in the entire image of the form based on the feature information indicating the feature of the template image stored in the template image storage unit 101 and the feature information indicating the feature of the frame image. Generate a guide image that indicates the area. Here, the feature information is feature points and feature amounts obtained from the target image. A feature point is information indicating a characteristic point such as a corner in an image of interest, and a feature amount indicates what kind of feature the feature point of the image of interest has. Information. As an algorithm for calculating feature information, for example, KAZE (an example of an image feature extraction algorithm) is applied.

The guide image generator 20 generates guide images as follows. In other words, the guide image generation unit 20 performs a matching process of detecting pairs of portions having similar shapes based on the feature information of the template image and the feature information of the frame image.

Based on the matching result, the guide image generation unit 20 performs a process of converting the frame image so that it matches the size of the template image and the positions of the items and ruled lines. Here, the transforming process, for example, calculates a homography matrix from the matching result as follows, and performs perspective transformation for transforming the frame image based on the calculated homography matrix.

Here, the projective transformation matrix is a matrix calculated based on the coordinate information of the points paired by the matching process, and is a matrix corresponding to the parameters for perspective transformation. Perspective transformation is a transformation that rotates the frame image so that it matches the size of the template image and the positions of the items and ruled lines. It is a conversion that converts a frame image that has been enlarged or reduced into an image that can be seen when viewed from the front. Therefore, the coordinate system of the frame image after perspective transformation and the coordinate system of the template image match.

Further, the guide image generation unit 20 calculates two coordinates of the upper left and lower right of the frame image after perspective transformation, for example, using the lower left of the template image as a reference. The frame image after perspective transformation and the guide image have the same coordinate system. Therefore, the guide image generating unit 20 sets two coordinate positions of the upper left and lower right of the calculated frame image after perspective transformation in the guide image, and hatches a square so as to match the set two coordinate positions. Arrange objects and combine them with the guide image. This quadrilateral shaded object corresponds to the marking indicating the area where the character information described above is obtained. The guide image generation unit 20 generates a new guide image by combining square hatched objects.

Based on the feature information of the template image stored in the template feature information storage unit 104 and the feature information of the frame image, the combined image generation unit 21 converts the coordinate system of the frame image to that of the template image in the same manner as the guide image generation unit 20. A combined image is generated by superimposing and combining the frame image on the template image, matching the coordinate system.

The communication processing unit 17 transmits and receives information to and from devices external to the terminal device 1 , such as the form data management device 3 and the form information server device 4 , via the communication network 5 .

The recognition data management unit 16 selects information to be stored in the form data management device 3 from information stored in the frame image storage unit 103 , the form information storage unit 105 , and the combined image storage unit 107 of the storage unit 10 . The recognition data management unit 16 associates the selected information with the user identification information assigned in advance to the terminal device 1 and transmits the information to the form data management device 3 through the communication processing unit 17 . Here, the user identification information is information that individually identifies a user. 3, the user ID (IDentification) at the time of logging in may be used.

The form data management device 3 includes a recording processing section 31 and a storage section 32 . The recording processing unit 31 receives information transmitted by the recognition data management unit 16 of the terminal device 1 through the communication network 5 and writes the received information into the storage unit 32 .

The storage unit 32 stores a form information total table 321 shown in FIG. The form information total table 321 includes "user identification information", "form identification information", "combined image information", "combined image configuration information (frame identification information)", "pre-conversion frame image information", "item number", It has an item of "acquired information".

The user identification information, the form identification information, and the data of the combined image received from the recognition data management unit 16 are written in the items of "user identification information", "form identification information", and "combined image information" by the recording processing unit 31. . In the item of "combined image configuration information (frame identification information)", the recording processing unit 31 writes the frame identification information associated with the frame image received from the recognition data management unit 16 in ascending numerical order. In the item of "frame image information before conversion", the data of the frame image before perspective change corresponding to the frame identification information recorded in the "combined image configuration information (frame identification information)" is written by the recording processing unit 31. .

In the items of "item number" and "acquisition information", the item of "item number" and "acquisition information" of the form information table 1051 corresponding to the frame identification information recorded in the "combined image configuration information (frame identification information)". Information” item is written by the recording processing unit 31 .

The form information server device 4 includes, in its internal storage area, a form template image that differs for each form type, and definition information defined for each form type, for each form identification information indicating the type of form. A form definition table 1021 is stored in advance.

(Display processing of guide image by terminal device)
Next, referring to FIGS. 9 to 16, display processing for displaying a guide image of an area in which character information has been acquired in the entire area of the form in the process of continuously photographing a part of the form. explain.

In response to the user's operation, the terminal device 1 executes the form photographing AP. The operation unit 11 detects, for example, a user's operation on the touch pad, and acquires form identification information indicating the type of form selected by the user. The operation unit 11 outputs to the communication processing unit 17 instruction information for acquiring the template image corresponding to the captured form identification information and the form definition table 1021 . Upon receiving the instruction information from the operation unit 11 , the communication processing unit 17 generates a request signal including the form identification information included in the instruction information, and transmits the generated request signal to the form information server device 4 .

When the form information server device 4 receives a request signal containing the form identification information from the communication processing unit 17 via the communication network 5, the form template image and the form definition table corresponding to the form identification number included in the request signal are generated. 1021 to generate a response signal. The form information server device 4 transmits the generated response signal to the communication processing unit 17 via the communication network 5 .

Upon receiving the response signal including the template image and the form definition table 1021 , the communication processing unit 17 reads the template image from the received response signal and writes the read template image to the template image storage unit 101 . Here, it is assumed that the image data of the form template shown in FIG. 2 is written as the template image 70 . Further, the communication processing unit 17 reads the form definition table 1021 from the received response signal, and writes the read definition information table to the form definition information storage unit 102 (step Sa1).

When the template image 70 is written in the template image storage unit 101 by the communication processing unit 17, the guide image generation unit 20 calculates the feature information of the template image 70 stored in the template image storage unit 101, and uses the calculated feature information. It is written in the template feature information storage unit 104 . The guide image generation unit 20 reads the template image 70 from the template image storage unit 101, and writes the read template image 70 to the guide image storage unit 106 as the initial image data of the guide image (step Sa2).

When the document photographing AP is executed, the display processing unit 13 generates a photographed image display area 51 and a guide image display area 52 on the screen 50 . The user presses, for example, the operation button 55 while referring to the preview display in the captured image display area 51 on the terminal device 1 . Upon detecting that the operation button 55 has been pressed, the operation unit 11 outputs instruction information for starting imaging to the imaging unit 12 (step Sa3).

The photographing unit 12 receives the instruction information to start photographing and starts photographing frame images. When photographing is started, the user moves the terminal device 1 while referring to the preview display in the photographed image display area 51 of the terminal device 1, with a part of the document as the photographing target. While moving, the user shoots frame images like shooting a moving image without performing an operation such as pressing the shutter button. As described above, the photographing unit 12 photographs a part of the form at 10 to 30 frames per second and outputs the frame image to the frame image acquisition unit 14 . Upon acquiring the frame image output by the imaging unit 12, the frame image acquisition unit 14 generates new frame identification information, associates the generated frame identification information with the acquired frame image, and writes the generated frame identification information to the frame image storage unit 103. Memorize.

The character recognition unit 15 reads from the frame image storage unit 103 the latest frame image and the frame identification information corresponding to the frame image among the unprocessed frame images that have not been processed in the flow chart shown in FIG. Sa4). The character recognition unit 15 performs character recognition processing on the read frame image, and detects character information and coordinate information corresponding to the character information from the frame image (step Sa5).

The character recognition unit 15 determines whether or not the character recognition process has succeeded, for example, based on whether or not a combination of one or more character information and coordinate information has been detected from one frame image (step Sa6). . If the character recognition unit 15 does not detect a combination of one or more character information and coordinate information from one frame image, the character recognition unit 15 determines that the character recognition processing was not successful (step Sa6, No), and determines that the processing target frame The image is deleted from the frame image storage unit 103 . The process returns to step Sa4, and the character recognition unit 15 reads the next frame image to be processed.

On the other hand, when detecting one or more combinations of character information and coordinate information from one frame image, the character recognition unit 15 determines that the character recognition processing has succeeded (step Sa6, Yes). If the character recognition unit 15 determines that the character recognition processing has succeeded, the character recognition unit 15 recognizes the item name and the item corresponding to the item name from the character information based on the detected character information, the coordinate information, and the form definition table 1021 . Recorded information is detected (step Sa7).

The character recognition unit 15 creates a new form information table 1051 in the form information storage unit 105 and writes the frame identification information in the item of “frame identification information” of the generated form information table 1051 . The character recognition unit 15 writes the acquired item number in the item of "item number", and writes and stores the item recording information corresponding to the item number in the item of "acquired information" (step Sa8). For example, in the case of the frame image 61 shown in FIG. 5, the character recognition unit 15 generates a form information table 1051-1 shown in FIG.

The character recognition unit 15 determines whether or not the item names and item record information are complete in the generated form information table 1051 (step Sa9). For example, if the generated form information table 1051 cannot acquire all the item record information corresponding to the item name, the character recognition unit 15 determines that the item name and the item record information are not complete.

When the character recognition unit 15 determines that the item names and the item recording information are not complete in the generated form information table 1051 (step Sa9, No), the character recognition unit 15 deletes the generated form information table 1051 from the form information storage unit 105, The frame image to be processed is deleted from the frame image storage unit 103 (step Sa10).

On the other hand, when the character recognition unit 15 determines that the item names and the item recording information are complete in the generated form information table 1051 (step Sa9, Yes), the character recognition unit 15 recognizes the frame of the frame image to be processed. Information is output to the guide image generation unit 20 . Upon receiving the frame identification information from the character recognition unit 15, the guide image generation unit 20 starts guide image generation processing (step Sa11).

(Guide image generation processing)
FIG. 10 is a flowchart showing the flow of guide image generation processing by the guide image generation unit 20. As shown in FIG. The guide image generation unit 20 reads the frame image corresponding to the frame identification information received from the character recognition unit 15 from the frame image storage unit 103 (step Sb1). The guide image generator 20 calculates feature information of the read frame image (step Sb2).

The guide image generation unit 20 performs matching processing by comparing the calculated feature information of the frame image and the feature information of the template image stored in the template feature information storage unit 104 (step Sb3). In the matching process, the guide image generating unit 20, for example, as shown in FIG. Detect as a pair. The guide image generation unit 20 also detects the coordinates of the intersection of the ruled lines 601 of the frame image 61 and the coordinates of the intersection of the ruled lines 701 of the template image 70 as a pair.

The guide image generation unit 20 calculates a projective transformation matrix based on the matching result including many pairs detected as described above, perspectively transforms the frame image 61 using the computed projective transformation matrix, and obtains a frame image 61a. As shown in FIG. 12, the coordinate system is matched with the template image 70 (step Sb4). The guide image generation unit 20 calculates, for example, the upper left coordinate 610 and the lower right coordinate 611 of the frame image 61a with reference to the lower left of the template image 70 (step Sb5).

The guide image generation unit 20 reads the guide image 80 from the guide image storage unit 106 and sets the calculated upper left coordinate 610 and lower right coordinate 611 in the read guide image 80 . The guide image generation unit 20 arranges a rectangular hatched object 91 to match the set upper left coordinates 610 and lower right coordinates 611, and combines the arranged objects 91 with the guide image 80 to create a new guide image. Generate 80. The guide image generation unit 20 overwrites the guide image storage unit 106 with the generated new guide image 80 (step Sb6).

The guide image generation unit 20 outputs instruction information for displaying the guide image to the display processing unit 13 . The display processing unit 13 reads the guide image 80 from the guide image storage unit 106 upon receiving the instruction information for displaying the guide image from the guide image generation unit 20 . The display processing unit 13 reduces the read guide image 80 to fit the size of the guide image display area 52 on the screen 50, and displays the reduced guide image 53 as shown in FIG. 4 (step Sb7).

The guide image generator 20 determines whether or not the shaded objects are combined in all areas of the new guide image 80 (step Sb8). In practice, as shown in FIG. 13, the marginal portion of the form may not be included in the frame image. Therefore, the guide image generation unit 20 determines whether or not the shaded object is connected to the entire area of the guide image 80 excluding the predetermined marginal portion.

When the guide image generation unit 20 determines that the shaded objects are not combined in all areas of the new guide image 80 (step Sb8, No), the subroutine of the guide image generation processing is terminated, and the processing proceeds to , the process proceeds to step Sa12 in FIG.

When the guide image generation unit 20 determines that the shaded objects are combined in all areas of the new guide image 80 (step Sb8, No), the guide image generation unit 20 instructs the display processing unit 13 to display a completion message. Output information. When the display processing unit 13 receives the instruction information for displaying the completion message from the guide image generating unit 20, the display processing unit 13 displays a message on the screen 50 to indicate to the user that the acquisition of predetermined character information has been completed (step Sb9). After that, the subroutine of the guide image generation processing ends, and the processing proceeds to step Sa12 in FIG.

Returning to FIG. 9, the operation unit 11 determines whether or not the user has instructed to end the shooting (step Sa12). For example, when the user presses the operation button 55 during shooting by referring to the message indicating that the acquisition of character information displayed on the screen 50 in the process of step Sb9 is completed, or for some other reason, do. The operation unit 11 detects that the operation button 55 has been pressed and determines that the user has instructed to end the shooting.

When the operation unit 11 determines that the user has not instructed the end of shooting (step Sa12, No), the shooting unit 12 continues shooting, and the processing after step Sa4 is performed.

For example, when the image capturing unit 12 captures the frame image 62 shown in FIG. 14, in the next guide image generation process, the guide image generation unit 20 generates a guide image to which the square hatched object 92 shown in FIG. 15 is combined. Generate image 80 .

Further, when the user touches the region of the guide enlargement button 54 with a finger during the process from step Sa4 to step Sa12, the operation unit 11 detects an operation on the touch pad in that region. When the operation is detected, the operation unit 11 outputs instruction information for enlarging and displaying the guide image 53 to the display processing unit 13 . The display processing unit 13 reads out the guide image 80 from the guide image storage unit 106 when instructed to enlarge and display it, and displays the read out guide image 80 on the entire screen 50, for example. The displayed guide image 80 is enlarged or reduced by, for example, an enlargement/reduction operation of opening and closing two fingers on a touch pad of a general smart phone.

In step Sa12, when the operation unit 11 determines that the user has instructed to end shooting (Yes in step Sa12), the operation unit 11 outputs instruction information to end shooting to the shooting unit 12, and outputs the combined image to the combined image generation unit 21. Outputs instruction information to start the generation process. The photographing unit 12 receives instruction information to end photographing from the operation unit 11 and finishes photographing. The combined image generation unit 21 receives instruction information for starting the combined image generation process from the operation unit 11 and starts the combined image generation process (step Sa13).

(Combined image generation processing)
FIG. 16 is a flow chart showing the flow of combined image generation processing by the combined image generation unit 21 . Upon receiving instruction information for combined image generation processing from the operation unit 11, the combined image generation unit 21 reads the template image from the template image storage unit 101, and stores the read template image as initial image data of the combined image data. It is written in the section 107 (step Sc1).

The combined image generation unit 21 detects an unprocessed form information table 1051 that has not been processed according to the flowchart shown in FIG. The combined image generation unit 21 reads out the form information table 1051 first written in the form information storage unit 105 among the detected unprocessed form information tables 1051 (step Sc2).

Combined image generation unit 21 reads from frame image storage unit 103 the frame image corresponding to the frame identification information in the item "frame identification information" of read form information table 1051 (step Sc3). The combined image generator 21 calculates feature information of the read frame image (step Sc4).

The combined image generation unit 21 compares the calculated feature information of the frame image with the feature information of the template image stored in the template feature information storage unit 104, and performs matching processing as shown in FIG. 11 (step Sc5). ).

The combined image generation unit 21 calculates a projective transformation matrix corresponding to the matching result, and perspectively transforms the frame image using the calculated projective transformation matrix. For example, when the frame image to be processed is the frame image 61 shown in FIG. 5, the combined image generator 21 perspectively transforms the frame image 61 into a frame image 61a, and generates a template image 70 as shown in FIG. and coordinate system (step Sc6).

Combined image generator 21 reads out the combined image from combined image storage unit 107 . Since the coordinate systems of the template image 70 and the combined image match, the combined image generation unit 21 combines the perspective-transformed frame image in front of the combined image to generate a new combined image. Combined image generation unit 21 overwrites combined image storage unit 107 with the generated new combined image (step Sc7).

After completing the process of step Sc7, the combined image generation unit 21 determines whether or not the unprocessed form information table 1051 exists in the form information storage unit 105 (step Sc8). When the combined image generation unit 21 determines that the unprocessed form information table 1051 exists in the form information storage unit 105 (step Sc8, Yes), the process from step Sc2 is performed.

For example, when the combined image generation unit 21 selects the frame image 62 shown in FIG. 14 as the next processing target, in step Sc7, the combined image generation unit 21 transforms the frame image 62a obtained by perspectively transforming the frame image 62 into a combined image. , to generate a combined image 85 as shown in FIG.

On the other hand, when the combined image generation unit 21 determines that the unprocessed form information table 1051 does not exist in the form information storage unit 105 (Step Sc8, No), the combined image generation unit 21 generates the instruction information for displaying the combined image. is output to the display processing unit 13 . The combined image generation unit 21 also outputs instruction information for starting transmission processing to the recognition data management unit 16 .

Display processing unit 13 receives the instruction information for displaying the combined image from combined image generation unit 21 and reads out the combined image from combined image storage unit 107 . The display processing unit 13 displays the read combined image on the screen 50 (step Sc9). After that, the combined image processing ends, and the process proceeds to step Sa14 in FIG.

Returning to FIG. 9, when the recognition data management unit 16 receives the instruction information for starting transmission processing from the combined image generation unit 21, the recognition data management unit 16 sets the item “form identification information” in the form definition table 1021 stored in the form definition information storage unit 102. Read out the form identification information shown in . The recognition data management unit 16 reads out the combined image stored in the combined image storage unit 107 . The recognition data management unit 16 reads out all the form information tables 1051 stored in the form information storage unit 105 . The recognition data management unit 16 reads from the frame image storage unit 103 the frame image corresponding to the frame identification information indicated in the item “frame identification information” of the read form information table 1051 . The recognition data management unit 16 associates all the read information with the user identification information assigned in advance to the terminal device 1, and transmits the information to the form data management device 3 through the communication processing unit 17 (step Sa14). End the process.

When the recording processing unit 31 of the form data management device 3 receives the information transmitted by the recognition data management unit 16 of the terminal device 1 through the communication network 5, it writes the received information into each item of the form information total table 321 and stores it. .

For example, in the form information total table 321 shown in FIG. 8, it is assumed that information with frame identification information "1" and information with frame identification information "2" have the same item number. At this time, it is assumed that the item record information visible in the combined image 85 is determined as valid information. In the combined image 85 , frame images to be combined later are superimposed on top of each other. Therefore, the item record information corresponding to the item number belonging to the larger number of the frame identification information becomes effective information.

On the other hand, for example, suppose that the combined image generation unit 21 replaces the process of step Sc2 shown in FIG. . In this case, in the combined image 85, the frame images with the larger frame identification information numbers are superimposed and combined in order, so the frame images with the smaller frame identification information numbers appear on top. Therefore, the item record information corresponding to the item number belonging to the number with the smaller number of the frame identification information becomes effective information.

With the configuration of the above embodiment, the frame image acquiring unit 14 acquires frame images obtained by partially continuously capturing one form. The character recognition unit 15 performs character recognition of the character image in the frame image. The guide image generating unit 20 generates character information from the entire image of the form based on the characteristic information of the entire image of the form including the entire image of the form and the characteristic information of the frame image from which the character information is obtained by character recognition. A guide image for shooting guide is generated that indicates the region where the image is captured. As a result, when one document is partially continuously photographed, by referring to the guide image, it is possible to easily capture areas where character information has been acquired and areas where character information has not been acquired. It becomes possible to distinguish

In the above embodiment, the guide image generation unit 20 calculates the upper left and lower right coordinates of the frame image after perspective transformation, but the upper right and lower left coordinates may be calculated. Also, if the frame image is greatly tilted with respect to the template image 70, using the upper left and lower right coordinates or the upper right and lower left coordinates, the area of the quadrilateral shaded object is It can get too big or too small. In this case, a quadrilateral area that meets four points: the midpoint of the upper left and upper right vertices, the midpoint of the lower left and lower right vertices, the midpoint of the upper and lower vertices of the left, and the midpoint of the upper right upper limit It may be a shaded object area.

In the above embodiment, the guide image generation unit 20 stores the template image stored in the template image storage unit 101 in the guide image storage unit 106 as the initial image data of the guide image in the process of step Sa2 shown in FIG. I am trying to write. For the initial image data of the guide image, the guide image generation unit 20 generates a guide image 80a by combining hatched objects with respect to areas that do not need to be photographed in advance, as shown in FIG. good too. By doing so, in the processing of step Sb8 shown in FIG. 10, there is no need to consider the marginal area, and the number of times of photographing can be reduced.

In the above-described embodiment, the template image is acquired from the form information server device 4. However, as the initial image of the guide image, an image of the whole form photographed in advance by the terminal device 1 is used as the template image. It may be used instead. However, as the resolution of the image obtained by photographing the entire image of the form, a resolution that enables character recognition is not required, but at least a resolution that enables matching processing in steps Sb3 and Sc5 is required.

Further, in the above embodiment, for example, as shown in FIGS. 13 and 15, the shaded objects 91 and 92 are arranged. , or an object with a pattern other than hatching. In addition, instead of hiding the character information with an object, the original template image is displayed so that the original template image can be seen in the area where the object is placed by inverting display, that is, shading the place other than the place where the object is placed. You may do so.

Further, in the above embodiment, the operation of confirming that there is no omission in the item record information to be saved and permitting transmission by the user who refers to the combined image displayed on the screen 50 by the display processing unit 13 in step Sc9 is For example, the process of step Sa14 may be started when the touch pad is used. Also, if the user notices that there is an omission in the item record information to be saved in the combined image, the process does not proceed to step Sa14, but the user presses the operation button 55 to restart the processing from step Sa3. may

Further, in the above embodiment, the processing of steps Sb2 to Sb4 by the guide image generation unit 20 shown in FIG. 10 is the same processing as the processing of steps Sc4 to Sc6 by the combined image generation unit 21 shown in FIG. , the perspective-transformed frame image generated by the guide image generation unit 20 in the processing of steps Sb2 to Sb4 is stored in an internal storage area, and the combined image generation unit 21 generates the guide image instead of the processing of steps Sc4 to Sc6. The perspective-transformed frame image generated by the generation unit 20 may be used in step Sc7.

Further, in the configuration of the above embodiment, the character recognition unit 15 reads the frame image and the frame identification information from the frame image storage unit 103 in step Sa4 shown in FIG. If the processing speed is sufficiently high, the character recognition section 15 may directly acquire the frame image and the frame identification information output by the frame image acquisition section 14 .

Further, in the configuration of the above-described embodiment, it is assumed that some kind of information is always written in the entry fields corresponding to all the item names of the form, and that no item name is allowed to be left blank. On the other hand, if blanks are allowed in a certain item name in the form definition table 1021, the character recognition unit 15 performs step In the processing of Sa9, it is determined that the item record information exists.

Further, in the configuration of the above embodiment, in the process of step Sb8 shown in FIG. , it is assumed that all item names and item record information have been acquired when shaded objects are combined in all areas of the guide image 80 . However, it is conceivable that the captured frame image may have an unclear portion. For example, in the frame image 61 shown in FIG. 5, the item names and item record information of "Furigana" and "Date of Birth" can be acquired, but the item name of "Name" may not be acquired. In the above embodiment, since it is not determined whether all the item names included in the captured frame image have been acquired, even if the guide image generation unit 21 determines "Yes" in the determination of step Sb8, , there are cases where all item names cannot be acquired. In order to cope with such a case, for example, the following two means may be used.

As a first means, for example, it is assumed that the definition information of the form definition table 1021 includes information indicating the continuity of positions between item names. As described above, when the item name of "name" between the item names of "furigana" and "date of birth" cannot be obtained, the character recognition unit 15 refers to the form definition table 1021, Since the item name is missing, it may be determined that the item name and the item record information are not complete in the process of step Sa9 shown in FIG. By doing so, it is possible to arrange a shaded object in the area corresponding to the frame image 61 in the guide image so that it is not combined.

As a second means, when the result of the determination processing in step Sb8 shown in FIG. It may be determined whether or not all names have been acquired. When the guide image generation unit 20 determines that all the item names of the form have not been acquired, the item names that have not been acquired may be displayed on the screen 50 as a message. Also, if the coordinates of the range related to each item name in the form template are known, the item name and the shaded object of the range of the item record information corresponding to the item name can be removed. good. When the guide image generation unit 20 determines that all the item names of the form have been acquired, the guide image generation unit 20 performs the process of step Sb9 to display a message on the screen 50 indicating that the acquisition of the character information has been completed. can be

Further, in the configuration of the above embodiment, the combined image generation unit 21 excludes the form information table 1051 having overlapping item record information before or after the process of step Sc1 shown in FIG. The form information table 1051 may be selected. For example, the combined image generator 21 arbitrarily selects the form information table 1051 . If each piece of item record information recorded in the arbitrarily selected form information table 1051 is recorded in any of the other form information tables 1051, the combined image generation unit 21 converts the selected form information table 1051 into The process of excluding all the form information tables 1051 is repeated. By doing so, each of the form information tables 1051 remaining to be saved includes at least one piece of item record information that is not included in the other form information tables 1051 to be saved. By doing so, it is possible to reduce duplication of item record information between the form information tables 1051, and reduce the number of form information tables 1051 and frame images to be saved.

In the above embodiment, the terminal device 1 includes the character recognition unit 15, the recognition data management unit 16, the guide image generation unit 20, and the combined image generation unit 21, and is a single "image It functions as a processing device. On the other hand, for example, the character recognition unit 15 is provided in an external OCR server device connected to the communication network 5, and the recognition data management unit 16, the guide image generation unit 20, and the combined image generation unit 21 are installed in the communication network 5. It may be provided in an external web server device to be connected. These devices may be caused to perform the processing performed by the character recognition section 15, the recognition data management section 16, the guide image generation section 20, and the combined image generation section 21, respectively. In this case, the terminal device 1, the OCR server device, and the web server device together form an "image processing device" that generates a guide image and a combined image.

You may make it implement|achieve the terminal device 1 in embodiment mentioned above by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in this recording medium may be read into a computer system and executed. It should be noted that the "computer system" referred to here includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard discs incorporated in computer systems. Furthermore, "computer-readable recording medium" refers to a program that dynamically retains programs for a short period of time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It may also include something that holds the program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in that case. Further, the program may be for realizing a part of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array).

Although the embodiment of the present invention has been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and design and the like are included within the scope of the gist of the present invention.

DESCRIPTION OF SYMBOLS 1... Terminal device 3... Form data management apparatus 4... Form information server apparatus 5... Communication network 10... Storage part 11... Operation part 12... Photographing part 13... Display processing part 14... Frame image acquisition Unit 15 Character recognition unit 16 Recognition data management unit 17 Communication processing unit 20 Guide image generation unit 21 Combined image generation unit 31 Recording processing unit 32 Storage unit

Claims (6)

a frame image obtaining unit that obtains frame images obtained by partially continuously photographing one form;
a character recognition unit that performs character recognition of the character image in the frame image;
The character information is obtained for the overall image of the form based on the characteristic information of the entire image of the form including the overall image of the form and the characteristic information of the frame image from which the character information is obtained by the character recognition. a guide image generating unit that generates a guide image for shooting guide by combining the area information indicating the area that has been captured;
a display processing unit that displays the frame image in a first area of a screen and displays the guide image in a second area of the screen that is different from the first area;
An image processing device comprising: The image processing apparatus according to claim 1, wherein the entire image of the form is a template image of the form. 2. The image processing apparatus according to claim 1, wherein the entire image of the form is an image obtained by photographing the entire image of the form to be photographed when the frame image is photographed. The guide image generation unit determines whether or not the area information is combined in all of the predetermined areas in the guide image.
4. The image processing apparatus according to any one of claims 1 to 3, characterized by: a frame image acquisition process for acquiring frame images obtained by partially continuously photographing one form;
a character recognition process for character recognition of the character image in the frame image;
The character information is obtained for the overall image of the form based on the characteristic information of the entire image of the form including the overall image of the form and the characteristic information of the frame image from which the character information is obtained by the character recognition. a guide image generation process for generating a guide image for shooting guide by combining the area information indicating the area that has been captured;
a display processing step of displaying the frame image in a first area of a screen and displaying the guide image in a second area of the screen different from the first area;
A guide image generation method, comprising: the computer,
frame image acquisition means for acquiring frame images obtained by partially continuously photographing one form;
character recognition means for performing character recognition of character images in the frame image;
The character information is obtained for the overall image of the form based on the characteristic information of the entire image of the form including the overall image of the form and the characteristic information of the frame image from which the character information is obtained by the character recognition. guide image generating means for generating a guide image for shooting guide by combining the area information indicating the area indicated by the
display processing means for displaying the frame image in a first area of a screen and displaying the guide image in a second area different from the first area on the screen;
A program to function as

Images