JP2019207735A - Mobile terminal, image processing method, and program - Google Patents

Abstract

To allow for quickly and accurately determining form types.SOLUTION: An image processing method is provided, comprising: acquiring a frame captured by an image capturing unit; acquiring document image data of a document from the frame; determining whether or not at least one partial feature of a registered form matches a document partial feature in the document at a position corresponding to a position of the partial feature in the form; and acquiring a frame captured by the image capturing unit again when it is determined that the partial feature and the document partial feature do not match.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a mobile terminal, an image processing method, and a program.

  Conventionally, techniques for executing image matching have been disclosed.

  Here, a technique for performing rotation correction by matching a plurality of feature points of an input form with a plurality of feature points of a registered form is disclosed (see Patent Document 1).

  Further, a technique is disclosed in which document features are extracted from a document image whose size is matched with a registered image, and the extracted document features are sequentially matched with features of a registered image whose size is matched with the document image (Patent Literature). 2).

  Further, there is a technology for determining whether or not the driver's license image is a processable driver's license image, and outputting a message indicating that the driver's license image cannot be processed in the case of a driver's license image that cannot be processed. It is disclosed (see Patent Document 3).

  In addition, a technique is disclosed in which a digital image is discriminated from one or more known dimensions of a plurality of known document types to determine what type of document the digital image is (see Patent Document 4). .

JP 2012-98984 A JP 2001-325563 A JP-A-7-65118 Special table 2015-503813 gazette

  However, the conventional photographing apparatus (Patent Document 1 or the like) has a problem that it takes time for matching and high-precision form type determination cannot be performed at high speed.

  The disclosed technology performs matching of one partial feature point on the determination target image, and automatically performs re-photographing when the image quality of the determination target image is poor, and retrying the matching, The purpose is to perform high-precision form type determination at high speed.

  A mobile terminal according to an aspect of the disclosure includes a frame acquisition unit that acquires a frame obtained by shooting by a shooting unit, a document acquisition unit that acquires document image data of a document from the frame, and one partial feature of a registered form, Partial feature determination means for determining whether or not the document partial feature in the document at a position corresponding to the position of the partial feature in the form matches, and the partial feature determination means determines the partial feature and the Re-acquisition means for acquiring a frame by re-shooting by the shooting unit when it is determined that the document part feature does not match.

  In addition, an image processing method according to an aspect of the disclosure includes a frame acquisition step of acquiring a frame obtained by shooting by a shooting unit, a document acquisition step of acquiring document image data of a document from the frame, and one part of a registered form In the partial feature determination step for determining whether or not the feature matches the document partial feature in the document at a position corresponding to the position of the partial feature in the form, the partial feature determination step A re-acquisition step of acquiring a frame by re-shooting by the shooting unit when it is determined that the partial feature and the document partial feature do not match.

  Further, a program according to an aspect of the disclosure includes a frame acquisition step of acquiring a frame obtained by shooting by a shooting unit, a document acquisition step of acquiring document image data of a document from the frame, and one partial feature of a registered form The partial feature determination step for determining whether or not the document partial feature in the document at a position corresponding to the position of the partial feature in the form matches, and the partial feature in the partial feature determination step And a re-acquisition step of acquiring a frame obtained by re-shooting by the shooting unit when it is determined that the document part feature does not match the document part feature.

  According to the disclosed aspect, the target form is automatically determined from the photographed image, and the image is acquired only when the image quality is such that OCR recognition is possible. You can shoot the desired original.

FIG. 1 is a block diagram illustrating an example of a configuration of a mobile terminal according to the present embodiment. FIG. 2 is a flowchart illustrating an example of processing in the mobile terminal according to the present embodiment. FIG. 3 is a diagram illustrating an example of a screen display in the present embodiment. FIG. 4 is a flowchart illustrating an example of processing in the mobile terminal according to the present embodiment. FIG. 5 is a diagram illustrating an example of partial feature determination in the present embodiment. FIG. 6 is a diagram illustrating an example of overall feature determination in the present embodiment. FIG. 7 is a flowchart illustrating an example of processing in the mobile terminal according to the present embodiment.

  Hereinafter, an embodiment of the present disclosure will be described in detail based on the drawings. In addition, this invention is not limited by this embodiment.

[Configuration of this embodiment]
Hereinafter, an example of the configuration of the mobile terminal 100 according to the embodiment of the present invention will be described with reference to FIG. 1, and then the processing and the like of the present embodiment will be described in detail. FIG. 1 is a block diagram illustrating an example of a configuration of the mobile terminal 100 according to the present embodiment.

  However, the embodiment described below exemplifies the mobile terminal 100 for embodying the technical idea of the present invention, and is not intended to specify the present invention for this mobile terminal 100, and claims. The present invention is equally applicable to the mobile terminals 100 of other embodiments included in the scope of the above.

  In addition, the form of function distribution in the mobile terminal 100 exemplified in the present embodiment is not limited to the following, and may be configured to be functionally or physically distributed / integrated in arbitrary units within a range where similar effects and functions can be achieved. be able to.

  Here, the mobile terminal 100 is a portable information processing having portability such as a tablet terminal, a mobile phone, a smartphone, a PHS, a PDA, a notebook personal computer, or a wearable computer such as a glasses type or a watch type. It may be a device.

  First, as shown in FIG. 1, the mobile terminal 100 is generally configured to include a control unit 102, a storage unit 106, a photographing unit 110, an input / output unit 112, a sensor unit 114, and a communication unit 116.

  Here, although omitted in FIG. 1, in the present embodiment, an input / output interface unit (not shown) for connecting the input / output unit 112 and the control unit 102 may be further provided. Each unit of the mobile terminal 100 is connected to be communicable via an arbitrary communication path.

  Here, the communication unit 116 is a network interface (NIC (Network Interface Controller), etc.), Bluetooth (registered trademark), infrared communication, etc. for transmitting and receiving IP data by wired communication and / or wireless communication (WiFi, etc.). May be an interface that performs wireless communication.

  Here, the mobile terminal 100 may be communicably connected to an external device via a network using the communication unit 116.

  Further, the sensor unit 114 detects a physical quantity and converts it into a signal (digital signal) of another medium. Here, the sensor unit 114 includes a proximity sensor, a direction sensor, a magnetic field sensor, a linear acceleration sensor, a luminance sensor, a gyro sensor, a pressure sensor, a gravity sensor, an acceleration sensor, an atmospheric pressure sensor, and / or a temperature sensor. Also good.

  The input / output unit 112 performs data input / output (I / O). Here, the input / output unit 112 may be, for example, a key input unit, a touch panel, a control pad (for example, a touch pad and a game pad), a mouse, a keyboard, and / or a microphone.

  Further, the input / output unit 112 may be a display unit (for example, a display, a monitor, a touch panel, or the like including a liquid crystal or an organic EL) that displays a display screen of an application or the like.

  The input / output unit 112 may be an audio output unit (for example, a speaker) that outputs audio information as audio. The input / output unit (touch panel) 112 may include a sensor unit 114 that detects physical contact and converts it into a signal (digital signal).

  The photographing unit 110 obtains continuous (moving image) image data (frames) by continuously photographing a subject (for example, a document or the like). For example, the imaging unit 110 may acquire video data. In addition, the imaging unit 110 may acquire ancillary data.

  Here, the imaging unit 110 may be a camera or the like that includes an imaging device such as a charge coupled device (CCD) and / or a complementary metal oxide semiconductor (CMOS).

  The photographing unit 110 may acquire still image data by taking a still image of the subject. Here, the frame may be uncompressed image data. The frame may be high-resolution image data.

  Here, the high resolution may be full high vision, 4K resolution, super high vision (8K resolution), or the like. Further, the photographing unit 110 may shoot moving images at 24 fps, 30 fps, or the like. Further, the photographing unit 110 may be a camera.

  The storage unit 106 is a storage unit, and for example, a memory such as a RAM / ROM, a fixed disk device such as a hard disk, an SSD (Solid State Drive), and / or an optical disk can be used.

  The storage unit 106 stores various databases, tables, buffers, and / or files (such as the feature data file 106a and the image data file 106b). Here, the storage unit 106 may store a computer program or the like for giving a command to a CPU (Central Processing Unit) and performing various processes.

  Among each component of the storage unit 106, the feature data file 106a stores (registers) feature data related to the features of a plurality of forms. Here, the feature may be a partial feature of the form and / or an overall feature.

  Here, the partial feature may be composed of a ruled line of a form, a character, and / or a characteristic image. Here, the characteristic image may be a pattern image, a character image, a face image, a background image, or the like. The overall feature may indicate a plurality of partial features.

  The image data file 106b stores image data (such as a frame). Here, the image data file 106b may store document image data, region image data of a partial region of the document, and / or divided region image data of a divided region of the document.

  The image data file 106b may store position data such as a document and / or a divided area. The image data file 106b may store character data corresponding to the image data.

  For example, the image data file 106b may store form data identified from the image data and / or character data related to characters described in the form. The image data file 106b may store video data. The image data file 106b may store ancillary data.

  In addition, the control unit 102 is a CPU, GPU (Graphics Processing Unit), DSP (Digital Signal Processor), LSI (Large Scale Integration), ASIC (Application Specific Integrate) and ASIC (Integrated Specimen Integrate Integrate / Circuit Integrate). And a tangible controller including an FPGA (Field-Programming Gate Array).

  In addition, the control unit 102 has an internal memory for storing a control program, a program defining various processing procedures, and necessary data, and performs information processing for executing various processes based on these programs. Do.

  Here, the control unit 102 conceptually has a frame acquisition unit 102a, a document acquisition unit 102b, a partial feature determination unit 102c, an overall feature determination unit 102d, a re-acquisition unit 102e, an image display unit 102f, and an OCR unit 102g. Is provided.

  The frame acquisition unit 102a acquires image data (frame). Here, the frame acquisition unit 102a may acquire image data obtained by photographing by the photographing unit 110. The frame acquisition unit 102a may acquire non-compressed and high-resolution image data.

  The frame acquisition unit 102a may acquire image data (frame) corresponding to one frame by controlling continuous image shooting or moving image shooting by the shooting unit 110. The frame acquisition unit 102a may acquire image data by controlling still image shooting by the shooting unit 110. The frame acquisition unit 102a may acquire ancillary data.

  The document acquisition unit 102b acquires document image data of a document. Here, the document acquisition unit 102b may acquire document image data of a document from a frame. Further, the document acquisition unit 102b may acquire region image data of a partial region of the document from the frame. Here, the document may be a rectangular document.

  Further, the document acquisition unit 102b may acquire the divided region image data of the divided region of the document from the frame. Here, the document acquisition unit 102b may detect the position data of the document from the frame. Here, the document acquisition unit 102b may detect the corner coordinates (four points) of the document from the frame.

  The document acquisition unit 102b may detect the layout of the document from the frame. Further, the document acquisition unit 102b may detect the position data of the document from the frame using the edge detection method and / or the feature point matching method, and acquire the document image data based on the position data of the document. Good.

  The partial feature determination unit 102c determines whether or not the partial feature of the form registered in the feature data file 106a matches the original partial feature in the original at a position corresponding to the position of the partial feature in the form. Determine.

  Here, the partial feature determination unit 102c has one partial feature of the form registered in the feature data file 106a and a document partial feature in the document at a position corresponding to the position of the partial feature in the form. It may be determined whether or not they match.

  In addition, the partial feature determination unit 102c matches one partial feature of the form registered in the feature data file 106a with the original partial feature in the document at a position corresponding to the position of the partial feature in the form. By comparing the degree with a threshold value, it may be determined whether or not the partial feature matches the original document partial feature.

  In addition, the partial feature determination unit 102c also includes the position (coordinates), the color gradation value, the number, and / or the ruled lines, characters, and / or characteristic images constituting the partial features of the form registered in the feature data file 106a. Based on the size, whether or not the partial feature and the document partial feature match by comparing the matching degree between the partial feature and the document partial feature in the document at the position corresponding to the position of the partial feature with a threshold. It may be determined.

  Here, the size may be a length, a size, an angle, and / or a width, or may be a concept (for example, a shape) expressed by a combination thereof.

  When the partial feature is a ruled line, the partial feature determination unit 102c is based on the position of the coordinate of the intersection of the ruled line, the gradation value of the ruled line color, the number of ruled lines, and / or the length and / or width of the ruled line. Thus, the degree of coincidence between the partial feature and the document partial feature may be compared with a threshold value.

  In the case where the partial feature is a character, the partial feature determination unit 102c determines the character position, the character color gradation value, the number of characters, and / or the size, length, and / or width of the character region. The degree of coincidence between the partial feature and the document partial feature may be compared with a threshold.

  When the partial feature is a characteristic image (for example, a pattern image), the position of the characteristic image, the color gradation value of the characteristic image, the number of characteristic images, and / or the size of the characteristic image The degree of coincidence between the partial feature and the original document partial feature may be compared with a threshold value based on the length, width, and / or shape.

  The overall feature determination unit 102d determines whether the overall feature of the form registered in the feature data file 106a matches the overall document feature of the document.

  Here, if the partial feature determination unit 102c determines that the partial feature and the document partial feature match, the overall feature determination unit 102d and the overall feature of the document registered in the feature data file 106a and the original document original feature And whether or not match.

  Here, if the partial feature determination unit 102c determines that the partial feature and the document partial feature match, the overall feature determination unit 102d determines a plurality of partial features of the form registered in the feature data file 106a and the It may be determined whether or not the document partial feature in the document at a position corresponding to the position of each of the partial features matches.

  The re-acquisition unit 102e acquires a frame used for feature determination, or acquires image data (frame) obtained by re-shooting by the shooting unit 110. Here, the reacquisition unit 102e may further output pseudo shutter operation information.

  Here, when the partial feature determination unit 102c determines that the partial feature and the document partial feature do not match, the re-acquisition unit 102e may acquire a frame obtained by re-shooting by the shooting unit 110.

  Further, the re-acquisition unit 102e may acquire a frame obtained by re-shooting by the shooting unit 110 when the overall feature determination unit 102d determines that the overall feature and the entire document feature do not match.

  The re-acquisition unit 102e may acquire a frame obtained by re-shooting by the shooting unit 110 when the overall feature determination unit 102d determines that the plurality of partial features do not match the document partial feature.

  In addition, the re-acquisition unit 102e causes the partial feature determination unit 102c to store one partial feature of all the forms registered in the feature data file 106a and the position in the document corresponding to the position of the partial feature in the form. If it is determined that the document part feature does not match, a frame obtained by re-shooting by the shooting unit 110 may be acquired.

  Further, the re-acquisition unit 102e acquires a frame used for feature determination when the partial feature determination unit 102c determines that the partial feature and the document partial feature match, or acquires a frame obtained by re-shooting by the shooting unit 110. In this case, pseudo shutter operation information may be output.

  Here, when the form identified by the feature determination is a small form such as a license or a business card, the reacquisition unit 102e may acquire the frame used for the feature determination as it is.

  On the other hand, when the form identified by the feature determination is a form that requires high resolution, such as A4, the reacquisition part 102e obtains a new high-resolution and uncompressed frame by re-shooting by the shooting unit 110. May be.

  In addition, the reacquisition unit 102e may acquire a new frame by controlling shooting (such as continuous image shooting, moving image shooting, or still image shooting) by the shooting unit 110. The reacquisition unit 102e may further store the acquired frame in the image data file 106b.

  The image display unit 102f displays image data. Here, the image display unit 102f may display a frame and / or document image data.

  Further, the image display unit 102f may display the image data on the input / output unit 112. The image display unit 102f may display character data.

  The OCR unit 102g performs OCR processing on the image data and acquires character data. Here, the OCR unit 102g may perform character image data by performing OCR processing on document image data or frames. The OCR unit 102g may further store the acquired character data in the image data file 106b.

[Process of this embodiment]
An example of the process executed by the mobile terminal 100 having the above-described configuration will be described with reference to FIGS.

[Form Feature Judgment Processing (Part 1)]
First, with reference to FIG. 2 and FIG. 3, an example of a form feature determination process when there is one type of registered form in the present embodiment will be described. FIG. 2 is a flowchart illustrating an example of processing in the mobile terminal 100 of the present embodiment.

  As shown in FIG. 2, first, the frame acquisition unit 102a activates the photographing unit (camera) 110 and controls continuous image photographing or moving image photographing by the photographing unit 110 (step SA-1).

  Then, the frame acquisition unit 102a inputs (acquires) a frame of an image captured by the camera 110 (Step SA-2).

  Then, the document acquisition unit 102b cuts out the document image data of the rectangular document from the frame acquired by the frame acquisition unit 102a using the edge detection method and / or the feature point matching method (step SA-3).

  Then, the partial feature determination unit 102c detects a document partial feature in the document at a position corresponding to the position of one partial feature of the form registered in the feature data file 106a (step SA-4).

  Here, in the feature data file 106a, position information and feature information (ruled line, character image, text, picture image, face image, etc.) may be registered in advance for each form. Here, the color may be a gradation value such as RGB, HSV, Lab, or YUV.

  That is, in the present embodiment, unique features and area information that are not available may be registered in advance as form data.

  Then, the partial feature determination unit 102c compares the degree of coincidence between one partial feature of the form registered in the feature data file 106a and the detected document partial feature with a threshold value, thereby determining the partial feature and the document partial feature. Are matched (rectangular document is the target form) (step SA-5).

  As described above, in the present embodiment, the extracted image data is subjected to feature detection, and consistency with the feature of the form registered in advance in the feature data file 106a is determined.

  In this embodiment, the consistency is determined by determining whether the position of the intersection coordinates, the color gradation value, the number of ruled lines, the length or width, the number of characters, and the like satisfy a threshold value. Alternatively, it may be determined whether the images match.

  That is, in the present embodiment, partial feature determination may be performed based on the position (coordinates, etc.), color, number, and / or size (length, size, and / or width, etc.) of partial features.

  When the partial feature determination unit 102c determines that the rectangular document is not the target form (step SA-5: No), the partial feature determination unit 102c makes an image reacquisition request and shifts the processing to step SA-1.

  On the other hand, if the partial feature determination unit 102c determines that the rectangular document is the target form (step SA-5: Yes), the process shifts to step SA-6.

  In other words, in this embodiment, if matching with a registered form can be achieved, matched form type information may be set, and the process may be shifted to subsequent image processing (OCR analysis or the like).

  Then, the reacquisition unit 102e acquires a frame used for feature determination, or controls shooting by the camera 110 to acquire a new high-resolution and uncompressed frame, and stores it in the image data file 106b. At this time, the pseudo shutter operation information is output to the input / output unit 112 (step SA-6).

  In other words, in the present embodiment, if the target form can be determined, the pseudo shutter may be automatically released to obtain the image data.

  Then, the OCR unit 102g performs character recognition by image processing (OCR processing) on the document image data or the frame included in the frame acquired by the re-acquisition unit 102e, and acquires character data (step SA- 7).

  The image display unit 102f then sends the form name of the form identified by the feature determination, the character data acquired by the OCR unit 102g, and the document image data included in the frame acquired by the re-acquisition unit 102e to the input / output unit 112. Output (display) (step SA-8), the process is terminated.

  Here, an example of the screen display in the present embodiment will be described with reference to FIG. FIG. 3 is a diagram illustrating an example of a screen display in the present embodiment.

  As shown in FIG. 3, in the present embodiment, the form name (driver's license) of the form identified by the feature determination, the icon of the form, and the information of the characters (name, date of birth) described in the form , Address, expiration date, and / or license number) and document image data may be displayed on the input / output unit (display) 112.

  In the present embodiment, the user's desired data processing is performed by transferring the read information (data transmission) to a service (system) corresponding to the form by a user operation on the screen after display. realizable.

  As described above, in this embodiment, high-speed retry processing is realized by performing partial feature determination.

[Form Feature Judgment Processing (Part 2)]
Next, an example of a form feature determination process when there are a plurality of types of registered forms in this embodiment will be described with reference to FIGS. FIG. 4 is a flowchart showing an example of processing in the mobile terminal 100 of the present embodiment.

  As shown in FIG. 4, first, the frame acquisition unit 102a activates the photographing unit (camera) 110 and controls continuous image photographing or moving image photographing by the photographing unit 110 (step SB-1).

  Then, the frame acquisition unit 102a inputs (acquires) a frame of an image captured by the camera 110 (step SB-2).

  Then, the document acquisition unit 102b cuts out the document image data of the rectangular document from the frame acquired by the frame acquisition unit 102a using the edge detection method and / or the feature point matching method (step SB-3).

  Then, the partial feature determination unit 102c detects a document partial feature in the document at a position corresponding to the position of one partial feature of the form registered in the feature data file 106a (step SB-4).

  Then, the partial feature determination unit 102c compares the degree of coincidence between one partial feature of the form registered in the feature data file 106a and the detected document partial feature with a threshold value, thereby determining the partial feature and the document partial feature. Are matched (rectangular document is the target form) (step SB-5).

  As described above, in the present embodiment, since the feature amount of the form is detected and determined, even forms of the same size can be identified with high accuracy.

  If the partial feature determination unit 102c determines that the rectangular document is not the target form (step SB-5: No), the process shifts to step SB-6.

  Then, the partial feature determination unit 102c determines whether or not the determination as to whether one partial feature of all the forms registered in the feature data file 106a matches the detected document partial feature is complete. (Step SB-6).

  When the partial feature determination unit 102c determines that the determination as to whether one partial feature of all registered forms matches the detected document partial feature is complete (step SB-6: Yes), an image reacquisition request is made, and the process proceeds to step SB-1.

  On the other hand, when the partial feature determination unit 102c determines that the determination as to whether one partial feature of all registered forms matches the detected document partial feature has not been completed (step SB-). 6: No), the type of the form registered in the feature data file 106a to be compared is changed, and the process proceeds to step SB-4.

  Here, an example of partial feature determination in the present embodiment will be described with reference to FIG. FIG. 5 is a diagram illustrating an example of partial feature determination in the present embodiment.

  As shown in FIG. 5, in the present embodiment, the consistency of the clipped document image data with the partial feature of the form registered in advance in the feature data file 106 a may be determined.

  As described above, in this embodiment, the consistency determination is a partial feature determination, so even if there are a plurality of registered forms, the determination can be performed at high speed, and the retry process at the time of error can also be performed at high speed. it can.

  Returning to FIG. 4, when the partial feature determination unit 102c determines that the rectangular document is the target form (step SB-5: Yes), the process proceeds to step SB-7. In other words, in the present embodiment, if matching with the registered form is achieved, the process is shifted to the overall feature detection.

  Then, the entire feature determination unit 102d detects the entire document feature in the document (for example, the document partial feature in the document at a position corresponding to the position of the plurality of partial features registered in the feature data file 106a). (Step SB-7).

  Then, the overall feature determination unit 102d matches the overall feature of the form registered in the feature data file 106a and determined to be the target form by the partial feature determination unit 102c with the overall document feature of the document ( It is determined whether or not the rectangular document is a target form (step SB-8).

  Here, with reference to FIG. 6, an example of the overall feature determination in the present embodiment will be described. FIG. 6 is a diagram illustrating an example of overall feature determination in the present embodiment.

  In the present embodiment, in the overall feature determination, the overall feature of the entire image may be compared with the overall feature of the registered form. Here, as in the partial feature determination, the comparison may be performed based on the feature data registered in the feature data file 106a.

  As shown in FIG. 6, in this embodiment, for example, in the case of a driver's license, layout determination such as name, date of birth, address, expiration date, and license number may be performed.

  Returning to FIG. 4, if the overall feature determination unit 102d determines that the rectangular document is not the target form (step SB-8: No), it makes an image reacquisition request and shifts the processing to step SB-1.

  In other words, in the present embodiment, the partial feature determination is consistent with some partial features of the specific form, but if the overall feature determination is not consistent, the first sequence of image data acquisition (re-execution) Start over from (Shooting).

  On the other hand, if the overall feature determination unit 102d determines that the rectangular document is the target form (step SB-8: Yes), the overall feature determination unit 102d shifts the process to step SB-9.

  That is, in this embodiment, when consistency is obtained in the overall feature determination, matched form type information is set, and the process is shifted to image processing (OCR analysis or the like). Here, the overall feature determination unit 102d may acquire information necessary for image processing for each of the various forms.

  Then, the reacquisition unit 102e acquires a frame used for feature determination, or controls shooting by the camera 110 to acquire a new high-resolution and uncompressed frame, and stores it in the image data file 106b. At that time, the pseudo shutter operation information is output to the input / output unit 112 (step SB-9).

  Then, the OCR unit 102g performs character recognition by image processing (OCR processing) on the document image data or the frame included in the frame acquired by the re-acquisition unit 102e, and acquires character data (step SB- 10).

  Then, the image display unit 102f outputs the form name identified by the characteristic determination, the character data acquired by the OCR unit 102g, and the document image data included in the frame acquired by the re-acquisition unit 102e to the input / output unit 112 ( Display) (step SB-11), and the process ends.

  Thus, in the present embodiment, it is possible to perform form identification with high accuracy and high speed by performing feature determination in two stages.

[Form Feature Judgment Processing (Part 3)]
Then, with reference to FIG. 7, an example of a form feature determination process when there are a plurality of types of registered forms in the present embodiment will be described. FIG. 7 is a flowchart illustrating an example of processing in the mobile terminal 100 of the present embodiment.

  First, the processing from step SC-1 to step SC-7 shown in FIG. 7 is the same as the processing from step SB-1 to step SB-7 shown in FIG.

  Then, the overall feature determination unit 102d matches the overall feature of the form registered in the feature data file 106a and determined to be the target form by the partial feature determination unit 102c with the overall document feature of the document ( It is determined whether or not the rectangular document is a target form (step SC-8).

  When the overall feature determination unit 102d determines that the rectangular document is the target form (step SC-8: Yes), the overall feature determination unit 102d shifts the process to step SC-9.

  Here, the processing from step SC-9 to step SC-11 shown in FIG. 7 is the same as the processing from step SB-9 to step SB-11 shown in FIG.

  On the other hand, if it is determined that the rectangular document is not the target form (step SC-8: No), the overall feature determination unit 102d makes an image reacquisition request and shifts the processing to step SC-12.

  Then, the reacquisition unit 102e activates the photographing unit (camera) 110 and controls continuous image photographing or moving image photographing by the photographing unit 110 (step SC-12).

  Then, the reacquisition unit 102e inputs (acquires) a frame of an image captured by the camera 110 (step SC-13).

  Then, the document acquisition unit 102b cuts out the document image data of the rectangular document from the frame acquired by the frame acquisition unit 102a using the edge detection method and / or the feature point matching method (step SC-14).

  Then, the entire feature determination unit 102d detects the entire document feature in the document (step SC-15).

  Then, the overall feature determination unit 102d matches the overall feature of the form registered in the feature data file 106a and determined to be the target form by the partial feature determination unit 102c with the overall document feature of the document ( It is determined whether or not the rectangular document is a target form (step SC-16).

  When the overall feature determination unit 102d determines that the rectangular document is not the target form (step SC-16: No), the overall feature determination unit 102d makes an image reacquisition request and shifts the processing to step SC-12.

  On the other hand, if the overall feature determination unit 102d determines that the rectangular document is the target form (step SC-16: Yes), the overall feature determination unit 102d shifts the process to step SC-9.

  As described above, in the present embodiment, when the medium to be imaged is clear once in the partial feature determination, the partial feature determination process can be passed in the second and subsequent processes.

  As a result, in the present embodiment, it is possible to reduce the time required for the partial feature determination process from the second time, and it is possible to perform a faster retry process.

  As described above, in the present embodiment, a rectangular region of a document area is cut out from high-resolution image data extracted by one frame from the moving image data of the camera, feature detection is performed on the cut-out document region, and the form type If there is no match, the image data may be acquired again.

  Here, in the present embodiment, two-stage feature determination may be performed in order to perform determination with high speed and high accuracy. In the first stage, a comparison with the characteristics of a specific area set for each form may be performed.

  Here, the specific area and feature set for each form may be unique for each form type. In the second stage, feature comparison of the entire form may be performed. Thereby, in this embodiment, high-speed processing and highly accurate form identification can be performed.

  In recent years, with the widespread use of mobile terminals such as smartphones and tablets, there is an increasing demand for document scanning with cameras of mobile terminals. This is because the use of a mobile terminal is easy and there is an advantage that a dedicated terminal is not required unlike a conventional scanner.

  Conventionally, there is a mechanism for automatically photographing a document using a mobile terminal. When performing personal authentication using a camera of a mobile terminal, it is necessary to photograph only a document for personal authentication purposes.

  However, conventionally, there has been a problem of shooting an unnecessary document or shooting an image that cannot be recognized by OCR.

  Therefore, in the present embodiment, a method is proposed in which it is determined whether or not the target document being shot by the camera is a target document, and when it is determined that the document is OCR-recognizable, it is automatically shot. Yes.

[Other Embodiments]
The embodiments of the present invention have been described so far, but the present invention may be implemented in various different embodiments other than the above-described embodiments within the scope of the technical idea described in the claims. Is.

  For example, the mobile terminal 100 may perform processing in a stand-alone form, performs processing in response to a request from a client terminal (which is a separate housing from the mobile terminal 100), and outputs the processing result to the client terminal You may make it return to.

  In addition, among the processes described in the embodiment, all or part of the processes described as being automatically performed can be performed manually, or all of the processes described as being performed manually are all performed. Alternatively, a part can be automatically performed by a known method.

  In addition, processing procedures, control procedures, specific names, information including parameters such as registered data or search conditions for each processing, screen examples, or database configuration shown in the description and drawings are specially noted. It can be changed arbitrarily except for.

  In addition, regarding the mobile terminal 100, each illustrated component is functionally conceptual and does not necessarily need to be physically configured as illustrated.

  For example, all or some of the processing functions provided in each device of the mobile terminal 100, particularly the processing functions performed by the control unit 102, are realized by the CPU and a program interpreted and executed by the CPU. Alternatively, it may be realized as hardware by wired logic.

  Note that the program is recorded on a non-transitory computer-readable recording medium including a programmed instruction for causing a computer to execute the method according to the present invention, which will be described later, and the mobile terminal 100 as necessary. Read mechanically. That is, in the storage unit 106 such as a ROM or an HDD, computer programs for performing various processes by giving instructions to the CPU in cooperation with an OS (Operating System) are recorded. This computer program is executed by being loaded into the RAM, and constitutes a control unit in cooperation with the CPU.

  The computer program may be stored in an application program server connected to the mobile terminal 100 via an arbitrary network, and may be downloaded in whole or in part as necessary. .

  In addition, the program according to the present invention may be stored in a computer-readable recording medium, and may be configured as a program product. Here, the “recording medium” means a memory card, USB memory, SD card, flexible disk, magneto-optical disk, ROM, EPROM, EEPROM, CD-ROM, MO, DVD, and Blu-ray (registered trademark). It includes any “portable physical medium” such as Disc.

  The “program” is a data processing method described in an arbitrary language or description method, and may be in any format such as source code or binary code. Note that the “program” is not necessarily limited to a single configuration, and functions are achieved in cooperation with a separate configuration such as a plurality of modules and libraries or a separate program represented by the OS. Including things. In addition, a well-known structure and procedure can be used about the specific structure for reading a recording medium in each apparatus shown in embodiment, a reading procedure, or the installation procedure after reading.

  Various databases and the like stored in the storage unit 106 are storage means such as a memory device such as a RAM or ROM, a fixed disk device such as a hard disk, a flexible disk, and / or an optical disk. Various programs, tables, databases, and / or web page files used may be stored.

  The mobile terminal 100 may be configured as an information processing apparatus such as a known personal computer, or may be configured by connecting an arbitrary peripheral device to the information processing apparatus. The mobile terminal 100 may be realized by installing software (including programs, data, and the like) that causes the information processing apparatus to implement the method of the present invention.

  Furthermore, the specific form of the distribution / integration of the devices is not limited to that shown in the figure, and all or a part of them may be functionally or physically in arbitrary units according to various additions or according to functional loads. Can be distributed and integrated. That is, the above-described embodiments may be arbitrarily combined and may be selectively implemented.

  As described above, the mobile terminal, the image processing method, and the program can be implemented in many industrial fields, particularly in the image processing field that handles images read by a camera, and are extremely useful.

DESCRIPTION OF SYMBOLS 100 Mobile terminal 102 Control part 102a Frame acquisition part 102b Document acquisition part 102c Partial feature determination part 102d Whole feature determination part 102e Re-acquisition part 102f Image display part 102g OCR part 106 Storage part 106a Feature data file 106b Image data file 110 Shooting part 112 Input / output unit 114 Sensor unit 116 Communication unit

Claims (9)

A frame acquisition means for acquiring a frame by shooting of the shooting unit;
Document acquisition means for acquiring document image data of a document from the frame;
Partial feature determination means for determining whether or not the registered partial feature of the document matches the document partial feature in the document at a position corresponding to the position of the partial feature in the document;
When the partial feature determination unit determines that the partial feature and the document partial feature match, the overall feature determination determines whether or not the overall feature of the form matches the overall document feature of the document. Means,
When the partial feature determination unit determines that the partial feature and the document partial feature do not match, or when the overall feature determination unit determines that the overall feature and the document total feature do not match, Re-acquisition means for acquiring a frame by re-shooting of the shooting unit;
Mobile terminal equipped with. The partial feature is a feature of a part of a plurality of items of the form, and the overall feature is a layout feature of the plurality of items.
The mobile terminal according to claim 1. The partial features are composed of ruled lines, characters, and / or characteristic images.
The mobile terminal according to claim 1. The partial feature determination unit compares the degree of coincidence between the partial feature of the registered form and the document partial feature in the document at a position corresponding to the position of the partial feature in the form with a threshold value. To determine whether or not the partial feature and the document partial feature match.
The mobile terminal according to claim 1. The partial feature determination means is configured to determine the degree of coincidence based on the position, color gradation value, number and / or size of ruled lines, characters and / or characteristic images constituting the partial feature of the registered form. Is compared with the threshold value to determine whether the partial feature matches the original document partial feature,
The mobile terminal according to claim 4. The re-acquisition means outputs pseudo-shutter operation information when acquiring a frame by re-shooting of the shooting unit,
The mobile terminal according to claim 1. An OCR unit that performs OCR processing on the document image data and obtains character data;
The mobile terminal according to claim 1. A frame acquisition step of acquiring a frame by shooting of the shooting unit;
A document acquisition step of acquiring document image data of the document from the frame;
A partial feature determination step for determining whether or not a partial feature of the registered form matches a document partial feature in the document at a position corresponding to the position of the partial feature in the form;
If it is determined in the partial feature determination step that the partial feature and the document partial feature match, an overall feature for determining whether or not the overall feature of the form and the original document feature of the document match. A determination step;
When it is determined in the partial feature determination step that the partial feature and the document partial feature do not match, or in the overall feature determination step, it is determined that the overall feature and the document original feature do not match A re-acquisition step of acquiring a frame by re-shooting of the shooting unit;
An image processing method including: A frame acquisition step of acquiring a frame by shooting of the shooting unit;
A document acquisition step of acquiring document image data of the document from the frame;
A partial feature determination step for determining whether or not a partial feature of the registered form matches a document partial feature in the document at a position corresponding to the position of the partial feature in the form;
If it is determined in the partial feature determination step that the partial feature and the document partial feature match, an overall feature for determining whether or not the overall feature of the form and the original document feature of the document match. A determination step;
When it is determined in the partial feature determination step that the partial feature and the document partial feature do not match, or in the overall feature determination step, it is determined that the overall feature and the document original feature do not match A re-acquisition step of acquiring a frame by re-shooting of the shooting unit;
A program that causes a computer to execute.

Images