TWI745724B - Mobile Document Recognition System - Google Patents

Abstract

A mobile document recognition system includes a mobile device. The mobile device includes an image capture module, a display module, and an image recognition module. The image recognition module uses deep learning technology and includes multiple classifiers corresponding to multiple categories for image recognition. The image capturing module is used for capturing to generate a preview image. A user selects the classifier corresponding to the category to which a written document belongs to perform image recognition on the preview image. When the classifier recognizes the preview image that the written document belongs to the category, the prompt message of the display module is used to notify the user of the successful recognition, so that the user generates a capture command to the preview image Save it as a document image to be identified, and re-identify the document image to be identified.

Description

Mobile Document Recognition System

The present invention relates to an image recognition system, in particular to a mobile document recognition system.

With the popularization of mobile devices, business personnel of existing financial or insurance companies have also widely used smartphones as a tool to assist in completing transactions. For example, when a business person negotiates a loan contract with a customer, and the signing process is to be executed, the business person will need to use a smartphone to take pictures of the customer's relevant documents, such as identity cards, land ownership certificates, financial proofs, etc. File, and then generate multiple file images. Then, the smart phone transmits the document images to a server system via the network, so that the undertaking staff located in the company can read the document images to complete the case review. In this way, the inconvenience and time-consuming caused by traditional physical file transfer can be solved. However, when the images of the files uploaded by the smart phone are wrong or unclear, the business person also needs to receive instructions from the undertaker before knowing that it must be re-shot, which also causes another kind of time delay and delays. Customer perception is not good. Therefore, how to determine the correctness of the document image to reduce the number of repairs and improve customer service Service becomes a problem to be solved.

Therefore, the purpose of the present invention is to provide a mobile document identification system that can run on a mobile device.

Therefore, according to the present invention, a mobile document recognition system is provided, which is suitable for a user and includes a mobile device. The mobile device includes an image capture module, a display module, and an image recognition module.

The image capture module is used to capture images to generate a preview image, and the user generates a capture command so that the image capture module saves the preview image including a written document as a waiting Identify document images. The display module is electrically connected to the image capturing module to display the preview image, and also displays a prompt box and a prompt message. The image recognition module is electrically connected to the image capture module and the display module, and uses deep learning technology for image recognition, and includes a plurality of classifiers, each of which corresponds to a plurality of categories. Image recognition.

Wherein, the written document belongs to one of these categories, and the user selects one of the classifiers corresponding to the category of the written document as a selected classifier. The selected classifier of the image recognition module performs image recognition on the preview image, so that when the selected classifier recognizes the preview image that the written document belongs to one of the categories, by the display module The prompt message informs the user of the successful identification, so that the user generates the retrieval command.

The selected classifier also performs image recognition on the document image to be recognized again, and when successfully recognizing that the written document belongs to one of the categories, the document image to be recognized is stored as a recognized document image.

In some embodiments, when the selected classifier determines that the preview image does not include the written document, the prompt message of the display module is used to notify the user to place the image corresponding to the written document in the prompt Within the box. When the selected classifier cannot identify the preview image that the written document belongs to one of the categories, the prompt message of the display module is used to notify the user that the written document is incorrect. When the selected classifier is unable to identify the document to be identified as belonging to one of the categories, the user is notified by the prompt message of the display module that the document is incorrect.

In some implementation aspects, when the selected classifier determines that the preview image does not contain the written document, the display module further displays the prompt box in red. When the selected classifier recognizes the preview image that the written document belongs to one of the categories, the display module also displays the prompt box in green.

In other implementations, when each of the classifiers performs image recognition, the preview image or the document image to be recognized is first scaled down so that the length and width of the image are less than or equal to a preset size, and then A black background image is added to the image, and the part of the image to be predicted and identified is adjusted to be in the middle of the entire image. Then, after normalizing the color value of each pixel in the image, a preprocessed image is obtained. Every A first Convolutional Neural Network (CNN) model of the classifier receives the preprocessed image for identification, and then obtains a probability value.

In some embodiments, the selected classifier determines whether the preview image corresponding to the preprocessed image or the document image to be recognized belongs to one of the categories according to the probability value.

In some implementation aspects, the first convolutional neural network model of each classifier is based on a layout, a font, a spacing, a stamp, and a stamp of the written document in the preprocessed image. The characteristics of at least one of the color and a special pattern are used for image recognition.

In other embodiments, the mobile document recognition system further includes a server system, and the mobile device establishes a connection with the server system to send the recognized document image to the server system for storage.

In other embodiments, the mobile document recognition system further includes a servo system, the servo system includes an image recognition unit, and the image recognition unit also uses deep learning technology for image recognition. The mobile device establishes a connection with the server system to send the recognized document image to the server system, and the image recognition unit of the server system performs image recognition on the recognized document image, and successfully recognizes the written document When it belongs to one of these categories, the recognized document image is stored.

In some embodiments, the image recognition unit of the servo system The element includes a plurality of classifiers corresponding to the categories, and a second convolutional neural network model of each classifier corresponds to the first convolutional neural network model of the same category, and each of the first convolutional neural network models The neural network model is related to the corresponding second convolutional neural network model. The storage space occupied by each classifier of the image recognition module of the mobile device is smaller than that of the corresponding classifier of the image recognition unit of the servo system.

In other implementation aspects, the categories include at least one of an identity category, a land category, a housing category, a financial capability category, a trading category, and an interest payment category.

The effect of the present invention is that the multiple classifiers included in the image recognition module respectively correspond to multiple types of different written documents, so that the user first selects the corresponding classifier according to the type of the written document to be recognized, and It can effectively and accurately identify the category of the written document. More importantly: By optimizing the design of each classifier, it can effectively reduce the storage space occupied, and even when the mobile device is not connected to the Internet, it can operate independently and execute the image correctly. Recognition function.

1: mobile device

11: Image capture module

12: Display module

13: Image recognition module

131: Classifier

132: Classifier

2: Servo system

21: Image recognition unit

211: Classifier

212: Classifier

9: Internet

Other features and effects of the present invention will be clearly presented in the embodiments with reference to the drawings, in which: FIG. 1 is a block diagram illustrating an embodiment of the mobile document identification system of the present invention.

Before the present invention is described in detail, it should be noted that in the following description, similar elements are denoted by the same numbers.

Referring to FIG. 1, an embodiment of the mobile document identification system of the present invention is suitable for a user and includes a mobile device 1 and a servo system 2. The servo system 2 includes an image recognition unit 21 that uses deep learning technology for image recognition. In more detail, the image recognition unit 21 includes a plurality of classifiers 211, 212, and a first Convolutional Neural Network (CNN) model of each of the classifiers 211, 212 is used to identify a plurality of A written document of one of the corresponding categories.

In this embodiment, the categories include an identity category, a land category, a house category, a financial capability category, a transaction category, and an interest category, but are not limited to this. For example, the written documents belonging to the identity category are the front and back of the ID card; the written documents belonging to the land category are the land title; the written documents belonging to the house category are the building ownership; the written documents belonging to the financial category It is a financial proof and a list of property ownership; the written document that belongs to the transaction category is the sales contract; the written document that belongs to the interest payment category is the interest payment certificate for the past 6 months.

The mobile device 1 is, for example, a smart phone, a tablet computer, or other portable electronic equipment, and includes an image capture module 11, a display module 12, And an image recognition module 13. The image capturing module 11 is used for capturing images to generate a preview image, and by the user generating a capturing command, the image capturing module 11 will include the preview of the written document to be recognized The image is saved as a document image to be recognized. The display module 12 is electrically connected to the image capturing module 11 to display the preview image, and also displays a prompt box and a prompt message.

The image recognition module 13 is electrically connected to the image capture module 11 and the display module 12, and also uses deep learning technology for image recognition, and includes a plurality of classifiers 131, 132, the classifiers 131, 132 Respectively correspond to these categories for image recognition. It should be specially reminded that: in this embodiment, the number of such categories is 6, therefore, the number of such classifiers is also 6, and for the convenience of description, only two are exemplified in Figure 1. The classifiers 131, 132 or 211, 212 do not indicate the actual number of classifiers.

More specifically, the mobile device 1 is pre-installed with an application program (APP). When the user wants to identify the written document (for example, the client's ID card), first click to execute the application, and select the One of the classifiers 131 and 132 corresponding to the category to which the written document belongs is used as a selected classifier, for example, the classifier corresponding to the identity category. Then, the user points the camera lens of the image capturing module 11 of the mobile device 1 at the written document, and after the user selects the selected classifier, the image recognition module 13 starts the preview The image is used for image recognition.

When the selected classifier determines that the preview image does not contain the written document, Then, the prompt message and the prompt box of the display module 12 notify the user to place the image corresponding to the written document in the prompt box. For example, the prompt message is "please aim at the watermark text in the prompt box", and the prompt box is displayed in red.

When the selected classifier cannot recognize the preview image that the written document belongs to the category (such as the identity category), the prompt message of the display module 12 is used to notify the user that the written document is incorrect.

When the selected classifier recognizes the preview image that the written document belongs to the category (such as the identity category), the prompt message and the prompt box of the display module 12 are used to notify the user of the successful identification, and then This allows the user to generate the capture command, such as pressing a corresponding shutter button of the mobile device 1. For example, the prompt message is "This may be the front of a correct ID card", and the prompt box is displayed in green.

After the user generates the capturing command, the image capturing module 11 generates the document image to be recognized, and the selected classifier further performs image recognition on the document image to be recognized. When the selected classifier successfully recognizes that the written document belongs to the category (such as the identity category), the document image to be recognized is stored as a recognized document image. Conversely, when the selected classifier fails to recognize that the written document belongs to the category (such as the identity category) from the image of the document to be recognized, the prompt message of the display module 12 is used to notify the user that the written document is incorrect .

In more detail, when each of the classifiers 131 and 132 performs image recognition, the preview image or the document image to be recognized is scaled down first, so that the length of the image and the The width is less than or equal to a preset size to improve the overall recognition efficiency; then add a black background image to the image, and adjust the part of the image to be predicted and recognized in the middle of the entire image to improve the accuracy of the recognition; After the color value of each pixel in is normalized, a preprocessed image is obtained. The preset size is, for example, 120 pixels (pixel/px), and normalizing the color value is to convert it to a value between 0 and 1, so as to improve the accuracy of recognition.

A second convolutional neural network model of each of the classifiers 131, 132 receives the pre-processed image, and according to the pre-processed image, the written document’s one format, one font, one spacing, one stamp, one The characteristics of at least one of the color and a special pattern are used for image recognition to obtain a probability value. The probability value is between 0 and 1. The selected classifier determines whether the preview image corresponding to the preprocessed image or the document image to be identified belongs to the category corresponding to the classifiers 131 and 132 according to the magnitude of the probability value. For example, when the probability value is greater than a preset threshold, it is determined to belong to the category corresponding to the classifiers 131 and 132. The preset threshold is, for example, 0.5 or 0.8. The higher the preset threshold, the higher the The image must have more clear features before it can be judged as belonging to this category.

After the written document is successfully identified as belonging to the category (such as the identity category), the document image to be identified is stored as the identified document image. The mobile device 1 establishes a connection with the server system 2 via a network 9 to send the recognized document image to the server system 2, and the image recognition unit 21 of the server system 2 corresponds to The first convolutional neural network model of the classifier 211, 212 of the category (such as the identity category) performs image recognition on the recognized document image, and successfully identifies that the written document belongs to the category (if the identity category) ), save the recognized document image.

It should be particularly emphasized that the second convolutional neural network model belonging to the same category is related to the corresponding first convolutional neural network model. For example, the classifiers 211 and 212 of the image recognition unit 21 of the servo system 2 adopt the related technologies of Keras and Google Xception to establish the first convolutional neural network models, and the mobile device 1 The classifiers 131, 132 of the image recognition module 13 adopt the related technologies of Keras, Google Xception, and CoreML to lighten and optimize the first convolutional neural network models, and establish the second convolutional neural network models. Convolutional neural network model. Refer to the following table to illustrate the comparison results of the recognition accuracy caused by different degrees of lightweight and optimization of the original Google Xception model with a total of 15 layers, but it is not limited to this. In more detail, the deep learning development tool Keras has relevant settings during model building that can be used for model optimization, for example, model pruning (Pruning or Network reduction). Pruning is due to the large number of nodes in the neural network algorithm and the huge amount of calculation. Therefore, in order to accelerate the use of the model, the nodes with smaller weights are usually adjusted, such as directly reducing the weight to 0, and Can reduce the amount of calculations. In the optimization of Google Xception layer reduction, the related technology of Ensemble learning is used to complete the learning task by constructing and combining multiple models to determine which layer should be removed each time. (Block) will not cause the accuracy to drop too much.

In this embodiment, the servo system 2 uses the first convolutional neural network models with 15 layers, and the mobile device 1 uses 12 removed layers and 3 layers With the second convolutional neural network model, the classifiers 131 and 132 of the mobile device 1 occupy only 0.221MB of storage space, compared to the classifiers 211 and 212 of the server system 2 that require 239MB of storage space. Not only the storage space used is relatively small and it is suitable for use on the mobile device 1, but it also does not have an unacceptable impact on the accuracy of identification, and it can take into account both the use capacity and the correctness at the same time.

In addition, it should be noted that: in this embodiment, the servo system 2 will perform image recognition again on the recognized document image through the image recognition unit 21, and In other embodiments, the servo system 2 may not perform image recognition again on the recognized document image, but directly store the recognized document image.

In summary, multiple classifiers are used to identify written documents belonging to different categories, and by lightening and optimizing the convolutional neural network model, it not only effectively reduces the storage space occupied, but also facilitates installation In a mobile device, it can also take into account the correctness of recognition, so that even when the mobile device is not connected to the Internet, it can still use its image recognition module to independently recognize the category of the written document, so it can indeed be achieved. The purpose of the present invention.

However, the above are only examples of the present invention. When the scope of implementation of the present invention cannot be limited by this, all simple equivalent changes and modifications made in accordance with the scope of the patent application of the present invention and the content of the patent specification still belong to Within the scope covered by the patent of the present invention.

1: mobile device

11: Image capture module

12: Display module

13: Image recognition module

131: Classifier

132: Classifier

2: Servo system

21: Image recognition unit

211: Classifier

212: Classifier

9: Internet

Claims (10)

A mobile document recognition system is suitable for a user and includes a mobile device. The mobile device includes: an image capturing module for capturing images to generate a preview image, and generating a preview image by the user The capture command causes the image capture module to store the preview image containing a written document as a document image to be recognized; a display module is electrically connected to the image capture module to display the preview image, and return Display a prompt box and a prompt message; and an image recognition module, which is electrically connected to the image capture module and the display module, and uses deep learning technology for image recognition, and includes multiple classifiers , The classifiers respectively correspond to a plurality of categories for image recognition, where the written document belongs to one of these categories, and the user selects one of the classifiers corresponding to the category of the written document as A selected classifier, the selected classifier of the image recognition module performs image recognition on the preview image, so that when the selected classifier recognizes the preview image that the written document belongs to one of the categories, by The prompt message of the display module informs the user of the successful identification, so that the user generates the capture command. The selected classifier also performs image identification again on the to-be-identified document image, and the written document is successfully identified When it belongs to one of these categories, the document image to be identified is stored as an identified document image. The mobile document identification system according to claim 1, wherein: when the selected classifier determines that the preview image does not contain the written document, the prompt message of the display module is used to notify the user that the corresponding written document is The image is placed in the prompt box, and when the selected classifier cannot recognize the preview image that the written document belongs to one of the categories, the user is notified of the written document by the prompt message of the display module The document is incorrect. When the selected classifier fails to identify the document to be identified as one of the categories, the user will be notified by the prompt message of the display module that the document is incorrect. . The mobile document identification system according to claim 2, wherein: when the selected classifier determines that the preview image does not contain the written document, the display module also displays the prompt box in red, and when the selected classifier responds to the When the preview image recognizes that the written document belongs to one of these categories, the display module also displays the prompt box in green. The mobile document recognition system according to claim 2, wherein when each of the classifiers performs image recognition, the preview image or the document image to be recognized is scaled down first, so that the length and width of the image are less than or equal to a preset Size, and then add a black base map to the image, and adjust the part of the image to be predicted and identified in the middle of the entire image, and then normalize the color value of each pixel in the image to obtain a preprocessed image , A first convolutional neural network (Convolutional Neural Network, CNN) model of each classifier receives the preprocessed image for identification, and then Obtain a probability value. The mobile document identification system according to claim 4, wherein the selected classifier determines whether the preview image corresponding to the preprocessed image or the document image to be identified belongs to one of the categories according to the probability value By. The mobile document identification system according to claim 5, wherein the first convolutional neural network model of each classifier is based on a layout, a font, and a spacing of the written document in the preprocessed image The characteristics of at least one of, a stamp, a color, and a special pattern are used for image recognition. The mobile document identification system according to claim 5 further includes a server system, and the mobile device establishes a connection with the server system to send the identified document image to the server system for storage. The mobile document recognition system according to claim 5 further includes a servo system, the servo system includes an image recognition unit, the image recognition unit also uses deep learning technology for image recognition, the mobile device is connected with the servo system Line to send the recognized document image to the servo system, and the image recognition unit of the servo system performs image recognition on the recognized document image, and when it successfully recognizes that the written document belongs to one of the categories To save the recognized document image. The mobile document recognition system according to claim 8, wherein the image recognition unit of the servo system includes a plurality of classifiers corresponding to the categories, and a second convolutional neural network model of each of the classifiers corresponds to the same A first convolutional neural network model of the category, each of the first convolutional neural network model is related to the corresponding second convolutional neural network model, and the mobile device The storage space occupied by each classifier of the image recognition module is smaller than that of the corresponding classifier of the image recognition unit of the servo system. The mobile document identification system according to claim 1, wherein the categories include at least one of an identity category, a land category, a house category, a financial capability category, a transaction category, and an interest category.

Images