CN116403226A - Unconstrained fold document image correction method, system, equipment and storage medium - Google Patents

Abstract

The invention discloses a method, a system, equipment and a storage medium for correcting unconstrained fold document images, which are one-to-one corresponding schemes, and the related schemes can solve the problem that the application scene of the existing scheme is limited, namely, deformed document images without document boundaries or only containing partial document boundaries can not be corrected. Meanwhile, the invention also improves the correction and recovery effects on the image with the complete document boundary. Compared with the traditional method, the method has no constraint on any form of the input fold document image, can more robustly and accurately correct various deformation document images shot in daily life, can be widely applied to portable equipment with cameras such as smart phones, and has wider application scenes and higher accuracy. Therefore, the invention greatly promotes the popularization of the digitization of the document image and provides powerful technical support for the digitization conversion of the paper document.

Description

Unconstrained wrinkled document image correction method, system, device and storage medium

technical field

The present invention relates to the technical field of wrinkled document image correction, in particular to an unconstrained wrinkled document image correction method, system, device and storage medium.

Background technique

With the rapid progress and popularity of portable cameras and smart phones, more and more people choose to use them to scan paper documents instead of using dedicated flatbed scanners as before. However, due to many uncertain factors in the shooting environment, such as uncertain camera position, uncertain lighting environment, and uncertain types and degrees of paper deformation, document images captured by these devices often have varying degrees of various distortion and deformation. This makes the processing of downstream tasks, such as automated text recognition, content analysis, editing, and comprehension, much more difficult. At the same time, it is not conducive to the dissemination and exchange of information and knowledge in daily life. In order to solve this problem, wrinkled document image correction has become an important research topic in the field of computer vision.

Traditional solutions are mainly based on 3D reconstruction techniques. These methods usually need to rely on additional hardware devices (such as laser scanners, depth cameras, etc.), or take multi-view images around the wrinkled paper to reconstruct the three-dimensional structure of the paper, and then perform flattening and correction on this basis. However, the popularization and use of these technologies has been greatly limited due to high hardware costs or cumbersome shooting requirements.

Currently, many smartphones have built-in document correction algorithms. Most of these algorithms are based on projection transformation technology: first, detect the four straight edges or four corner points of the paper document in the captured document image to form a quadrilateral area where the document is located; then apply projection transformation technology to map it into a regular rectangular image, thus Correction of the captured document image is completed. However, this solution requires that a complete document must appear in the captured image, and if the document itself is deformed, it cannot be corrected and restored, which will affect the effect. This limitation also brings inconvenience: many times, users may only focus on some areas of the document.

In recent years, deep learning has been introduced into the field of wrinkled document image correction. Compared with traditional methods, deep learning-based methods require less computational overhead while achieving similar performance. The neural network learns the ability to correct wrinkled documents by training with a large number of deformed-undistorted image pairs synthesized by the rendering engine. In the inference stage, a single wrinkled RGB document image is input, and the neural network can output a pixel-by-pixel coordinate mapping matrix, which samples the pixels in the wrinkled document area in the input image into the empty image to obtain a complete rectified image.

Generally speaking, whether it is the built-in document correction algorithm of the smartphone or the existing deep learning method, they mainly have the following defects:

(1) The current document image correction algorithm based on deep learning can generally only correct wrinkled document images with complete boundaries, that is, the input image must contain a complete document. However, in practical application scenarios, users may only want to follow or share some regions or text in a document. Therefore, captured images may have missing document boundaries. In addition, document images captured by mobile phones often have missing edges. In this case, the existing document image correction method will be invalid, and a normal correction result cannot be obtained. The current technical solutions lack effective research on the rectification of document images without document boundaries or only contain part of document boundaries, and further exploration and improvement are needed.

(2) The applicable scenarios of the document image correction algorithm built into the current smart phone are limited. These algorithms only work on complete, undistorted document images, that is, the paper document is free of folds, bends and wrinkles and appears intact in the captured image. In simple terms, these algorithms just switch the imaging projection plane of the paper document to a regular rectangular shape. Once the shape of the paper document is not a regular quadrilateral, these algorithms cannot normally complete the document image correction.

(3) With existing document image correction algorithms based on deep learning, the corrected document image still has a certain degree of distortion. This is because, during model training, only document images with complete boundaries are considered, and document images with no document boundaries or only partial document boundaries are ignored. Incorporating the latter into model training can effectively improve the accuracy and robustness of the model. The reason is that adding document images with no document boundaries or only part of the document boundaries to the training can improve the generalization of the model and allow the model to learn more effectively how to use the only features such as deformed text lines in the image to correct the image.

In view of this, the present invention is proposed.

Contents of the invention

The purpose of the present invention is to provide a method, system, device and storage medium for unconstrained wrinkled document image correction, which can correct deformed document images without document boundaries or only contain part of document boundaries, and can also improve the correction effect of images with complete document boundaries . In a word, the present invention can effectively correct and restore various deformed document images, and can effectively improve the practicability and practicality of document image correction, regardless of the constraints of document boundary integrity and deformation degree. Apply effects.

The purpose of the present invention is achieved by the following technical solutions:

An unconstrained wrinkled document image correction method, comprising:

modeling the pixel mapping relationship from the wrinkled document image to the undeformed document image, and generating sample pairs, wherein each sample pair includes an unconstrained wrinkled document image block and a coordinate mapping matrix from an unconstrained wrinkled document image block to an undeformed document image block;

Construct an unconstrained document image correction network, and use multiple samples to train the formed training data set;

The unconstrained wrinkled document image is input to the trained unconstrained document image correction network to obtain a predicted coordinate mapping matrix, and the unconstrained wrinkled document image is corrected by using the predicted coordinate mapping matrix to obtain a corrected image.

An unconstrained wrinkled document image correction system comprising:

The pixel mapping relationship modeling and sample pair generation unit is used to model the pixel mapping relationship from the wrinkled document image to the undeformed document image, and generate sample pairs, wherein each sample pair includes an unconstrained wrinkled document image block and an unconstrained wrinkled document A coordinate mapping matrix from the image block to the image block of the non-deformed document;

The network construction and training unit is used to construct an unconstrained document image correction network, and use multiple samples to train the formed training data set;

The image correction unit is configured to input the unconstrained wrinkled document image to the trained unconstrained document image correction network to obtain a predicted coordinate mapping matrix, and use the predicted coordinate mapping matrix to correct the unconstrained wrinkled document image to obtain the corrected image.

A processing device comprising: one or more processors; memory for storing one or more programs;

Wherein, when the one or more programs are executed by the one or more processors, the one or more processors are made to implement the aforementioned method.

A readable storage medium stores a computer program, and implements the aforementioned method when the computer program is executed by a processor.

It can be seen from the above-mentioned technical solution provided by the present invention that it can solve the problem of limited application scenarios of the existing solution, that is, it cannot correct deformed document images without document boundaries or only containing part of document boundaries. At the same time, the present invention also improves the rectification and restoration effect for images with complete document boundaries. Compared with traditional methods, the present invention does not have any formal constraints on the input wrinkled document images, and can more robustly and accurately correct various deformed document images captured in daily life. The present invention can be widely used in smartphones, etc. Portable devices with cameras have wider application scenarios and higher accuracy. Therefore, the invention will greatly promote the popularization of document image digitization, and provide strong technical support for the digitization conversion of paper documents.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For Those of ordinary skill in the art can also obtain other drawings based on these drawings on the premise of not paying creative efforts.

FIG. 1 is a flow chart of an unconstrained wrinkled document image correction method provided by an embodiment of the present invention;

FIG. 2 is a schematic diagram of modeling the pixel mapping relationship between an input deformed document image and an output undeformed document provided by an embodiment of the present invention;

FIG. 3 is a flow chart of realizing distorted image correction based on an unconstrained document image correction network provided by an embodiment of the present invention;

FIG. 4 is a schematic diagram of an unconstrained wrinkled document image correction system provided by an embodiment of the present invention;

Fig. 5 is a schematic diagram of a processing device provided by an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

First, the terms that may be used in this article are explained as follows:

The terms "comprising", "comprising", "containing", "having" or other descriptions with similar meanings shall be construed as non-exclusive inclusions. For example: including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.), should be interpreted as including not only a certain technical feature element explicitly listed, but also other technical feature elements not explicitly listed in the art.

The term "consisting of" means excluding any technical characteristic elements not explicitly listed. If this term is used in a claim, the term will make the claim closed so that it does not contain technical characteristic elements other than those expressly listed, except for conventional impurities related to them. If the term only appears in a certain clause of a claim, it only limits the elements explicitly listed in the clause, and the elements stated in other clauses are not excluded from the entire claim.

A method, system, device and storage medium for unconstrained wrinkled document image correction provided by the present invention will be described in detail below. The content not described in detail in the embodiments of the present invention belongs to the prior art known to those skilled in the art. In the embodiment of the present invention, if no specific conditions are indicated, it is carried out according to the conventional conditions in the art or the conditions suggested by the manufacturer.

Embodiment one

An embodiment of the present invention provides an unconstrained wrinkled document image correction method, as shown in Figure 1, which mainly includes:

Step 1. By modeling the pixel mapping relationship between wrinkled document images and undeformed document images, sample pairs are generated, wherein the sample pairs include unconstrained wrinkled document image blocks and coordinate mapping from unconstrained wrinkled document image blocks to undeformed document image blocks matrix.

As shown in Figure 2, the preferred implementation of this step is as follows:

(1) Global correction. A wrinkled document image with a complete boundary is obtained, and then the wrinkled document image with a complete boundary is rectified into a non-deformed document image by using its corresponding coordinate mapping matrix.

In the example of the present invention, the wrinkled document image with complete boundaries and its coordinate mapping matrix are all from the existing public data set. The coordinate mapping matrix describes the coordinate mapping relationship between each pixel of the wrinkled document image and the corresponding undeformed document image. That is, the pixel position of each pixel in the undeformed document image in the wrinkled document image.

(2) Modeling of local coordinate mapping relationship. An image block of an area randomly intercepted in the wrinkled document image with complete boundaries is called an unconstrained wrinkled document image block, and the corresponding area in the undeformed document image is found according to the coordinate mapping matrix of the area, which is called an undeformed document image Block, and then intercept the matrix of the same area in the coordinate mapping matrix, that is, the coordinate mapping matrix from the unconstrained wrinkled document image block to the undeformed document image block.

As shown in Figure 2, the dotted box in the lower left corner is an image of a region randomly intercepted, that is, the unconstrained wrinkled document image block, and the dotted line box in the lower right corner is the corresponding undeformed document image block. Since the image of an area is randomly intercepted here, the wrinkled document image without document boundary or complete document boundary is obtained, of course, it can also be a wrinkled document image with complete document boundary, so it is called unconstrained fold Document image block.

In the embodiment of the present invention, for each wrinkled document image with a complete boundary, the unconstrained wrinkled document image block and the unconstrained wrinkled document image block can be obtained by modeling the pixel mapping relationship between the wrinkled document image and the undeformed document image The coordinate mapping matrix to the image block of the undeformed document, the two form a sample pair.

In the embodiment of the present invention, a training data set can be formed by a plurality of sample pairs; wherein, for each wrinkled document image with a complete boundary, after performing global correction, local coordinate mapping relationship modeling can be performed one or more times to obtain a or multiple sample pairs; of course, multiple wrinkled document images with complete boundaries can be modeled as shown in FIG. 2 to obtain corresponding sample pairs. The number of specific sample pairs can be set according to the actual situation or experience.

It should be noted that Figure 2 mainly presents the principle of modeling the pixel mapping relationship between a wrinkled document image and a non-deformed document image. Considering privacy issues, the text in the document image has been blurred, but this does not affect the implementation of the scheme. In actual application, the sharpness of the document image will not be adjusted.

Step 2. Construct an unconstrained document image correction network, and use multiple samples to train the formed training data set.

In the embodiment of the present invention, the document image correction network may be a fully convolutional neural network, such as a UNet network, which mainly includes a feature extractor and a feature decoder.

During training, the input is the unconstrained wrinkled document image block in the sample pair, feature extraction is performed through the feature extractor, and the predicted coordinate mapping matrix is output through the feature decoder, and the unconstrained wrinkled document image block in the sample pair is converted to the undeformed document The coordinate mapping matrix of the image block is used as supervision information, and a loss function is constructed with the predicted coordinate mapping matrix to train the unconstrained document image rectification network.

The training process can be implemented with reference to conventional technologies, and the present invention does not repeat it. When the set stop conditions are met (for example, the number of training times reaches the set number of times, or the loss function converges, etc.), the training is stopped.

Step 3: Input the unconstrained wrinkled document image into the trained unconstrained document image correction network to obtain a predicted coordinate mapping matrix, and use the predicted coordinate mapping matrix to correct the unconstrained wrinkled document image to obtain a corrected image.

In the embodiment of the present invention, the unconstrained wrinkled document image can be a deformed image I _d in any wrinkled form, as shown in Figure 3, it can be the wrinkled document image with complete boundaries shown in part (a), it can be (b) The wrinkled document image without document boundary as shown in part (c) can also be the wrinkled document image without complete document boundary shown in part (c); feature extraction and feature decoding are performed through the trained unconstrained document image correction network, Output the predicted coordinate mapping matrix f _b , and then use an upsampling algorithm (eg, bilinear interpolation algorithm) to correct the unconstrained wrinkled document image through the predicted coordinate mapping matrix f _b to obtain a corrected image I _r .

The above solutions provided by the embodiments of the present invention can solve the problem of limited application scenarios of the existing solutions, that is, it is impossible to correct deformed document images without document boundaries or only containing part of document boundaries. At the same time, the present invention also improves the rectification and restoration effect for images with complete document boundaries. Compared with traditional methods, the present invention does not have any formal constraints on the input wrinkled document images, and can more robustly and accurately correct various deformed document images captured in daily life. The present invention can be widely used in smartphones, etc. Portable devices with cameras have wider application scenarios and higher accuracy. Therefore, the invention will greatly promote the popularization of document image digitization, and provide strong technical support for the digitization conversion of paper documents.

Embodiment two

An embodiment of the present invention provides an unconstrained wrinkled document image correction system, as shown in Figure 4, the system mainly includes:

The pixel mapping relationship modeling and sample pair generation unit is used to model the pixel mapping relationship from the wrinkled document image to the undeformed document image, and generate sample pairs, wherein each sample pair includes an unconstrained wrinkled document image block and an unconstrained wrinkled document A coordinate mapping matrix from the image block to the image block of the non-deformed document;

The network construction and training unit is used to construct an unconstrained document image correction network, and use multiple samples to train the formed training data set;

The image correction unit is configured to input the unconstrained wrinkled document image to the trained unconstrained document image correction network to obtain a predicted coordinate mapping matrix, and use the predicted coordinate mapping matrix to correct the unconstrained wrinkled document image to obtain the corrected image.

In the embodiment of the present invention, the modeling of the pixel mapping relationship between the wrinkled document image and the undeformed document image, and generating the sample pair include:

Obtaining a wrinkled document image with a complete boundary, and then using its corresponding coordinate mapping matrix to correct the wrinkled document image with a complete boundary to a non-deformed document image;

An image block of an area randomly intercepted in the wrinkled document image with complete boundaries is called an unconstrained wrinkled document image block, and the corresponding area in the undeformed document image is found according to the coordinate mapping matrix of the area, which is called an undeformed document image Block, and then intercept the matrix of the same area in the coordinate mapping matrix, that is, the coordinate mapping matrix from the unconstrained wrinkled document image block to the undeformed document image block;

The obtained unconstrained wrinkled document image block and the coordinate mapping matrix from the unconstrained wrinkled document image block to the undeformed document image block form a sample pair.

In the embodiment of the present invention, the construction of an unconstrained document image correction network, and using multiple samples to train the formed training data set includes:

Construct an unconstrained document image rectification network including feature extractor and feature decoder;

During training, the input is the unconstrained wrinkled document image block in the sample pair, feature extraction is performed through the feature extractor, and the predicted coordinate mapping matrix is output through the feature decoder, and the unconstrained wrinkled document image block in the sample pair is converted to the undeformed document The coordinate mapping matrix of the image block is used as supervision information, and a loss function is constructed with the predicted coordinate mapping matrix to train the unconstrained document image rectification network.

In the embodiment of the present invention, the correction of the unconstrained wrinkled document image by using the predicted coordinate mapping matrix, and obtaining the corrected image includes:

Using an up-sampling algorithm, the unconstrained wrinkled document image is corrected by predicting a coordinate mapping matrix to obtain a corrected image.

Embodiment three

The present invention also provides a processing device, as shown in FIG. 5 , which mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the When the one or more processors execute, the one or more processors implement the methods provided in the foregoing embodiments.

Further, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected through a bus.

In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example:

The input device can be a touch screen, an image acquisition device, a smart phone, a physical button or a mouse, etc.;

The output device can be a display terminal;

The memory may be random access memory (Random Access Memory, RAM), or non-volatile memory (non-volatile memory), such as disk memory.

Embodiment four

The present invention also provides a readable storage medium storing a computer program, and when the computer program is executed by a processor, the methods provided in the foregoing embodiments are realized.

The readable storage medium in the embodiment of the present invention is used as a computer-readable storage medium, and may be set in the foregoing processing device, for example, as a memory in the processing device. In addition, the readable storage medium may also be various media capable of storing program codes, such as a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a magnetic disk, or an optical disk.

The above is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person familiar with the technical field can easily conceive of changes or changes within the technical scope disclosed in the present invention. Replacement should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.

Claims (10)

1. An unconstrained wrinkle document image correction method, is characterized in that, comprises: modeling the pixel mapping relationship from the wrinkled document image to the undeformed document image, and generating sample pairs, wherein each sample pair includes an unconstrained wrinkled document image block and a coordinate mapping matrix from an unconstrained wrinkled document image block to an undeformed document image block; Construct an unconstrained document image correction network, and use multiple samples to train the formed training data set; The unconstrained wrinkled document image is input to the trained unconstrained document image correction network to obtain a predicted coordinate mapping matrix, and the unconstrained wrinkled document image is corrected by using the predicted coordinate mapping matrix to obtain a corrected image. 2. A method for correcting unconstrained wrinkled document images according to claim 1, wherein said modeling the pixel mapping relationship between wrinkled document images and undeformed document images, and generating sample pairs include: Obtaining a wrinkled document image with a complete boundary, and then using its corresponding coordinate mapping matrix to correct the wrinkled document image with a complete boundary to a non-deformed document image; An image block of an area randomly intercepted in the wrinkled document image with complete boundaries is called an unconstrained wrinkled document image block, and the corresponding area in the undeformed document image is found according to the coordinate mapping matrix of the area, which is called an undeformed document image Block, and then intercept the matrix of the same area in the coordinate mapping matrix, that is, the coordinate mapping matrix from the unconstrained wrinkled document image block to the undeformed document image block; The obtained unconstrained wrinkled document image block and the coordinate mapping matrix from the unconstrained wrinkled document image block to the undeformed document image block form a sample pair. 3. A kind of unconstrained wrinkled document image rectification method according to claim 1, it is characterized in that, said construction unconstrained document image rectification network, and utilize a plurality of samples to train the training data set that forms comprises: Construct an unconstrained document image rectification network including feature extractor and feature decoder; During training, the input is the unconstrained wrinkled document image block in the sample pair, feature extraction is performed through the feature extractor, and the predicted coordinate mapping matrix is output through the feature decoder, and the unconstrained wrinkled document image block in the sample pair is converted to the undeformed document The coordinate mapping matrix of the image block is used as supervision information, and a loss function is constructed with the predicted coordinate mapping matrix to train the unconstrained document image rectification network. 4. A method for correcting an unconstrained wrinkled document image according to claim 1, wherein said correcting said unconstrained wrinkled document image by using said predicted coordinate mapping matrix, and obtaining a corrected image comprises: Using an up-sampling algorithm, the unconstrained wrinkled document image is corrected by predicting a coordinate mapping matrix to obtain a corrected image. 5. An unconstrained wrinkled document image correction system, characterized in that it comprises: The pixel mapping relationship modeling and sample pair generation unit is used to model the pixel mapping relationship from the wrinkled document image to the undeformed document image, and generate sample pairs, wherein each sample pair includes an unconstrained wrinkled document image block and an unconstrained wrinkled document A coordinate mapping matrix from the image block to the image block of the non-deformable document; The network construction and training unit is used to construct an unconstrained document image correction network, and use multiple samples to train the formed training data set; The image correction unit is configured to input the unconstrained wrinkled document image to the trained unconstrained document image correction network to obtain a predicted coordinate mapping matrix, and use the predicted coordinate mapping matrix to correct the unconstrained wrinkled document image to obtain the corrected image. 6. A system for correcting unconstrained wrinkled document images according to claim 5, wherein said modeling the pixel mapping relationship between wrinkled document images and undeformed document images, and generating sample pairs include: Obtaining a wrinkled document image with a complete boundary, and then using its corresponding coordinate mapping matrix to correct the wrinkled document image with a complete boundary to a non-deformed document image; An image block of an area randomly intercepted in the wrinkled document image with complete boundaries is called an unconstrained wrinkled document image block, and the corresponding area in the undeformed document image is found according to the coordinate mapping matrix of the area, which is called an undeformed document image Block, and then intercept the matrix of the same area in the coordinate mapping matrix, that is, the coordinate mapping matrix from the unconstrained wrinkled document image block to the undeformed document image block; The obtained unconstrained wrinkled document image block and the coordinate mapping matrix from the unconstrained wrinkled document image block to the undeformed document image block form a sample pair. 7. A kind of unconstrained wrinkled document image correction system according to claim 5, it is characterized in that, said construction unconstrained document image correction network, and utilize a plurality of samples to train the training data set that forms comprises: Construct an unconstrained document image rectification network including feature extractor and feature decoder; During training, the input is the unconstrained wrinkled document image block in the sample pair, feature extraction is performed through the feature extractor, and the predicted coordinate mapping matrix is output through the feature decoder, and the unconstrained wrinkled document image block in the sample pair is converted to the undeformed document The coordinate mapping matrix of the image block is used as supervision information, and a loss function is constructed with the predicted coordinate mapping matrix to train the unconstrained document image rectification network. 8. An unconstrained wrinkled document image correction system according to claim 5, wherein said correcting said unconstrained wrinkled document image by using said predicted coordinate mapping matrix, and obtaining a corrected image comprises: Using an up-sampling algorithm, the unconstrained wrinkled document image is corrected by predicting a coordinate mapping matrix to obtain a corrected image. 9. A processing device, comprising: one or more processors; a memory for storing one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors are made to implement the method according to any one of claims 1-4. 10. A readable storage medium storing a computer program, wherein the method according to any one of claims 1-4 is implemented when the computer program is executed by a processor.

Images