CN112991470B

CN112991470B - Certificate photo background color checking method and system under complex background

Info

Publication number: CN112991470B
Application number: CN202110170353.6A
Authority: CN
Inventors: 郭大勇; 张海龙; 兰永
Original assignee: Shanghai Tongban Information Service Co ltd
Current assignee: Shanghai Tongban Information Service Co ltd
Priority date: 2021-02-08
Filing date: 2021-02-08
Publication date: 2023-12-26
Anticipated expiration: 2041-02-08
Also published as: CN112991470A

Abstract

The application discloses a certificate background color checking method and system under a complex background, wherein the method comprises the following steps: based on an image segmentation model of the neural network, performing pixel-level segmentation processing on the certificate size, and obtaining background coordinates of the image; segmenting the background part to obtain at least one background sub-image conforming to a preset size; performing color recognition on each background sub-image, and performing color purity recognition at the same time; and (3) judging by integrating the color recognition results and the color purity recognition results of the background sub-images, and judging that the background colors of the certificate candid are compliant when the colors of all the background sub-images are the preset background colors and the pure colors. The method and the device are used for identifying the pure-color background color of the photo, removing the photo such as pure color, complex background and the like, improving the intelligent examination efficiency of the photo background color in the government affair service process and improving the intelligent degree of the business service.

Description

Certificate photo background color checking method and system under complex background

Technical Field

The invention relates to the technical field of image processing of photos, in particular to a certificate photo background color checking method and a certificate photo background color checking system under a complex background.

Background

In the government service working process, the electronic certificate is submitted to serve as an identification, and the background color of the electronic certificate is required to be standardized. And when the background color of the document submitted by people is not standard, for example, the background color is not a designated color, the background color of the photo is not a solid color, and a complex background exists.

Background color screening and detection of the electronic certificate in the past needs manual one-by-one detection, can not automatically give out the detection result of the image, and has low efficiency and easy omission and misjudgment. Along with the development of computer technology, the degree of intellectualization is higher and higher in various fields, but a method for checking the background color of a certificate in the process of government service business is still blank.

Therefore, in the processing process of the certificate candid photograph, whether the solid background color of the candid photograph can be quickly, efficiently and accurately identified, and the intelligent examination efficiency of government affair service and the intelligent degree of office service can be influenced by the pictures such as the solid background and the complex background.

Disclosure of Invention

The invention aims to provide a certificate background color checking method and a certificate background color checking system under a complex background, so as to solve the problems in the technical background.

In order to achieve the above purpose, the present invention adopts the following technical scheme:

the first aspect of the present application provides a method for checking the background color of a document in a complex background, including:

based on an image segmentation model of the neural network, performing pixel-level segmentation processing on the certificate size, and obtaining background coordinates of the image;

segmenting the background part to obtain at least one background sub-image conforming to a preset size;

performing color recognition on each background sub-image, and performing color purity recognition at the same time;

and (3) judging by integrating the color recognition results and the color purity recognition results of the background sub-images, and judging that the background colors of the certificate candid are compliant when the colors of all the background sub-images are the preset background colors and the pure colors.

Preferably, the foreground is a character on the certificate photograph, and the foreground is the background.

Preferably, the image segmentation model is a neural network-based deep learning model.

Preferably, the neural network is obtained through pre-training, and the pre-training of the neural network includes:

step 1: collecting document size photographs, carrying out pixel-level segmentation and labeling on the document size photographs, and marking pixel points of a foreground outline, wherein the non-labeled pixel points are background pixel points, so that a training set is obtained;

step 2: building a network model, wherein the network model comprises the following modules:

the pre-training neural network ResNet module is used for obtaining feature vectors of the pictures and dividing a plurality of regions of interest (ROIs) according to the feature vectors;

the region generation network module is used for carrying out binary classification and class frame regression processing on the region of interest (ROI), and filtering the ROI according to a preset filtering standard;

the ROI alignment module is used for performing ROI alignment operation on the rest ROIs, corresponding unlabeled certificate images with pixel points of the feature vectors, then corresponding the feature vectors with the labeled pixel points, and inputting the feature vectors into the three-branch full convolution neural network to obtain three prediction results: classification, classification box regression and MASK;

step 3: taking the marked training set as the input of a network model, obtaining a MASK, and calculating the total loss function of the whole neural network: l=lcls+lbox+lmask, where L represents the total error, lcls represents the classification error, lbox represents the classification box position error, lmask represents the mask segmentation error;

performing iterative training by using the total loss function, and when the total loss function of the neural network reaches a preset minimum value, completing the training of the neural network to obtain a trained image segmentation model;

step 4: and inputting the unlabeled test image into the trained image segmentation model for testing, obtaining a mask of the foreground, and removing the rest pixel points of the foreground to obtain the background.

More preferably, the marking the certificate size comprises: the certificate inch comprises a foreground and a background, the pixel value of the foreground is set to be 0, the pixel value of the background is set to be 1, a two-dimensional matrix A is obtained, and then contour marking is carried out on all the certificate inch by using a marking tool.

More preferably, the preset filtering criteria include: and calculating the score between each region of interest (ROI) and the foreground of the corresponding label according to a preset standard, and taking the top 2000 of the scores from high to low and reserving the scores.

Preferably, the document size is subjected to pixel-level segmentation processing to obtain a two-dimensional matrix A comprising a foreground and a background, wherein a value 1 represents the background and a value 0 represents the foreground; the segmenting the background part to obtain at least one background sub-image conforming to a preset size comprises the following steps:

dividing the two-dimensional matrix A into at least 4 rectangular areas B with the same size along the transverse direction and the vertical direction;

finding the largest rectangle B which can be formed by the value 1 in each rectangle area B;

if the proportion of the value 1 in the rectangular area B, which is not in the range of the rectangle B, is larger than a preset threshold value, the largest rectangle c which can be formed by the value 1 is found in the area except the rectangle B in the rectangular area B;

respectively judging whether the number of pixel points in the rectangle b and the rectangle c is larger than or equal to a second preset threshold value, if so, dividing the rectangle b and/or the rectangle c into images to be processed; otherwise, the image is not divided into images to be processed;

and mapping rectangular coordinates of all the images to be processed into the certificate photo to obtain at least one background sub-image conforming to the preset size.

Preferably, the color recognition for each background sub-image includes:

when the background sub-image is a gray image, color judgment is directly carried out based on pixel values;

and when the background sub-image is a color image, converting the background sub-image from an RGB color space to an HSV color space, calculating a histogram of an H channel, and performing color judgment according to the maximum H value.

Preferably, the identifying the color purity of each background sub-image includes:

when the background sub-image is a color image, converting into a gray image;

carrying out two-dimensional discrete Fourier transform on the gray level image, intercepting high-frequency components of frequency, and carrying out sharpening treatment on the gray level image;

performing binarization processing on the sharpened image to obtain a binarized image;

and performing line detection on the binarized image by using Hough transformation, and judging the background sub-image as a non-solid background image if the lines are detected.

More preferably, when the background sub-image is a color image, the converting to a gray-scale image includes: in a color image, a color is formed by mixing three primary colors of R, G and B in proportion, an imaging basic unit is a pixel, 3 blocks of pixels are needed to be expressed, R, G and B are respectively represented, if 8 bits represent one color, a certain primary color with different brightness is distinguished by 0-255, a gray image is represented by black with different saturation degrees, for example, the gray degree is represented by 8 bits 0-255, each pixel point only needs one gray value, 8 bits are needed, the conversion between the RGB value and gray is actually the conversion of the brightness feeling perceived by human eyes, and the conversion formula is as follows: gray=0.299×r+0.587×g+0.144×b, according to the formula, R, G and B values of each pixel point in the color image are sequentially read, gray value calculation is performed, gray values are assigned to corresponding positions of the new image, and conversion is completed after all the pixel points are traversed once.

More preferably, the performing two-dimensional discrete fourier transform on the gray image, cutting out a high-frequency component of the frequency, and performing sharpening on the gray image includes:

transforming the gray image from the space domain to the frequency domain, and then performing frequency domain filtering treatment;

the frequency domain filtering adopts high-pass filtering, namely, high-frequency components of the frequency are intercepted, and then inverse Fourier transformation is carried out on the high-frequency components, so that a sharpened image is obtained.

In the above, the high frequency component corresponds to a place where the intensity change is severe in the gray-scale image, that is, an edge portion.

More preferably, the binarization process uses canny edge detection, returning sharp edges detected in the image.

Further, the canny edge detection adopts double-threshold processing and edge connection.

Still further, the dual thresholding and edge connection includes: the edges are connected into contours in the high-threshold image, when the end points of the contours are reached, points meeting the low threshold are found in the neighborhood of the break points, and new edges are collected according to the points until the edges of the whole image are closed.

More preferably, the hough transform includes Huo Fuxian transform and hough circle transform, respectively detecting straight lines and curves.

Further, the Huo Fuxian transform employs an accumulated probability hough transform PPHT.

A second aspect of the present application provides a document inch background color inspection system in a complex background, comprising:

the image segmentation module is used for carrying out pixel-level segmentation processing on the input certificate and obtaining the background coordinates of the image;

the background segmentation module is used for segmenting the background part according to the background coordinates acquired by the image segmentation module to acquire at least one background sub-image conforming to the preset size;

the color recognition module is used for carrying out color recognition on each background sub-image segmented by the background segmentation module;

the color purity identification module is used for identifying the color purity of each background sub-image split by the background splitting module;

and the statistics judging module is used for judging by integrating the color identification result and the color purity identification result of each background sub-image, and judging that the background color of the document is compliant when the colors of all the background sub-images are all the preset background colors and are all the pure colors.

Compared with the prior art, the technical scheme of the invention has the following beneficial effects:

the certificate photo background color checking method and system under the complex background are used for identifying the pure color background color of the photo picture, eliminating pictures such as pure color and complex background, and can improve the intelligent checking efficiency of photo background color in the government service process and the intelligent degree of business service.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application, illustrate and explain the application and are not to be construed as limiting the application. In the drawings:

FIG. 1 is a schematic flow chart of a document background color inspection method under a complex background of the present application;

FIG. 2 is a general flow chart of a document inch background color inspection method according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a process for obtaining a background of an original image according to an embodiment of the present application;

FIG. 4 is a schematic illustration of a flow of forming an image segmentation model of the present application;

FIG. 5 is a schematic diagram illustrating a processing procedure for segmenting a background to obtain n background sub-images conforming to a preset size in an embodiment of the present application;

FIG. 6 is a schematic diagram of a process flow of any background sub-image in an embodiment of the present application;

FIG. 7 is a schematic diagram of a process for processing a background sub-image according to an embodiment of the present application;

FIG. 8 is a schematic diagram of a document background color inspection system in a complex background of the present application.

Detailed Description

In order to make the objects, technical solutions and effects of the present invention clearer and more obvious, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

It is noted that the terms "first," "second," and the like in the description and claims of the present invention and in the foregoing figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order, and it is to be understood that the data so used may be interchanged where appropriate. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Referring to fig. 1, fig. 1 is a schematic flow chart of a document background color inspection method under a complex background of the present application.

The certificate background color inspection method under the complex background mainly comprises the following steps:

step A1: based on an image segmentation model of the neural network, performing pixel-level segmentation processing on the certificate size, and obtaining background coordinates of the image;

step A2: segmenting the background part to obtain at least one background sub-image conforming to a preset size;

step A3: performing color recognition on each background sub-image, and performing color purity recognition at the same time;

step A4: and (3) judging by integrating the color recognition results and the color purity recognition results of the background sub-images, and judging that the background colors of the certificate candid are compliant when the colors of all the background sub-images are the preset background colors and the pure colors.

The foreground is a character on the certificate, and the foreground is a background. The image segmentation model is a deep learning model based on a neural network.

Examples:

the present embodiment is specifically described with reference to a background color inspection process for an unoptionally small document. The original image is hereinafter referred to as an indication of the document to be detected, and the foreground is a person on the indication of the document, and the foreground is the background.

Referring to fig. 2, the method for checking the color of the document background in the complex background of the embodiment specifically includes the following steps:

first step S1: and obtaining the background coordinates of the original image through the image segmentation model.

The image segmentation model is a deep learning model based on a neural network, and after data set preparation, network model construction and model training, the model can be used for predicting an original image, and each pixel point on the original image is divided into a foreground or a background.

Wherein, data set preparation: a large number of certificate inch photographs are collected, pixel-level segmentation and labeling are carried out on the certificate inch photographs, and pixel points of a foreground outline are labeled, and non-labeled background pixel points are labeled, so that a training set is obtained. Referring to fig. 3, dots of the humanoid portion are marked with foreground portions and black portions are background portions.

Wherein, the network model: the system mainly comprises a pre-training neural network ResNet module, a region generation network module, a ROIAlign module and three full convolution neural networks which are respectively output (as shown in figure 4). The pre-training neural network ResNet module is used for obtaining feature vectors of the pictures and dividing a plurality of regions of interest (ROIs) according to the feature vectors; the region generation network module is used for carrying out binary classification and class frame regression processing on the region of interest (ROI), and filtering the ROI according to a preset filtering standard; the ROI alignment module is used for performing ROI alignment operation on the rest ROIs, corresponding unlabeled certificate images with pixel points of the feature vectors, then corresponding the feature vectors with the labeled pixel points, and inputting the feature vectors into the three-branch full convolution neural network to obtain three prediction results: classification, classification box regression, and MASK.

Wherein, model training: taking the marked training set as the input of a network model, obtaining a MASK, and calculating the total loss function of the whole neural network: l=lcls+lbox+lmask, where L represents the total error, lcls represents the classification error, lbox represents the classification box position error, lmask represents the mask segmentation error; and performing iterative training by using the total loss function, and when the total loss function of the neural network reaches a preset minimum value, finishing the training of the neural network to obtain a trained image segmentation model. And inputting the unlabeled test image into the trained image segmentation model for testing, obtaining a mask of the foreground, and removing the rest pixel points of the foreground to obtain the background.

Second step S2: the background portion is segmented to obtain n background sub-images (image 1, image 2, … …, and image n, n is a positive integer) conforming to a preset size.

Through the first step S1, a two-dimensional matrix a including a foreground and a background in the certificate snapshot is obtained, wherein a value 1 represents the background, and a value 0 represents the foreground, and only the coordinates of the value 1 in the matrix need to be processed. The method specifically comprises the following steps:

step S201: the two-dimensional matrix A is divided into a plurality of rectangular areas B with the same size along the transverse direction and the vertical direction. Referring to the first diagram in fig. 5, in this embodiment, the two-dimensional matrix a is equally divided into six rectangular areas B, and each rectangular area B includes a foreground and a background.

Step S202: the largest rectangle B which can be formed by the value 1 is found in each rectangle area B, and if the proportion of the value 1 in the rectangle area B which is not in the range of the rectangle B is larger than a preset threshold value, the largest rectangle c which can be formed by the value 1 is found in the area except the rectangle B in the rectangle area B. The rectangles B1, B2, B3, B4, and B5 illustrated in the second drawing in fig. 5 are the largest rectangle B that can be constituted by the value 1 found in each rectangular area B; rectangle c1 is the largest rectangle c that can be constituted by the value 1 found in the area other than rectangle b.

Step S203: filtering the obtained rectangles b and c based on a second preset threshold, respectively judging whether the number of pixel points in the rectangles b and c is larger than or equal to the second preset threshold, and if so, dividing the rectangles b and/or c into images to be processed for reservation; otherwise, it will not remain. For example, the rectangle b5 illustrated in the second drawing in fig. 5, if the number of pixels constituting the rectangle does not reach the second preset threshold, is determined to be unsatisfactory, and will not be reserved as an image to be processed. In this embodiment, it is assumed that the value set by the second preset threshold is smaller than the number of pixels of the rectangle b5, and the rectangle b5 is reserved.

Step S204: and mapping the coordinates of the matrix b and the matrix c meeting the requirements into the original certificate inch, so as to obtain a plurality of background sub-images meeting the preset size. Referring to the rectangle illustrated in the third graph in fig. 5, after the background portion is segmented, the obtained background sub-images conforming to the preset size are respectively an image 1, an image 2, an image 3, an image 4, an image 5 and an image 6.

Third step S3: and simultaneously performing color recognition processing and color purity recognition processing on each background sub-image.

This step may be illustrated with reference to the process flow diagram of fig. 6.

(1) The color identification process specifically comprises the following steps:

there are two possibilities for the color recognition process.

Step S301: in a first possibility, the background sub-image is a gray-scale image, and the color judgment is directly performed based on the pixel value.

Step S302: and the second possibility is that the background sub-image is a color image, the background sub-image is converted into an HSV color space from an RGB color space, and then the color judgment is carried out by inquiring a preset table.

The color of the image is difficult to judge in the RGB space, the HSV color space is a hexagonal pyramid model, and the color can be judged directly through the high and low threshold values. HSV respectively represents hue H, saturation S and brightness V, the value mode of the three values is a mode of taking the value, namely the value H is the value with the largest occurrence number in an H dimension matrix, and S, V is the same.

The conversion formula for converting the RGB color space into the HSV color space is as follows:

let (r, g, b) be the red, green and blue coordinates of one color, respectively, their values being real numbers between 0 and 1;

let max be equal to the largest of r, g, b, i.e. max=max (r, g, b);

let min be equal to the minimum of r, g, b, i.e. min = min (r, g, b); then the first time period of the first time period,

v＝max

step S303: the color cn of the background sub-image is output, n represents the number of the background sub-images, and is a positive integer, and cn represents the color of the nth background sub-image.

(2) The color purity identifying process (see also the processing procedure example of fig. 7) specifically includes the following steps:

step S304: and when the background sub-image is a color image, converting into a gray image.

Step S305: and carrying out two-dimensional discrete Fourier transform on the gray level image, intercepting high-frequency components of the frequency, and carrying out sharpening treatment on the gray level image.

The image is a digital image in a space domain, is converted into a frequency domain through Fourier transformation, and is subjected to frequency domain filtering; the frequency domain filtering adopts high-pass filtering, namely, high-frequency components of the frequency are intercepted, and then inverse Fourier transformation is carried out on the high-frequency components, so that a sharpened image can be obtained, wherein the high-frequency components in the image are places with intense intensity changes in the image, namely, edge parts.

Step S306: and carrying out binarization processing on the sharpened image to obtain a binarized image. The binarization adopts canny edge detection, which returns obvious edges detected in the image, and adopts double-threshold processing and edge connection.

Step S307: and (3) carrying out line detection on the binarized image by using Hough transformation, and judging that the background sub-image is a non-solid background image if the line is detected, or else, judging that the background sub-image is a solid background image.

Wherein the hough transform component Huo Fuxian transforms and the hough round transform detect straight lines and curves, respectively. Huo Fuxian transform adopts the cumulative probability Hough transform (PPHT), and is more efficient to execute than the standard Hough transform.

Step S308: and outputting the color purity judgment result of the background sub-image, outputting T if the background sub-image is a pure color background image, and outputting F if the background sub-image is not the pure color background image.

Fourth step S4: and integrating the color output result and the color purity output result of each background sub-image, and finally determining whether the picture is a pure background color picture.

When the colors of all the background sub-images are all the preset background colors and are all pure colors, namely c1=c2= … =cn, and the output of the purity detection is all T, judging that the background colors of the document candid images are compliant, and the background colors are c1.

In addition, the application further provides a certificate background color checking system under a complex background, referring to fig. 8, specifically including:

the image segmentation module 1 is used for carrying out pixel level segmentation processing on the input certificate, and obtaining the background coordinates of the image;

the background segmentation module 2 is used for segmenting the background part according to the background coordinates acquired by the image segmentation module 1 to acquire at least one background sub-image conforming to the preset size;

the color recognition module 3 is used for performing color recognition on each background sub-image segmented by the background segmentation module 2;

the color purity recognition module 4 is used for recognizing the color purity of each background sub-image segmented by the background segmentation module 2;

and the statistics judging module 5 is used for judging by integrating the color identification result and the color purity identification result of each background sub-image, and judging that the background color of the certificate is compliant when the colors of all the background sub-images are the preset background colors and are pure colors.

In summary, the certificate background color inspection method and system under the complex background are used for identifying the pure color background color of the photo and removing the photo such as pure color and complex background, so that the intelligent inspection efficiency of the photo background color in the government service process can be improved, and the intelligent degree of the business service can be improved.

The above description of the specific embodiments of the present invention has been given by way of example only, and the present invention is not limited to the above described specific embodiments. Any equivalent modifications and substitutions for the present invention will occur to those skilled in the art, and are also within the scope of the present invention. Accordingly, equivalent changes and modifications are intended to be included within the scope of the present invention without departing from the spirit and scope thereof.

Claims

1. A document background color checking method under complex background is characterized by comprising the following steps:

the color recognition results and the color purity recognition results of all the background sub-images are synthesized to judge, and when the colors of all the background sub-images are all the preset background colors and are all the pure colors, the background color compliance of the certificate candid is judged;

the method comprises the steps of performing pixel-level segmentation processing on a certificate size to obtain a two-dimensional matrix A comprising a foreground and a background, wherein a value 1 represents the background, and a value 0 represents the foreground; the segmenting the background part to obtain at least one background sub-image conforming to a preset size comprises the following steps:

2. The method for checking the background color of a document in a complex background according to claim 1, wherein the neural network is obtained through pre-training, and the pre-training of the neural network comprises:

the ROIAlign module is configured to perform ROIAlign operation on the remaining ROIs, correspond unlabeled document size images to pixels of the feature vectors, then correspond the feature vectors to the labeled pixels, and input the result to the three-branch full convolution neural network to obtain three prediction results: classification, classification box regression and MASK;

3. The method for checking the background color of a document in a complex background according to claim 1, wherein said performing color recognition on each background sub-image comprises:

4. The method for checking the color of a document against a background in a complex background according to claim 1, wherein said identifying the color purity of each background sub-image comprises:

when the background sub-image is a color image, converting into a gray image;

5. The method for checking the background color of a document in a complex background according to claim 4, wherein the step of performing two-dimensional discrete fourier transform on the gray image, cutting out a high-frequency component of the frequency, and performing sharpening processing on the gray image comprises:

6. The method for background color inspection of documents in a complex background of claim 4, wherein the hough transform comprises Huo Fuxian transform and hough circle transform, respectively detecting straight lines and curves.

7. The method of claim 6, wherein the Huo Fuxian transform uses a cumulative probability hough transform PPHT.

8. The method for checking the background color of a document in a complex background according to claim 4, wherein the binarizing process uses canny edge detection to return the detected sharp edges in the image; the canny edge detection adopts double-threshold processing and edge connection.

9. A document inch background color inspection system under a complex background, comprising:

the image segmentation module is used for carrying out pixel level segmentation processing on the input certificate, and obtaining the background coordinates of the image, wherein a two-dimensional matrix A comprising a foreground and a background is obtained after the pixel level segmentation processing, a value 1 in the two-dimensional matrix A represents the background, and a value 0 represents the foreground;

the background segmentation module is used for segmenting the background part according to the background coordinates acquired by the image segmentation module to acquire at least one background sub-image conforming to a preset size, and comprises the following steps:

mapping rectangular coordinates of all the images to be processed into a certificate inch picture to obtain at least one background sub-image conforming to a preset size;