CN112991470A

CN112991470A - Method and system for checking photo background color of certificate under complex background

Info

Publication number: CN112991470A
Application number: CN202110170353.6A
Authority: CN
Inventors: 郭大勇; 张海龙; 兰永
Original assignee: Shanghai Tongban Information Service Co ltd
Current assignee: Shanghai Tongban Information Service Co ltd
Priority date: 2021-02-08
Filing date: 2021-02-08
Publication date: 2021-06-18
Anticipated expiration: 2041-02-08
Also published as: CN112991470B

Abstract

The application discloses a method and a system for checking the background color of document photo under a complex background, wherein the method comprises the following steps: based on an image segmentation model of a neural network, performing pixel-level segmentation processing on the image of the certificate to obtain a background coordinate of the image; segmenting the background part to obtain at least one background sub-image which accords with a preset size; carrying out color identification on each background sub-image and simultaneously carrying out color purity identification; and judging by integrating the color identification result and the color purity identification result of each background sub-image, and judging the background color compliance of the certificate photo when the colors of all the background sub-images are preset background colors and are pure colors. The method and the device are used for identifying the pure background color of the photo picture and arranging pictures except the pure color, the complex background and the like, so that the intelligent examination efficiency of the photo background color in the government affair service process can be improved, and the intelligent degree of the office affair service is improved.

Description

Method and system for checking photo background color of certificate under complex background

Technical Field

The invention relates to the technical field of image processing of photos, in particular to a method and a system for checking the background color of a document photo under a complex background.

Background

The electronic certificate photo needs to be submitted as an identity certificate in the government affair service transaction process, and the background color of the electronic certificate photo has a standardization requirement. The background color of the inch photo of the certificate submitted by people is irregular, for example, the background color is not a specified color, the background color of the photo is not a pure color, a complex background exists and the like.

Background color screening and detection of traditional electronic certificate photo needs manual work for detecting one by one, detection results of images cannot be automatically given, efficiency is low, and omission and misjudgment are prone to happen. With the development of computer technology, the degree of intelligence is higher and higher in various fields, but a method for checking background colors of certificates in the government affairs service transaction process has some blanks.

Therefore, whether the pure background color of the photo picture can be quickly, efficiently and accurately identified in the process of processing the document photo, and except the pictures with the pure color, the complex background and the like, the intelligent examination efficiency of government affair services and the intelligent degree of affair handling services can be influenced.

Disclosure of Invention

The invention aims to provide a method and a system for checking the photo background color of a certificate under a complex background, so as to solve the problems in the technical background.

In order to achieve the purpose, the invention adopts the following technical scheme:

the first aspect of the application provides a method for checking the photo background color of a certificate under a complex background, which comprises the following steps:

based on an image segmentation model of a neural network, performing pixel-level segmentation processing on the image of the certificate to obtain a background coordinate of the image;

segmenting the background part to obtain at least one background sub-image which accords with a preset size;

carrying out color identification on each background sub-image and simultaneously carrying out color purity identification;

and judging by integrating the color identification result and the color purity identification result of each background sub-image, and judging the background color compliance of the certificate photo when the colors of all the background sub-images are preset background colors and are pure colors.

Preferably, the foreground is a person on the photo of the document, and the foreground and the background are all background.

Preferably, the image segmentation model is a deep learning model based on a neural network.

Preferably, the neural network is obtained through pre-training, and the pre-training of the neural network includes:

step 1: collecting the certificate photo, carrying out pixel-level segmentation and labeling on the certificate photo, marking out pixel points of the foreground outline, and obtaining a training set if the pixel points are not marked as background pixel points;

step 2: building a network model, wherein the network model comprises the following modules:

the pre-training neural network ResNet module is used for obtaining a feature vector of the picture and dividing a plurality of interested regions ROI according to the feature vector;

the region generation network module is used for carrying out binary classification and category frame regression processing on the ROI and filtering the ROI according to a preset filtering standard;

and the ROI Align module is used for carrying out ROI Align operation on the rest ROI, corresponding the unmarked certificate photo and the pixel point of the feature vector, then corresponding the feature vector and the marked pixel point, and then inputting the feature vector into a three-branch full convolution neural network to obtain three prediction results: classification, classification box regression and MASK;

and step 3: and taking the marked training set as the input of the network model to obtain a MASK, and calculating the total loss function of the whole neural network: l ═ Lcls + Lbox + Lmask, where L denotes total error, Lcls denotes classification error, Lbox denotes classification frame position error, and Lmask denotes mask segmentation error;

performing iterative training by using the total loss function, and finishing the neural network training when the total loss function of the neural network reaches a preset minimum value to obtain a trained image segmentation model;

and 4, step 4: inputting the unmarked test image into the trained image segmentation model for testing to obtain the mask of the foreground, and removing the remaining pixel points of the foreground to obtain the background.

More preferably, the marking of the document photo comprises: the certificate photo comprises a foreground and a background, the pixel value of the foreground is set to be 0, the pixel value of the background is set to be 1, a two-dimensional matrix A is obtained, and then all the certificate photos are subjected to contour labeling by using a labeling tool.

More preferably, the preset filtering criteria include: and calculating the score between each ROI and the corresponding labeled foreground according to a preset standard, and taking the first 2000 reservations from high to low according to the score.

Preferably, after the pixel-level segmentation processing is performed on the image of the certificate, a two-dimensional matrix a comprising a foreground and a background is obtained, wherein a value 1 represents the background, and a value 0 represents the foreground; the segmenting the background part to obtain at least one background sub-image meeting the preset size comprises the following steps:

dividing the two-dimensional matrix A into at least 4 rectangular areas B with the same size along the transverse direction and the vertical direction;

finding the largest rectangle B which can be formed by the numerical value 1 in each rectangular area B;

if the proportion that the numerical value 1 in the rectangular area B is not in the range of the rectangle B is larger than the preset threshold value, finding the largest rectangle c which can be formed by the numerical value 1 in the area except the rectangle B in the rectangular area B;

respectively judging whether the number of pixel points in the rectangle b and the rectangle c is larger than or equal to a second preset threshold value, if so, dividing the rectangle b and/or the rectangle c into images to be processed; otherwise, the images are not divided into images to be processed;

and mapping the rectangular coordinates of all the images to be processed into the certificate photo to obtain at least one background sub-image which accords with the preset size.

Preferably, the color identifying each background sub-image includes:

when the background sub-image is a gray image, color judgment is directly carried out based on pixel values;

and when the background sub-image is a color image, converting the background sub-image from an RGB color space to an HSV color space, calculating a histogram of an H channel, and judging the color according to the maximum H value.

Preferably, the identifying the color purity of each background sub-image includes:

when the background sub-image is a color image, converting the background sub-image into a gray image;

carrying out two-dimensional discrete Fourier transform on the gray level image, intercepting high-frequency components of frequency and carrying out sharpening processing on the gray level image;

carrying out binarization processing on the sharpened image to obtain a binarized image;

and (4) performing line detection on the binary image by using Hough transform, and if the line is detected, judging that the background sub-image is a non-pure-color background image.

More preferably, when the background sub-image is a color image, the converting into a grayscale image includes: in a color image, a color is formed by proportionally mixing three primary colors of R, G and B, the basic unit of an image is a pixel, one pixel needs 3 blocks to represent R, G and B respectively, if 8 bits represent one color, certain primary colors with different brightness are distinguished by 0-255, a gray image uses black with different saturation to represent each image point, for example, 8 bits 0-255 are used to digitally represent the degree of gray, each pixel point only needs one gray value, 8 bits are needed, and the conversion of RGB value and gray is actually the conversion of human eyes for the perception of brightness of the color, and the conversion formula is as follows: and (3) sequentially reading the R, G and B values of each pixel point in the color image according to the formula, calculating the Gray value, assigning the Gray value to the corresponding position of the new image, and traversing all the pixel points once to finish the conversion.

More preferably, the performing two-dimensional discrete fourier transform on the grayscale image, and intercepting the high-frequency component of the frequency to perform sharpening processing on the grayscale image includes:

transforming the gray level image from a space domain to a frequency domain, and then performing frequency domain filtering processing;

the frequency domain filtering adopts high-pass filtering, namely intercepting high-frequency components of frequency, and then carrying out inverse Fourier transform on the high-frequency components to obtain a sharpened image.

In the above, the high frequency component corresponds to a place where the intensity change is severe in the grayscale image, that is, an edge portion.

More preferably, the binarization processing employs canny edge detection, returning to the apparent edges detected in the image.

Further, the canny edge detection employs dual threshold processing and edge joining.

Still further, the dual-thresholding and edge-linking, comprising: connecting edges into a contour in the high-threshold image, searching a point meeting a low threshold in the neighborhood of a breakpoint when the end point of the contour is reached, and collecting a new edge according to the point until the edge of the whole image is closed.

More preferably, the hough transform includes a hough line transform and a hough circle transform, and the straight line and the curve are detected respectively.

Further, the hough line transformation adopts cumulative probability hough transformation PPHT.

A second aspect of the present application provides a document photo background color inspection system under a complex background, comprising:

the image segmentation module is used for carrying out pixel level segmentation processing on the input certificate photo to obtain the background coordinates of the image;

the background segmentation module is used for segmenting the background part aiming at the background coordinate acquired by the image segmentation module to acquire at least one background sub-image which accords with the preset size;

the color identification module is used for carrying out color identification on each background sub-image segmented by the background segmentation module;

the color purity identification module is used for identifying the color purity of each background sub-image segmented by the background segmentation module;

and the statistical judgment module is used for comprehensively judging the color identification result and the color purity identification result of each background sub-image, and judging the background color compliance of the certificate photo when the colors of all the background sub-images are preset background colors and are pure colors.

Compared with the prior art, the technical scheme of the invention has the following beneficial effects:

the document photo background color inspection method and system under the complex background are used for identifying the pure background color of the photo, arranging pictures except the pure color, the complex background and the like, improving the intelligent examination efficiency of the photo background color in the government affair service process and improving the intelligent degree of the affair service.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application and are not intended to limit the application. In the drawings:

FIG. 1 is a schematic flow chart of a document photo-background color inspection method under a complex background according to the present application;

FIG. 2 is a schematic overall flow chart of a document photo background color checking method in one embodiment of the present application;

FIG. 3 is a schematic diagram of a process for obtaining a background of an original image according to an embodiment of the present application;

FIG. 4 is a schematic flow chart of the formation of an image segmentation model of the present application;

fig. 5 is a schematic diagram of a processing procedure for segmenting a background to obtain n background sub-images meeting a preset size in an embodiment of the present application;

FIG. 6 is a flow chart illustrating the processing of any background sub-image in one embodiment of the present application;

FIG. 7 is a schematic diagram of a process for processing a background sub-image according to an embodiment of the present application;

FIG. 8 is a schematic diagram of a document photo background color inspection system in a complex background according to the present application.

Detailed Description

In order to make the objects, technical solutions and effects of the present invention clearer and clearer, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order, it being understood that the data so used may be interchanged under appropriate circumstances. Furthermore, the terms "comprises," "comprising," and any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Referring to fig. 1, fig. 1 is a schematic flow chart of a document photo background color checking method under a complex background of the present application.

The method for checking the photo background color of the certificate under the complex background mainly comprises the following steps:

step A1: based on an image segmentation model of a neural network, performing pixel-level segmentation processing on the image of the certificate to obtain a background coordinate of the image;

step A2: segmenting the background part to obtain at least one background sub-image which accords with a preset size;

step A3: carrying out color identification on each background sub-image and simultaneously carrying out color purity identification;

step A4: and judging by integrating the color identification result and the color purity identification result of each background sub-image, and judging the background color compliance of the certificate photo when the colors of all the background sub-images are preset background colors and are pure colors.

Wherein, the foreground is a figure on the photo of the certificate, and the foreground and the background are all background. The image segmentation model is a deep learning model based on a neural network.

Example (b):

this embodiment is specifically described by taking a background color check process of an unconventional document photo as an example. The original image is the photo of the certificate to be detected, the foreground is a figure on the photo of the certificate, and the foreground and the background are all the same.

Referring to fig. 2, the method for checking the background color of the document photo under the complex background of the embodiment includes the following steps:

first step S1: and acquiring the background coordinates of the original image through the image segmentation model.

The image segmentation model is a deep learning model based on a neural network, and after data set preparation, network model building and model training, the model can be used for predicting an original image, and each pixel point on the original image is divided into a foreground or a background.

Wherein the data set preparation: a large number of certificate photo images are collected, pixel-level segmentation and labeling are carried out on the certificate photo images, pixel points of foreground contours are labeled, and unmarked pixel points are background pixel points, so that a training set is obtained. Referring to fig. 3, dots of the human-shaped portion are marked as foreground portions, and black portions are background portions.

Wherein, the network model: the method mainly comprises a pre-training neural network ResNet module, an area generation network module, a ROIAlign module and three full convolution neural networks which are respectively output (as shown in figure 4). The pre-training neural network ResNet module is used for obtaining a feature vector of the picture and dividing a plurality of interested regions ROI according to the feature vector; the region generation network module is used for carrying out binary classification and category frame regression processing on the ROI and filtering the ROI according to a preset filtering standard; and the ROI Align module is used for carrying out ROI Align operation on the rest ROI, corresponding the unmarked certificate photo and the pixel point of the feature vector, then corresponding the feature vector and the marked pixel point, and then inputting the feature vector into a three-branch full convolution neural network to obtain three prediction results: classification, classification box regression, and MASK.

Wherein, the model training: and taking the marked training set as the input of the network model to obtain a MASK, and calculating the total loss function of the whole neural network: l ═ Lcls + Lbox + Lmask, where L denotes total error, Lcls denotes classification error, Lbox denotes classification frame position error, and Lmask denotes mask segmentation error; and performing iterative training by using the total loss function, and finishing the neural network training when the total loss function of the neural network reaches a preset minimum value to obtain a trained image segmentation model. Inputting the unmarked test image into the trained image segmentation model for testing to obtain the mask of the foreground, and removing the remaining pixel points of the foreground to obtain the background.

Second step S2: the background portion is segmented to obtain n background sub-images (image 1, image 2, … …, image n, n is a positive integer) meeting the preset size.

Through the first step S1, a two-dimensional matrix a including a foreground and a background in the certificate photo is obtained, where a value 1 represents the background and a value 0 represents the foreground, and we only need to process the coordinate where the value 1 is located in the matrix. The method specifically comprises the following steps:

step S201: and the two-dimensional matrix A is divided into a plurality of rectangular areas B with the same size along the transverse direction and the vertical direction. Referring to the first diagram in fig. 5, in the present embodiment, the two-dimensional matrix a is equally divided into six rectangular areas B, and each rectangular area B includes a foreground and a background.

Step S202: and finding the largest rectangle B which can be formed by the value 1 in each rectangular area B, and if the proportion that the value 1 in the rectangular area B is not in the range of the rectangle B is greater than a preset threshold value, finding the largest rectangle c which can be formed by the value 1 in the area except the rectangle B in the rectangular area B. The rectangles B1, B2, B3, B4 and B5 illustrated in the second diagram in fig. 5 are the largest rectangles B that can be formed by the value 1 found in each rectangular area B; rectangle c1 is the largest rectangle c that can be formed by the value 1 found in the area other than rectangle b.

Step S203: filtering the obtained rectangles b and c based on a second preset threshold, respectively judging whether the number of pixel points in the rectangles b and c is larger than or equal to the second preset threshold, and if so, dividing the rectangles b and/or c into images to be processed for reservation; otherwise it will not be retained. For example, the rectangle b5 illustrated in the second diagram in fig. 5 is determined to be unsatisfactory if the number of pixel points constituting the rectangle does not reach the second preset threshold, and will not be retained as the image to be processed. In this embodiment, it is assumed that the value set by the second preset threshold is smaller than the number of the pixels of the rectangle b5, and the rectangle b5 is reserved.

Step S204: and mapping the coordinates of all the matrixes b and c meeting the requirements to the original certificate photo to obtain a plurality of background sub-images in accordance with the preset size. Referring to the third diagram in fig. 5, the rectangles are illustrated, that is, the sub-images of the background that meet the preset size and are obtained after the background is partially cut are image 1, image 2, image 3, image 4, image 5 and image 6, respectively.

Third step S3: and simultaneously performing color recognition processing and color purity recognition processing on each background sub-image.

This step may be referred to in the process flow diagram example of fig. 6.

(1) The color identification processing specifically comprises the following steps:

in the color recognition process, there are two possibilities.

Step S301: in a first possibility, the background sub-image is a grayscale image, and the color determination is performed directly based on the pixel value.

Step S302: in a second possibility, if the background sub-image is a color image, the background sub-image is converted from an RGB color space to an HSV color space, and then a preset table is queried to perform color judgment.

The color of the image is difficult to judge in the RGB space, and the HSV color space is a hexagonal cone model and can be directly judged by passing through a high threshold and a low threshold. HSV represents hue H, saturation S, and lightness V, respectively, and the value of the three values is a mode of mode selection, that is, the H value is the value with the largest occurrence frequency in the H dimensional matrix, S, V is the same.

The conversion formula for converting the RGB color space into the HSV color space is as follows:

let (r, g, b) be the red, green and blue coordinates of a color, respectively, whose values are real numbers between 0 and 1;

let max be equal to the maximum of r, g, b, i.e., max ═ max (r, g, b);

setting min equal to the minimum of r, g and b, namely, min equals to min (r, g and b); then the process of the first step is carried out,

v＝max

step S303: and outputting the color cn of the background sub-image, wherein n represents the number of the background sub-images and is a positive integer, and cn represents the color of the nth background sub-image.

(2) The color purity recognition processing (see the processing example of fig. 7), specifically includes the following steps:

step S304: and when the background sub-image is a color image, converting the background sub-image into a gray-scale image.

Step S305: and carrying out two-dimensional discrete Fourier transform on the gray level image, and intercepting the high-frequency component of the frequency to carry out sharpening processing on the gray level image.

The image is a digital image on a space domain, is transformed into a frequency domain through Fourier transform, and then is subjected to frequency domain filtering; the frequency domain filtering adopts high-pass filtering, namely high-frequency components of frequency are intercepted and subjected to inverse Fourier transform, and a sharpened image is obtained, wherein the high-frequency components in the image are places with severe intensity changes in the image, namely edge parts.

Step S306: and carrying out binarization processing on the sharpened image to obtain a binarized image. The binarization adopts canny edge detection, returns the obvious edge detected in the image, and adopts dual-threshold processing and edge connection.

Step S307: and (4) performing line detection on the binary image by using Hough transform, if the line is detected, judging that the background sub-image is a non-pure-color background image, and otherwise, judging that the background sub-image is a pure-color background image.

The Hough transform is divided into Hough line transform and Hough circle transform, and straight lines and curves are detected respectively. The accumulative probability Hough transform (PPHT) adopted by the Hough line transform has higher execution efficiency than the standard Hough transform.

Step S308: and outputting the color purity judgment result of the background sub-image, outputting T if the background sub-image is a pure color background image, and otherwise, outputting F.

Fourth step S4: and finally determining whether the picture is a pure background color picture or not by integrating the color output result and the color purity output result of each background sub-picture.

And when the colors of all the background sub-images are the preset background colors and are all pure colors, namely c 1-c 2- … -cn and the output of the purity detection is all T, judging that the background color of the certificate photo is in compliance and the background color is c 1.

In addition, the present application further provides a document photo background color inspection system under a complex background, referring to fig. 8, which specifically includes:

the image segmentation module 1 is used for carrying out pixel level segmentation processing on the input certificate photo to obtain the background coordinates of the image;

the background segmentation module 2 is used for segmenting the background part aiming at the background coordinates acquired by the image segmentation module 1 to acquire at least one background sub-image which accords with the preset size;

the color recognition module 3 is used for carrying out color recognition on each background sub-image segmented by the background segmentation module 2;

the color purity identification module 4 is used for identifying the color purity of each background sub-image segmented by the background segmentation module 2;

and the statistical judgment module 5 is used for comprehensively judging the color identification result and the color purity identification result of each background sub-image, and judging the background color compliance of the certificate photo when the colors of all the background sub-images are preset background colors and are pure colors.

To sum up, the document photo background color inspection method and the document photo background color inspection system under the complex background are used for identifying the pure color background color of the photo picture and arranging pictures except the pure color, the complex background and the like, so that the intelligent examination efficiency of the photo background color in the government affair service process can be improved, and the intelligent degree of the office affair service is improved.

The embodiments of the present invention have been described in detail, but the embodiments are merely examples, and the present invention is not limited to the embodiments described above. Any equivalent modifications and substitutions to those skilled in the art are also within the scope of the present invention. Accordingly, equivalent changes and modifications made without departing from the spirit and scope of the present invention should be covered by the present invention.

Claims

1. A method for checking the background color of a photo of a certificate under a complex background is characterized by comprising the following steps:

2. The method for checking the photo background color of the document under the complex background as claimed in claim 1, wherein the neural network is obtained through pre-training, and the pre-training of the neural network comprises:

the ROIAlign module is used for carrying out ROIAlign operation on the rest ROI, enabling the size of the unlabeled certificate to correspond to the pixel point of the feature vector, enabling the feature vector to correspond to the labeled pixel point, and then inputting the feature vector to the three-branch full convolution neural network to obtain three prediction results: classification, classification box regression and MASK;

3. The method for checking the background color of the document photo under the complex background as claimed in claim 1, wherein the document photo is subjected to pixel-level segmentation processing to obtain a two-dimensional matrix A comprising a foreground and a background, wherein a value 1 represents the background, and a value 0 represents the foreground; the segmenting the background part to obtain at least one background sub-image meeting the preset size comprises the following steps:

4. The method for checking the print background color of the document under the complicated background according to claim 1, wherein the color recognition of each background sub-image comprises:

5. The method for checking the print background color of the document under the complicated background according to claim 1, wherein the identifying the color purity of each background sub-image comprises:

6. The method for checking the print background color of the certificate under the complicated background as claimed in claim 5, wherein said performing two-dimensional discrete Fourier transform on the gray image, and intercepting the high frequency component of the frequency to perform sharpening process on the gray image comprises:

7. The method for checking the photo background color of the certificate under the complex background as claimed in claim 5, wherein the Hough transform comprises Hough line transform and Hough circle transform, and the Hough line transform and the Hough circle transform are used for respectively detecting straight lines and curves.

8. The method for checking the document photo background color under the complex background as claimed in claim 7, wherein the hough line transformation employs cumulative probability hough transformation PPHT.

9. The method for checking the document photo background color under the complex background according to claim 5, wherein the binarization processing adopts canny edge detection to return the obvious edge detected in the image; the canny edge detection employs dual threshold processing and edge joining.

10. A document photo background color inspection system under a complex background, comprising: