CN110705531B

CN110705531B - Missing character detection and missing character detection model establishing method and device

Info

Publication number: CN110705531B
Application number: CN201910936228.4A
Authority: CN
Inventors: 肖航; 张子昊
Original assignee: Beijing Orion Star Technology Co Ltd
Current assignee: Beijing Orion Star Technology Co Ltd
Priority date: 2019-09-29
Filing date: 2019-09-29
Publication date: 2022-03-18
Anticipated expiration: 2039-09-29
Also published as: CN110705531A

Abstract

The application discloses missing character detection and missing character detection model establishing methods and devices, belongs to the technical field of image processing, and comprises the following steps: the method comprises the steps of obtaining an image to be detected, inputting the image to be detected into a missing character detection model, generating a first target image according to probability information of each pixel in the image to be detected, which is output by the missing character detection model, extracting the outline of the first target image, determining the number of code-spraying characters in the image to be detected according to the extracted outline information, and determining that the code-spraying characters in the image to be detected are missing if the number of the code-spraying characters is smaller than a preset number, wherein the probability information of each pixel output by the missing character detection model comprises the probability that each pixel belongs to the code-spraying characters.

Description

Missing character detection and missing character detection model establishing method and device

Technical Field

The application relates to the technical field of image processing, in particular to missing character detection and a method and a device for establishing a missing character detection model.

Background

In industrial production, before products are shipped out, character information such as production date or batch number is printed on outer packages of the products, and the characters are generally generated by an ink-jet printer.

At present, the scheme adopted for detecting the missing of code-spraying characters is as follows: after the image is obtained, calculating the pixel area of the code spraying character in the image, and if the pixel area of the code spraying character is determined to be smaller than a preset threshold value, determining that the code spraying character in the image is lost; and if the pixel area of the code spraying character is not smaller than the preset threshold value, the code spraying character in the image is considered to be not lost. In this scheme, the image can not have the change in size, because the pixel area of spouting the code character also can change after the image size changes, it will no longer be effective to predetermine the threshold value to, predetermine that the threshold value sets up and causes the hourglass to examine easily, predetermine that the threshold value sets up and causes the false positive detection easily excessively, the rate of accuracy that detects also is difficult to guarantee.

Disclosure of Invention

The embodiment of the application provides a missing character detection method and a missing character detection model establishing method and device, and aims to solve the problems that in the prior art, the size requirement of an image to be detected is strict and the accuracy rate is difficult to guarantee when missing character detection is carried out.

In a first aspect, a missing character detection method provided in an embodiment of the present application includes:

acquiring an image to be detected;

inputting the image to be detected into an established missing character detection model to determine probability information of each pixel in the image to be detected, wherein the probability information of each pixel comprises first probability information, and the first probability information comprises the probability that the pixel belongs to code-spraying characters;

generating a first target image according to first probability information of each pixel in the image to be detected, which is output by the missing character detection model;

extracting the outline of the first target image, and determining code spraying character information in the image to be detected according to the extracted outline information, wherein the code spraying character information comprises the number of code spraying characters;

and if the number of the code spraying characters is smaller than the preset number, determining that the code spraying characters in the image to be detected are missing.

In a second aspect, a method for building a missing character detection model provided in an embodiment of the present application includes:

acquiring an image sample, wherein the image sample comprises at least one code spraying character;

inputting the image sample into a deep learning network model to determine probability information of each pixel in the image sample, wherein the probability information of each pixel comprises first probability information, and the first probability information comprises the probability that the pixel belongs to a code-sprayed character;

determining a first loss value of the deep learning network model according to first probability information of each pixel in the image sample output by the deep learning network model and a first label image generated in advance;

and adjusting parameters of the deep learning network model according to the first loss value, and establishing a missing character detection model.

In a third aspect, an apparatus for detecting missing characters provided in an embodiment of the present application includes:

the acquisition module is used for acquiring an image to be detected;

the determining module is used for inputting the image to be detected into the established missing character detection model so as to determine probability information of each pixel in the image to be detected, wherein the probability information of each pixel comprises first probability information, and the first probability information comprises the probability that the pixel belongs to code-spraying characters;

the generating module is used for generating a first target image according to first probability information of each pixel in the image to be detected, which is output by the missing character detection model;

and the processing module is used for extracting the outline of the first target image, determining code spraying character information in the image to be detected according to the extracted outline information, wherein the code spraying character information comprises the number of code spraying characters, and determining that the code spraying characters in the image to be detected are missing if the number of the code spraying characters is determined to be smaller than the preset number.

In a fourth aspect, an apparatus for building a missing character detection model provided in an embodiment of the present application includes:

the device comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring an image sample which comprises at least one code spraying character;

a probability determining module, configured to input the image sample into a deep learning network model to determine probability information of each pixel in the image sample, where the probability information of each pixel includes first probability information, and the first probability information includes a probability that the pixel belongs to a code-sprayed character;

a loss value determining module, configured to determine a first loss value of the deep learning network model according to first probability information of each pixel in the image sample output by the deep learning network model and a first label image generated in advance;

and the adjusting module is used for adjusting the parameters of the deep learning network model according to the first loss value and establishing a missing character detection model.

In a fifth aspect, an electronic device provided in an embodiment of the present application includes: at least one processor, and a memory communicatively coupled to the at least one processor, wherein:

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform any of the methods described above.

In a sixth aspect, embodiments of the present application provide a computer-readable medium storing computer-executable instructions for performing any one of the methods described above.

In the embodiment of the application, after an image to be detected is obtained, the image to be detected is input into an established missing character detection model to determine probability information of each pixel in the image to be detected, a first target image is generated according to the probability that each pixel in the image to be detected output by the missing character detection model belongs to a code-spraying character, then the first target image is subjected to contour extraction, code-spraying character information in the image to be detected is determined according to the extracted contour information, such as the number of the code-spraying characters, and if the number of the code-spraying characters is determined to be smaller than a preset number, the code-spraying characters in the image to be detected are determined to be missing, wherein the probability information output by the missing character detection model comprises the probability that each pixel belongs to the code-spraying character, so that the missing detection is performed on the code-spraying characters by taking the pixels as a unit, the detection granularity can be refined to a pixel level, and the detection accuracy is higher, and the change of the size of the image to be detected is not easy to influence the detection effect.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:

fig. 1 is a flowchart of a missing character detection method according to an embodiment of the present disclosure;

fig. 2 is a flowchart of another missing character detection method according to an embodiment of the present disclosure;

fig. 3 is a flowchart of a method for establishing a missing character detection model according to an embodiment of the present disclosure;

FIG. 4 is a flowchart of determining a first loss value of a deep learning network model according to an embodiment of the present disclosure;

fig. 5 is a flowchart of a method for building a missing character detection model according to an embodiment of the present application;

FIG. 6 is a schematic diagram of an image sample provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of an initial image provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of an image sample generated from an initial image according to an embodiment of the present application;

FIG. 9 is a schematic diagram of an embodiment of the present application for labeling an image sample;

FIG. 10 is a schematic diagram of a first label image provided by an embodiment of the present application;

FIG. 11 is a schematic diagram of a second label image provided in an embodiment of the present application;

fig. 12 is a schematic structural diagram of a deep learning network model according to an embodiment of the present disclosure;

FIG. 13 is a diagram illustrating a process for detecting missing characters according to an embodiment of the present disclosure;

fig. 14 is a schematic diagram of a missing character detection result according to an embodiment of the present disclosure;

FIG. 15 is a diagram illustrating another missing character detection result according to an embodiment of the present disclosure;

fig. 16 is a schematic hardware structural diagram of an electronic device for implementing a missing character detection method and/or a missing character detection model building method according to an embodiment of the present disclosure;

fig. 17 is a schematic structural diagram of a missing character detection apparatus according to an embodiment of the present application;

fig. 18 is a schematic structural diagram of a missing character detection model building apparatus according to an embodiment of the present application.

Detailed Description

In order to solve the problems that the size requirement of an image to be detected is strict and the accuracy rate is difficult to guarantee when missing character detection is performed in the prior art, the embodiment of the application provides a method and a device for establishing a missing character detection model and a missing character detection model.

The preferred embodiments of the present application will be described below with reference to the accompanying drawings of the specification, it should be understood that the preferred embodiments described herein are merely for illustrating and explaining the present application, and are not intended to limit the present application, and that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.

Referring to fig. 1, fig. 1 is a flowchart of a missing character detection method provided in an embodiment of the present application, where the method includes the following steps:

s101: and acquiring an image to be detected.

In practical application, the character information of the product such as production date, production batch and the like is generally sprayed at a fixed position, so that the image to be detected can be obtained only by carrying out image acquisition on the fixed position of the product.

S102: inputting an image to be detected into the established missing character detection model to determine probability information of each pixel in the image to be detected, wherein the probability information of each pixel comprises first probability information, and the first probability information comprises the probability that the pixel belongs to the code-spraying character.

The missing character detection model established in the embodiment of the application can determine the probability that each pixel in the image to be detected belongs to the code-spraying character, so that whether the code-spraying character is missing or not can be distinguished from the pixel level, and the detection accuracy is improved.

S103: and generating a first target image according to the first probability information of each pixel in the image to be detected, which is output by the missing character detection model.

The first target image corresponds to pixels in the image to be detected one by one, and the first target image comprises information whether each pixel in the image to be detected belongs to code spraying characters.

During specific implementation, the size relation between the probability that each pixel in an image to be detected belongs to a code spraying character and a preset probability can be determined, and then a first target image is generated according to the size relation between the probability that each pixel in the image to be detected belongs to the code spraying character and the preset probability, wherein if the probability that each pixel in the image to be detected belongs to the code spraying character is larger than the preset probability, the pixel corresponding to the pixel in the first target image is the mark value of the code spraying character; and if the probability that the pixel belongs to the code spraying character is not greater than the preset probability, the pixel corresponding to the pixel in the first target image is a preset value representing the background.

For example, a first gray image may be obtained, where the first gray image corresponds to a pixel in an image to be detected one to one, and then, if it is determined that the probability that the pixel belongs to a code-spraying character is greater than a preset probability, for example, 0.5, for each pixel in the image to be detected, the pixel corresponding to the pixel is marked as a mark value, for example, "1", of the code-spraying character on the first gray image; if the probability that the pixel belongs to the code-spraying character is determined to be not greater than the preset probability, the pixel corresponding to the pixel is marked as a preset value such as '0' representing the background on the first gray-scale image, so that the code-spraying characters in the image to be detected are drawn on the first gray-scale image, and the first gray-scale image does not contain objects except the code-spraying characters, and the code-spraying character information in the image to be detected can be conveniently determined subsequently, therefore, the first gray-scale image with the value of each pixel modified can be used as the first target image.

S104: and extracting the outline of the first target image, and determining code spraying character information in the image to be detected according to the extracted outline information, wherein the code spraying character information comprises the number of code spraying characters.

In specific implementation, a cv2.findContours function of an opencv library can be used for extracting the outline of the first target image to obtain the outline information of the code-sprayed character, and then the code-sprayed character information in the image to be detected, such as the number of the code-sprayed characters, is analyzed and determined according to the obtained outline information of the code-sprayed character, wherein the outline information of the code-sprayed character comprises position information of each outline pixel point of the code-sprayed character.

S105: and if the number of the code spraying characters is smaller than the preset number, determining that the code spraying characters in the image to be detected are missing.

In industrial production, the number of the code spraying characters sprayed on the outer package of the product is generally fixed, so that if the number of the code spraying characters is determined to be smaller than the preset number, the code spraying characters in the image to be detected are lost.

In addition, in industrial production, the code-spraying characters generally consist of fixed rows such as N rows of code-spraying points, that is, each code-spraying character comprises N rows of code-spraying points, if each row of code-spraying points is regarded as one type, each code-spraying character comprises N types of code-spraying points under the condition that the code-spraying points are not lost, and N is an integer greater than 1.

Based on the above, the embodiment of the application further provides a missing character detection method for detecting whether the code-sprayed character in the image is missing or not and detecting whether the code-sprayed dot in the image is missing or not. Specifically, fig. 2 is a flowchart of another missing character detection method provided in the embodiment of the present application, including the following steps:

s201: the method comprises the steps of obtaining an image to be detected, wherein the image to be detected comprises at least one code spraying character, each code spraying character comprises N types of code spraying points when not missing, and N is an integer.

Assuming that N is 7, that is, each code spraying character in the image to be detected includes 7 lines of code spraying points, and the 1 st type code spraying point and the 2 nd type code spraying point … … are sequentially performed according to the sequence from top to bottom of the number of code spraying points, where the label value of the 1 st type code spraying point may be "1", and the label value of the 2 nd type code spraying point may be "2" … … and the label value of the 7 th type code spraying point may be "7".

S202: inputting an image to be detected into the established missing character detection model to determine probability information of each pixel in the image to be detected, wherein the probability information of each pixel comprises first probability information and second probability information.

Here, the first probability signal includes a probability that the pixel belongs to a code-sprayed character; the second probability information comprises the probability that the pixel belongs to various code spraying points and the probability that the pixel belongs to the background, wherein the background refers to all objects except the code spraying points in the image to be detected. That is to say, the pixels in the image to be detected are divided into N code spraying point classes and 1 background class, the pixels in the image to be detected which do not belong to the code spraying points are classified as the background class, and the background class can be marked as "0" by using a preset value.

S203: and generating a first target image according to the first probability information of each pixel in the image to be detected, which is output by the missing character detection model.

The implementation process of this step is referred to as the implementation process of S103, and is not described herein again.

S204: and extracting the outline of the first target image, and determining code spraying character information in the image to be detected according to the extracted outline information, wherein the code spraying character information comprises the number of code spraying characters and position information of each code spraying character.

In the step S104, each pixel representing the contour is analyzed, so that not only the number of the code-spraying characters in the image to be detected can be determined, but also the position information of each code-spraying character can be determined.

S205: judging whether the number of the code spraying characters is smaller than a preset number, if so, entering S206; otherwise, S207 is entered.

S206: and determining that the code spraying characters in the image to be detected are missing.

S207: and generating a second target image according to second probability information of each pixel in the image to be detected, which is output by the missing character detection model.

The second target image corresponds to pixels in the image to be detected one by one, and the second target image comprises category information of the code spraying point to which each pixel in the image to be detected belongs.

In specific implementation, each pixel in the image to be detected is taken as the category to which the pixel belongs according to the category to which each pixel in the image to be detected belongs, and then a second target image is generated according to the category to which each pixel in the image to be detected belongs, wherein if the category to which the pixel belongs is the background, the pixel corresponding to the pixel in the second target image is a preset value representing the background; if the category to which the pixel belongs is the ith type code spraying point, the pixel corresponding to the pixel in the second target image is the label value of the category to which the ith type code spraying point belongs, i is more than or equal to 1 and less than or equal to N, and i is an integer.

For example, a second gray scale image may be obtained first, where the second gray scale image corresponds to a pixel in the image to be detected one to one, and then, for each pixel in the image to be detected, the category with the highest probability in the second probability information of the pixel is taken as the category to which the pixel belongs, and if the category to which the pixel belongs is the background, the pixel corresponding to the pixel is marked as a preset value representing the background, such as "0"; if the type of the pixel belongs to the ith type code spraying, the pixel corresponding to the pixel is marked as a marking value of the type of the ith type code spraying point on the second gray scale image, so that the image of each code spraying point in the image to be detected is drawn on the second gray scale image, and the second gray scale image does not contain objects except the code spraying point, and whether the code spraying point in the image to be detected is lost or not is conveniently judged subsequently, therefore, the second gray scale image with the modified pixel value can be used as a second target image.

S208: and determining whether the code spraying points of the code spraying characters in the image to be detected are lost or not according to the position information of the code spraying characters and the second target image.

Specifically, a pixel area corresponding to the code-spraying character in the second target image is determined according to the position information of each code-spraying character, and if the type of the code-spraying points of each pixel in the pixel area is less than N, the code-spraying points of the code-spraying character in the image to be detected are determined to be missing; and if the type of the code spraying points of each pixel in the pixel area is not less than N, determining that the code spraying points of the code spraying character in the image to be detected are not missing, wherein the type of the code spraying points of each pixel in the pixel area is equal to the type of the marking value which indicates the type of the code spraying points in the pixel area.

Therefore, what each code spraying character in the image to be detected is does not need to be determined, and whether the code spraying points in the code spraying character are missing or not can be determined only by judging whether the code spraying point category contained in each code spraying character reaches N or not, and the detection speed is high.

In the above process, when it is determined that the code spraying character in the image to be detected is missing or the code spraying point of the code spraying character is missing, alarm information can be sent so as to remind relevant personnel to take treatment of expressions in time. In addition, for each identified object (including code spraying characters and code spraying points), the object can be marked in the image to be detected according to the position information of the object so as to display the detection result in a display manner, and the user experience is better.

Corresponding to the above embodiment, as shown in fig. 3, a flowchart of a method for establishing a missing character detection model provided in the embodiment of the present application is shown, where the method includes the following steps:

s301: and acquiring an image sample, wherein the image sample comprises at least one code spraying character.

In practical application, although code-spraying character missing or code-spraying point missing images can occur on a production line, the process of collecting the images is long, and the missing character detection model is not favorable for rapidly establishing, so that the method for generating the image sample is further provided.

Specifically, an initial image is obtained, the initial image includes at least one code-spraying character, and then the following steps are executed in a circulating manner: and shielding the code-sprayed characters in the initial image according to position marking information of the code-sprayed characters in the initial image, and taking the initial image subjected to shielding treatment as an image sample when the cycle number reaches a set number, wherein the sizes of the shielding areas selected each time can be the same or different, and the missing condition of the code-sprayed characters in the finally generated image sample can be close to the real missing condition of the code-sprayed characters when the sizes of the shielding areas selected each time are different.

S302: inputting the image sample into a deep learning network model to determine probability information of each pixel in the image sample, wherein the probability information of each pixel comprises first probability information, and the first probability information comprises the probability that the pixel belongs to the code-sprayed character.

In specific implementation, after the image sample is input into the deep learning network model, the deep learning network model can be controlled to perform feature extraction on the image sample, and then probability information of each pixel in the image sample is predicted according to the extracted image feature.

S303: and determining a first loss value of the deep learning network model according to the first probability information of each pixel in the image sample output by the deep learning network model and the first label image generated in advance.

The first label image corresponds to pixels in the image sample one by one, and the first label image contains information whether each pixel in the image sample belongs to code-spraying characters.

In particular, the first label image may be generated according to the following steps:

acquiring outline marking information of each code-sprayed character in an image sample, wherein the outline marking information comprises position information of each outline marking point in the code-sprayed character, and further generating a first label image according to the position information of each outline marking point in each code-sprayed character in the image sample, wherein for each code-sprayed character in the image sample, a character outline is formed by pixels, corresponding to each outline marking point in the code-sprayed character, in the first label image, pixels in the character outline are marking values of the code-sprayed character, and pixels, which do not fall into any character outline, in the second label image are preset values for representing a background.

For example, a first gray image can be obtained, the first gray image corresponds to pixels in an image sample one by one, and contour label information of each code-sprayed character in the image sample is obtained, the contour label information includes position information of each contour label point in the code-sprayed character, then, for each code-sprayed character, according to the position information of each contour label point in the code-sprayed character, a pixel corresponding to the contour label point is determined on the first gray image, a character contour formed by the pixels corresponding to each contour label point on the first gray image is determined, and further, a value of each pixel in the character contour is modified to be a label value of the code-sprayed character such as "1", and a pixel in a second label image which does not fall into any character contour is marked to be a preset value representing a background such as "0", so that the code-sprayed character in the image sample is equivalently drawn in the first gray image, and the first gray image does not contain other objects, so that whether the code spraying characters in the image sample are missing or not can be judged conveniently, and therefore the first gray image with the value of each pixel modified can be used as the first label image.

In practical application, the area occupied by code-spraying characters in an image sample is generally smaller, and the area occupied by non-code-spraying characters is larger, so that the first loss value of the deep learning network model is controlled by most of pixel points of the non-code-spraying characters, and the recognition effect on the code-spraying characters is poorer.

Considering that the value of the pixel in the area occupied by the code-spraying character in the first label image is the marking value, and the value of the pixel in the area occupied by the non-code-spraying character is the preset value, when the marking value is larger than the preset value, if the error value of each pixel is multiplied by the value of the corresponding pixel in the first label image, the control of the pixel point of the non-code-spraying character on the first loss value of the deep learning network model can be greatly weakened. Therefore, in the embodiment of the application, the first loss value of the deep learning network model is determined according to the probability that each pixel in the image sample belongs to the code-sprayed character and the value of the corresponding pixel in the first label image.

Specifically, referring to fig. 4, a flowchart for determining a first loss value of a deep learning network model provided in an embodiment of the present application includes the following steps:

s401 a: for each pixel in the image sample, determining a first loss value of the pixel according to the probability that the pixel belongs to the code-spraying character and the value of the corresponding pixel in the first label image, determining the product of the first loss value of the pixel and the value of the corresponding pixel in the first label image, and updating the first loss value of the pixel into a product result.

For example, a first loss value bce loss (x) of the ith pixel in the image sample is determined according to the following formula_i,y_i)：

bce loss(x_i,y_i)＝-[y_ilogx_i+(1-y_i)log(1-x_i)]；

Wherein x is_iProbability y of i-th pixel in image sample belonging to code-sprayed character_iAnd the value of the ith pixel in the first label image is taken.

Then, the updated first loss value of the ith pixel in the image sample is outer loss (x)_i,y_i) Comprises the following steps:

our bce loss(x_i,y_i)＝bce loss(x_i,y_i)*y_i。

s402 a: and determining the average value of the first loss values of all pixels in the image sample as the first loss value of the deep learning network model.

That is to say that the first and second electrodes,

where M represents the number of pixels of the image sample.

S304: and adjusting parameters of the deep learning network model according to the first loss value of the deep learning network model, and establishing a missing character detection model.

In specific implementation, parameters of the deep learning network model can be adjusted by using a gradient descent method until the first loss value of the deep learning network model is determined to be smaller than the preset loss value, and the current deep learning network model is used as the established missing character detection model.

As shown in fig. 5, a flowchart of a method for building a missing character detection model according to an embodiment of the present application is provided, where the method includes the following steps:

s501: and acquiring an image sample, wherein the image sample comprises at least one code spraying character, each code spraying character comprises N types of code spraying points when not missing, and N is an integer greater than 1.

Assuming that N is 7, that is, there are 7 lines of code spraying points in the code spraying character in the image sample, the code spraying character is sequentially divided into a type 1 code spraying point and a type 2 code spraying point … …, where the type to which the type 1 code spraying point belongs may be labeled as "1", and the type to which the type 2 code spraying point belongs may be labeled as "2" … …, and the type to which the type 7 code spraying point belongs may be labeled as "7", according to the order from top to bottom of the number of code spraying points.

In addition, the implementation process of generating the image sample may refer to the implementation process of S301, and is not described herein again.

S502: the image sample is input into a deep learning network model to determine probability information of pixels in the image sample, wherein the probability information of each pixel comprises first probability information and second probability information.

Here, the first probability signal includes a probability that the pixel belongs to a code-sprayed character; the second probability information includes the probability that the pixel belongs to various types of code spraying points and the probability that the pixel belongs to the background.

S503: and determining a first loss value of the deep learning network model according to the first probability information of each pixel in the image sample output by the deep learning network model and the first label image generated in advance.

The implementation process of this step can refer to the implementation process of S303, which is not described herein again.

S504: and determining a second loss value of the deep learning network model according to second probability information of each pixel in the image sample output by the deep learning network model and a pre-generated second label image.

The second label image corresponds to pixels in the image sample one by one, and the second label image comprises category information of the code spraying point to which each pixel in the image sample belongs.

In particular, the second label image may be generated according to the following steps:

acquiring contour marking information of each code spraying point in an image sample, wherein the contour marking information comprises position information of each contour marking point of the code spraying points, and further generating a second label image according to the position information of each contour marking point in each code spraying point in the image sample, wherein for each code spraying point in the image sample, a code spraying point contour is formed by pixels, corresponding to each contour marking point in the code spraying points, in the second label image, pixels in the code spraying point contour are marking values of the type to which the code spraying points belong, and pixels, which do not fall into any code spraying point contour, in the second label image are preset values representing a background.

For example, a second gray scale image is obtained, the second gray scale image corresponds to pixels in an image sample one by one, and contour label information of each code spraying point in the image sample is obtained, the contour label information includes position information of each contour label point of the code spraying point, for each code spraying point, according to the position information of each contour label point in the code spraying point, a pixel corresponding to the contour label point is determined on the second gray scale image, a code spraying point contour formed by the pixels corresponding to each contour label point on the second gray scale image is determined, the value of each pixel in the code spraying point contour is modified to be the label value of the code spraying point category to which the code spraying point belongs, and the value of the pixel in the second gray scale image which does not fall into any code spraying point contour is modified to be a preset value representing the background, such as "0", which is equivalent to that each code spraying point in the image sample is drawn in the second gray scale image, and the second gray image does not contain other objects, so that whether the code spraying points in the image sample are missing or not can be judged conveniently in the follow-up process, and therefore the second gray image with the modified pixel value can be used as a second label image.

Further, a second loss value of the deep learning network model is determined according to the second probability information of each pixel in the image sample and the second label image.

Specifically, for each pixel in the image sample, the class with the highest probability in the second probability information of the pixel is taken as the class to which the pixel belongs, and the second loss value of the pixel is determined according to the class to which the pixel belongs and the class to which the corresponding pixel in the second label image belongs.

For example, for the ith pixel in the image sample, assuming that the ith pixel is predicted by the deep learning network model to belong to the jth class of code spraying point, and the ith pixel is marked in the second label image to belong to the kth class of code spraying point, the Cross Entropy loss function (Cross Entropy) can be used to calculate the second loss value Cross Entropy loss (i, y ') of the ith pixel'_i)：

CrossEntropyLoss(i,y'_i)＝-y'_i log(p_i)；

Wherein p is_iThe probability that the ith pixel predicted by the deep learning network model belongs to the jth class code spraying point is obtained; y 'when j-k, i.e. the prediction category and annotation category, coincide'_i1, y 'when j ≠ k, i.e., the prediction class and annotation class do not agree'_i＝0，1≤i≤N，1≤j≤N。

Then, an average value of the second loss values of the pixels in the image sample may be calculated, and the average value may be determined as the second loss value of the deep learning network model.

That is to say that the first and second electrodes,

where M represents the number of pixels of the image sample.

S505: and determining a total loss value of the deep learning network model according to the first loss value and the second loss value, and adjusting parameters of the deep learning network model according to the total loss value.

For example, the total Loss value Loss of the deep learning network model can be calculated according to the following formula:

the weight is w, outer bce Loss + (1-w), cross enhanced Loss, where w is a predetermined weight, for example, w is 0.5.

Further, parameters of the deep learning network model can be adjusted by using a gradient descent method until the total loss value of the deep learning network model is determined to be smaller than a preset loss value, and the current deep learning network model is used as the established missing character detection model.

It should be noted that in the flow shown in fig. 5, there is no strict precedence relationship between S503 and S504.

The embodiments of the present application will be described with reference to specific embodiments.

The main idea of the embodiment of the application is to classify the code spraying points in each code spraying character, for example, each code spraying character is composed of 7 rows of code spraying points from top to bottom, the code spraying points are classified into 7 categories, the 1 st row of code spraying points is the 1 st category, and the 2 nd row of code spraying points is the 2 nd category. On the basis of the class division, a missing character detection model is used for carrying out classification prediction on the code spraying points in the code spraying characters, the class number of the code spraying points contained in each predicted code spraying character is counted, if the class number does not reach 7, the code spraying points in a certain row of the code spraying character are missing, and otherwise, the code spraying points in the code spraying character are not missing.

Specifically, the specific flow of the embodiment of the present application is as follows.

The first step is as follows: an image sample is acquired.

As shown in fig. 6, an image sample of a zip-top can with code spraying characters on the bottom is obtained in the embodiment of the present application.

In order to reduce the time cost for obtaining the image sample, the embodiment of the application can also automatically generate the image sample with missing code spraying. Referring to fig. 7, in specific implementation, a small tube bottom background image is collected from an initial image as a pickup frame, as indicated by a rectangular frame, then the size of the pickup frame is adjusted, the pickup frame is randomly covered to the area where the code-sprayed characters are located in fig. 7 after the pickup frame is adjusted to a suitable size, and if the pickup frame is covered on the code-sprayed characters, an image sample with missing characters is generated; if the code-spraying points are covered, image samples with missing code-spraying points are generated, and fig. 8 shows a covering result.

And secondly, generating a label image.

Labeling the detection object by using a rectangular frame on the image sample, wherein the detection object comprises the code spraying characters in the image sample and the code spraying points in each code spraying character, the labeling effect is shown in fig. 9, and the position information of each labeled contour point is stored in a json file.

In the embodiment of the present application, because the positions of the code spraying characters and the code spraying points in the image sample are predicted at the same time, two label images are generated in advance: a first label image and a second label image.

First, a first label image is generated.

Specifically, contour marking information indicating a code-sprayed character contour in an image sample in a son file is acquired, for each contour marking point, a pixel point corresponding to the contour marking point is determined in a gray image of a black background (the pixel value is 0) according to the position information of the contour marking point, a closed area (namely, a character contour) formed by the pixel points corresponding to the contour marking points is filled with white pixels (the pixel value is 1), and a first label image is obtained, as shown in fig. 10.

Then, a second label image is generated.

Specifically, contour marking information of each code spraying point representing each code spraying character in an image sample in a json file is obtained, for each code spraying point, a pixel point corresponding to the contour marking point is determined on a gray-scale image of a black background (the pixel value is 0) according to position information of the contour marking point of the code spraying point, a closed area (namely, a code spraying point contour) formed by the pixel points corresponding to the contour marking point is filled with a marking value of the type to which the code spraying point belongs, wherein if the code spraying point is a type 1 code spraying point, pixels in the closed area are filled with '1', if the code spraying point is a type 2 code spraying point, pixels in the closed area are filled with '2', … … if the code spraying point is a type 7 code spraying point, pixels in the closed area are filled with '7', and finally a second label image is obtained, as shown in fig. 11.

It should be noted that the image shown in fig. 11 represents that pixels in an image sample are classified, where 1 background class (pixels that do not belong to any code-spraying point are all classified as background classes), 7 code-spraying point classes, and a total of 8 classes of pixels.

And thirdly, training a missing character detection model.

The structure of the deep learning Network model adopted in the embodiment of the application is shown in fig. 12, and in specific implementation, an image sample is input into the deep learning Network model, the deep learning Network model performs Feature extraction of different scales on the image sample by using a Feature Pyramid Network (FPN), Feature maps of different scales are fused, convolution operation is performed on the fused Feature maps, and probability information of each pixel in the image sample is output, where the probability information of each pixel includes first probability information and second probability information, where the first probability information includes a probability that the pixel belongs to a code-spraying character, and the second probability information includes a probability that the pixel belongs to various code-spraying points and a probability that the pixel belongs to a background.

Further, a loss value is calculated according to probability information of each pixel in an image sample output by the deep learning network model and a label image generated in advance.

Specifically, a first loss value of the deep learning network model is determined according to first probability information and a first label image of each pixel in an image sample output by the deep learning network model.

Here, the loss function used is bce loss, and specifically, a first loss value bce loss (x) of the ith pixel in the image sample is determined according to the following formula_i,y_i)：

bce loss(x_i,y_i)＝-[y_ilogx_i+(1-y_i)log(1-x_i)]；

Because the number of pixels belonging to the code-spraying character in the image sample is less, and the number of pixels not belonging to the code-spraying character is more, the calculated loss value is controlled by most of the pixels not belonging to the code-spraying character, which can cause the detection effect of the deep learning network model on the code-spraying character to be poor, for this reason, the embodiment of the application improves the loss function, and the loss function of the ith pixel in the image sample after improvement is:

our bce loss(x_i,y_i)＝bce loss(x_i,y_i)*y_i。

the value of the pixel belonging to the code-spraying character in the first label image is "1", and the value of the pixel not belonging to the code-spraying character is "0", so that after the loss value of the ith pixel in the image sample is obtained, the loss value of the ith pixel and the value of the pixel in the first label image are calculated, the loss value of the non-hair pixel can be reduced as much as possible, the influence of the non-hair pixel on the final loss value is weakened, and the contribution of the hair pixel on the final loss value is improved.

Then, calculating an average value of our bce loss of each pixel point to obtain a first loss value of the deep learning network model, namely:

where M represents the number of pixels of the image sample.

And determining a second loss value of the deep learning network model according to the second probability information of each pixel in the image sample and the second label image.

Specifically, for the ith pixel in the image sample, assuming that the ith pixel is predicted by the deep learning network model to belong to the jth class of code spraying point, the ith pixel is marked in the second label image to belong to the kth class of code spraying point, and the Cross EntropyLoss (i, y ') is used for calculating the second loss value of the ith pixel'_i)：

CrossEntropyLoss(i,y'_i)＝-y'_i log(p_i)；

That is to say that the first and second electrodes,

where M represents the number of pixels of the image sample.

Finally, the two loss values are added together in a weighted manner to obtain the total loss, for example, by using the following formula:

Loss＝0.5×BCELoss+0.5×CrossEntropyLoss。

furthermore, convolution parameters in the convolution kernel are reversely corrected according to the Loss value, and the smaller the Loss value is, the more the prediction result is the same as the label, and the fewer the pixels with wrong prediction are. And when the loss value is smaller than a certain value, the deep learning network model is considered to be capable of automatically completing the classification of different pixels in the image sample, and the training is finished.

And fourthly, applying a character missing detection model.

As shown in fig. 13, in specific implementation, an acquired image to be detected is input into a missing character detection model to obtain probability information of each pixel of the image to be detected, where the probability information of each pixel includes first probability information and second probability information, where the first probability information includes a probability that the pixel belongs to a code-sprayed character; the second probability information includes the probability that the pixel belongs to various types of code spraying points and the probability that the pixel belongs to the background.

Further, a first target image is generated according to first probability information of each pixel in the image to be detected, which is output by the missing character detection model, the first target image is subjected to contour extraction, code spraying character information in the image to be detected is determined according to the extracted contour information, the code spraying character information comprises the number of code spraying characters and position information of each code spraying character, and if the number of the code spraying characters is determined to be smaller than the preset number, the code spraying characters in the image to be detected are determined to be missing.

In particular implementation, the first target image is generated according to the following steps:

determining the size relation between the probability that each pixel belongs to the code spraying character and a preset probability in an image to be detected, and further generating a first target image according to the size relation between the probability that each pixel in the image to be detected belongs to the code spraying character and the preset probability, wherein if the probability that each pixel in the image to be detected belongs to the code spraying character is greater than the preset probability such as 0.5, the pixel corresponding to the pixel in the first target image is the labeled value of the code spraying character such as '1'; otherwise, the pixel in the first target image corresponding to the pixel is a preset value representing the background, such as "0".

In addition, if the number of the code-spraying characters is determined to be not less than the preset number, a second target image can be generated according to second probability information of each pixel in the image to be detected, which is output by the missing character detection model,

in particular implementation, the second target image is generated according to the following steps:

taking the category with the highest probability in the second probability information of each pixel in the image to be detected as the category to which the pixel belongs, and further generating a second target image according to the category to which each pixel in the image to be detected belongs, wherein if the category to which the pixel belongs is the background, the pixel corresponding to the pixel in the second target image is a preset value representing the background; if the type of the pixel belongs to the ith type code spraying point, the pixel corresponding to the pixel in the second target image is the label value of the ith type code spraying point, i is more than or equal to 1 and less than or equal to N, and i is an integer.

Further, whether the code spraying points of the code spraying characters in the image to be detected are missing or not can be determined according to the second target image and the position information of the code spraying characters.

Specifically, a pixel area corresponding to the code-spraying character in the second target image is determined according to the position information of each code-spraying character, and if the type of the code-spraying points (namely, the value type of the code-spraying points) contained in the pixel area is less than 7, the code-spraying points of the code-spraying character in the image to be detected are determined to be missing; and if the type of the code spraying points contained in the pixel area is not less than 7, determining that the code spraying points of the code spraying characters in the image to be detected are not missing.

Referring to fig. 14, fig. 14 shows two prediction results, wherein a rectangular frame represents a predicted code-sprayed character, each number represents a class of code-sprayed points, wherein (i), (ii), (iv), (c), and (c) 7 classes of code-sprayed points are identified in the left rectangular frame, which represents that the code-sprayed character on the left side has no code-sprayed point missing phenomenon; only 4 types of code spraying points are identified in the right rectangular frame, code spraying points of the fourth type are not included, and the code spraying point loss phenomenon of the code spraying characters on the right side is represented.

Under the assumption of normal conditions, 13 code-spraying characters are arranged in the first line of the image to be detected, 13 code-spraying characters are arranged in the second line of the image to be detected, and the code-spraying missing detection result is assumed to be shown in fig. 15, wherein 11 code-spraying characters are arranged in the first line, 7 code-spraying characters are arranged in the second line, which indicates that 2 characters are missing in the first line in fig. 15, 6 characters are missing in the second line, in addition, the code-spraying dot missing phenomenon exists in 6 code-spraying characters in 11 code-spraying characters in the first line in 15, and the code-spraying dot missing phenomenon exists in 2 code-spraying characters in 7 code-spraying characters in the second line.

In the embodiment of the application, each pixel in the image to be detected is classified based on the segmented deep learning network model, so that the probability that the pixel belongs to the code spraying character can be obtained, the probability that the pixel belongs to various code spraying points can be identified, and then the code spraying character in the image to be detected is determined to be missing according to the probability information, whether the code spraying point of each code spraying character is missing or not is determined, and the identification precision is higher. In addition, the algorithm is strong and good in adaptability, can cope with the change of different scenes, and is good in robustness.

Referring to fig. 16, a schematic structural diagram of an electronic device provided in the embodiment of the present application includes a transceiver 1601, a processor 1602 and other physical devices, where the processor 1602 may be a Central Processing Unit (CPU), a microprocessor, an application specific integrated circuit, a programmable logic circuit, a large scale integrated circuit, or a digital processing unit. The transceiver 1601 is used for data transmission and reception between an electronic device and other devices.

The electronic device may further comprise a memory 1603 for storing software instructions to be executed by the processor 1602, but may also store other data required by the electronic device, such as identification information of the electronic device, encryption information of the electronic device, user data, and the like. Memory 1603 may be a volatile memory (volatile memory), such as a random-access memory (RAM); memory 1603 may also be a non-volatile memory such as, but not limited to, a read-only memory (ROM), a flash memory (flash memory), a Hard Disk Drive (HDD) or a solid-state drive (SSD), or memory 1603 is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Memory 1603 may be a combination of the above.

The embodiment of the present application does not limit the specific connection medium among the processor 1602, the memory 1603 and the transceiver 1601. In fig. 16, the embodiment of the present application is described by taking only the case where the memory 1603, the processor 1602, and the transceiver 1601 are connected by the bus 1604 as an example, the bus is shown by a thick line in fig. 16, and the connection manner between other components is merely schematically described and is not limited thereto. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 16, but this is not intended to represent only one bus or type of bus.

The processor 1602 may be dedicated hardware or a processor running software, and when the processor 1602 runs software, the processor 1602 reads software instructions stored in the memory 1603 and executes the missing character detection method and/or the missing character detection model building method referred to in the foregoing embodiments under the driving of the software instructions.

When the method provided in the embodiments of the present application is implemented in software or hardware or a combination of software and hardware, a plurality of functional modules may be included in the electronic device, and each functional module may include software, hardware or a combination of software and hardware.

Specifically, referring to fig. 17, a schematic structural diagram of a missing character detection apparatus provided in the embodiment of the present application includes an obtaining module 1701, a determining module 1702, a generating module 1703, and a processing module 1704.

An obtaining module 1701 for obtaining an image to be detected;

a determining module 1702, configured to input the image to be detected into the established missing character detection model to determine probability information of each pixel in the image to be detected, where the probability information of each pixel includes first probability information, and the first probability information includes a probability that the pixel belongs to a code-spraying character;

a generating module 1703, configured to generate a first target image according to first probability information of each pixel in the image to be detected, which is output by the missing character detection model;

a processing module 1704, configured to extract a contour of the first target image, determine, according to the extracted contour information, code-spraying character information in the image to be detected, where the code-spraying character information includes the number of code-spraying characters, and if it is determined that the number of the code-spraying characters is smaller than a preset number, determine that the code-spraying characters in the image to be detected are missing.

Optionally, the generating module 1703 is specifically configured to:

for each pixel in the image to be detected, determining the size relation between the probability that the pixel belongs to the code spraying character and a preset probability;

generating a first target image according to the size relation between the probability that each pixel in the image to be detected belongs to the code spraying character and a preset probability; for each pixel in the image to be detected, if the probability that the pixel belongs to the code-spraying character is greater than the preset probability, the pixel corresponding to the pixel in the first target image is a mark value of the code-spraying character; otherwise, the pixel corresponding to the pixel in the first target image is a preset value representing the background.

Optionally, when each code-spraying character in the image to be detected is not missing, the image to be detected contains N types of code-spraying points, the probability information of each pixel further includes second probability information, the second probability information includes the probability that the pixel belongs to each type of code-spraying points and the probability that the pixel belongs to the background, the code-spraying character information further includes position information of each code-spraying character, N is an integer greater than 1, and

the generating module 1703 is further configured to generate a second target image according to second probability information of each pixel in the image to be detected, which is output by the missing character detection model, if it is determined that the number of the code-sprayed characters is not less than the preset number;

the processing module 1704 is further configured to determine whether code-spraying points of each code-spraying character in the image to be detected are missing according to the position information of each code-spraying character and the second target image.

Optionally, the generating module 1703 is specifically configured to:

for each pixel in the image to be detected, taking the class with the highest probability in the second probability information of the pixel as the class to which the pixel belongs;

generating a second target image according to the category of each pixel in the image to be detected; for each pixel in the image to be detected, if the type of the pixel is a background, the pixel corresponding to the pixel in the second target image is a preset value representing the background; if the category to which the pixel belongs is the ith type code spraying point, the pixel corresponding to the pixel in the second target image is the label value of the category to which the ith type code spraying point belongs, i is more than or equal to 1 and less than or equal to N, and i is an integer.

Optionally, the processing module 1704 is specifically configured to:

determining a pixel area corresponding to each code-spraying character in the second target image according to the position information of each code-spraying character;

if the type of the code spraying points contained in the pixel area is smaller than N, determining that the code spraying points of the code spraying characters in the image to be detected are missing;

and if the type of the code spraying points contained in the pixel area is not less than N, determining that the code spraying points of the code spraying characters in the image to be detected are not missing.

Referring to fig. 18, a schematic structural diagram of an apparatus for building a missing character detection model provided in the embodiment of the present application includes an obtaining module 1801, a probability determining module 1802, a loss value determining module 1803, and an adjusting module 1804.

An obtaining module 1801, configured to obtain an image sample, where the image sample includes at least one code-spraying character;

a probability determination module 1802, configured to input the image sample into a deep learning network model to determine probability information of each pixel in the image sample, where the probability information of each pixel includes first probability information, and the first probability information includes a probability that the pixel belongs to a code-sprayed character;

a loss value determining module 1803, configured to determine a first loss value of the deep learning network model according to first probability information of each pixel in the image sample output by the deep learning network model and a first label image generated in advance;

an adjusting module 1804, configured to adjust parameters of the deep learning network model according to the first loss value, and establish a missing character detection model.

Optionally, a generating module 1805 is further included, configured to generate the first label image according to the following steps:

acquiring contour marking information of each code-sprayed character in the image sample, wherein the contour marking information comprises position information of each contour marking point in the code-sprayed character;

and generating the first label image according to the position information of each contour marking point in each code-sprayed character in the image sample, wherein for each code-sprayed character in the image sample, a character contour is formed by pixels, corresponding to each contour marking point in the code-sprayed character, in the first label image, pixels in the character contour are marking values of the code-sprayed character, and pixels, which do not fall into any character contour, in the second label image are preset values representing a background.

Optionally, the labeled value of the code-spraying character is greater than the preset value representing the background, and the loss value determining module 1803 is specifically configured to:

for each pixel in the image sample, determining a first loss value of the pixel according to the probability that the pixel belongs to the code-spraying character and the value of the corresponding pixel in the first label image, and updating the first loss value of the pixel into the product of the first loss value of the pixel and the value of the corresponding pixel in the first label image;

and determining the average value of the first loss values of the pixels in the image sample as the first loss value of the deep learning network model.

Optionally, if each code-sprayed character in the image sample does not lack, N types of code-sprayed dots are included, the probability information of each pixel further includes second probability information, the second probability information includes the probability that the pixel belongs to each type of code-sprayed dot and the probability that the pixel belongs to the background, the code-sprayed character information further includes position information of each code-sprayed character, N is an integer greater than 1, and

the generating module 1805 is further configured to determine a second loss value of the deep learning network model according to second probability information of each pixel in the image sample output by the deep learning network model and a second label image generated in advance;

the adjusting module 1804 is further configured to determine a total loss value of the deep learning network model according to the first loss value and the second loss value, and adjust a parameter of the deep learning network model according to the total loss value.

Optionally, the generating module 1805 is further configured to generate the second label image according to the following steps:

acquiring contour marking information of each code spraying point in the image sample, wherein the contour marking information comprises position information of each contour marking point of the code spraying points;

and generating the second label image according to the position information of each contour mark point in each code spraying point in the image sample, wherein for each code spraying point in the image sample, a code spraying point contour is formed by pixels corresponding to each contour mark point in the code spraying points in the second label image, the pixels in the code spraying point contour are mark values of the type to which the code spraying points belong, and the pixels in the second label image which do not fall into any code spraying point contour are preset values representing the background.

Optionally, the loss value determining module 1803 is specifically configured to:

for each pixel in the image sample, taking the class with the highest probability in the second probability information of the pixel as the class to which the pixel belongs, and determining a second loss value of the pixel according to the class to which the pixel belongs and the class to which the corresponding pixel in the second label image belongs;

and determining a second loss value of the deep learning network model according to the second loss value of each pixel.

Optionally, the obtaining module 1801 is specifically configured to:

acquiring an initial image, wherein the initial image comprises at least one code spraying character;

and (3) circularly executing: shielding the code-sprayed characters in the initial image according to position marking information of the code-sprayed characters in the initial image, and taking the initial image subjected to shielding treatment as the image sample when the cycle times reach the set times;

wherein the size of the occlusion region is the same or different each time occlusion processing is performed.

The division of the modules in the embodiments of the present application is schematic, and only one logical function division is provided, and in actual implementation, there may be another division manner, and in addition, each functional module in each embodiment of the present application may be integrated in one processor, may also exist alone physically, or may also be integrated in one module by two or more modules. The coupling of the various modules to each other may be through interfaces that are typically electrical communication interfaces, but mechanical or other forms of interfaces are not excluded. Thus, modules described as separate components may or may not be physically separate, may be located in one place, or may be distributed in different locations on the same or different devices. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode.

A computer-readable medium is provided in an embodiment of the present application, and stores computer-executable instructions for performing any one of the methods described above.

In some possible embodiments, the missing character detection and missing character detection model establishing method provided by the present application may also be implemented in the form of a program product, which includes program code for causing an electronic device to perform any one of the methods described above when the program product is run on the electronic device.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The program product for missing character detection, the creation of a missing character detection model of embodiments of the present application may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a computing device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device over any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., over the internet using an internet service provider).

It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. Indeed, the features and functions of two or more units described above may be embodied in one unit, according to embodiments of the application. Conversely, the features and functions of one unit described above may be further divided into embodiments by a plurality of units.

Further, while the operations of the methods of the present application are depicted in the drawings in a particular order, this does not require or imply that these operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the application.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims

1. A missing character detection method, comprising:

acquiring an image to be detected;

if the number of the code-spraying characters is smaller than the preset number, determining that the code-spraying characters in the image to be detected are missing;

the method comprises the following steps that when each code spraying character in an image to be detected is not lost, N types of code spraying points are included, probability information of each pixel further comprises second probability information, the second probability information comprises the probability that the pixel belongs to the various code spraying points and the probability that the pixel belongs to the background, the code spraying character information further comprises position information of each code spraying character, N is an integer greater than 1, and the method further comprises the following steps:

if the number of the code-spraying characters is determined to be not smaller than the preset number, generating a second target image according to second probability information of each pixel in the image to be detected, which is output by the missing character detection model; and

and determining whether the code spraying points of the code spraying characters in the image to be detected are missing or not according to the position information of the code spraying characters and the second target image.

2. The method of claim 1, wherein generating a first target image based on the first probability information of each pixel in the image to be detected output by the missing character detection model comprises:

3. The method of claim 1, wherein generating a second target image according to second probability information of each pixel in the image to be detected output by the missing character detection model comprises:

generating a second target image according to the category of each pixel in the image to be detected; for each pixel in the image to be detected, if the type of the pixel is a background, the pixel corresponding to the pixel in the second target image is a preset value representing the background; if the type of the pixel belongs to the ith type code spraying point, the pixel corresponding to the pixel in the second target image is the label value of the ith type code spraying point, i is more than or equal to 1 and less than or equal to N, and i is an integer.

4. The method as claimed in claim 1 or 3, wherein determining whether the code-sprayed dots of each code-sprayed character in the image to be detected are missing according to the position information of each code-sprayed character and the second target image comprises:

if the type of the code spraying points of each pixel in the pixel area is smaller than N, determining that the code spraying points of the code spraying character in the image to be detected are missing;

and if the type of the code spraying points of each pixel in the pixel area is not less than N, determining that the code spraying points of the code spraying characters in the image to be detected are not missing.

5. A method for establishing a missing character detection model is characterized by comprising the following steps:

adjusting parameters of the deep learning network model according to the first loss value, and establishing a missing character detection model;

if each code-spraying character in the image sample contains N types of code-spraying points when not missing, the probability information of each pixel also comprises second probability information, the second probability information comprises the probability that the pixel belongs to each type of code-spraying points and the probability that the pixel belongs to the background, and N is an integer greater than 1, the method also comprises the following steps:

determining a second loss value of the deep learning network model according to second probability information of each pixel in the image sample output by the deep learning network model and a pre-generated second label image; and

adjusting parameters of the deep learning network model according to the first loss value, including:

and determining a total loss value of the deep learning network model according to the first loss value and the second loss value, and adjusting parameters of the deep learning network model according to the total loss value.

6. The method of claim 5, wherein the first label image is generated according to the steps of:

and generating the first label image according to the position information of each contour marking point in each code-sprayed character in the image sample, wherein for each code-sprayed character in the image sample, a character contour is formed by pixels, corresponding to each contour marking point in the code-sprayed character, in the first label image, pixels in the character contour are marking values of the code-sprayed character, and pixels, which do not fall into any character contour, in the first label image are preset values representing a background.

7. The method of claim 6, wherein the marking value of the code-sprayed character is greater than the preset value representing the background, and determining a first loss value of the deep learning network model according to the first probability information of each pixel in the image sample output by the deep learning network model and a first label image generated in advance comprises:

8. The method of claim 5, wherein the second label image is generated according to the steps of:

9. The method according to claim 5 or 8, wherein determining a second loss value of the deep learning network model according to the second probability information of each pixel in the image sample output by the deep learning network model and a pre-generated second label image comprises:

10. The method of any one of claims 5-8, wherein acquiring an image sample comprises:

11. A missing character detection apparatus, comprising:

the acquisition module is used for acquiring an image to be detected;

the processing module is used for extracting the outline of the first target image, determining code spraying character information in the image to be detected according to the extracted outline information, wherein the code spraying character information comprises the number of code spraying characters, and determining that the code spraying characters in the image to be detected are missing if the number of the code spraying characters is determined to be smaller than the preset number;

the image to be detected contains N types of code spraying points when each code spraying character is not lost, the probability information of each pixel further comprises second probability information, the second probability information comprises the probability that the pixel belongs to various types of code spraying points and the probability that the pixel belongs to the background, the code spraying character information further comprises position information of each code spraying character, N is an integer greater than 1, and

the generating module is further configured to generate a second target image according to second probability information of each pixel in the image to be detected, which is output by the missing character detection model, if it is determined that the number of the code-sprayed characters is not less than the preset number;

the processing module is further configured to determine whether code spraying points of each code spraying character in the image to be detected are missing according to the position information of each code spraying character and the second target image.

12. The apparatus of claim 11, wherein the generation module is specifically configured to:

for each pixel in the image to be detected, determining the size relation between the probability that the pixel belongs to the code spraying character and a preset probability; generating a first target image according to the size relation between the probability that each pixel in the image to be detected belongs to the code spraying character and a preset probability; for each pixel in the image to be detected, if the probability that the pixel belongs to the code-spraying character is greater than the preset probability, the pixel corresponding to the pixel in the first target image is a mark value of the code-spraying character; otherwise, the pixel corresponding to the pixel in the first target image is a preset value representing the background.

13. The apparatus of claim 11, wherein the generation module is specifically configured to:

for each pixel in the image to be detected, taking the class with the highest probability in the second probability information of the pixel as the class to which the pixel belongs; generating a second target image according to the category of each pixel in the image to be detected; for each pixel in the image to be detected, if the type of the pixel is a background, the pixel corresponding to the pixel in the second target image is a preset value representing the background; if the category to which the pixel belongs is the ith type code spraying point, the pixel corresponding to the pixel in the second target image is the label value of the category to which the ith type code spraying point belongs, i is more than or equal to 1 and less than or equal to N, and i is an integer.

14. The apparatus of claim 11 or 13, wherein the processing module is specifically configured to:

determining a pixel area corresponding to each code-spraying character in the second target image according to the position information of each code-spraying character; if the type of the code spraying points contained in the pixel area is smaller than N, determining that the code spraying points of the code spraying characters in the image to be detected are missing; and if the type of the code spraying points contained in the pixel area is not less than N, determining that the code spraying points of the code spraying characters in the image to be detected are not missing.

15. An apparatus for building a missing character detection model, comprising:

the adjusting module is used for adjusting parameters of the deep learning network model according to the first loss value and establishing a missing character detection model;

if each code-spraying character in the image sample contains N types of code-spraying points when not missing, the probability information of each pixel also comprises second probability information, the second probability information comprises the probability that the pixel belongs to the various code-spraying points and the probability that the pixel belongs to the background, and N is an integer greater than 1, then the image sample contains N types of code-spraying points when not missing

The generating module is further used for determining a second loss value of the deep learning network model according to second probability information of each pixel in the image sample output by the deep learning network model and a pre-generated second label image;

the adjusting module is further configured to determine a total loss value of the deep learning network model according to the first loss value and the second loss value, and adjust a parameter of the deep learning network model according to the total loss value.

16. The apparatus of claim 15, further comprising a generation module to generate the first label image according to the steps of:

17. The apparatus of claim 16, wherein the labeled value of the inkjet-coded character is greater than the preset value indicative of the background, and the loss value determining module is specifically configured to:

for each pixel in the image sample, determining a first loss value of the pixel according to the probability that the pixel belongs to the code-spraying character and the value of the corresponding pixel in the first label image, and updating the first loss value of the pixel into the product of the first loss value of the pixel and the value of the corresponding pixel in the first label image; and determining the average value of the first loss values of the pixels in the image sample as the first loss value of the deep learning network model.

18. The apparatus of claim 15, wherein the generation module is further configured to generate the second label image according to:

19. The apparatus of claim 15 or 18, wherein the loss value determination module is specifically configured to:

for each pixel in the image sample, taking the class with the highest probability in the second probability information of the pixel as the class to which the pixel belongs, and determining a second loss value of the pixel according to the class to which the pixel belongs and the class to which the corresponding pixel in the second label image belongs; and determining a second loss value of the deep learning network model according to the second loss value of each pixel.

20. The apparatus of any one of claims 15-18, wherein the acquisition module is specifically configured to:

acquiring an initial image, wherein the initial image comprises at least one code spraying character; and (3) circularly executing: shielding the code-sprayed characters in the initial image according to position marking information of the code-sprayed characters in the initial image, and taking the initial image subjected to shielding treatment as the image sample when the cycle times reach the set times; wherein the size of the occlusion region is the same or different each time occlusion processing is performed.

21. An electronic device, comprising: at least one processor, and a memory communicatively coupled to the at least one processor, wherein:

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 10.

22. A computer-readable medium having stored thereon computer-executable instructions for performing the method of any one of claims 1 to 10.