CN117291926B

CN117291926B - Character defect detection method, apparatus, and computer-readable storage medium

Info

Publication number: CN117291926B
Application number: CN202311589481.XA
Authority: CN
Inventors: 吴雨培; 富奕通
Original assignee: Beijing Aqiu Technology Co ltd
Current assignee: Beijing Aqiu Technology Co ltd
Priority date: 2023-11-27
Filing date: 2023-11-27
Publication date: 2024-03-05
Anticipated expiration: 2043-11-27
Also published as: CN117291926A

Abstract

The invention discloses a character defect detection method, character defect detection equipment and a computer readable storage medium, and belongs to the technical field of image processing. The method comprises the following steps: determining the category, the position and the size of each character in the initial image based on the character recognition model; extracting a character image corresponding to the character according to the category, the position and the size; and determining the character defect type corresponding to each character image through a character defect classification model. The invention aims to improve the accuracy of character defect detection.

Description

Character defect detection method, apparatus, and computer-readable storage medium

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to a method and apparatus for detecting character defects, and a computer readable storage medium.

Background

Character defect detection, which is a key image processing technology, is widely used in various industrial scenes. The method mainly aims at detecting and identifying character defects such as incomplete, reprinting, deformation and the like, and verifying the character quality so as to ensure the definition and the readability of the characters and further meet strict product quality requirements.

In the related art, a deep learning method is generally adopted to detect character defects, taking a convolutional neural network as an example, in the training process, the convolutional neural network can extract local features of an image based on convolution operation and pooling operation, and capture features of different scales through a hierarchical structure and convolution layer stacking, so that certain background and context information are indirectly considered, and the improvement of model performance is realized.

However, because of global concerns, convolutional neural networks tend to ignore local quality problems of characters, such as when a character is slightly flawed or otherwise slightly defective, which can be ignored by the convolutional neural network and still be identified as a flawless character.

Disclosure of Invention

The invention mainly aims to provide a character defect detection method, equipment and a computer readable storage medium, and aims to solve the technical problem that the accuracy of the existing character defect detection method is low.

In order to achieve the above object, the present invention provides a character defect detection method comprising the steps of:

determining the category, the position and the size of each character in the initial image based on the character recognition model;

Extracting a character image corresponding to the character according to the category, the position and the size;

and determining the character defect type corresponding to each character image through a character defect classification model.

Optionally, after the step of extracting the character image corresponding to the character according to the category, the position and the size, the method includes:

calculating a histogram corresponding to the character image, and determining a gray value threshold according to the distribution condition of the histogram;

comparing each pixel in the character image with the gray value threshold value, and generating a binarized image according to a comparison result;

calculating the average character sizes of all the binarized images, and obtaining the standard character sizes and standard image sizes required by the character defect classification model;

dividing the standard character size by the average character size to obtain a scaling factor, and multiplying the character size of each binarized image by the scaling factor;

and if the image size of the binarized image does not meet the standard image size, performing pigment filling on the edge of the binarized image according to the standard image size to generate a new character image.

Optionally, the step of determining the character defect type corresponding to each character image through the character defect classification model includes:

determining the prediction scores of the character images on various types through a convolution layer, a batch normalization layer, a nonlinear activation layer and a full connection layer, and obtaining corresponding index results after exponential function processing;

summarizing the prediction scores of the character images on all the types, and obtaining a total index result after the index function processing;

dividing the index result by the total index result, determining the confidence degree of the character image on each type, and determining the type with the highest confidence degree as the character defect type.

selecting one or more target character defect classification submodules from a character defect classification submodule set according to the defect mode corresponding to the category;

and determining the character defect type of the character image by using the target character defect classification sub-module.

Optionally, before the step of determining the category, the position and the size of each character in the initial image based on the character recognition model, the method includes:

Determining the category, the position and the size of each character in the sample image based on the character recognition model;

extracting a sample character image corresponding to the character according to the category, the position and the size;

selecting the sample character image in the font file, and generating a rendering character image according to the corresponding setting parameters;

randomly selecting partial pixels in the rendered character image to erode and/or expand to obtain an irregular character image;

preprocessing the irregular character image to generate a corresponding training character image, and receiving labeling information of a user based on the character image;

determining the character defect type corresponding to each training character image through an initial character defect classification model;

and calculating a loss function according to the character defect type and the labeling information, and adjusting the character defect classification model according to the loss function.

Optionally, the step of preprocessing the irregular character image to generate a corresponding training character image includes:

performing binarization and size normalization processing on the irregular character image to obtain a standard character image;

randomly selecting character pixels in the standard character image, and covering the character pixels by using a geometric shape consistent with the color of background pixels to obtain a character shielding image;

Performing corrosion operation on the character shielding image by using specific structural elements to obtain a corresponding character ink-lacking image;

randomly generating line segments meeting the size requirement, rotating the line segments at random angles, and then shielding the standard character image to generate a character breaking image;

randomly shifting the standard character image in the transverse and longitudinal directions, and overlapping the shifted image to the standard character image to generate a character reprinting image;

dividing the standard character image into two parts along the transverse and longitudinal directions, randomly selecting one part for shrinkage or stretching, and splicing the shrunk or stretched image with the other part to generate a character deformation image;

and summarizing the character shielding image, the character ink-missing image, the character word-breaking image, the character reprinting image and the character deformation image which meet the standard requirements to obtain a training character image.

Optionally, after the step of summarizing the character shielding image, the character ink-missing image, the character word breaking image, the character reprinting image and the character deformation image which meet the specification requirements, the step of obtaining a training character image includes:

Randomly scaling the size of the training character image to be within a first preset range;

randomly translating the position coordinates of the zoomed training character image into a second preset range;

randomly rotating the training character image within a third preset range by taking the image center of the translated training character image as an axis;

and if the rotated training character image does not meet the size requirement, filling or cutting is carried out.

Optionally, after the step of determining the character defect type corresponding to each character image through the character defect classification model, the method includes:

screening adjacent character images from all the character images with defects, and determining the defect types of the adjacent character images;

and if the defect types are the same, sending the adjacent character images to a review interface of the user, and receiving a review result fed back by the user.

In addition, in order to achieve the above object, the present invention also provides a character defect detecting apparatus comprising: the character defect detection device comprises a memory, a processor and a character defect detection program which is stored in the memory and can run on the processor, wherein the character defect detection program is configured to realize the steps of the character defect detection method.

Further, a computer-readable storage medium has stored thereon a character defect detection program which, when executed by a processor, implements the steps of the character defect detection method.

In one technical scheme provided by the invention, the category, the position and the size of each character in an initial image are determined based on a character recognition model, then the character image is extracted based on the parameters, and is input into a character defect classification model to determine the corresponding character defect type. By carrying out defect detection on a single character image, defects can be more accurately positioned and identified without being interfered by other parts in the whole image, so that the model is more focused on the local information of the current character, and further the accuracy and reliability of defect detection are improved.

Drawings

FIG. 1 is a flowchart of a first embodiment of a character defect detection method according to the present invention;

FIG. 2 is a flowchart of step S13 in a first embodiment of the character defect detection method according to the present invention;

FIG. 3 is a flowchart of step S13 in a first embodiment of the character defect detection method according to the present invention;

FIG. 4 is a flowchart of a second embodiment of a character defect detection method according to the present invention;

FIG. 5 is a simplified flowchart illustrating a character defect detection method according to a second embodiment of the present invention;

FIG. 6 is a character image before preprocessing and a character image after preprocessing in a second embodiment of the character defect detecting method of the present invention;

FIG. 7 is a flowchart of a third embodiment of a character defect detection method according to the present invention;

FIG. 8 is a detailed flowchart of a training portion of a third embodiment of the character defect detection method of the present invention;

FIG. 9 is an image of a training character in a third embodiment of the character defect detection method of the present invention;

fig. 10 is a character image enhanced by applying affine transformation data in a third embodiment of the character defect detecting method of the present invention;

FIG. 11 is a diagram showing a comparison of different training data source character defect classification model indexes and a comparison of different defect data source character defect classification model indexes according to a third embodiment of the character defect detection method of the present invention;

fig. 12 is a schematic structural diagram of a character defect detecting device in a hardware running environment according to an embodiment of the present invention.

The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.

Detailed Description

It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

In industrial production, key information of a product, such as a lot number, a date of production, etc., is often required to be displayed in the form of characters on the product. The characters are added by etching, printing and other technical means, however, defects such as character defects, reprinting, deformation and the like can occur in the process.

Character defect detection, which is a key image processing technology, is widely used in various industrial scenes. The method mainly aims at detecting and identifying character defects such as incomplete, reprinting, deformation and the like, and verifying the character quality so as to ensure the definition and the readability of the characters and further meet strict product quality requirements. Currently, the main character defect detection techniques include:

template matching-based method: the predefined character template is compared with the image to be detected to determine if the character is defective. In practice, the template is typically compared to the image at various locations and the degree of match is calculated, and when the degree of match is below a certain threshold, it is determined that a defect is present. However, this approach has some limitations. Firstly, the method is very sensitive to the changes of factors such as the size, the direction, the shooting condition and the like of the characters, and has poor processing effect due to certain affine transformation such as rotation, scaling and the like between a template and an image; second, specific defect types of characters, such as defects, deformations, and the like, cannot be accurately determined and classified only by setting a threshold value.

Feature extraction-based method: defects of the character are detected by extracting morphological features of the character, such as inflection points, edges, and the like. However, the main challenge of this approach is to carefully design and select features. Feature design usually requires expertise and practical experience, and is adjusted according to specific requirements of tasks and scenes, for example, in a task of identifying character defects, features reflecting character integrity need to be designed; in a scene handling different lighting conditions, it is necessary to design features that are resistant to lighting variations. The selection and combination of features requires complex optimization algorithms and thus may be limited in versatility.

Deep learning-based method: in particular strategies using convolutional neural networks have shown great capability. The method can learn and extract complex features of the image according to the fitting target to perform character recognition, and is expected not to be recognized when the characters have defects. However, this approach may not only focus on the character's own features, but also on the background and other contextual information when processing the image. While this aids in understanding the overall image, global concerns may make the model ignore local quality issues with the character. For example, convolutional neural networks may ignore these problems when characters are slightly incomplete or otherwise slightly defective, yet still recognize them as non-defective. On the other hand, while some researchers have attempted to detect defects in images directly using a deep-learned target detection strategy, which treats characters as part of the background, it is often more challenging to detect defects directly than to identify characters due to complex background interference and multiple defect types.

Aiming at the problems, particularly the problem of low accuracy of the existing character defect detection method, the invention firstly identifies the parameters of each character from the initial image, extracts the character image according to the parameters, and finally determines the defect type corresponding to each character image, thereby aiming at improving the detection accuracy.

In order to better understand the above technical solution, exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

Referring to fig. 1, fig. 1 is a schematic flow chart of a first embodiment of a character defect detection method according to the present invention.

In this embodiment, the character defect detection method includes:

step S11: determining the category, the position and the size of each character in the initial image based on the character recognition model;

it will be appreciated that the character recognition model is an artificial intelligence technique for obtaining parameters of each character in an initial image, where the initial image is a product image collected on an industrial production line, and the characters are identification information, such as letters, numbers, chinese characters, symbols, and the like, included in the product image. The parameters of the character specifically include a category, a position, a size, etc., wherein the category refers to a type of the character, such as which of letters, numbers, etc.; the position refers to specific position information of the character in the whole initial image and can be expressed by coordinates; the dimensions refer to the size of the character and can be generally represented by the width and height of the character.

Alternatively, the approximate location of each character in the initial image is located using an edge detection algorithm, a contour detection algorithm, or the like. On this basis, the category, position and size of each character are determined by using a trained character recognition model.

Or extracting character features by using a gray level histogram, a local binary pattern and other methods, inputting the features into a classifier for classification, and determining the category to which the features belong; using edge detection algorithm, contour detection algorithm and other methods to find out the boundary frame or contour of the character, and determining the position of the character in the initial image by analyzing the position information of the boundary frame or contour; the width and height of the bounding box or outline are used to represent the size of the character.

Step S12: extracting a character image corresponding to the character according to the category, the position and the size;

it will be appreciated that the direct use of convolutional neural networks for defect identification throughout the original image is not immune to background and context information. Therefore, character extraction needs to be performed on the whole initial image, each character in the initial image is independent to form an independent character image, and therefore follow-up defect classification models can concentrate on specific characteristics and possible defects of each character and cannot be interfered by context information around the characters.

Firstly, classifying characters according to categories to obtain a number group, a letter group and the like, and then extracting the characters in each character group in turn. Taking a digital group as an example, determining the position coordinate of a first character, determining the size of an extraction frame according to the size of the first character, and extracting the first character by taking the position coordinate as the center of the extraction frame to obtain a corresponding character image.

Note that since different types of characters may have different shapes, sizes, and features, different extraction boxes may be set for each group, and features of each type of character may be better captured. For the digital group, a rectangular extraction box is used, for example, since most have a more pronounced rectangular or square shape; for a circular set of symbols, such as "@", a circular extraction box may be used; for square symbol groups, such as "#", square extraction boxes can be used; for the handwritten character set, since the shape difference is large, an n-sided extraction box may be used, and the value of n may be determined according to the edge feature points of the character.

According to the scheme, the specific feature extraction method is selected for each category, so that feature information of each category can be better reserved, feature mixing among different categories is avoided, pertinence is improved, and accuracy and effectiveness are improved. Moreover, the character category can be conveniently expanded and updated by extracting the characters in batches according to the category, and when a new character category needs to be added, only the character of the category needs to be extracted and classified, and all the characters do not need to be reprocessed.

Step S13: and determining the character defect type corresponding to each character image through a character defect classification model.

It is understood that the character defect type refers to a defect or error type that may exist in a character image, including but not limited to character occlusion, character deficiency, character breaking, character reprinting, character morphing, character flash, character blurring, and the like.

Optionally, each character image is input into a trained character defect classification model to obtain a corresponding character defect type, and common classification models such as a support vector machine, a random forest and a convolutional neural network are not particularly limited in this embodiment. If the convolutional neural network large model is adopted to classify the defects of the independent character images, compared with the traditional machine vision algorithm, the accuracy of detection can be remarkably improved, meanwhile, the dependence on professional knowledge is reduced, and non-professional staff can conveniently detect the defects.

Optionally, referring to fig. 2, step S13 includes:

step S131: determining the prediction scores of the character images on various types through a convolution layer, a batch normalization layer, a nonlinear activation layer and a full connection layer, and obtaining corresponding index results after exponential function processing;

Step S132: summarizing the prediction scores of the character images on all the types, and obtaining a total index result after the index function processing;

step S133: dividing the index result by the total index result, determining the confidence degree of the character image on each type, and determining the type with the highest confidence degree as the character defect type.

It can be appreciated that the character defect classification model is formed by a series of convolutional layers, batch normalization layers and nonlinear activation layers which are linearly stacked, and finally outputs confidence of each type through the full connection layers and normalization functions.

Optionally, the convolution layer can extract the features of the character image, and the convolution operation is performed on the image through the sliding window to obtain a series of feature images, wherein the feature information representing different positions and scales in the image can be used for subsequent classification and identification; the batch normalization layer can normalize the output of the convolution layer, so that the mean value of each characteristic channel is close to 0, the variance is close to 1, the convergence speed of the model can be accelerated, and the stability and generalization capability of the model are improved; the nonlinear activation layer introduces nonlinear transformation, specifically, a weight vector and input features are linearly transformed on each neuron of the fully connected layer, and then nonlinear transformation is performed through a nonlinear activation function such as ReLU, so that the original prediction score of the character image in each type can be obtained.

Further, the scheme adopts a Softmax function as a normalization function, performs exponential function processing on the original prediction scores of each type to obtain corresponding exponential results, and collects the prediction scores of all types on the basis of the exponential function processing to obtain total exponential results. And finally, calculating the confidence coefficient of the character image on each type according to the following formula, and determining the type with the highest confidence coefficient as the character defect type.

Wherein,representing the model's original predictive score for the i-th type,/for the i-th type>Is the result of subjecting this fraction to an exponential function,/->For total index result, < >>For the i-th type of confidence, this function translates the original score of the model output into probabilities for each class to ensure that all output values are between 0 and 1 and their sum is 1, making the model output easier to interpret.

After calculating the confidence of each defect type, the type with the highest confidence is determined as the character defect type.

Optionally, referring to fig. 3, step S13 includes:

step S134: selecting one or more target character defect classification submodules from a character defect classification submodule set according to the defect mode corresponding to the category;

Step S135: and determining the character defect type of the character image by using the target character defect classification sub-module.

It will be appreciated that each character has different specific properties and possible defect patterns, and thus a classification model can be built separately for each character. This means that a plurality of models, each of which is specially responsible for identifying defects of one character, will be trained, so that it is possible to consider specific properties and defect patterns of each character, thereby improving the accuracy of classification.

Alternatively, in the process of recognizing the character category by the character recognition model, recognition can be performed based on specific properties of the character, for example, alphabetic characters generally have different shapes and line combinations, such as straight lines, curved lines, sharp angles, and the like; numerical characters are typically composed of straight lines and curved lines, with a significant degree of distinction in shape, and numerical characters are typically of relatively uniform height and width compared to alphabetic characters, and they are typically simpler in shape than alphabetic characters, without falling portions, such as letters "g" and "p", or rising portions, such as letters "b" and "d"; symbolic characters, such as punctuation marks, mathematical symbols, monetary symbols, etc., typically have unique combinations of shapes and lines.

Optionally, preset defect modes corresponding to various types in advance, for example, defect modes corresponding to alphabetic characters are connecting lines and deformation defects; the Chinese character may have radical missing or misplacement, i.e. structural error defect, and in the case of complex characters, fuzzy unclear condition, i.e. fuzzy defect, so that the defect modes corresponding to the Chinese character are wiring, deformation, structural error and fuzzy defect.

Further, according to the defect mode, one or more target character defect classification submodules are selected from the character defect classification submodule set. If aiming at the alphabetic characters, selecting a connecting sub-module and a deformation sub-module; and selecting a connecting sub-module, a deformation sub-module, a structural error sub-module and a fuzzy sub-module aiming at the digital characters.

Further, each character image is input into each trained sub-module to detect whether defects exist and the specific defect type.

It should be noted that defect classification may also be performed in a comparably classified manner. Specifically, two pictures are input into the model at a time, and one picture is a non-defective character image and is used as a reference template; the other is a character image of the type of defect to be identified. The task of the model is to compare the two pictures, and to determine whether the character image to be recognized is defective or not, and the type of the defect. This approach translates the problem into a more intuitive comparison task that may help to improve the performance of the model.

In addition, after determining the defect type of each character, the method further comprises:

It will be appreciated that an art word is a special font design, typically used for decorative, design or personalization purposes, in which there may be an overlapping, linking or overlapping effect between characters, which is intended to be, and not a true character defect.

Optionally, selecting adjacent character images from all the character images with defects, determining the defect types of the adjacent character images, if the defect types are the same, describing that the defect types are the deliberately designed artistic fonts, and therefore, sending the deliberately designed artistic fonts to a review interface of a user, carrying out review by the user, and receiving a review result fed back by the user.

In one technical scheme provided by the embodiment, firstly, the category, the position and the size of each character in an initial image are determined based on a character recognition model, then, a character image is extracted based on the parameters, and is input into a character defect classification model to determine the corresponding character defect type. By carrying out defect detection on a single character image, defects can be more accurately positioned and identified without being interfered by other parts in the whole image, so that the model is more focused on the local information of the current character, and further the accuracy and reliability of defect detection are improved.

Further, referring to fig. 4, a second embodiment of the character defect detection method of the present invention is proposed. Based on the embodiment shown in fig. 1, after the step of extracting the character image corresponding to the character according to the category, the position and the size, the method includes:

step S21: calculating a histogram corresponding to the character image, and determining a gray value threshold according to the distribution condition of the histogram;

step S22: comparing each pixel in the character image with the gray value threshold value, and generating a binarized image according to a comparison result;

it will be appreciated that after each character image is extracted from the initial image, problems such as noise, spots, and the like may occur due to interference in the image acquisition device or transmission process, and these problems may affect the accuracy of the subsequent classification result, so that preprocessing is required for all the character images before inputting the defect detection classification model.

Referring to fig. 5, a simplified flowchart is provided for the whole scheme, and the preprocessing of the scheme includes binarization, character size adjustment and image size normalization.

It will be appreciated that binarizing an image is the process of converting the gray value of the image into a binary value, and specific methods include global thresholding, adaptive thresholding, oxford binarization, histogram-based methods, and the like. The purpose of binarization is to simplify the information in the image into black and white two colors, and the binarization can simplify the complex information into binary form, so as to highlight the structural characteristics of the character, remove the background noise and facilitate the subsequent image processing and analysis. Compared with other deep learning methods, the method can understand the whole image, ignore interference of other factors such as background and the like, and focus on the local quality problem of the characters.

Alternatively, taking the global thresholding method as an example, the character image is first converted into a grayscale image, i.e., a color image is converted into a single-channel grayscale image. And then calculating a gray level histogram of the gray level image, and counting the number of pixels of each gray level. On the basis, a proper gray value threshold value is selected according to the gray histogram through an Otsu algorithm, an Otsu algorithm and the like.

Further, comparing the gray value of each pixel point in the character image with a gray value threshold, setting the pixel point larger than the threshold as white, namely 1, setting the pixel point smaller than the threshold as black, namely 0, and finally summarizing to generate a binarized image, wherein white represents the character, and black represents the background.

Step S23: calculating the average character sizes of all the binarized images, and obtaining the standard character sizes and standard image sizes required by the character defect classification model;

step S24: dividing the standard character size by the average character size to obtain a scaling factor, and multiplying the character size of each binarized image by the scaling factor;

it will be appreciated that the size of the current character image may be different from the size of the character image during training, and the performance of the model may be affected by the character size, so that it is necessary to align the size of the current character image with the size during training.

On the one hand, the average character size of all the characters is calculated, namely, the size of a white area in the binarized image is calculated, and the average character size is obtained through averaging. On the other hand, the standard character size of the character defect classification model in the training process is obtained.

Further, the standard character size is divided by the average character size to obtain a scaling factor γ. Next, the character size of each binarized image is multiplied by a scaling factor γ so as to be identical to the size of the character image at the time of training.

Step S25: and if the image size of the binarized image does not meet the standard image size, performing pigment filling on the edge of the binarized image according to the standard image size to generate a new character image.

Finally, the image sizes of all the binarized images are unified to a standard image size, such as 64×64 pixels, which is generally larger than the actual character size.

Alternatively, if the image size of the binarized image does not satisfy the standard image size, an edge portion of a phase difference between the two is determined, and then the edge portion of the binarized image is filled with black pigment to generate a new character image, referring to fig. 6, a character image before preprocessing and a character image after preprocessing, respectively.

In the technical scheme provided by the embodiment, the character image is subjected to the preprocessing steps of binarization, character size adjustment and image size normalization in sequence, noise and detail information can be removed, the shape and outline of the character are highlighted, the size and proportion of the character and the image are unified, the influence of size difference on subsequent processing and analysis is eliminated, and the stability and reliability of character defect recognition results are improved.

Further, referring to fig. 7, a third embodiment of the character defect detection method of the present invention is proposed. Based on the embodiment shown in fig. 1, before the step of determining the category, the position and the size of each character in the initial image based on the character recognition model, the method includes:

step S31: determining the category, the position and the size of each character in the sample image based on the character recognition model;

step S32: extracting a sample character image corresponding to the character according to the category, the position and the size;

the scheme can be divided into two parts, namely training and reasoning, the detailed flow of the training part is shown in figure 8, and the detailed flow of the reasoning part is shown in figure 5.

Optionally, a training sample is formed based on images of various characters collected by an industrial production line and corresponding manual labeling information, then the training sample is input into a preset character recognition model for training, the model can output the category, position and size of each character, then the output result is compared with the manual labeling information, and then the character recognition model is continuously optimized according to the comparison result.

Further, the specific steps of extracting the sample character image corresponding to the character according to the category, the position and the size are the same as those of the first embodiment, and will not be described herein.

Step S33: selecting the sample character image in the font file, and generating a rendering character image according to the corresponding setting parameters;

it will be appreciated that the real data collected is often not sufficiently diverse, it is difficult to cover all scenes and fonts, such as fonts for different lines, characters may differ, and it is difficult to collect all classes of characters due to time constraints. This makes it difficult to make the character defect classification model trained based on real data alone versatile. Therefore, the scheme uses a character body file such as a ttf file to generate character images so as to simulate characters of different scenes and different production lines.

Optionally, after the FreeType library is used for loading the ttf file, a sample character image, namely a character to be rendered, is selected, and then the character is rendered into a bitmap at the center position of the background image according to corresponding setting parameters, such as the size, the position, the spacing and the like of the character, and is saved as an image, so that a rendered character image is obtained.

Step S34: randomly selecting partial pixels in the rendered character image to erode and/or expand to obtain an irregular character image;

Step S35: preprocessing the irregular character image to generate a corresponding training character image, and receiving labeling information of a user based on the character image;

it will be appreciated that in actual production, the characters eventually presented are often not regular due to variations in the external environment, materials and character printing equipment. Thus, further processing of the generated characters is required to simulate these irregularities.

Optionally, part of pixels in the rendered character image are randomly selected to erode and/or expand, for example, 50% of pixels of the number of character outlines are randomly selected to erode or expand for 2-3 pixels, so as to simulate the irregularity of the character edges, and the effect is as in the second graph in fig. 4, so that an irregular character image is obtained. This method allows a large amount of training data to be generated in a short time, thereby improving training efficiency and performance of the model.

Optionally, the step S35 includes:

step S351: performing binarization and size normalization processing on the irregular character image to obtain a standard character image;

it will be appreciated that in order to increase the diversity of training samples and simulate a real production environment, defect simulation is performed on character images to simulate defects of characters, such as shielding, ink shortage, word breaking, reprinting, deformation, and the like, which may occur on a production line.

Optionally, the global thresholding method is used to binarize the character image, and then size normalization processing is performed, including the character size and the image size, and specific steps are the same as those of the second embodiment, which are not described herein again, so as to finally obtain the standard character image.

Step S352: randomly selecting character pixels in the standard character image, and covering the character pixels by using a geometric shape consistent with the color of background pixels to obtain a character shielding image;

optionally, the step of emulating character occlusion is as follows:

character pixels in the standard character image, such as white pixels, are randomly selected, and then covered with geometric shapes consistent with the color of background pixels, such as black circles, ovals, rectangles, random polygons and the like, so as to obtain the simulated character shielding image.

For circular coverage, the radius is calculated as follows:

where radius_pixel represents the radius of a circle in pixels. The value is the smaller of the height and width of the image multiplied by a number selected uniformly and randomly from the range 0.1, 0.15.

The ellipse is covered like a circle, the major axis and the minor axis of the ellipse are selected by using the formula 1 respectively, and then the generated ellipse is randomly rotated within the range of [0 DEG, 180 DEG ] by taking the center as the axis.

For rectangular coverage, the specific calculation modes of the length and the width are as follows:

where h and w represent the rectangular height and width in pixels. Their value is the smaller of the height and width of the image multiplied by a number chosen uniformly randomly from the range of 0.1,0.4. Then, the generated rectangle is rotated around the center as an axis at random in the interval of [0 DEG, 180 DEG ] and then covered on the image.

For random polygon coverage, a polygenerator is used to generate a number of polygon vertices within the image, and a closed figure of these vertices is filled in on the image.

Step S353: performing corrosion operation on the character shielding image by using specific structural elements to obtain a corresponding character ink-lacking image;

optionally, the step of emulating the character ink deficiency is as follows:

based on the character shielding image, the specific structural elements, such as the rectangle of 3*3, are used for corroding the character shielding image, so that the corresponding character ink-missing image is obtained.

Step S354: randomly generating line segments meeting the size requirement, rotating the line segments at random angles, and then shielding the standard character image to generate a character breaking image;

optionally, the step of emulating a character breaking is as follows:

Segments meeting the size requirement, such as a width of 1 or 2, are randomly generated, wherein the length of the segments is random, but needs to meet the requirement of not exceeding the height and width of the image. Then, the standard character image is shielded after rotating by [0 DEG, 180 DEG ] within any angle in the interval by taking the center as the axis, and the character word-breaking image is generated.

Step S355: randomly shifting the standard character image in the transverse and longitudinal directions, and overlapping the shifted image to the standard character image to generate a character reprinting image;

optionally, the step of reprinting the simulated character is as follows:

the random shift applied to the image in the lateral and longitudinal directions, respectively, may be set to a range of 0.1, 0.2, which means that the shift amount may be between 10% and 20% of the width or height of the character. And then superposing the offset image on the standard character image to generate a character reprinting image so as to simulate the position offset of the printing equipment and the product, thereby leading to the situation that the characters are repeatedly printed.

Step S356: dividing the standard character image into two parts along the transverse and longitudinal directions, randomly selecting one part for shrinkage or stretching, and splicing the shrunk or stretched image with the other part to generate a character deformation image;

It will be appreciated that when characters are printed on flexible materials, such as paper labels, woven bags, etc., the characters may be deformed due to the fact that the surface of the material may not be fully flattened or there may be an inclination between the material and the printing device.

Optionally, the step of simulating character deformation is as follows:

the image is equally divided into two parts along the transverse direction or the longitudinal direction, one part of the image is randomly selected and subjected to longitudinal or transverse shrinkage or stretching, the range can be set to be [0.5, 0.7] and [1.3, 1.5], which means that 50-70% shrinkage or 130-150% stretching is carried out on one part of the character, the modified image is spliced with the other part, redundant or missing parts are cut or filled, so that the image size is kept consistent, and finally, the character deformation image is obtained.

Step S357: and summarizing the character shielding image, the character ink-missing image, the character word-breaking image, the character reprinting image and the character deformation image which meet the standard requirements to obtain a training character image.

Optionally, performing standard detection on the simulated defect image, if for a character shielding image, if the ratio of the number of white pixels of the covered image to the number of white pixels of the original image is less than 0.9, considering that the defect simulation is qualified, and repeatedly generating the defect until the defect is qualified for an unqualified picture; for the simulated ink-deficiency image, judging whether the ratio of the number of the white pixels of the covered image to the number of the white pixels of the original image is smaller than a threshold value or not, and judging whether defect simulation is qualified or not; and judging whether the generated defects are qualified or not according to whether the number of the connected domains is larger than that of the original image for the character breaking image. Finally, the simulated defect images meeting the specification requirements are summarized, and training character images are generated as shown in fig. 9.

In addition, after various simulated defect images are generated, the following steps can be performed:

It will be appreciated that affine transformations, such as rotation, scaling, and translation transformations, are employed on the simulated defect image to increase its diversity. By data enhancement, more training data can be generated from a relatively limited training sample, which will help the model better understand and learn the various possible variations of the character to improve the generalization ability of the model and thus the accuracy of character defect detection.

Optionally, the size of the training character image is randomly scaled to a first preset range, such as applying a scaling transform with a range of [0.9, 1.1], i.e., the size of the image is randomly adjusted to between 90% and 110% of the original size.

Optionally, the position coordinates of the scaled training character image are randomly translated into a second preset range, such as applying a translation transformation with a range of [ -0.1, 0.1], i.e. the positions of the image on the x-axis and the y-axis are randomly moved by-10% to 10% of the original positions, respectively, and positive and negative represent directions.

Optionally, the image center of the translated training character image is taken as an axis, and random rotation is performed in a third preset range, for example, rotation transformation with the application range of [ -5 degrees, 5 degrees ] is performed, that is, the image center is taken as an axis, and random rotation is performed in [ -5 degrees, 5 degrees ]. As shown in fig. 10, in order to apply the character image enhanced by affine transformation data, it can be seen from these figures that the main features of the character remain despite the transformation, which is very useful for training the model recognition and handling various situations that may occur in actual production.

It is noted that these transformations may result in the image size no longer meeting the requirements. For example, rotation or translation may cause portions of the image to move out of the original boundaries, while scaling may change the overall size of the image. In this case, the transformed image is appropriately processed to ensure that its size is satisfactory. If the image becomes smaller, a black fill needs to be added around it to increase its size to a range that satisfies the requirements. If the image becomes large, then it is necessary to cut out the size that meets the requirements starting from the center.

In general, the training set of character defect classification models is mainly composed of binarized and size-normalized character images, and images based on these images after defect simulation and data enhancement. Each defect type, defect free, character incomplete, character reprint, character morph, etc., contains the same number of samples. The final training set contains a total of 40 ten thousand images.

Further, in order to improve the accuracy of character defect detection, a sample equalization strategy may be employed to ensure that the number of each character in the training set is the same. The sample equalization strategy is to solve the problem of data imbalance, and when the sample number of each category in the training data is too large, the model may bias to the category with more samples, so that the identification effect of the category with less samples is poor. Through a sample equalization strategy, the sample quantity of each category can be similar by sampling a few categories or undersampling a majority category, so that the recognition capability of the model on each category is improved.

Step S36: determining the character defect type corresponding to each training character image through an initial character defect classification model;

step S37: and calculating a loss function according to the character defect type and the labeling information, and adjusting the character defect classification model according to the loss function.

Optionally, determining the character defect type corresponding to each training character image according to the initial character defect classification model, comparing the character defect type with the labeling information of the user based on the character images, further calculating a loss function, and adjusting the parameters of the character defect classification model according to the loss function.

As shown in fig. 11, the above table is a comparison of indexes of character defect classification models of different training data sources, and it is known that the training data sources of the character defect classification models have significant influence on model performance. The character image training model generated using the multiple font files may achieve higher accuracy than using only real character images from the same batch of a single production line. This is because the character images of various fonts provide more diversified data, which can help the model learn richer features, thereby improving the recognition ability of various defects.

As shown in FIG. 11, the following table shows the effect of different defect sources on model performance for character defect classification model index comparison of different defect data sources. The simulation defect is used as training data, and compared with the defect of using only real data, the accuracy of the model can be obviously improved. This is because the defect morphology of the real data is more single, and the simulated defects can cover more scenes, thereby enhancing the generalization capability of the model.

In the technical scheme provided by the embodiment, a training process of an initial character defect classification model is provided, more rendered character images are generated based on sample images in the training process, and corroded and/or expanded to simulate irregularities in actual production, so that font images with different styles, shapes and appearances can be generated, the diversity of training data sets is enlarged, and the robustness, generalization capability and accuracy of the model can be improved, so that the model is better adapted to different font styles and changes.

Referring to fig. 12, fig. 12 is a schematic diagram of a character defect detecting device in a hardware running environment according to an embodiment of the present invention.

As shown in fig. 12, the character defect detecting apparatus may include: a processor 1001, such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, a memory 1005. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a WIreless interface (e.g., a WIreless-FIdelity (WI-FI) interface). The Memory 1005 may be a high-speed random access Memory (Random Access Memory, RAM) Memory or a stable nonvolatile Memory (NVM), such as a disk Memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.

It will be appreciated by those skilled in the art that the structure shown in fig. 12 does not constitute a limitation of the character defect detection apparatus, and may include more or fewer components than shown, or may combine certain components, or may have a different arrangement of components.

As shown in fig. 12, an operating system, a data storage module, a network communication module, a user interface module, and a character defect detection program may be included in the memory 1005 as one type of storage medium.

In the character defect detecting apparatus shown in fig. 12, the network interface 1004 is mainly used for data communication with other apparatuses; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the character defect detecting apparatus of the present invention may be provided in a character defect detecting apparatus which calls a character defect detecting program stored in the memory 1005 through the processor 1001 and performs the character defect detecting method provided by the embodiment of the present invention.

An embodiment of the present invention provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of any of the embodiments of the character defect detection method described above.

Since the embodiments of the computer readable storage medium portion and the embodiments of the method portion correspond to each other, the embodiments of the computer readable storage medium portion are referred to the description of the embodiments of the method portion, and are not repeated herein.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.

The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.

From the above description of embodiments, it will be clear to a person skilled in the art that the above embodiment method may be implemented by means of software plus a necessary general hardware platform, but may of course also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) as described above, comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present invention.

The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.

Claims

1. A character defect detection method, characterized in that the character defect detection method comprises the steps of:

Calculating a loss function according to the character defect type and the labeling information, and adjusting the character defect classification model according to the loss function;

if the image size of the binarized image does not meet the standard image size, pigment filling is carried out on the edge of the binarized image according to the standard image size, and a new character image is generated;

Determining the character defect type corresponding to each character image through a character defect classification model;

2. The character defect detection method of claim 1, wherein the step of determining the type of the character defect corresponding to each of the character images through the character defect classification model includes:

3. The character defect detection method of claim 1, wherein the step of determining the type of the character defect corresponding to each of the character images through the character defect classification model includes:

4. The character defect detection method of claim 1, wherein the step of preprocessing the irregular character image to generate a corresponding training character image includes:

5. The method for detecting character defects according to claim 4, wherein the step of summing the character masking image, the character ink-missing image, the character breaking image, the character reprinting image, and the character morphing image, which satisfy the specification requirements, includes, after the step of obtaining the training character image:

6. A character defect detecting apparatus, characterized by comprising: a memory, a processor, and a character defect detection program stored on the memory and executable on the processor, the character defect detection program configured to implement the steps of the character defect detection method of any one of claims 1 to 5.

7. A computer-readable storage medium, wherein a character defect detection program is stored on the computer-readable storage medium, which when executed by a processor, implements the steps of the character defect detection method according to any one of claims 1 to 5.