CN116433561A - Image processing method, electronic device, storage medium, and program product - Google Patents
Image processing method, electronic device, storage medium, and program product Download PDFInfo
- Publication number
- CN116433561A CN116433561A CN202111670966.2A CN202111670966A CN116433561A CN 116433561 A CN116433561 A CN 116433561A CN 202111670966 A CN202111670966 A CN 202111670966A CN 116433561 A CN116433561 A CN 116433561A
- Authority
- CN
- China
- Prior art keywords
- uniform
- uniformity
- pixel
- image
- pixels
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 42
- 239000013598 vector Substances 0.000 claims abstract description 146
- 238000012549 training Methods 0.000 claims abstract description 49
- 238000010586 diagram Methods 0.000 claims abstract description 42
- 238000004590 computer program Methods 0.000 claims abstract description 25
- 238000000605 extraction Methods 0.000 claims description 45
- 238000000034 method Methods 0.000 claims description 44
- 238000012545 processing Methods 0.000 claims description 20
- 230000002776 aggregation Effects 0.000 claims description 7
- 238000004220 aggregation Methods 0.000 claims description 7
- 238000005070 sampling Methods 0.000 claims description 4
- 230000000007 visual effect Effects 0.000 abstract description 11
- 238000013527 convolutional neural network Methods 0.000 description 15
- 230000008569 process Effects 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 6
- 238000005259 measurement Methods 0.000 description 6
- 238000010801 machine learning Methods 0.000 description 4
- 241000208125 Nicotiana Species 0.000 description 3
- 235000002637 Nicotiana tabacum Nutrition 0.000 description 3
- 239000003086 colorant Substances 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000000691 measurement method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 239000004753 textile Substances 0.000 description 1
- 230000007723 transport mechanism Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
Abstract
The invention provides an image processing method, electronic equipment, a computer readable storage medium and a computer program product, wherein the image processing method is used for outputting a uniformity thermodynamic diagram capable of accurately reflecting the uniformity of pixels in an image by extracting the characteristic vectors of the pixels in the image to be processed and measuring the uniformity of the vectors based on a uniform pixel characteristic vector set obtained by training in advance, so that a user can conveniently identify and optimize a non-uniform area, and visual characteristics are not required to be set manually in advance, so that the image processing method has wider application fields.
Description
Technical Field
The present invention relates to the field of artificial intelligence, and in particular, to an image processing method, an electronic device, a computer readable storage medium, and a computer program product.
Background
Computer vision is an important branch in the field of artificial intelligence, and refers to that a computer can replace human eyes and brains to perform visual recognition on specific targets, for example, the color uniformity, brightness uniformity or texture uniformity of a product is identified by the computer in industry, so as to detect the quality or flaws of the product.
However, in the related art, when the computer performs visual recognition and judgment, it is required to rely on the visual features designed manually in advance to perform comparison, so that universality of different application scenes is not strong, and at the same time, after recognition and comparison, the computer can only output a single judgment result, for example, whether a product has a flaw or not, but the specific position of the flaw of the product cannot be displayed, so that a user is difficult to optimize the flaw of the product, and the use feeling of the user is poor.
Disclosure of Invention
The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.
The embodiment of the invention provides an image processing method, electronic equipment, a computer readable storage medium and a computer program product, which can identify and mark target pixels of an image and improve the image processing capability.
In a first aspect, an embodiment of the present invention provides an image processing method, including: acquiring an image to be processed; acquiring a characteristic vector of a pixel in the image to be processed; acquiring a uniform pixel characteristic vector set obtained by pre-training; obtaining the vector uniformity of the pixels according to the uniform pixel feature vector set and the feature vector of the pixels; and obtaining a uniformity thermodynamic diagram of the image to be processed according to the vector uniformity of the pixels.
In a second aspect, an embodiment of the present invention further provides an electronic device, including: a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the image processing method as described above when executing the computer program.
In a third aspect, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for performing the image processing method as above.
In a fourth aspect, embodiments of the present invention further provide a computer program product comprising a computer program or computer instructions, characterized in that the computer program or the computer instructions are stored in a computer readable storage medium, from which the computer program or the computer instructions are read by a processor of a computer device, the processor executing the computer program or the computer instructions, causing the computer device to perform the image processing method as described above.
The embodiment of the invention comprises the following steps: obtaining feature vectors of pixels in the image to be processed and the image to be processed, obtaining a uniform pixel feature vector set obtained through training in advance, obtaining the vector uniformity of the pixels according to the uniform pixel feature vector set and the feature vectors of the pixels, and finally obtaining a uniformity thermodynamic diagram of the image to be processed according to the vector uniformity of the pixels. According to the scheme provided by the embodiment of the invention, the characteristic vector of the pixel in the image to be processed is extracted, and the vector uniformity is measured based on the uniform pixel characteristic vector set obtained by training in advance, so that the uniformity thermodynamic diagram capable of accurately reflecting the pixel uniformity in the image is output, the non-uniform area can be conveniently identified and optimized by a user, and the visual characteristics are not required to be set manually in advance, so that the image processing method has wider application scenes.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate and do not limit the invention.
FIG. 1 is a flow chart of an image processing method provided by an embodiment of the present invention;
FIG. 2 is a flow chart of one particular method of step S400 of FIG. 1;
FIG. 3 is a flow chart of an image processing method provided by an embodiment of the present invention;
FIG. 4 is a flow chart of an image feature extraction model training process provided by one embodiment of the present invention;
FIG. 5 is a schematic diagram of an image feature extraction model feature extraction network according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a target area provided by one embodiment of the present invention;
FIG. 7 is a flow chart of an image processing method provided by an embodiment of the present invention;
FIG. 8 is a schematic diagram showing the effect of an image processing method according to an embodiment of the present invention;
fig. 9 is a schematic diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
It should be noted that although a logical order is illustrated in the flowchart, in some cases, the steps illustrated or described may be performed in an order different from that in the flowchart. The terms first, second and the like in the description and in the claims and in the above-described figures, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
The identification and processing of the images have important practical significance in industrial machine vision, for example, in the tobacco industry, the tobacco leaves are subjected to grade discrimination according to the color uniformity of the tobacco leaves, namely the color uniformity is identified; in the quality inspection process of automobiles and electronic parts, product flaw detection, namely brightness uniformity identification, is carried out according to the brightness uniformity of the image; in the textile industry, the identification of the texture uniformity is performed by judging the potential cloth flaw area according to the texture uniformity of the image.
However, in the related art, the specific area in the image needs to be identified according to specific visual features, such as color, brightness or texture, and most of these visual features need to be designed manually in advance, for example, color uniformity is determined based on the pixel statistics of color models such as Lab, HSV, etc.; and extracting texture features by using texture feature descriptors such as a gray level co-occurrence matrix, LBP and the like, and judging texture uniformity according to the feature statistical characteristics. The image recognition method for carrying out the non-uniform area by manually predefining the visual characteristics has low universality when carrying out product quality or flaw detection, and can not display non-uniform parts in the image at the pixel level, so that some tiny flaws can be difficult to observe by naked eyes, and inconvenience is brought to actual use of users.
The invention provides an image processing method, electronic equipment, a computer readable storage medium and a computer program product, which are used for acquiring feature vectors of an image to be processed and pixels in the image to be processed, acquiring a uniform pixel feature vector set obtained through training in advance, then obtaining the vector uniformity of the pixels according to the uniform pixel feature vector set and the feature vectors of the pixels, and finally obtaining a uniformity thermodynamic diagram of the image to be processed according to the vector uniformity of the pixels. According to the scheme provided by the embodiment of the invention, the characteristic vector of the pixel in the image to be processed is extracted, and the vector uniformity is measured based on the uniform pixel characteristic vector set obtained by training in advance, so that the uniformity thermodynamic diagram capable of accurately reflecting the pixel uniformity in the image is output, the non-uniform area can be conveniently identified and optimized by a user, and the visual characteristics are not required to be set manually in advance, so that the image processing method has wider application scenes.
Embodiments of the present invention will be further described below with reference to the accompanying drawings.
As shown in fig. 1, fig. 1 is a flowchart of an image processing method according to an embodiment of the present invention, where the image processing method may be applied to a terminal, for example, an electronic device such as a mobile phone, a computer, a camera, and the like. In the example of fig. 1, the image processing method may include, but is not limited to, steps S100 to S500.
Step S100: and acquiring an image to be processed.
It should be noted that, the spatial resolution of any picture to be processed is not limited, that is, no matter whether the picture to be processed is a high-resolution high-definition image or a low-resolution blurred image, the implementation of the image processing method in the embodiment of the present application is not affected.
Step S200: and obtaining the characteristic vector of the pixel in the image to be processed.
It should be noted that, in order to obtain the feature vector of the pixel in the image to be processed, the image to be processed may be input into the image feature extraction model, and the output of the model is the feature vector of the pixel in the image to be processed. Based on different scenes or requirements, the method can set up to perform feature extraction on the complete image to be processed, and can also perform feature extraction on the local image to be processed, and the local image feature extraction can reduce the feature extraction time, so that the feature extraction efficiency is improved.
It should be further noted that, in some application scenarios, various typical non-uniform images and corresponding feature vectors thereof may be obtained in advance by a machine learning model or a manual processing manner, so that when an image to be processed is obtained, the image to be processed may be classified or judged by a person or a model to obtain the corresponding feature vectors thereof. The feature vector acquisition mode is simpler and faster, and is particularly suitable for application scenes which do not support running the image feature extraction model at any time.
Step S300: and obtaining a uniform pixel characteristic vector set obtained through pre-training.
The uniform pixel feature vector set is a pixel feature vector set outputted by a model after training an image feature extraction model in machine learning, and may be obtained by training in advance. The training samples used to train the model may be full images or partial images, and the resolution of the training samples may or may not be the same.
It should be noted that, uniformity of the image may be in various aspects, such as color uniformity, brightness uniformity, texture uniformity, etc. in the image, and the image processing method provided in this embodiment can implement image uniformity recognition under various conditions and various types, and is further described below using three examples.
Example one:
in order to obtain an image feature extraction model capable of extracting color features in an image, firstly, a plurality of complete images with uniform colors or partial image blocks of the complete images are used as training samples, a pre-trained ResNet50 convolutional neural network model on an image Net large-scale data set is trained, at this time, the ResNet50 is an image feature extraction model, feature extraction can be carried out on pixel features with uniform colors through the model, and further, the feature extraction can be represented through a uniform pixel feature vector set.
Example two:
in order to obtain an image feature extraction model capable of extracting texture features in an image, a plurality of complete images or partial image blocks of the complete images with uniform textures are used as training samples, and a pre-trained DenseNet convolutional neural network model is trained, at this time, the DenseNet is an image feature extraction model, and pixel features with uniform textures can be extracted through the model, and further represented through a uniform pixel feature vector set.
Example three:
in order to obtain an image feature extraction model capable of extracting brightness features in an image, a plurality of complete images or partial image blocks of the complete images with uniform brightness are adopted as training samples, other pretrained convolutional neural network models are subjected to deep learning, feature extraction can be performed on pixel features with brightness textures through the model, and further the pixel features are represented through a uniform pixel feature vector set.
It will be appreciated by those skilled in the art that the above two embodiments use a deep convolutional neural network model as the image feature extraction model because it performs well in terms of image recognition, but other machine learning models, such as a deep convolutional neural network, may also be applied to the image processing method provided in the embodiments of the present application.
It should be noted that the uniform pixel feature vector set may be a set of image pixel feature vectors directly extracted after model training, or may be a set of image pixel features with a smaller number obtained after clustering of image pixel features extracted after model training.
In an embodiment, to improve the characterizability of the uniform pixel feature vectors, feature aggregation is performed on the output features of the output layer of the image feature extraction model.
Step S400: and obtaining the vector uniformity of the pixels according to the uniform pixel characteristic vector set and the characteristic vector of the pixels.
It should be noted that, there are various methods for measuring the similarity between features, which mainly depend on the calculated relationship between feature vectors. For the measurement of the feature similarity, the smaller the feature similarity, the larger the difference between the two feature vectors is, the larger the difference between the image pixels is, otherwise, the larger the feature similarity is, the smaller the difference between the two feature vectors is, and the smaller the difference between the image pixels is. For example, in the case of identifying and processing whether the color, brightness, texture, etc. on the image are uniform, the similarity between the feature vector of the pixel on the image to be processed and the feature vector in the uniform pixel feature vector set is based on the similarity comparison, and when the similarity is high, it is indicated that the pixel on the image to be processed is similar in feature to the uniform pixel, and therefore, the pixel may be identified as the pixel having uniform color, brightness, texture.
In one embodiment, the image processing method provides a measurement method for obtaining pixel uniformity by calculating cosine uniformity, as shown in fig. 2, and specifically includes the following steps:
referring to fig. 2, in an embodiment, further describing step S400, step S400 may include, but is not limited to, the following steps:
step S411: and calculating to obtain a vector uniformity set of the pixels according to the feature vectors of the pixels and each uniform pixel feature vector in the uniform pixel feature vector set, wherein the vector uniformity set comprises vector uniformity elements.
Step S412: and determining the vector uniformity element with the largest value in the vector uniformity set of the pixels as the vector uniformity of the pixels.
In one embodiment, the vector uniformity of the pixels may be cosine uniformity, assuming that the eigenvector of the pixel p in the image to be processed is v p The uniform pixel characteristic vector set obtained according to training is { v 0 ,v 1 ,L,v N The cosine uniformity of pixel p can be calculated according to the following formula:
in the above formula (1), s p Representing cosine uniformity, v q Represents the uniform pixel feature vectors in the uniform pixel feature vector set, and N represents the number of vectors in the uniform pixel feature vector set.
It should be noted that the above calculation process is only cosine uniformity calculation performed on one pixel in the image to be processed, and in practical application of the image processing method, the above calculation is performed on a plurality of pixels, even all pixels, in the image to be processed. The cosine value between the pixel p and each feature vector in the uniform pixel feature vector set is calculated, and the maximum cosine similarity is selected as the cosine similarity of the pixel p, so that the feature vector which can represent the pixel p most can be obtained. It will be appreciated that cosine uniformity is only one representation of the distance measure between the two vectors, and that in practical applications further calculations of cosine uniformity may be required to obtain the actual uniformity value for the pixels in the uniformity thermodynamic diagram.
Step S500: and obtaining a uniformity thermodynamic diagram of the image to be processed according to the vector uniformity of the pixels.
It should be noted that, the uniformity thermodynamic diagram is a diagram which can reflect the uniformity of pixels in an image in units of pixels, and the color shade of each pixel in the diagram can reflect the uniformity of the pixels in the image to be processed corresponding to the color shade. The cosine uniformity is adopted to calculate the uniformity of the pixels, and after the uniformity of the pixels is obtained, the display mode of the thermodynamic diagram can be manually set according to actual requirements. For example, if it is desired to represent the non-uniform region with a dark color and the uniform region with a light color, for the cosine uniformity metric method, dark color for pixels with small cosine uniformity values and light color for pixels with large cosine uniformity values are required. The pixel depth is set according to the user's requirement, and the difference between the pixel depths is the uniformity value obtained by measuring the pixel depths by using different uniformity methods.
It should be noted that the pixel uniformity thermodynamic diagram may be a global pixel thermodynamic diagram of the image to be processed, or may be a local pixel uniformity thermodynamic diagram. The output pixel uniformity thermodynamic diagram can help a user to better observe the distribution of the target pixels.
It should be noted that, the uniformity thermodynamic diagram of a pixel is a generic term of a thermodynamic diagram representing uniformity of an image, and in practical applications, a non-uniformity thermodynamic diagram may be used to represent uniformity of an image, where the generation method and principle are the same as that of the uniformity thermodynamic diagram.
In the embodiments described below, the uniformity thermodynamic diagram is further identified to provide a more prominent display of non-uniform pixels. As shown in fig. 3, at least the following steps are included:
step S100: and acquiring an image to be processed.
Step S200: and obtaining the characteristic vector of the pixel in the image to be processed.
Step S300: and obtaining a uniform pixel characteristic vector set obtained through pre-training.
Step S410: and obtaining cosine uniformity of the pixel according to the uniform pixel characteristic vector set and the characteristic vector of the pixel.
In this embodiment, cosine uniformity is used to measure the uniformity of the pixel.
Step S510: and obtaining a uniformity thermodynamic diagram of the image to be processed according to the cosine uniformity of the pixels.
Step S600: and determining non-uniform pixels in the pixels according to the uniformity thermodynamic diagram and preset uniformity judging conditions. In this embodiment, when the uniformity measurement is performed using cosine similarity, the non-uniformity determination condition is whether cosine uniformity corresponding to a pixel is smaller than a first preset pixel uniformity threshold, specifically, by comparing the computed cosine uniformity of a plurality of pixels with the first preset pixel uniformity threshold, for each pixel, if the remaining chord uniformity is greater than the first preset pixel uniformity threshold, the pixel is a uniform pixel; if the remaining chord uniformity is less than the first predetermined pixel uniformity threshold, the pixel is a non-uniform pixel.
In this embodiment, whether the uniformity of each pixel is smaller than the preset pixel uniformity threshold is determined to identify abnormal pixels, i.e., to identify non-uniform pixels.
Step S700: and carrying out identification processing on the non-uniform pixels. There are various methods for the identification processing of the non-uniform pixels, and any method that can distinguish between uniform pixels and non-uniform pixels belongs to the identification processing of non-uniform pixels.
It will be appreciated that the identification of non-uniform pixels is to highlight non-uniform pixels on the image to be processed, such as by circling or highlighting the outline of non-uniform pixels. It should be noted that the pixel uniformity thermodynamic diagram may be a global pixel thermodynamic diagram of the image to be processed, or may be a local pixel thermodynamic diagram. The output pixel thermodynamic diagram can help a user to better observe the distribution of the target pixels.
According to the image processing method, the image characteristics of the image to be processed are extracted, and the non-uniform pixels which do not meet the characteristic requirements in the image to be processed can be identified by comparing the extracted image characteristics with the uniform pixel characteristic vectors obtained through pre-training, so that visual characteristics are not required to be set manually in advance, and the image processing method has wider application scenes; meanwhile, the identified non-uniform pixels can be identified, so that a user can conveniently identify and optimize the non-uniform pixels, and the image processing capability is comprehensively improved.
As shown in fig. 4, fig. 4 is a flowchart of an image feature extraction model training process according to an embodiment of the present invention, which at least includes the following steps:
step S310: and obtaining a training sample, wherein the training sample is a uniform picture.
It should be noted that, the training samples may be some complete images or may be a partial image block of the images, and at the same time, the spatial resolutions of the different training sample images may not be uniform. In the model training process, the same batch of samples can be repeatedly input, or the same batch of samples can be batched, and samples in different batches can be input in each iteration.
Step S320: inputting the training sample into an image feature extraction model to obtain a sample uniform pixel feature vector set; the image feature extraction model is a convolutional neural network model.
In one embodiment, a convolutional neural network is used as an image feature extraction model, which has a strong feature extraction capability compared with other machine learning models. Features of image distribution pixels are extracted using a convolutional neural network model pre-trained on a large-scale dataset, imageNet, COCO, etc. The feature extraction method without supervision learning can avoid manual data tagging, so that the workload of manual labeling can be greatly reduced, and a pre-training model on any large-scale data set can be used for the image uniformity recognition scheme provided by the embodiment of the invention.
It should be noted that, in the image feature extraction process, feature dimensions are very high, in order to improve the characterization capability of depth features, feature receptive fields need to be increased, output features of different convolution layers are aggregated, and a convolution neural network topology diagram adopting feature aggregation is shown in fig. 5.
In this embodiment, the image to be processed is input to the input layer 101, the output layers 102 to 105 are the function of feature extraction in the convolutional neural network, and feature aggregation is performed by adopting the middle three output layers of the convolutional neural network, wherein assuming that the output layer 3 104 is the j-th output layer, the output layer 2103 is the j-1 layer, and the output layer 4 105 is the j+1 layer.
Sample x i Through the process ofThe feature map output by the j-th layer after the reasoning of the pre-training convolutional neural network model is expressed as follows:
in the above formula (2), C j 、H j 、W j The number, the height and the width phi of the channels respectively representing the characteristic diagram ij Representing the output of sample i at layer j.
In this embodiment, feature aggregation is performed using the middle three-layer output of the pre-training model, and the aggregate features are expressed as:
F(x i )＝f agg ({φ ij-1 ,φ ij ,φ ij+1 }),j∈{2} (3)
in the above formula (3), f agg (. Cndot.) is an aggregate operator that first references the output features phi of output layer 3 104 ij Up-sampling 2x is performed on the output characteristics phi of the output layer 4 105 ij+1 Up-sampling 4x and then comparing the interpolated feature map with the feature map phi of the output layer 2103 ij-1 Channel combination is carried out, so that feature aggregation of three different layers is completed; and finally, up-sampling the aggregation characteristics to ensure that the output spatial resolution is consistent with the input image resolution.
Presuming training sample x i Is of spatial resolution H i ×W i Depth features extracted by pre-training convolutional neural networks are expressed as:
F i ＝f(x i ) (4)
in the above formula (4), F i Representing depth feature representation extracted by training samples through a pre-training convolutional neural network, wherein f (·) represents a feature extraction operator, and the number of output channels of the feature extraction operator f (·) is C f The space dimension is H i ×W i The feature space dimension is consistent with the spatial resolution of the input picture, i.e. each pixel can be saved with one C f Feature vector of dimension is expressed, and after feature extraction, sample x i Can obtain H i ×W i C of uniform pixels f The dimension depth feature vector, the feature vector for each pixel is expressed as:
in the above formula (5), v k A depth feature vector representing a pixel,
representing the dimension of the depth feature vector of the pixel.
Through the feature extraction, feature vectors of a large number of pixels can be obtained, and assuming that the number of pixels is M, that is, the dimension of the preset target clustering feature vector is M, the sample uniform pixel feature vector set can be expressed as:
Ω＝{v 0 ,v 1 ,…,v M } (6)
in the above formula (6), Ω represents a sample uniform pixel feature vector set, v M Representing a sample uniform pixel feature vector.
Step S330: and clustering elements in the sample uniform pixel feature vector set to obtain the uniform pixel feature vector set.
In general, as more training samples exist, the image resolution of the training samples is larger, the number of pixels is large, and in order to facilitate the subsequent pixel-level uniformity measurement, the feature sets are required to be clustered, so that the number of the feature vector pixels is reduced.
In this embodiment, the kmeans++ clustering algorithm performs feature set clustering, and assuming that the number of uniform pixels of the final clustering is N (N < < M), the final obtained pixel feature vector set may be expressed as:
Ω′＝{v 0 ,v 1 ,…,v N } (7)
in the above formula (7), Ω' represents a uniform set of pixel feature vectors after clustering.
As shown in fig. 6, a schematic representation of the target area is provided, each square grid representing a pixel, in this embodiment, a uniform texture pixel is identified and labeled. It can be seen that the pixels A, B, C, D, E, F are non-uniform texture pixels, wherein the pixels B, C, D, E, F are adjacent to each other, forming a pixel area. In some application scenarios, the user may not care about some non-uniform pixel areas with smaller areas, so that only those pixel areas with areas exceeding a preset area can be identified by setting the areas of the pixel areas in advance, and the pixel areas meeting the identification conditions are defined as target areas. In this embodiment, the target area is marked by using a contour line, so that the target area is highlighted, and the user can observe the target area conveniently.
In another embodiment, the number of the target areas and the number of the pixels contained in each target area can be output and displayed, so that a user can know the image condition more accurately, and the user can optimize the image and even the product conveniently.
Since the image processing method provided in the present application may be implemented by a pre-configured image processing model, in the following, in one embodiment, a specific flow of implementing the image processing method provided in the present application by using an image processing model with a cosine structure is described, as shown in fig. 7, and specifically includes the following steps:
step S810: and loading an image processing model, wherein the image processing model comprises a trained image feature extraction model and a feature vector set of uniform pixels.
It can be understood that the feature vector set of the uniform pixels is generally stored through a disk to form a model file, and is loaded when needed; the image feature extraction is trained by inputting a plurality of sample images with uniform colors, uniform textures or uniform brightness in advance, and the trained image feature extraction model can extract features of the input image to be processed. The image feature extraction model may exist as a functional module in the image processing model.
In this embodiment, 4 uniform pictures are selected as training samples, and the resolution of the images is 1080x1920, and it is noted that the uniform sample in this embodiment may be a local image block of the uniform picture, and the sizes of different training pictures may be inconsistent.
Step S820: video frames are input to an image processing model.
In this embodiment, offline video is loaded from disk and video frames are read as input source, i.e. image to be processed.
It is worth noting that the network video stream may also be read as an input source by rtmp or rtsp.
Step S830: and extracting feature vectors of pixels in the image to be processed by the image feature extraction model through feature vector extraction of the input video frame.
In this embodiment, any pretrained model on a large-scale dataset may be used for the uniform pixel feature extraction of the present invention, and in this embodiment, the pretrained ResNet50 convolutional neural network model on an ImageNet dataset is selected to extract features of uniformly distributed pixels, and obtain feature vectors of the pixels.
Sample x in this embodiment i Is 1080x1920, and depth features extracted by pre-training convolutional neural networks are expressed as:
F i ＝f(x i )
the number of output channels of the depth feature extraction operator f (·) is 3584, the spatial dimension is 1080x1920, and the feature spatial dimension is consistent with the spatial resolution preservation of the input picture, i.e. each pixel can be represented by a feature vector of 3584 dimensions. The feature vector of m= 8294400 uniform pixels can be obtained by 4 uniform sample pictures of 1080x1920, the dimension of each feature vector is 3584, and the sample uniform pixel feature vector set can be expressed as:
Ω＝{v 0 ,v 1 ,…,v M }
in this embodiment, in order to reduce the calculation amount of the uniformity of the subsequent feature, the feature set is clustered, and the order of magnitude of the feature vector pixels is reduced. The eigenvector set clustering is carried out by using a kmeans++ clustering algorithm, the number of the finally clustered uniform pixels is N=100, and the finally obtained clustered sample uniform pixel eigenvector set can be expressed as follows:
Ω′＝{v 0 ,v 1 ,…,v N }
similar to the uniform sample learning process, in this embodiment, the spatial resolution of the video frame y is assumed to be 1080×1920, and the depth feature F extracted by the pre-training model y Expressed as:
F y (i,j)＝f(y),(1≤i≤1080,1≤j≤1920)
wherein F is y Is 3584 x 1080x 1920.
Step S840: and carrying out uniformity calculation on the feature vector of each pixel in the video frame by using the image processing model.
According to the sample uniform pixel characteristic vector set omega' trained to each uniform pixel, the depth characteristic map F of the image to be processed y Performing measurement of feature uniformity, namely uniformity measurement, and obtaining a uniformity thermodynamic diagram of each pixel position:
S y (i,j),(1≤i≤1080,1≤j≤1920)
S y the value range of (2) is [0,1 ]]0 indicates the worst uniformity and 1 indicates the best uniformity.
In this embodiment, the feature uniformity measurement adopts a vector cosine uniformity measurement method, and the pixel p to be measured and the depth feature vector v thereof are measured p According to the training, the obtained sample uniform pixel characteristic vector set { v } 0 ,v 1 ,L,v N The uniformity value of pixel p can be expressed as:
n=100 in this embodiment.
Step S850: displaying the corresponding pixel uniformity thermodynamic diagram of the video frame.
Step S860: and extracting non-uniform pixels with uniformity not reaching the standard, and circling the outline of the non-uniform pixels.
As shown in fig. 8, an effect schematic diagram of processing an image by using the image processing method according to the above embodiment is provided, it can be seen that a non-uniformity thermodynamic diagram corresponding to the image to be processed is output, in the non-uniformity thermodynamic diagram, 1 indicates that uniformity is the worst, 0 indicates that uniformity is the best, that is, the darker the color is, the more uneven the description is, the lighter the color is, the more uniform the user can easily see non-uniform pixels or regions in the image, which is intuitive and effective.
In other embodiments, the non-uniform region profile may be superimposed in the uniformity thermodynamic diagram and the number of non-uniform regions displayed for more obvious identification.
According to the image processing method, the image characteristics of the image to be processed are extracted, and the non-uniform pixels which do not meet the characteristic requirements in the image to be processed can be identified by comparing the extracted image characteristics with the uniform pixel characteristic vectors obtained through pre-training, so that visual characteristics are not required to be set manually in advance, and the image processing method has wider application scenes; meanwhile, the identified non-uniform pixels can be identified, so that a user can conveniently identify and optimize the non-uniform pixels, and the image processing capability is comprehensively improved.
In addition, an embodiment of the present invention further provides an electronic device 110, where the electronic device 110 includes: the processor 111, the memory 112 and a computer program stored on the memory and executable on the processor, the processor being capable of executing the computer program to carry out the image processing method as described above.
The memory, as a non-transitory computer readable storage medium, may be used to store non-transitory software programs as well as non-transitory computer executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory optionally includes memory remotely located relative to the processor, the remote memory being connectable to the processor through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The non-transitory software programs and instructions required to implement the image processing methods of the above embodiments are stored in the memory, and when executed by the processor, the image processing methods of the above embodiments are performed, for example, the method steps S100 to S500 in fig. 1, the method steps S411 to S412 in fig. 2, the method steps S100 to S700 in fig. 3, the method steps S310 to S330 in fig. 4, and the method steps S810 to S860 in fig. 7 described above are performed.
The network element embodiments described above are merely illustrative, in that the elements illustrated as separate components may or may not be physically separate, i.e., may be located in one place, or may be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
Furthermore, an embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions that are executed by a processor or a controller, for example, by one of the processors in the network element embodiment, and may cause the processor to perform the image processing method in the embodiment, for example, perform the method steps S100 to S500 in fig. 1, the method steps S411 to S412 in fig. 2, the method steps S100 to S700 in fig. 3, the method steps S310 to S330 in fig. 4, and the method steps S810 to S860 in fig. 7 described above.
Furthermore, an embodiment of the present invention provides a computer program product including a computer program or computer instructions, characterized in that the computer program or the computer instructions are stored in a computer-readable storage medium, from which a processor of a computer device reads the computer program or the computer instructions, the processor executing the computer program or the computer instructions, causes the computer device to perform the image processing method as described above, for example, to perform the method steps S100 to S500 in fig. 1, the method steps S411 to S412 in fig. 2, the method steps S100 to S700 in fig. 3, the method steps S310 to S330 in fig. 4, and the method steps S810 to S860 in fig. 7 described above.
Those of ordinary skill in the art will appreciate that all or some of the steps, systems, and methods disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as known to those skilled in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Furthermore, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
While the preferred embodiment of the present invention has been described in detail, the present invention is not limited to the above embodiment, and various equivalent modifications and substitutions can be made by those skilled in the art without departing from the spirit of the present invention, and these equivalent modifications and substitutions are intended to be included in the scope of the present invention as defined in the appended claims.
Claims (13)
1. An image processing method, comprising:
acquiring an image to be processed;
acquiring a characteristic vector of a pixel in the image to be processed;
acquiring a uniform pixel characteristic vector set obtained by pre-training;
obtaining the vector uniformity of the pixels according to the uniform pixel feature vector set and the feature vector of the pixels;
and obtaining a uniformity thermodynamic diagram of the image to be processed according to the vector uniformity of the pixels.
2. The method according to claim 1, wherein the method further comprises:
determining non-uniform pixels in the pixels according to the uniformity thermodynamic diagram and a preset non-uniformity judging condition;
and carrying out identification processing on the non-uniform pixels.
3. The method of claim 1, wherein the obtaining the vector uniformity of the pixel from the uniform set of pixel feature vectors and the feature vector of the pixel comprises:
according to the feature vector of the pixel and each uniform pixel feature vector in the uniform pixel feature vector set, calculating to obtain a vector uniformity set of the pixel, wherein the vector uniformity set comprises vector uniformity elements;
and determining the vector uniformity element with the largest numerical value in the vector uniformity set of the pixel as the vector uniformity of the pixel.
4. The method according to claim 2, wherein the non-uniformity determination condition includes: the vector uniformity corresponding to the non-uniform pixels is smaller than a preset pixel uniformity threshold;
the determining non-uniform pixels in the pixels according to the uniformity thermodynamic diagram and a preset non-uniformity judging condition comprises:
comparing the vector uniformity of the pixel with the preset pixel uniformity threshold;
determining vector uniformity with a value smaller than the preset pixel uniformity threshold as target vector uniformity;
and determining the pixels corresponding to the uniformity of the target vector as non-uniform pixels.
5. The method of claim 2, wherein the identifying the non-uniform pixels comprises:
judging whether a non-uniform area exists or not, wherein the non-uniform area comprises non-uniform pixels with the number larger than a preset pixel number threshold;
and when judging that the non-uniform area exists, carrying out identification processing on the non-uniform area.
6. The method of claim 5, wherein the identifying the non-uniform region comprises:
combining the non-uniform pixels to obtain a non-uniform region;
highlighting the contour of the non-uniform region on the image to be processed.
7. The method according to any one of claims 2 to 4, wherein said identifying said non-uniform pixels comprises:
highlighting the non-uniform pixels on the image to be processed.
8. The method of claim 1, wherein the uniform set of pixel feature vectors is derived by:
obtaining a training sample, wherein the training sample is a uniform image sample;
inputting the training sample into the image feature extraction model to obtain a sample uniform pixel feature vector set;
and clustering elements in the sample uniform pixel feature vector set to obtain the uniform pixel feature vector set.
9. The method of claim 8, wherein the image feature extraction model comprises a first output layer, a second output layer, and a third output layer;
inputting the training sample to the image feature extraction model to obtain a sample uniform pixel feature vector set, wherein the method comprises the following steps:
inputting the training sample into the image feature extraction model to obtain a first feature map output by the first output layer, a second feature map output by the second output layer and a third feature map output by the third output layer;
upsampling the second feature map to obtain a second upsampled feature map;
upsampling the third feature map to obtain a third upsampled feature map
Channel combination is carried out on the first feature map, the second upsampling feature map and the third upsampling feature map, so that an aggregation feature map is obtained;
up-sampling the aggregated feature map according to the image resolution of the training sample to obtain an aggregated feature vector set;
and determining the aggregated feature vector set as the sample uniform pixel feature vector set.
10. The method of claim 8, wherein clustering the elements in the sample uniform pixel feature vector set to obtain the uniform pixel feature vector set comprises:
according to the preset dimension of the clustering feature vector, carrying out feature clustering on the elements in the sample uniform pixel feature vector set to obtain a clustered feature vector set;
and determining the clustered feature vector set as the sample uniform pixel feature vector set.
11. An electronic device, comprising: memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the image processing method according to any one of claims 1 to 10 when executing the computer program.
12. A computer-readable storage medium storing computer-executable instructions for performing the image processing method of any one of claims 1 to 10.
13. A computer program product comprising a computer program or computer instructions, characterized in that the computer program or the computer instructions are stored in a computer readable storage medium, from which the computer program or the computer instructions are read by a processor of a computer device, which processor executes the computer program or the computer instructions, so that the computer device performs the image processing method according to any one of claims 1 to 10.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111670966.2A CN116433561A (en) | 2021-12-31 | 2021-12-31 | Image processing method, electronic device, storage medium, and program product |
| PCT/CN2022/142861 WO2023125671A1 (en) | 2021-12-31 | 2022-12-28 | Image processing method, electronic device, storage medium and program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111670966.2A CN116433561A (en) | 2021-12-31 | 2021-12-31 | Image processing method, electronic device, storage medium, and program product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116433561A true CN116433561A (en) | 2023-07-14 |
Family
ID=86998017
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111670966.2A Pending CN116433561A (en) | 2021-12-31 | 2021-12-31 | Image processing method, electronic device, storage medium, and program product |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN116433561A (en) |
| WO (1) | WO2023125671A1 (en) |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2678665A4 (en) * | 2011-02-24 | 2017-07-05 | 3M Innovative Properties Company | System for detection of non-uniformities in web-based materials |
| CN111598827A (en) * | 2019-02-19 | 2020-08-28 | 富泰华精密电子（郑州）有限公司 | Appearance flaw detection method, electronic device and storage medium |
| CN113205474A (en) * | 2020-01-15 | 2021-08-03 | 华为技术有限公司 | Screen detection and screen detection model training method, device and equipment |
| CN111461182B (en) * | 2020-03-18 | 2023-04-18 | 北京小米松果电子有限公司 | Image processing method, image processing apparatus, and storage medium |
| WO2021191908A1 (en) * | 2020-03-25 | 2021-09-30 | Yissum Research Development Company Of The Hebrew University Of Jerusalem Ltd. | Deep learning-based anomaly detection in images |
| CN113516735A (en) * | 2021-01-12 | 2021-10-19 | 腾讯科技（深圳）有限公司 | Image processing method, image processing device, computer readable medium and electronic equipment |
| CN113205176B (en) * | 2021-04-19 | 2022-09-06 | 重庆创通联达智能技术有限公司 | Method, device and equipment for training defect classification detection model and storage medium |
-
2021
- 2021-12-31 CN CN202111670966.2A patent/CN116433561A/en active Pending
-
2022
- 2022-12-28 WO PCT/CN2022/142861 patent/WO2023125671A1/en unknown
Also Published As
| Publication number | Publication date |
|---|---|
| WO2023125671A1 (en) | 2023-07-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112132156B (en) | Image saliency target detection method and system based on multi-depth feature fusion | |
| CN111723860A (en) | Target detection method and device | |
| Lei et al. | Scale insensitive and focus driven mobile screen defect detection in industry | |
| CN111091109B (en) | Method, system and equipment for predicting age and gender based on face image | |
| CN109410192B (en) | Fabric defect detection method and device based on multi-texture grading fusion | |
| CN111161222B (en) | Printing roller defect detection method based on visual saliency | |
| CN108961180B (en) | Infrared image enhancement method and system | |
| KR20200087297A (en) | Defect inspection method and apparatus using image segmentation based on artificial neural network | |
| CN110059666B (en) | Attention detection method and device | |
| CN109241867B (en) | Method and device for recognizing digital rock core image by adopting artificial intelligence algorithm | |
| CN110111347B (en) | Image sign extraction method, device and storage medium | |
| Samadzadegan et al. | Automatic detection and classification of damaged buildings, using high resolution satellite imagery and vector data | |
| CN112419270B (en) | No-reference image quality evaluation method and device under meta-learning and computer equipment | |
| CN114444565A (en) | Image tampering detection method, terminal device and storage medium | |
| CN113744142A (en) | Image restoration method, electronic device and storage medium | |
| CN113609984A (en) | Pointer instrument reading identification method and device and electronic equipment | |
| Wu et al. | Visual structural degradation based reduced-reference image quality assessment | |
| CN114841992A (en) | Defect detection method based on cyclic generation countermeasure network and structural similarity | |
| Salih et al. | Adaptive local exposure based region determination for non-uniform illumination and low contrast images | |
| CN112396564A (en) | Product packaging quality detection method and system based on deep learning | |
| CN116798041A (en) | Image recognition method and device and electronic equipment | |
| CN116862812A (en) | Infrared image processing method, device, computer equipment, storage medium and product | |
| CN116977895A (en) | Stain detection method and device for universal camera lens and computer equipment | |
| CN116433561A (en) | Image processing method, electronic device, storage medium, and program product | |
| Al-Otum | A novel set of image morphological operators using a modified vector distance measure with color pixel classification |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication |