CN112150414A - Target object detection method and device, electronic equipment and storage medium - Google Patents
Target object detection method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112150414A CN112150414A CN202010913275.XA CN202010913275A CN112150414A CN 112150414 A CN112150414 A CN 112150414A CN 202010913275 A CN202010913275 A CN 202010913275A CN 112150414 A CN112150414 A CN 112150414A
- Authority
- CN
- China
- Prior art keywords
- image
- network
- super
- training
- resolution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 50
- 238000003860 storage Methods 0.000 title claims abstract description 20
- 238000011156 evaluation Methods 0.000 claims abstract description 47
- 238000000034 method Methods 0.000 claims abstract description 37
- 238000000605 extraction Methods 0.000 claims abstract description 24
- 238000012549 training Methods 0.000 claims description 104
- 239000004831 Hot glue Substances 0.000 claims description 23
- 238000012163 sequencing technique Methods 0.000 claims description 23
- 230000008447 perception Effects 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 10
- 238000005286 illumination Methods 0.000 claims description 7
- 238000013441 quality evaluation Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 17
- 230000006870 function Effects 0.000 description 9
- 230000008569 process Effects 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 238000013461 design Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000005498 polishing Methods 0.000 description 3
- LAHWLEDBADHJGA-UHFFFAOYSA-N 1,2,4-trichloro-5-(2,5-dichlorophenyl)benzene Chemical compound ClC1=CC=C(Cl)C(C=2C(=CC(Cl)=C(Cl)C=2)Cl)=C1 LAHWLEDBADHJGA-UHFFFAOYSA-N 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 238000012856 packing Methods 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000011179 visual inspection Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
- G06T7/0006—Industrial image inspection using a design-rule based approach
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30141—Printed circuit board [PCB]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Image Analysis (AREA)
Abstract
The application discloses a target object detection method and device, electronic equipment and a storage medium. The method is used for solving the problem that due to the complex background of the object to be detected, the industrial camera cannot well acquire the image of the object to be detected, and the object to be detected cannot be accurately judged. In the embodiment of the application, an image of a target object is acquired by using an industrial camera with a single resolution, and the outer surface of the target object is provided with filaments with diameters smaller than a preset value; performing image enhancement operation on the image content of the filamentous object to obtain a super-resolution image; performing feature extraction on the super-resolution image to obtain an appointed evaluation index of the filament; and comparing the specified evaluation index of the target object with a preset standard index to obtain a detection result of the target object for the filiform object.
Description
Technical Field
The present disclosure relates to the field of information detection technologies, and in particular, to a method and an apparatus for detecting a target object, an electronic device, and a storage medium.
Background
The traditional industrial visual inspection process is to perform algorithm identification on images acquired by an industrial camera so as to meet the requirement of detecting an object to be detected. However, due to the complex background of the object to be measured, the industrial camera cannot acquire the image of the object to be measured well, and the object to be measured cannot be judged accurately. In response to this phenomenon, a high-resolution industrial camera or multiple industrial cameras are commonly used for joint acquisition in the related art, but this greatly increases the hardware cost.
Disclosure of Invention
The application aims to provide a target object detection method, a target object detection device, an electronic device and a storage medium, which are used for solving the following problems: the super-resolution image is provided for the target object to detect the target object.
In a first aspect, an embodiment of the present application provides a target object detection method, where the method includes:
acquiring an image of the target object, wherein the outer surface of the target object is provided with filaments with diameters smaller than a preset value;
performing image enhancement operation on the image content of the silk object in the image to obtain a super-resolution image;
performing feature extraction on the super-resolution image to obtain an appointed evaluation index of the filament;
and comparing the specified evaluation index of the target object with a preset standard index to obtain a detection result of the target object for the filiform object.
In one embodiment, said performing an image enhancement operation on image content of said filamentous object in said image comprises:
inputting the images into an evaluation and combining with super resolution to generate a confrontation network RankSRGAN, and obtaining the super resolution images subjected to image enhancement by the RankSRGAN.
In one embodiment, the ranksrnan includes an evaluation network Ranker and a super resolution countermeasure network srnan, and the method further includes:
training the RankSRGAN according to the following method:
constructing a first training sample set for training the Ranker network and a second training sample set for training the SRGAN network;
training the Ranker network by adopting the first training sample set so that the Ranker network outputs content sequencing loss rank-content loss;
inputting each second training sample in the second training sample set into a generator of the SRGAN network to obtain a super-resolution sample image output by the generator;
respectively inputting the super-resolution sample images into discriminators of the Ranker network and the SRGAN network to obtain a rank-content loss value comprising the Ranker network output;
and training the generator according to the loss value to obtain a RankSRGAN network for image enhancement.
In one embodiment, the constructing a first training sample set for training the Ranker network comprises:
acquiring a sample image set;
for each sample image in the sample image set, performing:
respectively generating SR images corresponding to the sample images by using a plurality of super-resolution SR methods; and forming an image pair by the sample image and each corresponding SR image;
respectively scoring each image pair by applying an image quality evaluation index NIQE to obtain a perception score of each image pair;
sequencing all the image pairs according to the perception scores of the image pairs to obtain sequencing serial numbers of the image pairs;
and constructing the first training sample set by taking the sequencing sequence number and the perception score of each image pair as the label of the corresponding image pair.
In one embodiment, the super-resolution image is subjected to feature extraction to obtain a specified evaluation index of the filament, and the method comprises the following steps:
and inputting the super-resolution image into a pre-trained index extraction model to obtain the specified evaluation index of the filament.
In one embodiment, the target object is a Printed Circuit Board (PCB), the filaments are hot melt adhesive filaments, and the specified evaluation index comprises at least one of the following: position, length, diameter, area.
In one embodiment, the sample image set includes sample images obtained by image acquisition of a sample object under multiple illumination conditions.
In a second aspect, an embodiment of the present application further provides a target object detection apparatus, including:
the image acquisition module is used for acquiring an image of the target object, and filaments with diameters smaller than a preset value are arranged on the outer surface of the target object;
the image enhancement module is used for carrying out image enhancement operation on the image content of the silk-like object in the image to obtain a super-resolution image;
the characteristic extraction module is used for extracting the characteristics of the super-resolution image to obtain the specified evaluation index of the silk object;
and the index comparison module is used for comparing the specified evaluation index of the target object with a preset standard index to obtain a detection result of the target object for the filiform object.
In one embodiment, the image enhancement module is to:
inputting the images into an evaluation and combining with super resolution to generate a confrontation network RankSRGAN, and obtaining the super resolution images subjected to image enhancement by the RankSRGAN.
In one embodiment, the ranksrnan includes an evaluation network Ranker and a super resolution countermeasure network srnan, and the apparatus further includes:
a training module comprising the following elements to train the RankSRGAN:
a training sample set constructing unit, configured to construct a first training sample set used for training the Ranker network and a second training sample set used for training the SRGAN network;
a Ranker network training unit, configured to train the Ranker network by using the first training sample set, so that the Ranker network outputs a content ranking loss rank-content loss;
a super-resolution sample image output unit, configured to input each second training sample in the second training sample set to a generator of the SRGAN network, so as to obtain a super-resolution sample image output by the generator;
a loss value obtaining unit, configured to input the super-resolution sample image to the Ranker network and the discriminator of the SRGAN network, respectively, to obtain a loss value including rank-content loss output by the Ranker network;
and the generator training unit is used for training the generator according to the loss value to obtain a RankSRGAN network for image enhancement.
In one embodiment, the training sample set constructing unit is configured to:
acquiring a sample image set;
for each sample image in the sample image set, performing:
respectively generating SR images corresponding to the sample images by using a plurality of super-resolution SR methods; and forming an image pair by the sample image and each corresponding SR image;
respectively scoring each image pair by applying an image quality evaluation index NIQE to obtain a perception score of each image pair;
sequencing all the image pairs according to the perception scores of the image pairs to obtain sequencing serial numbers of the image pairs;
and constructing the first training sample set by taking the sequencing sequence number and the perception score of each image pair as the label of the corresponding image pair.
In one embodiment, the feature extraction module is to:
and inputting the super-resolution image into a pre-trained index extraction model to obtain the specified evaluation index of the filament.
In one embodiment, the target object is a Printed Circuit Board (PCB), the filaments are hot melt adhesive filaments, and the specified evaluation index comprises at least one of the following: position, length, diameter, area.
In one embodiment, the sample image set includes sample images obtained by image acquisition of a sample object under multiple illumination conditions.
In a third aspect, another embodiment of the present application further provides an electronic device, including at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the methods provided by the embodiments of the first aspect of the present application.
In a fourth aspect, another embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and the computer program is configured to cause a computer to execute any one of the methods provided in the first aspect of the present application.
According to the embodiment of the application, the quality of the image of the target object is improved by adopting the super-resolution generation countermeasure network, the super-resolution image is obtained, the super-resolution image is detected by adopting the deep learning target detection framework, the detection result is obtained, the hardware cost is effectively saved, and the stability of the detection system is improved.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic view of an application scenario of a target object detection method provided in an embodiment of the present application;
fig. 2 is a flowchart of a target object detection method provided in an embodiment of the present application;
FIG. 3 is a schematic diagram of obtaining YOLO-V4 in the target object detection method according to the embodiment of the present application;
fig. 4 is a schematic diagram illustrating training of ranksrnan in a target object detection method provided in an embodiment of the present application;
fig. 5 is a schematic structural diagram of a RankSRGAN provided in an embodiment of the present application;
fig. 6 is a schematic diagram of constructing a first training sample set for training a Ranker network according to an embodiment of the present application;
fig. 7 is a schematic diagram illustrating a Ranker network trained by using a first training sample set in the target object detection method according to the embodiment of the present application;
fig. 8 is a schematic structural diagram of a generator provided in an embodiment of the present application;
fig. 9 is a schematic structural diagram of an apparatus of a target object detection method according to an embodiment of the present application;
fig. 10 is a schematic view of an electronic device of a target object detection method according to an embodiment of the present application.
Detailed Description
In the embodiment of the present application, the term "and/or" describes an association relationship of associated objects, and means that there may be three relationships, for example, a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
In the embodiments of the present application, the term "plurality" means two or more, and other terms are similar thereto.
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The inventor researches and discovers that the traditional industrial visual detection process is to perform algorithm identification on images acquired by an industrial camera so as to meet the requirement of detecting an object to be detected, however, due to the fact that the background of the object to be detected is complex, morphological characteristics such as length, color and diameter of a filament with the diameter smaller than a preset value on the object to be detected are difficult to control, the industrial camera cannot acquire the images of the object to be detected well, and accurate judgment on the object to be detected cannot be performed.
The inventor researches and finds that in order to solve the above problems, a high-resolution industrial camera or multiple industrial cameras are generally adopted for joint acquisition in the related art, but the hardware cost is greatly increased, and the detection system is complex and has poor stability.
In view of the above, the present application provides a target object detection method, an apparatus, an electronic device, and a storage medium, which are used to solve the above problems.
The invention conception of the application is as follows: when the outer surface of the target object has filaments with diameters smaller than the preset value, an industrial camera can be used to obtain an image of the target object, and it should be noted that the industrial camera used in the present application has low requirements on the resolution. After the image of the target object is obtained, image enhancement operation can be performed on the image content of the filamentous object to obtain a super-resolution image; then, performing feature extraction on the super-resolution image to obtain an appointed evaluation index of the filament; and comparing the specified evaluation index of the target object with a preset standard index to obtain a detection result of the target object for the filiform object.
The method for detecting the target object comprises the steps of carrying out image enhancement operation on image content of the filament, and effectively repairing and filling detail parts of the filament, so that the obtained super-resolution image can be comparable to the visual effect of a high-resolution industrial camera.
The target object detection method provided by the application is suitable for detecting the fine filaments in various scenes, such as but not limited to products including Printed Circuit Boards (PCBs), shoes with textures, chip electronic original devices, ornaments with textures, packing boxes with textures, furniture with textures, and the like.
The embodiments of the present application use a ranksrngan network (evaluation and super-resolution generation countermeasure network) when optimizing the image content of a thread-like object in an image, but the technical means of the present application is not limited to ranksrngan, and other neural networks capable of image enhancement processing, such as a super-resolution generation countermeasure network (srnan) and a generation countermeasure network (GAN), may be used.
The following describes a target object detection method in the embodiments of the present application in detail with reference to the drawings.
In an embodiment, as shown in fig. 1, an application scenario diagram illustrating a target object detection method by taking a PCB as an example in the embodiment of the present application is shown. The application scene comprises the following steps: PCB circuit board 101, electron device 102, hot melt adhesive silk 103. In the target object detection method provided by the embodiment of the application, an image of a target object is acquired from the PCB 101 by using an industrial camera with a single resolution, and image enhancement operation is performed on a hot melt adhesive wire on the PCB 101 to obtain a super-resolution image; extracting the characteristics of the super-resolution image to obtain an appointed evaluation index of the hot melt adhesive wire 103; and comparing the specified evaluation index of the target object with a preset standard index to obtain a detection result of the target object for the hot melt adhesive filament 103.
For convenience of understanding, in the embodiment of the present application, the target object is a PCB circuit board, the filament is a hot melt adhesive filament, and the specified evaluation index includes at least one of the following: position, length, diameter, area.
Of course, when detecting other target objects, a specific evaluation index for detection may be set according to actual requirements, which is not limited in the present application.
Fig. 2 shows a flowchart of an implementation process of the embodiment of the present application, which includes the following specific steps:
in step 201: acquiring an image of a target object, wherein the outer surface of the target object is provided with filaments with diameters smaller than a preset value;
in step 202: performing image enhancement operation on image content of the silk object in the image to obtain a super-resolution image;
in step 203: performing feature extraction on the super-resolution image to obtain an appointed evaluation index of the filament;
in one embodiment, when the super-resolution image is extracted, the super-resolution image needs to be input into a pre-trained index extraction model, so as to obtain the specified evaluation index of the filament.
For easy understanding, in the embodiment of the present application, the pre-trained index extraction model is YOLO-V4, but the technical means of the present application is not limited to YOLO-V4, and other models with index extraction are also applicable, as shown in fig. 3, the specific implementation steps of obtaining YOLO-V4 for a filament with a diameter smaller than a preset value are as follows:
in step 301: marking filaments with diameters smaller than a preset value in a pre-collected image sample set containing a target object image;
in step 302: and training and learning the filaments with the diameters smaller than the preset value in the image sample set by adopting a YOLO-V4 deep learning target detection framework so as to obtain a network model capable of performing feature extraction and target detection on the filaments with the diameters smaller than the preset value. That is, the trained YOLO-V4 model can accurately separate the silk from the background of the silk, and can extract specified evaluation indexes such as the position, size and dimension of the silk.
In step 204: and comparing the specified evaluation index of the target object with a preset standard index to obtain a detection result of the target object for the filiform object.
In some embodiments, the image enhancement operation may be performed as: the images are input into an evaluation and super-resolution generation confrontation network RankSRGAN, and a super-resolution image enhanced by an image generator of the RankSRGAN is obtained. So as to reconstruct and repair the image part of the silk object by utilizing the characteristic that the RankSRGAN can simulate to produce high-quality images.
The RankSRGAN comprises an evaluation network Ranker and a super-resolution countermeasure network SRGAN, and is trained according to the steps shown in fig. 4, wherein the specific implementation steps are as follows:
in step 401: constructing a first training sample set for training a Ranker network and a second training sample set for training a SRGAN network;
in the embodiment of the present application, as shown in fig. 6, a first training sample set for training a Ranker network is constructed according to the following operations:
in step 601: acquiring a sample image set;
the sample image set comprises a sample image set obtained by carrying out image acquisition on a sample object under various illumination conditions. The specific implementation process is as follows:
the design method comprises the steps of designing a polishing scheme of the PCB by utilizing a structural auxiliary light source, determining the design basis according to the practical application scene of the PCB, and mainly controlling the structure, the color, the brightness, the stroboscopic control and the like of an industrial detection light source. Through the design of the polishing scheme, the characteristic difference between the hot melt adhesive wire and other backgrounds of the PCB is effectively increased. And the method can enable the RankSRGAN network to learn the characteristics of the filaments in different lighting environments.
In this embodiment of the present application, the first training sample set and the second training sample set may be the same as or different from images in the sample image set, but both the first training sample set and the second training sample set include sample images in the sample image set, which is not limited in this embodiment of the present application.
In step 602: for each sample image in the sample image set, performing the following operations;
in step 603: respectively generating SR images corresponding to the sample images by using a plurality of super-resolution SR methods; and a sample image and each corresponding SR image form an image pair;
in step 604: respectively scoring each Image pair by applying an Image Quality evaluation index (NIQE) to obtain a perception score of each Image pair;
in step 605: sequencing all the image pairs according to the perception scores of the image pairs to obtain sequencing serial numbers of the image pairs;
in step 606: and constructing a first training sample set by taking the sequencing sequence number and the perception score of each image pair as the labels of the corresponding image pair.
In step 402: the Ranker network is trained using a first set of training samples such that the Ranker network outputs a content-ordering loss (rank-content loss).
In the embodiment of the present application, as shown in fig. 7, the specific implementation steps of training the Ranker network by using the first training sample set are as follows:
in step 701: respectively inputting a sample image in an image pair in a first training sample set and an SR image corresponding to each sample image into two branches of a Ranker network, and generating a quality score aiming at the sample image and the SR image corresponding to each sample image;
in step 702: calculating loss (margin-ranking loss) by using an optimization objective function according to the quality score;
in step 703: calculating gradient according to margin-ranking loss and applying back propagation to update parameters of a Ranker network;
in step 704: and stopping updating after the parameters reach the preset requirements, and taking the Ranker network with the parameters as the trained Ranker network.
In the embodiment of the present application, the trained Ranker network has the capability of ranking image pairs according to perceptual scores, and rank-content loss can be generated according to the input images.
In step 403: inputting each second training sample in the second training sample set into a generator of the SRGAN network to obtain a super-resolution sample image output by the generator;
in step 404: respectively inputting the super-resolution sample images into discriminators of a Ranker network and an SRGAN network to obtain a rank-content loss value containing Ranker network output;
in step 405: and training the generator according to the loss value to obtain a RankSRGAN network for image enhancement.
In the embodiment of the present application, as shown in fig. 8, the generator includes a perceptual loss (VGG loss), a countermeasure loss (GAN loss), and a rank-content loss.
For convenience of understanding, the following overall flow of the target object detection method provided in the embodiments of the present application is exemplified by taking a PCB as an example:
in one embodiment, before detecting the hot melt adhesive wire of the PCB, YOLO-V4 and RankSRGAN are trained on the hot melt adhesive wire;
1. training of YOLO-V4
Before training YOLO-V4, firstly, collecting hot melt adhesive wires of a PCB (printed Circuit Board) by using an industrial camera, marking the hot melt adhesive wires, and forming an image sample set by the marked hot melt adhesive wires;
training and learning the hot melt adhesive filaments in the image sample set by adopting YOLO-V4;
and obtaining a YOLO-V4 model capable of accurately extracting the hot melt adhesive filaments of the PCB.
2. Training RankSRGAN
The PCB is subjected to a polishing scheme design by utilizing a structural auxiliary light source, and the characteristic difference between the hot melt adhesive wire and other backgrounds of the PCB is increased;
acquiring a sample image set, wherein the sample image set comprises a sample image set obtained by carrying out image acquisition on a sample object under various illumination conditions;
aiming at each sample image in the sample image set, a plurality of super-resolution SR methods are applied to respectively generate SR images corresponding to the sample images; and a sample image and each corresponding SR image form an image pair;
respectively scoring each image pair by applying NIQE to obtain a perception score of each image pair;
sequencing all the image pairs according to the perception scores of the image pairs to obtain sequencing serial numbers of the image pairs;
constructing a first training sample set by using the sequencing sequence number and the perception score of each image pair as labels of the corresponding image pair (a second training sample set and a first training sample set used for training the SRGAN network can be the same or different, and the obtaining method is the same as that of the first training sample set, and is not repeated herein);
respectively inputting a sample image in an image pair in a first training sample set and an SR image corresponding to each sample image into two branches of a Ranker network, and generating a quality score aiming at the sample image and the SR image corresponding to each sample image;
calculating to obtain margin-ranking loss by using an optimization objective function according to the quality score;
calculating gradient according to margin-ranking loss and applying back propagation to update parameters of a Ranker network;
stopping updating after the parameters reach the preset requirements, and taking the Ranker network with the parameters as a trained Ranker network;
inputting each second training sample in the second training sample set into a generator of the SRGAN network to obtain a super-resolution sample image output by the generator;
respectively inputting the super-resolution sample images into discriminators of a Ranker network and an SRGAN network to obtain a rank-content loss value containing Ranker network output;
and training the generator according to the loss value to obtain a RankSRGAN network for image enhancement.
3. The hot melt adhesive filaments to be detected are detected by using the trained YOLO-V4 and RankSRGAN
Acquiring a PCB image to be detected; carrying out image enhancement operation on the PCB image by using the trained RankSRGAN to obtain a super-resolution image of the PCB;
and (3) performing feature extraction on the super-resolution image by using the trained YOLO-V4 to obtain at least one of the following indexes of the hot melt adhesive wire: position, length, diameter, area;
and comparing the index of the hot melt adhesive wire with a preset standard index to obtain whether the pollution of the hot melt adhesive wire on the PCB exceeds the standard or not.
Based on the same inventive concept, an embodiment of the present application further provides a target object detection apparatus, as shown in fig. 9, which is a schematic structural diagram of the apparatus, and the apparatus includes:
an image acquiring module 9001, configured to acquire an image of the target object, the target object having filaments with a diameter smaller than a preset value on an outer surface thereof;
an image enhancement module 9002, configured to perform an image enhancement operation on image content of the filament in the image to obtain a super-resolution image;
a feature extraction module 9003, configured to perform feature extraction on the super-resolution image to obtain an assigned evaluation index of the filament;
an index comparison module 9004, configured to compare the specified evaluation index of the target object with a preset standard index, to obtain a detection result of the target object for the filament.
In one embodiment, the image enhancement module is to:
inputting the images into an evaluation and combining with super resolution to generate a confrontation network RankSRGAN, and obtaining the super resolution images subjected to image enhancement by the RankSRGAN.
In one embodiment, the ranksrnan includes an evaluation network Ranker and a super resolution countermeasure network srnan, and the apparatus further includes:
a training module comprising the following elements to train the RankSRGAN:
a training sample set constructing unit, configured to construct a first training sample set used for training the Ranker network and a second training sample set used for training the SRGAN network;
a Ranker network training unit, configured to train the Ranker network by using the first training sample set, so that the Ranker network outputs a content ranking loss rank-content loss;
a super-resolution sample image output unit, configured to input each second training sample in the second training sample set to a generator of the SRGAN network, so as to obtain a super-resolution sample image output by the generator;
a loss value obtaining unit, configured to input the super-resolution sample image to the Ranker network and the discriminator of the SRGAN network, respectively, to obtain a loss value including rank-content loss output by the Ranker network;
and the generator training unit is used for training the generator according to the loss value to obtain a RankSRGAN network for image enhancement.
In one embodiment, the training sample set constructing unit is configured to:
acquiring a sample image set;
for each sample image in the sample image set, performing:
respectively generating SR images corresponding to the sample images by using a plurality of super-resolution SR methods; and forming an image pair by the sample image and each corresponding SR image;
respectively scoring each image pair by applying an image quality evaluation index NIQE to obtain a perception score of each image pair;
sequencing all the image pairs according to the perception scores of the image pairs to obtain sequencing serial numbers of the image pairs;
and constructing the first training sample set by taking the sequencing sequence number and the perception score of each image pair as the label of the corresponding image pair.
In one embodiment, the feature extraction module is to:
and inputting the super-resolution image into a pre-trained index extraction model to obtain the specified evaluation index of the filament.
In one embodiment, the target object is a PCB, the filament is a hot melt adhesive filament, and the specified evaluation index includes at least one of the following: position, length, diameter, area.
In one embodiment, the sample image set includes sample images obtained by image acquisition of a sample object under multiple illumination conditions.
Having described a target object detection method and apparatus according to an exemplary embodiment of the present application, an electronic device according to another exemplary embodiment of the present application is described next.
As will be appreciated by one skilled in the art, aspects of the present application may be embodied as a system, method or program product. Accordingly, various aspects of the present application may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.
In some possible implementations, an electronic device according to the present application may include at least one processor, and at least one memory. Wherein the memory stores program code which, when executed by the processor, causes the processor to perform the steps of the target object detection method according to various exemplary embodiments of the present application described above in the present specification. For example, the processor may perform the steps shown in FIG. 2.
The electronic apparatus 130 according to this embodiment of the present application is described below with reference to fig. 10. The electronic device 130 shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 10, the electronic device 130 is represented in the form of a general electronic device. The components of the electronic device 130 may include, but are not limited to: the at least one processor 131, the at least one memory 132, and a bus 133 that connects the various system components (including the memory 132 and the processor 131).
The memory 132 may include readable media in the form of volatile memory, such as Random Access Memory (RAM)1321 and/or cache memory 1322, and may further include Read Only Memory (ROM) 1323.
The electronic device 130 may also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), with one or more devices that enable a user to interact with the electronic device 130, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 130 to communicate with one or more other electronic devices. Such communication may occur via input/output (I/O) interfaces 135. Also, the electronic device 130 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) via the network adapter 136. As shown, network adapter 136 communicates with other modules for electronic device 130 over bus 133. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
In some possible embodiments, various aspects of a target object detection method provided by the present disclosure may also be implemented in the form of a program product including program code for causing a computer device to perform the steps in a target object detection method according to various exemplary embodiments of the present disclosure described above in this specification when the program product is run on the computer device, for example, the target object detection device may perform the steps as shown in fig. 2.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A program product for a target object detection method of an embodiment of the present disclosure may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on an electronic device. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the consumer electronic device, partly on the consumer electronic device, as a stand-alone software package, partly on the consumer electronic device and partly on a remote electronic device, or entirely on the remote electronic device or server. In the case of remote electronic devices, the remote electronic devices may be connected to the consumer electronic device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external electronic device (e.g., through the internet using an internet service provider).
It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. Indeed, the features and functions of two or more units described above may be embodied in one unit, in accordance with embodiments of the present disclosure. Conversely, the features and functions of one unit described above may be further divided into embodiments by a plurality of units.
Further, while the operations of the disclosed methods are depicted in the drawings in a particular order, this does not require or imply that these operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (16)
1. A target object detection method, the method comprising:
acquiring an image of the target object, wherein the outer surface of the target object is provided with filaments with diameters smaller than a preset value;
performing image enhancement operation on the image content of the silk object in the image to obtain a super-resolution image;
performing feature extraction on the super-resolution image to obtain an appointed evaluation index of the filament;
and comparing the specified evaluation index of the target object with a preset standard index to obtain a detection result of the target object for the filiform object.
2. The method according to claim 1, wherein said performing an image enhancement operation on image content of said filamentous object in said image comprises:
inputting the images into an evaluation and combining with super resolution to generate a confrontation network RankSRGAN, and obtaining the super resolution images subjected to image enhancement by the RankSRGAN.
3. The method of claim 2, wherein the RankSRGAN comprises an evaluation network Ranker and a super resolution countermeasure network SRGAN, the method further comprising:
training the RankSRGAN according to the following method:
constructing a first training sample set for training the Ranker network and a second training sample set for training the SRGAN network;
training the Ranker network by adopting the first training sample set so that the Ranker network outputs content sequencing loss rank-content loss;
inputting each second training sample in the second training sample set into a generator of the SRGAN network to obtain a super-resolution sample image output by the generator;
respectively inputting the super-resolution sample images into discriminators of the Ranker network and the SRGAN network to obtain a rank-content loss value comprising the Ranker network output;
and training the generator according to the loss value to obtain a RankSRGAN network for image enhancement.
4. The method of claim 3, wherein constructing a first set of training samples for training the Ranker network comprises:
acquiring a sample image set;
for each sample image in the sample image set, performing:
respectively generating SR images corresponding to the sample images by using a plurality of super-resolution SR methods; and forming an image pair by the sample image and each corresponding SR image;
respectively scoring each image pair by applying an image quality evaluation index NIQE to obtain a perception score of each image pair;
sequencing all the image pairs according to the perception scores of the image pairs to obtain sequencing serial numbers of the image pairs;
and constructing the first training sample set by taking the sequencing sequence number and the perception score of each image pair as the label of the corresponding image pair.
5. The method of claim 1, wherein the super-resolution image is subjected to feature extraction to obtain a specified evaluation index of the filament, comprising
And inputting the super-resolution image into a pre-trained index extraction model to obtain the specified evaluation index of the filament.
6. The method according to any one of claims 1 to 5, wherein the target object is a Printed Circuit Board (PCB), the filament is a hot melt adhesive filament, and the specified evaluation index comprises at least one of: position, length, diameter, area.
7. The method of claim 4, wherein the sample image set comprises sample images obtained by image capturing of the sample object under multiple illumination conditions.
8. A target object detection apparatus, characterized in that the apparatus comprises:
the image acquisition module is used for acquiring an image of the target object, and filaments with diameters smaller than a preset value are arranged on the outer surface of the target object;
the image enhancement module is used for carrying out image enhancement operation on the image content of the silk-like object in the image to obtain a super-resolution image;
the characteristic extraction module is used for extracting the characteristics of the super-resolution image to obtain the specified evaluation index of the silk object;
and the index comparison module is used for comparing the specified evaluation index of the target object with a preset standard index to obtain a detection result of the target object for the filiform object.
9. The apparatus of claim 8, wherein the image enhancement module is configured to:
inputting the images into an evaluation and combining with super resolution to generate a confrontation network RankSRGAN, and obtaining the super resolution images subjected to image enhancement by the RankSRGAN.
10. The apparatus of claim 9, wherein the RankSRGAN comprises an evaluation network Ranker and a super resolution countermeasure network SRGAN, the apparatus further comprising:
a training module comprising the following elements to train the RankSRGAN:
a training sample set constructing unit, configured to construct a first training sample set used for training the Ranker network and a second training sample set used for training the SRGAN network;
a Ranker network training unit, configured to train the Ranker network by using the first training sample set, so that the Ranker network outputs a content ranking loss rank-content loss;
a super-resolution sample image output unit, configured to input each second training sample in the second training sample set to a generator of the SRGAN network, so as to obtain a super-resolution sample image output by the generator;
a loss value obtaining unit, configured to input the super-resolution sample image to the Ranker network and the discriminator of the SRGAN network, respectively, to obtain a loss value including rank-content loss output by the Ranker network;
and the generator training unit is used for training the generator according to the loss value to obtain a RankSRGAN network for image enhancement.
11. The apparatus of claim 10, wherein the training sample set constructing unit is configured to:
acquiring a sample image set;
for each sample image in the sample image set, performing:
respectively generating SR images corresponding to the sample images by using a plurality of super-resolution SR methods; and forming an image pair by the sample image and each corresponding SR image;
respectively scoring each image pair by applying an image quality evaluation index NIQE to obtain a perception score of each image pair;
sequencing all the image pairs according to the perception scores of the image pairs to obtain sequencing serial numbers of the image pairs;
and constructing the first training sample set by taking the sequencing sequence number and the perception score of each image pair as the label of the corresponding image pair.
12. The apparatus of claim 8, wherein the feature extraction module is configured to:
and inputting the super-resolution image into a pre-trained index extraction model to obtain the specified evaluation index of the filament.
13. The apparatus according to any one of claims 8-12, wherein the target object is a printed circuit board, PCB, board, the filament is a hot melt adhesive filament, and the specified evaluation index comprises at least one of: position, length, diameter, area.
14. The apparatus of claim 11, wherein the sample image set comprises sample images obtained by image capturing of a sample object under multiple illumination conditions.
15. An electronic device, comprising:
a processor;
a memory configured to store the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the target object detection method of any one of claims 1-7.
16. A computer-readable storage medium, in which a computer program is stored, which, when executed by a processor of an electronic device, enables the electronic device to perform the target object detection method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010913275.XA CN112150414A (en) | 2020-09-03 | 2020-09-03 | Target object detection method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010913275.XA CN112150414A (en) | 2020-09-03 | 2020-09-03 | Target object detection method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112150414A true CN112150414A (en) | 2020-12-29 |
Family
ID=73889623
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010913275.XA Pending CN112150414A (en) | 2020-09-03 | 2020-09-03 | Target object detection method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112150414A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113095135A (en) * | 2021-03-09 | 2021-07-09 | 武汉理工大学 | System, method, device and medium for beyond-the-horizon target detection based on GAN |
-
2020
- 2020-09-03 CN CN202010913275.XA patent/CN112150414A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113095135A (en) * | 2021-03-09 | 2021-07-09 | 武汉理工大学 | System, method, device and medium for beyond-the-horizon target detection based on GAN |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109118482B (en) | Panel defect analysis method and device and storage medium | |
| CN107392218A (en) | A kind of car damage identification method based on image, device and electronic equipment | |
| CN111563509B (en) | Tesseract-based substation terminal row identification method and system | |
| CN107403424A (en) | A kind of car damage identification method based on image, device and electronic equipment | |
| CN111460967B (en) | Illegal building identification method, device, equipment and storage medium | |
| WO2022247005A1 (en) | Method and apparatus for identifying target object in image, electronic device and storage medium | |
| WO2020238256A1 (en) | Weak segmentation-based damage detection method and device | |
| CN110175519B (en) | Method and device for identifying separation and combination identification instrument of transformer substation and storage medium | |
| CN112613569B (en) | Image recognition method, training method and device for image classification model | |
| CN112508078B (en) | Image multitasking multi-label recognition method, system, equipment and medium | |
| CN109376628A (en) | A kind of picture quality detection method, device and storage medium | |
| CN116049397B (en) | Sensitive information discovery and automatic classification method based on multi-mode fusion | |
| CN111044149A (en) | Method and device for detecting temperature abnormal point of voltage transformer and readable storage medium | |
| CN105302906A (en) | Information labeling method and apparatus | |
| CN115690615B (en) | Video stream-oriented deep learning target recognition method and system | |
| CN109949313A (en) | A kind of real-time semantic segmentation method of image | |
| CN110555488A (en) | Image sequence auditing method and system, electronic equipment and storage medium | |
| CN112150414A (en) | Target object detection method and device, electronic equipment and storage medium | |
| CN114241384A (en) | Continuous frame picture marking method, electronic equipment and storage medium | |
| CN114693963A (en) | Recognition model training and recognition method and device based on electric power data feature extraction | |
| CN113591645A (en) | Power equipment infrared image identification method based on regional convolutional neural network | |
| JP7502570B2 (en) | Liquor product positioning method, liquor product information management method, and apparatus, device, and storage medium thereof | |
| KR20210037199A (en) | Apparatus for dividing, tagging an image and for detecting defect of facilities using the same | |
| CN110442807A (en) | A kind of webpage type identification method, device, server and storage medium | |
| CN115905016A (en) | BIOS Setup search function test method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |