WO2021197231A1 - 图像标注方法、装置及系统 - Google Patents
图像标注方法、装置及系统 Download PDFInfo
- Publication number
- WO2021197231A1 WO2021197231A1 PCT/CN2021/083332 CN2021083332W WO2021197231A1 WO 2021197231 A1 WO2021197231 A1 WO 2021197231A1 CN 2021083332 W CN2021083332 W CN 2021083332W WO 2021197231 A1 WO2021197231 A1 WO 2021197231A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- labeling
- image
- client
- annotation
- result
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 113
- 238000003062 neural network model Methods 0.000 claims abstract description 39
- 238000012545 processing Methods 0.000 claims abstract description 11
- 238000002372 labelling Methods 0.000 claims description 418
- 230000007547 defect Effects 0.000 claims description 171
- 238000001514 detection method Methods 0.000 claims description 23
- 238000009826 distribution Methods 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 19
- 238000012552 review Methods 0.000 claims description 19
- 238000003860 storage Methods 0.000 claims description 16
- 238000012549 training Methods 0.000 claims description 9
- 238000000605 extraction Methods 0.000 claims description 7
- 238000011176 pooling Methods 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 claims description 2
- 238000004519 manufacturing process Methods 0.000 description 25
- 230000008569 process Effects 0.000 description 20
- 239000002245 particle Substances 0.000 description 9
- 238000010586 diagram Methods 0.000 description 7
- 238000012550 audit Methods 0.000 description 6
- 230000002950 deficient Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000007689 inspection Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000005315 distribution function Methods 0.000 description 2
- 239000004576 sand Substances 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 239000000969 carrier Substances 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000037303 wrinkles Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/51—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
- G06T7/0008—Industrial image inspection checking presence/absence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/55—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/5866—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
- G06T7/001—Industrial image inspection using an image reference approach
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/262—Analysis of motion using transform domain methods, e.g. Fourier domain methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20048—Transform domain processing
- G06T2207/20056—Discrete and fast Fourier transform, [DFT, FFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30204—Marker
Definitions
- the present disclosure relates to the field of image processing technology, and in particular to an image labeling method, device and system.
- AOI Automated Optical Inspection
- the image acquisition device sends the detected bad images to the client, and the annotator needs to manually mark the defects of each bad image in the client to obtain the annotation result of the bad image.
- the present disclosure provides an image labeling method, device and system.
- the present disclosure provides an image labeling method, which is applied to a first client, and the method includes:
- the first labeling task assigned by the second client is received;
- the first labeling task includes the defect labeling of the first original image, and the defect labeling of the first original image Image labeling methods include a first labeling method, a second labeling method, and a third labeling method;
- the image annotation method is the first annotation method, determine the first annotation result corresponding to the first original image according to the neural network model;
- the image annotation method is the second annotation method, determine the second annotation result corresponding to the first original image according to an unsupervised algorithm model;
- the image annotation method is the third annotation method, receiving a third annotation result input by the user on the first original image;
- the target marking result is sent to the second client; the target marking result includes any one of the first marking result, the second marking result, and the third marking result.
- the present disclosure also discloses an image tagging device, which is applied to the first client, and the device includes:
- a working mode determining module configured to determine the working mode of the first client
- the first labeling task receiving module is configured to receive the first labeling task assigned by the second client when the working mode is the first mode; the first labeling task includes performing defect labeling on the first original image, so
- the image labeling methods for performing defect labeling on the first original image include a first labeling method, a second labeling method, and a third labeling method;
- the first annotation result determining module is configured to determine the first annotation result corresponding to the first original image according to the neural network model when the image annotation method is the first annotation method;
- the second annotation result determining module is configured to determine the second annotation result corresponding to the first original image according to the unsupervised algorithm model when the image annotation method is the second annotation method;
- the third annotation result determining module is configured to receive a third annotation result input by a user on the first original image when the image annotation method is the third annotation method;
- the target marking result sending module is configured to send the target marking result to the second client; the target marking result includes any of the first marking result, the second marking result, and the third marking result Either.
- the present disclosure also discloses an image labeling system, which includes a first client and a server;
- the first client includes the above-mentioned image tagging device
- the server is configured to receive the target marking result sent by the first client or the second client, and store the target marking result in a designated location.
- the present disclosure also discloses an electronic device installed with a first client, the electronic device including a processor, a memory, and a computer program stored on the memory and running on the processor, When the computer program is executed by the processor, the steps of the above-mentioned image labeling method are realized.
- the present disclosure also discloses a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned image labeling method are realized.
- the present disclosure also provides a computer program, including computer-readable code, which when the computer-readable code runs on an electronic device, causes the electronic device to execute the above-mentioned image labeling method.
- Fig. 1 shows a flowchart of an image labeling method according to an embodiment of the present disclosure
- Fig. 2 shows a connection structure diagram between a client and a server according to an embodiment of the present disclosure
- FIG. 3 shows a specific flowchart of an image labeling method according to an embodiment of the present disclosure
- FIG. 4 shows a processing flowchart when the first client is in the first mode according to an embodiment of the present disclosure
- Figure 5 shows a schematic diagram of a functional module interface of an embodiment of the present disclosure
- FIG. 6 shows a schematic diagram of a first image annotation interface according to an embodiment of the present disclosure
- FIG. 7 shows a schematic diagram of a second image annotation interface according to an embodiment of the present disclosure
- FIG. 8 shows a flowchart of another image labeling method according to an embodiment of the present disclosure.
- FIG. 9 shows a processing flowchart when the first client is in the second mode according to an embodiment of the present disclosure
- Fig. 10 shows a structural block diagram of an image labeling device according to an embodiment of the present disclosure
- FIG. 11 schematically shows a block diagram of an electronic device for executing the method according to the present disclosure.
- Fig. 12 schematically shows a storage unit for holding or carrying program codes for implementing the method according to the present disclosure.
- Step 101 Determine the working mode of the first client.
- the first client is an image tagging client, which mainly includes two working modes, namely a first mode and a second mode.
- the first client When the working mode of the first client is the first mode, the first client is mainly used to receive the first labeling task assigned by the second client, and mark the defects of the first original image according to the first labeling task; When the working mode of the client is in the second mode, the first client is mainly used to allocate the second labeling task to the third client, and receive the review result of the user reviewing the target labeling result returned by the third client.
- the first client in the embodiment of the present disclosure integrates two working modes, and the working mode of the first client is determined according to the user name, password and other information that the user logs in on the first client.
- the user logs in user name, password and other information on the first client.
- the first client judges whether the user has administrator rights based on the logged-in information. If the user has administrator rights, set the working mode of the first client It is the second mode; if you do not have the administrator authority, you cannot set the working mode of the first client to the second mode. At this time, the working mode of the first client is directly set to the first mode by default. Or, the first client sets the working mode of the first client according to the user's login user name, password and other information in the first client. If the information logged in the first client is the administrator information, the first client The working mode of the first client is set to the second mode; if the information logged on the first client is annotated personnel information, the working mode of the first client is set to the first mode.
- Step 102 When the working mode is the first mode, receive a first labeling task assigned by the second client; the first labeling task includes performing defect labeling on a first original image, and performing a defect labeling on the first original image.
- Image labeling methods for defect labeling include a first labeling method, a second labeling method, and a third labeling method.
- an image acquisition device in the process of manufacturing a product, is used to detect the product after each process.
- the image acquisition device can be a camera, a camera, an AOI device, etc., and the AOI device can detect the product in the manufacturing process.
- Perform an optical inspection to determine the difference between the product image and the standard product image, so as to identify it as a defect and take a photo, and then obtain a bad image with product defect graphics, which is convenient for subsequent identification and analysis of the defect category.
- the image acquisition device sends the detected bad image to the server, and the server receives the bad image sent by the image acquisition device, and stores the bad image in the product image database.
- the product image database can be a distributed file system (DFS) Or other data storage devices.
- the second client When it is determined that the working mode of the first client is the first mode, at this time, the second client is required to assign the first labeling task to the first client, and the second client needs to connect to the server, according to the product image database in the server The distribution characteristics of the pattern in the bad image are determined, and the image labeling method corresponding to the first original image that the first client needs to perform defect labeling is determined. Then, the second client assigns the first labeling task to the first client (at this time, the The working mode of the second client is the second mode), and the first client receives the first labeling task assigned by the second client.
- the first labeling task refers to the task that the first client needs to label the defects of the first original image.
- the image labeling methods for the first client to label the first original image include the first labeling method and the second labeling method. Method and the third labeling method; the first original image refers to the image that the first client needs to perform defect labeling in the defective image.
- first client and the second client may be different clients or the same client, which is not limited in the embodiment of the present disclosure.
- the second client when the second client assigns the first labeling task to the first client, it analyzes the distribution characteristics of the patterns in the bad images in the product image database on a factory basis, and determines the corresponding image labeling Way.
- the defective images in the product image database include defective images in the production process of Factory 1, Factory 2, and Factory 3, and the number of defective images in the production process of each factory is large, usually tens of thousands or hundreds of thousands.
- the second client analyzes the bad images in the production process of factory 1, and if the patterns in the bad images corresponding to factory 1 have the characteristics of array distribution, it determines that the image annotation method corresponding to the bad images in the production process of factory 1 is the second annotation
- the second labeling method refers to the automatic labeling method; accordingly, the second client analyzes the bad images in the production process of factory 2.
- the image labeling method corresponding to the bad images in the production process of factory 2 is determined to be the first labeling method.
- the first labeling method refers to the semi-automatic labeling method; the second client analyzes the factory 3 Bad images in the production process. If the differences between the bad images corresponding to factory 3 are large and the image background is complex, the image labeling method corresponding to the bad images in the production process of factory 3 is determined to be the third labeling method, and the third labeling method The method refers to the manual labeling method.
- Array distribution refers to the existence of multiple identical or similar structures in the bad image, and these multiple structures are periodically distributed in the bad image, such as the pixel structure fabricated on the substrate, generally, the circuit of small-sized products It is relatively dense, and the patterns in the bad images during the production process usually have the characteristics of array distribution.
- the client and server are connected through the Transmission Control Protocol (TCP) protocol.
- TCP Transmission Control Protocol
- the working mode of client 1 is set to the first mode, that is, client 1 is the first client
- the working mode of client 2 is set to the second mode, that is, client 2 is the second client
- the working mode of the client 3 and the client 4 is also set to the first mode, and the client 3 and the client 4 are also the first client.
- Client 2 connects to the server, and according to the distribution characteristics of the patterns in the bad images in the product image database in the server, determines the image labeling method corresponding to the first original image that client 1, client 3 and client 4 need to perform defect labeling ; Client 2 assigns the first labeling task to client 1.
- What client 1 needs to perform defect labeling is the bad image in the production process of factory 1, and the corresponding image labeling method is the second labeling method; client 2 A labeling task is assigned to client 3. What client 3 needs to label defects is a bad image in the production process of factory 2, and the corresponding image labeling method is the first labeling method; client 2 assigns the first labeling task to the client At terminal 4, what the client 4 needs to perform defect labeling is a bad image in the production process of factory 3, and the corresponding image labeling method is the third labeling method.
- Step 103 When the image annotation method is the first annotation method, determine the first annotation result corresponding to the first original image according to the neural network model.
- the first labeling method when it is determined that the image labeling method corresponding to the first original image that the first client needs to perform defect labeling is the first labeling method, that is, the semi-automatic labeling method, the first client is determined according to the pre-trained neural network model.
- the first labeling result is a semi-automatic labeling result
- the neural network model is a supervised model.
- the defect of each sample image is manually labeled to obtain the labeling result, and then the neural network model obtained by training according to the sample image and the labeling result of each sample image is used in the defect labeling In the process, the trained neural network model is then used to determine the first annotation result corresponding to the first original image. Therefore, the semi-automatic labeling method for defect labeling of the first original image requires a process of manual participation.
- the number of sample images is small, such as 100 or 200 sample images, so that It takes less time to mark the defects of the sample images, and when the semi-automatic markup method is used to mark the defects of the first original image, the manpower and time required for defect markup can be greatly saved.
- client 2 assigns the first labeling task to client 3.
- What client 3 needs to perform defect labeling is a bad image in the production process of factory 2, and the corresponding image labeling method is the first labeling method.
- the pre-trained neural network model determines the first annotation result corresponding to the bad image in the production process of factory 2.
- Step 104 When the image annotation method is the second annotation method, determine a second annotation result corresponding to the first original image according to an unsupervised algorithm model.
- the first client when it is determined that the image labeling method corresponding to the first original image that the first client needs to perform defect labeling is the second labeling method, that is, the automatic labeling method, the first client is directly determined according to the unsupervised algorithm model.
- the second labeling result is the automatic labeling result, and the unsupervised algorithm model does not require manual training.
- client 2 assigns the first labeling task to client 1, and client 1 needs to mark defective images in the production process of factory 1.
- the corresponding image labeling method is the second labeling method, and client 1 directly According to the unsupervised algorithm model, the second annotation result corresponding to the bad image in the production process of factory 1 is determined.
- Step 105 When the image tagging mode is the third tagging mode, receive a third tagging result input by the user on the first original image.
- the third labeling method that is, the manual labeling method
- the third image corresponding to the manual labeling module is manually labeled.
- a third annotation result is added to the first original image, and the first client receives the third annotation result input by the user on the first original image.
- the third labeling result is a manual labeling result.
- client 2 assigns the first labeling task to client 4, and what client 4 needs to perform defect labeling is a bad image in the production process of factory 3.
- the corresponding image labeling method is the third labeling method, and client 4 receives The third annotation result of the user input to the bad image in the production process of factory 3.
- Step 106 Send the target marking result to the second client; the target marking result includes any one of the first marking result, the second marking result, and the third marking result.
- the first client after obtaining the target marking result of the first original image, the first client sends the target marking result to the second client, and the second client receives the target marking result sent by the first client, and manages The employee reviews the target marking result sent by the first client and enters the review result on the second client.
- the second client sends the target marking result to the server.
- the second client returns the target marking result to the first client to perform defect marking again.
- the target labeling result is any one of the first labeling result, the second labeling result, and the third labeling result.
- the target labeling result sent to the second client is the first labeling result; when the first client needs to perform the defect labeling
- the image labeling method corresponding to the first original image is the second labeling method
- the target labeling result sent to the second client is the second labeling result; when the first client needs to perform defect labeling corresponding to the first original image
- the target labeling result sent to the second client is the third labeling result.
- the target marking result includes the defect category and defect location of the first original image.
- the first original image is analyzed to determine the image labeling method of the first original image, and the first original image is labeled according to the neural network model or the unsupervised algorithm model to realize automatic defect labeling.
- the defect labeling speed of the first original image is improved, and the manpower and time required for defect labeling are saved.
- the annotated target annotation result can be used as a sample in the subsequent training of the deep learning model.
- FIG. 3 there is shown a specific flowchart of an image labeling method according to an embodiment of the present disclosure, which is applied to the first client, and may specifically include the following steps:
- Step 301 Determine the working mode of the first client.
- Step 302 When the working mode is the first mode, receive a first labeling task assigned by the second client; the first labeling task includes performing defect labeling on a first original image, and performing a defect labeling on the first original image.
- Image labeling methods for defect labeling include a first labeling method, a second labeling method, and a third labeling method.
- AOI equipment in the process of manufacturing products, after each process, AOI equipment is used to perform optical inspection on the products in the manufacturing process to obtain bad images with product defect patterns.
- AOI equipment Send the detected bad images to the server through File Transfer Protocol (FTP).
- FTP File Transfer Protocol
- the second client is required to assign the first labeling task to the first client, and the second client needs to connect to the server, according to the product image database in the server
- the distribution characteristics of the patterns in the bad image determine the image labeling method corresponding to the first original image that the first client needs to perform defect labeling.
- the second client assigns the first labeling task to the first client, and the first client The terminal receives the first labeling task assigned by the second client.
- the first labeling task refers to the task that the first client needs to label the defects of the first original image.
- the image labeling methods for the first client to label the first original image include the first labeling method and the second labeling method. And the third labeling method.
- the first labeling method refers to the semi-automatic labeling method
- the second labeling method refers to the automatic labeling method
- the third labeling method refers to the manual labeling method.
- the corresponding image labeling method is the first labeling method; when the patterns in the first original image are distributed in an array, the corresponding image labeling method is the second labeling method ;
- the corresponding image labeling method is the third labeling method.
- the second client assigns the first labeling task to the first client 1, the first client 2, and the first client 3.
- the first client 1 needs to mark the defect of the first original image.
- the image annotation method is the automatic annotation method.
- the image annotation method that the first client 2 needs to annotate the first original image is the semi-automatic annotation method.
- the image annotation method that the first client 3 needs to annotate the first original image is the image annotation method. Manual labeling method.
- Step 303 When the image labeling method is the first labeling method, obtain a first labeling result obtained after the server performs defect labeling on the first original image in the server according to the neural network model selected by the user.
- the first original image that the first client needs to perform defect annotation is stored in the server, and the neural network model is also stored in the server.
- the neural network model is the target detection algorithm model.
- the server stores multiple categories of target detection algorithm models, and the target detection algorithm model of each category is based on pre-acquired multiple sample images belonging to the same category and each sample
- the image annotation result is obtained through training, and the number of sample images is less than the preset number.
- the preset number can be set manually, for example, the preset number is 100, 200, 300, 500, etc.
- the corresponding image labeling method is the first labeling method
- the corresponding image labeling method of the first original image is the first labeling
- the user selects the neural network model corresponding to the first original image that needs defect labeling in the first client.
- the first client sends the identification of the neural network model selected by the user to the server.
- the neural network model of the corresponding category stored in the server is obtained, and then the server performs defect annotation on the first original image in the server according to the neural network model selected by the user,
- the server sends the first labeling result to the first client, and the first client obtains the first labeling result obtained after the server performs defect labeling on the first original image according to the neural network model selected by the user.
- the first original image that the first client needs to mark the defect is stored in the server, and the first client connects to the server remotely to mark the defect of the first original image; of course, the first client can also receive the defect from the server. Download the first original image in the, and mark the defect in the first original image in the first client.
- the method of remotely connecting to the server for defect labeling can increase the speed of defect labeling compared to the method of downloading to the first client to perform defect labeling.
- the specific reason is that the first original image in the server is usually tens of thousands or a few. One hundred thousand sheets.
- the first client wants to download all the first original images, the first client needs a large storage capacity, and the network bandwidth and download time between the first client and the server also need to be very high. Requirements, and by connecting to the server remotely, the speed of defect marking can be increased, and the requirements of the first client on hardware storage devices and network bandwidth can also be reduced.
- step 303 may specifically include sub-step S11 to sub-step S15:
- Sub-step S11 when the image labeling method is the first labeling method, in the case of receiving the first operation of the user on the first labeling module in the functional module interface of the first client, display the The first image annotation interface corresponding to the first annotation module;
- Sub-step S12 in the case of receiving a second operation of the user on the configuration file in the first image annotation interface, connecting the first client to the server;
- Sub-step S13 sending a folder opening instruction to the server according to the image address of the first original image in the server, so as to open the folder in the server where the first original image is located;
- Sub-step S14 in the case of receiving the user's third operation on the model category in the first image annotation interface, determine the neural network model selected by the user;
- Sub-step S15 in the case of receiving the user's fourth operation on the image labeling option in the first image labeling interface, acquiring the server’s response to the first original image in the server according to the neural network model The first annotation result obtained after image defect annotation.
- the functional module interface 50 of the first client includes an administrator module 51, an automatic labeling mode 52, a semi-automatic labeling module 53, and a manual labeling module 54.
- the automatic labeling mode 52 is the second labeling module
- the semi-automatic labeling module 53 is The first labeling module and the manual labeling module 54 are the third labeling module.
- the first client When the user performs a trigger operation on the administrator module 51, the first client receives the user's trigger operation on the administrator module 51 in the function module interface 50 of the first client, and displays the management interface on the first client;
- the image labeling method corresponding to the first original image that a client needs to perform defect labeling is the first labeling method.
- the user can trigger the semi-automatic labeling module 53, and the first client receives the user’s request to the functional module interface 50 of the first client.
- the first operation of the semi-automatic labeling module 53 in, and then the first image labeling interface 60 corresponding to the semi-automatic labeling module 53 shown in FIG. 6 is displayed.
- the device corresponding to the first client is connected to a mouse, and the mouse clicks on the location of the administrator module 51 in the functional module interface 50 of the first client to trigger the administrator module 51, or the user touches Control the location of the administrator module 51 in the functional module interface 50 to implement the trigger operation on the administrator module 51; accordingly, the user's first operation of the semi-automatic marking module 53 in the functional module interface 50 of the first client is also It can be realized by mouse or finger touch.
- a configuration file option (not shown in FIG. 6) is displayed in the first image annotation interface 60, and the user can mark the first image in the interface 60
- the configuration file performs parameter configuration
- the first client receives the user's second operation on the configuration file in the first image annotation interface 60, and connects the first client to the server.
- the second client After the first client connects with the server, since the first original image is stored in the server, the second client also needs to send the image address of the first original image in the server to the first client.
- An image address is input in the first image annotation interface 60 of a client.
- the first client sends a folder opening instruction to the server according to the image address. After receiving the folder opening instruction, the server opens the file containing the first original image in the server. folder.
- the first image annotation interface 60 also displays an annotation model option 61. Then, the user selects an appropriate model category from the annotation model options 61 according to the first original image, and the first client receives the user's annotation of the first image in the interface 60 The third operation of the model category determines the neural network model selected by the user.
- the fourth operation is to send the identification of the neural network model selected by the user to the server.
- the server obtains the neural network model of the corresponding category stored in the server according to the identification of the neural network model selected by the user sent by the first client.
- the neural network model selected by the user performs defect annotation on the first original image in the server, and obtains the first annotation result.
- the server sends the first annotation result to the first client, and the first client obtains the neural network selected by the server according to the user.
- the first annotation result obtained after the network model performs defect annotation on the first original image.
- the specific process of the server performing defect annotation on the first original image in the server according to the neural network model selected by the user is as follows: performing feature extraction on the first original image in the server to obtain the feature image; extracting the target candidate area from the feature image; The feature image corresponding to the target candidate area is subjected to ROI Pooling processing to obtain a feature image of a preset size; the feature image of the preset size is classified to obtain the first annotation result.
- the first original image is input into the deep convolutional neural network, through the multi-layer convolution layer, activation layer, and pooling layer, the feature extraction of the first original image is realized, and the feature image corresponding to the entire first original image is obtained.
- the feature image corresponding to the first original image is input into the Regional Proposal Network (Region Proposal Network, RPN) to achieve the extraction of the candidate region of the feature image, and the target is filtered out by classifying the candidate region of the feature image Candidate area, and determine the coordinate position of the target candidate area;
- RPN Regional Proposal Network
- the target candidate area is filtered out by classifying the candidate region of the feature image Candidate area, and determine the coordinate position of the target candidate area
- the feature image corresponding to the target candidate area is performed ROI Pooling processing, that is, the target candidate area is divided into grids of preset size, each grid is mapped to the corresponding feature image, and the maximum value in each grid is extracted to obtain the feature image of preset size; finally, The feature image
- the target detection algorithm model Through the target detection algorithm model, it can be realized from the input of a first original image to the output of the position and category of the defect in the first original image. It has a good defect detection ability for bad images that do not have an array distribution. .
- the specific steps of the target detection algorithm model in the embodiments of the present disclosure are not limited to the above-mentioned feature extraction, target candidate region extraction, ROI Pooling processing, and the steps of classifying feature images of preset sizes, as long as they can All target detection algorithm models that implement defect annotation on the first original image are applicable.
- Step 304 When the image labeling method is the second labeling method, perform defect labeling on the first original image in the server according to the unsupervised algorithm model to obtain a second labeling result.
- the first original image that the first client needs to perform defect annotation is stored in the server, and the unsupervised algorithm model is stored in the first client.
- the unsupervised algorithm model is a fast Fourier transform algorithm model, that is, the fast Fourier transform algorithm model is stored in the first client.
- the corresponding image labeling method is the second labeling method
- the image labeling method corresponding to the first original image is the second labeling method
- defect labeling is performed on the first original image in the server to obtain the second labeling result.
- the Fourier transform is usually applied in the field of signal processing. It converts a time domain signal into the corresponding amplitude and phase at different frequencies.
- the frequency spectrum is the performance of the time domain signal in the frequency domain. It is used in image processing.
- the frequency of the image is an index that characterizes the intensity of the gray level change in the image, and is the gradient of the gray level in the plane space.
- the physical meaning of the Fourier transform is to transform the gray level distribution function of the image into the frequency distribution function of the image. Perform a two-dimensional Fourier transform on the first original image to obtain a spectrogram, which is the distribution map of the image gradient.
- the bright and dark bright spots obtained on the Fourier spectrogram are actually the strength of the difference between a certain point on the first original image and a neighboring point. , That is, the size of the gradient, that is, the size of the frequency of the point.
- the image gradient at that place will undergo abrupt changes, and it will be reflected in the Fourier spectrogram, then there will be bright spots; on the contrary, if the first original image If the original image has no defects, the gradient on the first original image is relatively uniform, and the spectrogram obtained after Fourier transform also has no bright spots.
- the defect location of the first original image can be directly completed based on the rule of the array distribution of the first original image, so as to quickly determine the second labeling result .
- the unsupervised algorithm model of the embodiment of the present disclosure is not limited to the fast Fourier transform algorithm model, of course, it can also be other models.
- step 304 may specifically include sub-step S21 to sub-step S24:
- Sub-step S21 when the image labeling method is the second labeling method, in the case of receiving the user's fifth operation on the second labeling module in the functional module interface of the first client, display the The second image annotation interface corresponding to the second annotation module;
- Sub-step S22 in the case of receiving the user's sixth operation on the configuration file in the second image annotation interface, connect the first client to the server;
- Sub-step S23 according to the image address of the first original image in the server, send a folder opening instruction to the server to open the folder where the first original image is located in the server;
- Sub-step S24 in the case of receiving the user's seventh operation on the image annotation option in the second image annotation interface, perform the operation on the first original image in the server according to the unsupervised algorithm model Defect marking, get the second marking result.
- the user can trigger the automatic labeling module 52, and the first client receives The user performs the fifth operation on the automatic labeling module 52 in the functional module interface 50 of the first client, and then displays the second image labeling interface 70 corresponding to the automatic labeling module 52 as shown in FIG. 7.
- a configuration file option (not shown in FIG. 7) is displayed in the second image tagging interface 70, and the user can tag the images in the second image tagging interface 70
- the first client receives the user's sixth operation on the configuration file in the second image annotation interface 70, and connects the first client to the server.
- the second client After the first client connects with the server, since the first original image is stored in the server, the second client also needs to send the image address of the first original image in the server to the first client. Enter the image address in the second image annotation interface 70 of a client, and the first client sends a folder opening instruction to the server according to the image address. After receiving the folder opening instruction, the server opens the file containing the first original image in the server. folder.
- the user can trigger the image annotation option (not shown in FIG. 7) in the second image annotation interface 70, and the first client receives the user's request for the image annotation option in the second image annotation interface 70
- the seventh operation is to perform defect annotation on the first original image in the server according to the unsupervised algorithm model stored in the first client to obtain the second annotation result.
- the second image annotation interface 70 is similar to the first image annotation interface 60, except that the annotation model option 61 does not need to be displayed on the second image annotation interface 70.
- Step 305 When the image annotation method is the third annotation method, receive a third annotation result input by the user on the first original image.
- the user when it is determined that the image labeling method corresponding to the first original image for which the first client needs to perform defect labeling is the third labeling method, the user can perform a trigger operation on the manual labeling module 54 , The first client receives the user's eleventh operation on the manual labeling module 54 in the functional module interface 50 of the first client, and then displays the third image labeling interface corresponding to the manual labeling module 54.
- the first original image, the category tree option corresponding to the defect category, the frame option corresponding to the defect position, and the saving option and marking option are displayed.
- the user selects the corresponding frame from the frame options according to actual needs and adds it to the defect position of the first original image.
- the defect category of the first original image select the corresponding defect category from the category tree options and add it to the first original image.
- the defect annotation of the first original image is implemented, and the first client receives the third annotation result input by the user on the first original image.
- the third annotation result can be saved by triggering the save option; or, when the user is not sure whether the third annotation result is correct, the third annotation result can be marked by triggering the marking option. , For subsequent re-determination or adjustment of the accuracy of the third annotation result.
- the specific process of manual annotation can refer to the specific process of the user inputting the third annotation result to the first original image in step 305.
- Step 306 Display the target annotation result corresponding to the first original image on the image annotation interface of the first client.
- the first client displays the target annotation result corresponding to the first original image on the image annotation interface of the first client.
- the target labeling result is any one of the first labeling result, the second labeling result, and the third labeling result.
- the image annotation interface is the first image annotation interface 60 as shown in FIG. 6, and the first annotation result is displayed in the area 62 in FIG. 6, and the first original According to the first annotation result corresponding to the image, it can be seen that the patterns in the first original image displayed in the area 62 are distributed in a non-displayed manner, and the defect positions in the first original image are displayed in the area 621.
- the image annotation interface is the second image annotation interface 70 as shown in FIG. 7.
- the second annotation result is displayed in the area 71 in FIG. 7, and the first original According to the second annotation result corresponding to the image, it can be seen that the patterns in the first original image displayed at area 71 are distributed in an array, and the patterns in the first original image displayed at area 71 are pixel structures, and each pixel structure is Array type distribution, the area 711 shows the defect position in the first original image.
- the image annotation interface is the third image annotation interface.
- the target marking result includes the defect category and defect location of the first original image.
- Defect categories include: particle, remain, line, hole, splash, wrinkle, film color, bubble, etc. More sub-categories can be divided based on each defect category. For example, particle categories can be divided into normal particles P1 (Normal Particle), loose sand particles P2 (Sand Particle), and sedimentary particles P3 (Wall Particle) based on particle defects of different shapes. )Wait.
- the defect position is used to indicate the specific coordinate position of the defect in the first original image, which includes the specific coordinate position on the circuit board where the defect is located, the relationship between the defect and the background circuit, for example, whether the defect completely falls on the gate or gate line ), whether the defect intersects the Gate, or the defect is located in the nearby area outside the Gate, etc.
- the target marking result may also include the size of the defect in the first original image, and the size of the defect includes the length of the defect or the area of the defect.
- Step 307 When the user's adjustment operation on the target marking result is received, adjust the target marking result.
- the user can preview the target annotation result corresponding to the first original image displayed on the image annotation interface of the first client.
- the user can check The target labeling result performs a trigger operation, and the first client receives the user's adjustment operation on the target labeling result, and then adjusts the target labeling result.
- the defect location and/or defect category can be adjusted.
- the user can trigger the category tree option 63 in the first image labeling interface 60, and select the correct defect category from the category tree option 63 to replace it.
- the user can trigger the category tree option 72 in the second labeling interface 70, and select from the category tree option 72
- the correct defect category replaces the originally marked defect category.
- step 307 the method further includes step S31: in the case of receiving the user's eighth operation of the save option in the image annotation interface, the adjusted The target marking result is saved.
- the save option in the image annotation interface can be triggered, and the first client receives the eighth of the save option in the image annotation interface by the user. Operate, and then save the target annotation result corresponding to the adjusted first original image.
- the user can trigger the save option 64 in the first image annotation interface 60 to change the adjusted first annotation The result is saved; or, as shown in FIG. 7, after the user finishes adjusting the second annotation result corresponding to a first original image, the user can trigger the save option 73 in the second annotation interface 70 to adjust Save the second annotation result afterwards.
- the save option can not only save the adjusted target annotation results of a first original image, but also click the save option after all the target annotation results corresponding to all the first original images in the first annotation task are adjusted. , To realize the saving of the target annotation results corresponding to all the adjusted first original images.
- step 306 the method further includes step S32: in the case of receiving the ninth operation of the user on the marking option in the image marking interface, marking the target marking result to obtain the to-be-processed Mark the result.
- the user can preview the target annotation result corresponding to the first original image displayed on the image annotation interface of the first client.
- the user can mark the image in the image annotation interface.
- the first client receives the user's ninth operation on the marking option in the image marking interface, and then marks the target marking result to obtain the pending marking result for subsequent re-determination of the target marking result If the target marking result is adjusted subsequently, step 307 may be performed.
- the user when the user is not sure whether the first annotation result corresponding to the first original image is correct, the user can trigger the marking option 65 in the first image annotation interface 60, and the first client will The annotation result is marked to obtain the annotation result to be processed; or, as shown in FIG. 7, when the user is not sure whether the second annotation result corresponding to the first original image is correct, the user can select the marking option 74 in the second annotation interface 70
- the first client marks the second marking result, and obtains the marking result to be processed.
- FIG. 6 also includes a file tree option 66, an upward jump option 67, a downward jump option 68, and a progress bar 69.
- the file tree option 66 includes a list of the first original image in the first labeling task; after clicking the up jump option 67, the first image of the previous first original image can be displayed in the area 62 of the first image labeling interface 60 Annotation result; after clicking the down jump option 68, the first annotation result of the next first original image can be displayed in the area 62 of the first image annotation interface 60; the progress bar 69 is used to display the progress of the first original image annotation ;
- Figure 7 also includes file tree options 75, upper jump options 76, down jump options 77 and progress bar 78, file tree options 75, up jump options 76, down jump options 77 and progress bar 78 specific functions The function is similar to the corresponding option in Fig. 6, in order to avoid repetition, it will not be repeated here.
- Step 308 Send the target marking result to the second client; the target marking result includes any one of the first marking result, the second marking result, and the third marking result.
- the adjusted target marking result is sent to the second client.
- the second client receives the target marking result sent by the first client,
- the administrator reviews the target marking result sent by the first client and enters the review result on the second client.
- the second client sends the target marking result to the server for storage.
- the review result is correct
- the second client returns the target marking result to the first client to remark the defect.
- step 308 includes sub-step S330, sub-step S340, and sub-step S350:
- Sub-step S330 in the case of receiving the user's tenth operation on the submission option in the image annotation interface, determine whether the first original image in the first annotation task has the to-be-processed annotation result;
- Sub-step S340 when there is no annotation result to be processed, send the target annotation result to the second client;
- sub-step S350 when the annotation result to be processed exists, a prompt box is displayed on the image annotation interface to prompt the user to adjust or save the annotation result to be processed.
- the submit option is displayed on the image annotation interface.
- the submit option is triggered, and the first client receives the user’s comment on the image annotation interface. Submit the tenth operation of the option, and then determine whether the first original image in the first labeling task has a labeling result to be processed.
- the target annotation result is sent to the second client; when the first original image in the first annotation task has the annotation result to be processed, the image A prompt box pops up on the annotation interface, prompting the user to adjust or save the annotation result to be processed. Or, the prompt box that pops up on the image annotation interface can also be used to prompt the user whether to continue sending the target annotation result.
- the first original image is analyzed to determine the image labeling method of the first original image, and the first original image is labeled according to the neural network model or the unsupervised algorithm model to realize automatic defect labeling.
- the defect marking speed of the first original image is improved, and the manpower and time required for defect marking are saved.
- the marked target marking result can be adjusted subsequently, so that the target marking result sent to the second client is more accurate.
- FIG. 8 there is shown a flowchart of another image labeling method according to an embodiment of the present disclosure, which is applied to a first client and specifically may include the following steps:
- Step 801 Determine the working mode of the first client.
- Step 802 When the working mode is the second mode, according to the distribution characteristics of the patterns in the second original image in the server, the second labeling task is allocated to the third client.
- the AOI device sends the detected bad image to the server.
- the first client needs to be the first
- the third client assigns the second labeling task
- the first client connects to the server, and according to the distribution characteristics of the pattern in the bad image in the product image database in the server, determines the second original image corresponding to the second original image that the third client needs to perform defect labeling Image labeling method, then, the first client assigns the second labeling task to the third client.
- the second labeling task refers to the task that the third client needs to perform defect labeling on the second original image.
- the image labeling methods used by the third client to label the second original image include the first labeling method and the second labeling method. And the third labeling method.
- the first labeling method refers to the semi-automatic labeling method
- the second labeling method refers to the automatic labeling method
- the third labeling method refers to the manual labeling method.
- the first client assigns the second labeling task to the third client 1, the third client 2, and the third client 3.
- the third client 1 needs to mark the defect of the second original image.
- the image annotation method is the automatic annotation method
- the image annotation method that the third client 2 needs to annotate the second original image is a semi-automatic annotation method
- the image annotation method that the third client 3 needs to annotate the second original image is Manual labeling method.
- Step 803 Receive a target labeling result obtained by the third client after labeling the second original image in the server according to the image labeling method.
- the third client assigns the second labeling task according to the first client to perform defect labeling on the second original image.
- the second original image refers to the defective image that the third client needs to perform defect labeling. image.
- the second original image and the first original image can be the same image, but because the client for defect annotation is different, it is divided into the first original image and the second original image.
- the first original image is the first client for annotation.
- the second original image was annotated by the third client.
- the third client is performing defect labeling on the second original image to obtain the target labeling result.
- the third client sends the target labeling result to the first client, and the first client receives that the third client reports to the server according to the image labeling method.
- the target labeling result obtained after labeling the second original image in the middle.
- Step 804 Receive a review result of the target annotation result input by the user on the first client.
- the user ie, the administrator
- reviews the target marking result sent by the third client and enters the review result for the target marking result on the first client, then the first client A client terminal receives the review result of the target marking result input by the user on the first client terminal.
- Step 805 When the review result is correct, send the target marking result to the server.
- the target marking result is sent to the server, and the server stores the target marking result sent by the first client in the designated Location.
- Step 806 When the review result is wrong, the target marking result is returned to the third client to perform defect marking again.
- the target marking result is returned to the third client to perform the defect again. Marking, the third client re-marks the defect of the second original image, and then sends it to the first client for review.
- the second original image is analyzed to determine the image labeling method of the second original image, and the second original image is marked with defects according to the neural network model or the unsupervised algorithm model to realize automatic defect labeling.
- the defect labeling speed of the second original image is improved, and the manpower and time required for defect labeling are saved.
- the image tagging device 1000 is applied to a first client, and the image tagging device 1000 includes:
- the working mode determining module 1001 is configured to determine the working mode of the first client
- the first labeling task receiving module 1002 is configured to receive the first labeling task assigned by the second client when the working mode is the first mode; the first labeling task includes performing defect labeling on the first original image,
- the image labeling methods for performing defect labeling on the first original image include a first labeling method, a second labeling method, and a third labeling method;
- the first annotation result determining module 1003 is configured to determine the first annotation result corresponding to the first original image according to the neural network model when the image annotation method is the first annotation method;
- the second annotation result determining module 1004 is configured to determine the second annotation result corresponding to the first original image according to the unsupervised algorithm model when the image annotation method is the second annotation method;
- the third annotation result determining module 1005 is configured to receive a third annotation result input by the user on the first original image when the image annotation method is the third annotation method;
- the target marking result sending module 1006 is configured to send the target marking result to the second client; the target marking result includes the first marking result, the second marking result, and the third marking result. Any of them.
- the first marking result determination module 1003 includes:
- the first labeling result determining submodule is configured to, when the image labeling mode is the first labeling mode, the acquiring server performs defect labeling on the first original image in the server according to the neural network model selected by the user The first annotation result obtained afterwards;
- the image labeling method is the first labeling method
- the neural network model is a target detection algorithm model
- the server stores how many Target detection algorithm models of three categories, and the target detection algorithm model of each category is obtained by training based on pre-acquired multiple sample images belonging to the same category and the labeling result of each sample image, the number of sample images Less than the preset amount.
- the second marking result determination module 1004 includes:
- the second labeling result determining submodule is configured to, when the image labeling mode is the second labeling mode, perform defect labeling on the first original image in the server according to the unsupervised algorithm model to obtain the first 2. Mark the result;
- the image labeling method is the second labeling method
- the unsupervised algorithm model is a fast Fourier transform algorithm model
- the fast Fourier transform The algorithm model is stored in the first client.
- the image tagging device 1000 further includes:
- the target annotation result display module is configured to display the target annotation result corresponding to the first original image on the image annotation interface of the first client;
- the target labeling result adjustment module is configured to adjust the target labeling result when the user's adjustment operation on the target labeling result is received.
- the target marking result includes the defect category and defect location of the first original image.
- the image tagging device 1000 further includes:
- the second labeling task allocation module is configured to allocate the second labeling task to the third client according to the distribution characteristics of the patterns in the second original image in the server when the working mode is the second mode;
- the second labeling task refers to the task that the third client needs to perform defect labeling on the second original image, and the image labeling method for the third client to perform defect labeling on the second original image includes the first labeling Method, second labeling method and third labeling method;
- the target marking result receiving module is configured to receive the target marking result obtained by the third client after marking the second original image in the server according to the image marking mode.
- the image tagging device 1000 further includes:
- An audit result receiving module configured to receive an audit result of the target annotation result input by the user on the first client
- the result sending module is configured to send the target marking result to the server when the audit result is correct;
- the result return module is configured to return the target marking result to the third client to perform defect marking again when the audit result is wrong.
- the description is relatively simple, and for related parts, please refer to the part of the description of the method embodiment.
- the first original image is analyzed to determine the image labeling method of the first original image, and the first original image is labeled according to the neural network model or the unsupervised algorithm model to realize automatic defect labeling.
- the defect labeling speed of the first original image is improved, and the manpower and time required for defect labeling are saved.
- the embodiment of the present disclosure also provides an image labeling system, including a first client and a server;
- the first client includes the above-mentioned image tagging device 900;
- the server is configured to receive the target marking result sent by the first client or the second client, and store the target marking result in a designated location.
- the server is further configured to train to obtain a target detection algorithm model according to a plurality of sample images belonging to the same category and the labeling result of each sample image obtained in advance, and to train the target detection algorithm model according to the target detection algorithm.
- Defect annotation is performed on the first original image in, and the first annotation result is obtained.
- the server when the first client is in the first mode, the server receives the target marking result sent by the second client; when the first client is in the second mode, the server receives the first A target marking result sent by the client.
- the specific training process of the target detection algorithm model is carried out in the server, and the defect labeling of the first original image according to the target detection algorithm model is also carried out in the server.
- the server is specifically configured to perform feature extraction on the first original image in the server to obtain a feature image; extract a target candidate region from the feature image; and determine a feature image corresponding to the target candidate region Perform ROI Pooling processing to obtain a feature image of a preset size; classify the feature image of the preset size to obtain the first annotation result.
- the first original image is analyzed to determine the image labeling method of the first original image, and the first original image is labeled according to the neural network model or the unsupervised algorithm model to realize automatic defect labeling.
- the defect labeling speed of the first original image is improved, and the manpower and time required for defect labeling are saved.
- an embodiment of the present disclosure also provides an electronic device, the electronic device is installed with a first client, the electronic device includes a processor, a memory, and is stored on the memory and can be stored on the processor.
- a running computer program that, when executed by the processor, implements the steps of the image labeling method described above.
- the embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the steps of the image labeling method described above.
- the device embodiments described above are merely illustrative.
- the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the objectives of the solutions of the embodiments. Those of ordinary skill in the art can understand and implement it without creative work.
- the various component embodiments of the present disclosure may be implemented by hardware, or by software modules running on one or more processors, or by a combination of them.
- a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the electronic device according to the embodiments of the present disclosure.
- DSP digital signal processor
- the present disclosure can also be implemented as a device or device program (for example, a computer program and a computer program product) for executing part or all of the methods described herein.
- Such a program for realizing the present disclosure may be stored on a computer-readable medium, or may have the form of one or more signals.
- Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.
- FIG. 11 shows an electronic device that can implement the method according to the present disclosure.
- the electronic device traditionally includes a processor 1010 and a computer program product in the form of a memory 1020 or a computer-readable medium.
- the memory 1020 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), EPROM, hard disk, or ROM.
- the memory 1020 has a storage space 1030 for executing program codes 1031 of any method steps in the above methods.
- the storage space 1030 for program codes may include various program codes 1031 respectively used to implement various steps in the above method. These program codes can be read from or written into one or more computer program products.
- These computer program products include program code carriers such as hard disks, compact disks (CDs), memory cards, or floppy disks.
- Such a computer program product is usually a portable or fixed storage unit as described with reference to FIG. 12.
- the storage unit may have storage segments, storage spaces, etc. arranged similarly to the storage 1020 in the electronic device of FIG. 11.
- the program code can be compressed in an appropriate form, for example.
- the storage unit includes computer-readable codes 1031', that is, codes that can be read by, for example, a processor such as 1010. These codes, when run by an electronic device, cause the electronic device to execute each of the methods described above. step.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- Quality & Reliability (AREA)
- Library & Information Science (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
Description
Claims (23)
- 一种图像标注方法,应用于第一客户端,所述方法包括:确定所述第一客户端的工作模式;当所述工作模式为第一模式时,接收第二客户端分配的第一标注任务;所述第一标注任务包括对第一原始图像进行缺陷标注,所述对第一原始图像进行缺陷标注的图像标注方式包括第一标注方式、第二标注方式和第三标注方式;当所述图像标注方式为所述第一标注方式时,根据神经网络模型,确定所述第一原始图像对应的第一标注结果;当所述图像标注方式为所述第二标注方式时,根据无监督算法模型,确定所述第一原始图像对应的第二标注结果;当所述图像标注方式为所述第三标注方式时,接收用户对所述第一原始图像输入的第三标注结果;将目标标注结果发送至所述第二客户端;所述目标标注结果包括所述第一标注结果、所述第二标注结果和所述第三标注结果中的任一者。
- 根据权利要求1所述的方法,所述当所述图像标注方式为所述第一标注方式时,根据神经网络模型,确定所述第一原始图像对应的第一标注结果的步骤,包括:当所述图像标注方式为所述第一标注方式时,获取服务器根据用户选择的神经网络模型对所述服务器中的所述第一原始图像进行缺陷标注后得到的第一标注结果。
- 根据权利要求2所述的方法,所述神经网络模型为目标检测算法模型,所述服务器中存储有多个类别的目标检测算法模型;每个类别的所述目标检测算法模型是根据预先获取的属于同一类别的多个样本图像以及每个所述样本图像的标注结果训练得到的,所述样本图像的数量小于预设数量。
- 根据权利要求1所述的方法,所述当所述图像标注方式为所述第二标注方式时,根据无监督算法模型,确定所述第一原始图像对应的第二标注结果的步骤,包括:当所述图像标注方式为所述第二标注方式时,根据所述无监督算法模型,对服务器中的所述第一原始图像进行缺陷标注,得到第二标注结果。
- 根据权利要求1所述的方法,所述无监督算法模型为快速傅立叶变换算法模型,且所述快速傅立叶变换算法模型存储在所述第一客户端中。
- 根据权利要求1所述的方法,在所述将目标标注结果发送至所述第二客户端的步骤之前,还包括:在所述第一客户端的图像标注界面显示所述第一原始图像对应的目标标注结果;在接收到用户对所述目标标注结果的调整操作的情况下,对所述目标标注结果进行调整。
- 根据权利要求6所述的方法,所述在接收到用户对所述目标标注结果的调整操作的情况下,对所述目标标注结果进行调整的步骤之后,还包括:在接收到用户对所述图像标注界面中的保存选项的第八操作的情况下,将调整后的所述目标标注结果进行保存。
- 根据权利要求6所述的方法,所述在所述第一客户端的图像标注界面显示所述第一原始图像对应的目标标注结果的步骤之后,还包括:在接收到用户对所述图像标注界面中的标记选项的第九操作的情况下,对所述目标标注结果进行标记,得到待处理标注结果。
- 根据权利要求8所述的方法,所述将目标标注结果发送至所述第二客户端的步骤,包括:在接收到用户对所述图像标注界面中的提交选项的第十操作的情况下,判断所述第一标注任务中的所述第一原始图像是否存在所述待处理标注结果;当不存在所述待处理标注结果时,将所述目标标注结果发送至所述第二客户端;当存在所述待处理标注结果时,在所述图像标注界面显示提示框,以提示用户对所述待处理标注结果进行调整或保存。
- 根据权利要求1所述的方法,所述目标标注结果包括第一原始图像的缺陷类别和缺陷位置。
- 根据权利要求1至10任一项所述的方法,在所述确定所述第一客户端的工作模式的步骤之后,还包括:当所述工作模式为第二模式时,根据服务器中的第二原始图像中的图案的分布特点,分配第二标注任务至第三客户端;所述第二标注任务包括对所述第二原始图像进行缺陷标注,所述对第二原始图像进行缺陷标注的图像标注方式包括第一标注方式、第二标注方式和第三标注方式;接收所述第三客户端根据所述图像标注方式对所述第二原始图像标注后得到的目标标注结果。
- 根据权利要求11所述的方法,所述根据服务器中的第二原始图像中的图案的分布特点,分配第二标注任务至第三客户端的步骤,包括:当所述第二原始图像的图案呈阵列式分布时,确定所述第二原始图像对应的图像标注方式为第二标注方式;当所述第二原始图像的图案呈非阵列式分布,且各个所述第二原始图像的差异性较小时,确定所述第二原始图像对应的图像标注方式为第一标注方 式;当各个所述第二原始图像的差异性较大,且所述第二原始图像的图像背景复杂时,确定所述第二原始图像对应的图像标注方式为第三标注方式。
- 根据权利要求11所述的方法,在所述接收所述第三客户端根据所述图像标注方式对所述第二原始图像标注后得到的目标标注结果的步骤之后,还包括:接收用户在所述第一客户端上输入的针对所述目标标注结果的审核结果;当所述审核结果无误时,将所述目标标注结果发送至所述服务器;当所述审核结果有误时,将所述目标标注结果返回至所述第三客户端重新进行缺陷标注。
- 根据权利要求11所述的方法,所述确定所述第一客户端的工作模式的步骤,包括:根据用户在所述第一客户端上的登陆信息,判断所述用户是否具有管理员权限;当所述用户具有所述管理员权限时,将所述第一客户端的工作模式确定为所述第二模式;当所述用户不具有所述管理员权限时,将所述第一客户端的工作模式确定为所述第一模式。
- 根据权利要求11所述的方法,所述服务器与所述第一客户端或所述第二客户端通过传输控制协议连接。
- 一种图像标注装置,应用于第一客户端,所述装置包括:工作模式确定模块,被配置为确定所述第一客户端的工作模式;第一标注任务接收模块,被配置为当所述工作模式为第一模式时,接收第二客户端分配的第一标注任务;所述第一标注任务包括对第一原始图像进行缺陷标注,所述对第一原始图像进行缺陷标注的图像标注方式包括第一标注方式、第二标注方式和第三标注方式;第一标注结果确定模块,被配置为当所述图像标注方式为所述第一标注方式时,根据神经网络模型,确定所述第一原始图像对应的第一标注结果;第二标注结果确定模块,被配置为当所述图像标注方式为所述第二标注方式时,根据无监督算法模型,确定所述第一原始图像对应的第二标注结果;第三标注结果确定模块,被配置为当所述图像标注方式为所述第三标注方式时,接收用户对所述第一原始图像输入的第三标注结果;目标标注结果发送模块,被配置为将目标标注结果发送至所述第二客户端;所述目标标注结果包括所述第一标注结果、所述第二标注结果和所述第三标注结果中的任一者。
- 根据权利要求16所述的装置,所述装置还包括:第二标注任务分配模块,被配置为当所述工作模式为第二模式时,根据服务器中的第二原始图像中的图案的分布特点,分配第二标注任务至第三客户端;所述第二标注任务包括对所述第二原始图像进行缺陷标注,所述对第二原始图像进行缺陷标准的图像标注方式包括第一标注方式、第二标注方式和第三标注方式;目标标注结果接收模块,被配置为接收所述第三客户端根据所述图像标注方式对所述第二原始图像标注后得到的目标标注结果。
- 一种图像标注系统,包括第一客户端和服务器;所述第一客户端包括如权利要求16或17所述的图像标注装置;所述服务器,被配置为接收所述第一客户端或第二客户端发送的目标标注结果,并将所述目标标注结果存储在指定位置。
- 根据权利要求18所述的系统,所述服务器,还被配置为:根据预先获取的属于同一类别的多个样本图像以及每个所述样本图像的标注结果训练得到目标检测算法模型,以及根据所述目标检测算法对所述服务器中的第一原始图像进行缺陷标注,得到第一标注结果。
- 根据权利要求19所述的系统,所述服务器,还被配置为:对所述服务器中的第一原始图像进行特征提取,得到特征图像;从所述特征图像中提取目标候选区域;对所述目标候选区域对应的特征图像进行ROI Pooling处理,得到预设尺寸的特征图像;对所述预设尺寸的特征图像进行分类,得到所述第一标注结果。
- 一种电子设备,所述电子设备安装有第一客户端,所述电子设备包括处理器、存储器以及存储在所述存储器上并可在所述处理器上运行的计算机程序,所述计算机程序被所述处理器执行时实现如权利要求1至15中任一项所述的图像标注方法的步骤。
- 一种计算机可读存储介质,所述计算机可读存储介质上存储计算机程序,所述计算机程序被处理器执行时实现如权利要求1至15中任一项所述的图像标注方法的步骤。
- 一种计算机程序,包括计算机可读代码,当所述计算机可读代码在电子设备上运行时,导致所述电子设备执行根据权利要求1-15中任一项所述的图像标注方法。
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/628,364 US20220414859A1 (en) | 2020-03-31 | 2021-03-26 | Image annotation method, device and system |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010247686.X | 2020-03-31 | ||
CN202010247686.XA CN113468350A (zh) | 2020-03-31 | 2020-03-31 | 一种图像标注方法、装置及系统 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021197231A1 true WO2021197231A1 (zh) | 2021-10-07 |
Family
ID=77865910
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2021/083332 WO2021197231A1 (zh) | 2020-03-31 | 2021-03-26 | 图像标注方法、装置及系统 |
Country Status (3)
Country | Link |
---|---|
US (1) | US20220414859A1 (zh) |
CN (1) | CN113468350A (zh) |
WO (1) | WO2021197231A1 (zh) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115048004A (zh) * | 2022-08-16 | 2022-09-13 | 浙江大华技术股份有限公司 | 标注方法及标注装置、电子设备和计算机可读存储介质 |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2002303580A (ja) * | 2001-04-03 | 2002-10-18 | Dainippon Printing Co Ltd | シート状製品の欠陥マーキング方法および装置 |
CN109118482A (zh) * | 2018-08-07 | 2019-01-01 | 腾讯科技(深圳)有限公司 | 一种面板缺陷分析方法、装置及存储介质 |
CN109299296A (zh) * | 2018-11-01 | 2019-02-01 | 郑州云海信息技术有限公司 | 一种交互式图像文本标注方法与系统 |
CN110567974A (zh) * | 2019-09-30 | 2019-12-13 | 华中科技大学 | 一种基于云端人工智能表面缺陷检测系统 |
CN110659661A (zh) * | 2018-06-29 | 2020-01-07 | 由田新技股份有限公司 | 瑕疵分类标记系统及瑕疵分类标记方法 |
-
2020
- 2020-03-31 CN CN202010247686.XA patent/CN113468350A/zh active Pending
-
2021
- 2021-03-26 WO PCT/CN2021/083332 patent/WO2021197231A1/zh active Application Filing
- 2021-03-26 US US17/628,364 patent/US20220414859A1/en active Pending
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2002303580A (ja) * | 2001-04-03 | 2002-10-18 | Dainippon Printing Co Ltd | シート状製品の欠陥マーキング方法および装置 |
CN110659661A (zh) * | 2018-06-29 | 2020-01-07 | 由田新技股份有限公司 | 瑕疵分类标记系统及瑕疵分类标记方法 |
CN109118482A (zh) * | 2018-08-07 | 2019-01-01 | 腾讯科技(深圳)有限公司 | 一种面板缺陷分析方法、装置及存储介质 |
CN109299296A (zh) * | 2018-11-01 | 2019-02-01 | 郑州云海信息技术有限公司 | 一种交互式图像文本标注方法与系统 |
CN110567974A (zh) * | 2019-09-30 | 2019-12-13 | 华中科技大学 | 一种基于云端人工智能表面缺陷检测系统 |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115048004A (zh) * | 2022-08-16 | 2022-09-13 | 浙江大华技术股份有限公司 | 标注方法及标注装置、电子设备和计算机可读存储介质 |
Also Published As
Publication number | Publication date |
---|---|
CN113468350A (zh) | 2021-10-01 |
US20220414859A1 (en) | 2022-12-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN106290378B (zh) | 缺陷分类方法和缺陷检查系统 | |
US20220075634A1 (en) | Detection of user interface controls via invariance guided sub-control learning | |
US20140164607A1 (en) | Dependency mapping among a system of servers, analytics and visualization thereof | |
US20150212929A1 (en) | Systems and methods for processing software application metadata associated with a software application | |
US20160147796A1 (en) | Automatic schema mismatch detection | |
US11194596B2 (en) | Simulating human interface interaction resilient to user interface changes | |
CN106164896B (zh) | 用于发现对手方关系的多维递归方法及系统 | |
US11397974B2 (en) | Method and system for assessing quality of commodities | |
US20240104148A1 (en) | Identifying web elements based on user browsing activity and machine learning | |
WO2021197231A1 (zh) | 图像标注方法、装置及系统 | |
CN107578438A (zh) | 圆识别方法、装置及电子设备 | |
CN111967576B (zh) | 一种基于深度学习的地球化学数据处理方法和系统 | |
CN113538154A (zh) | 风险对象的识别方法、装置、存储介质和电子设备 | |
CN108121648B (zh) | 一种界面错误监控方法 | |
US8601431B2 (en) | Method and system for identifying software applications for offshore testing | |
CN111738290B (zh) | 图像检测方法、模型构建和训练方法、装置、设备和介质 | |
JP7355299B2 (ja) | 学習用データセット生成システム、学習サーバ、及び学習用データセット生成プログラム | |
CN115905016A (zh) | 一种BIOS Setup搜索功能测试方法、装置、电子设备及存储介质 | |
US20220269901A1 (en) | Image processing method, apparatus, computer device and storage medium | |
US11575589B2 (en) | Network traffic rule identification | |
CN111598121B (zh) | 一种机柜设备检测方法 | |
EP4229564A1 (en) | Systems and methods for automatic detection of features on a sheet | |
CN110827261B (zh) | 图像质量检测方法及装置、存储介质及电子设备 | |
CN107491906A (zh) | 测绘产品检查方法及系统 | |
US20230154028A1 (en) | Image feature measurement |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21780408 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21780408 Country of ref document: EP Kind code of ref document: A1 |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 090523) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21780408 Country of ref document: EP Kind code of ref document: A1 |