WO2021082921A1

WO2021082921A1 - Back cover appearance defect detection method for electronic apparatus, and apparatus

Info

Publication number: WO2021082921A1
Application number: PCT/CN2020/120878
Authority: WO
Inventors: 徐鹏; 沈圣远; 常树林; 姚巨虎
Original assignee: 上海悦易网络信息技术有限公司
Priority date: 2019-10-28
Filing date: 2020-10-14
Publication date: 2021-05-06
Also published as: CN110827249A; JP2022539912A

Abstract

Provided are a back cover appearance defect detection method for an electronic apparatus and an apparatus. The invention comprises: acquiring an appearance image of an electronic apparatus; extracting a back cover appearance region image of the electronic apparatus from the appearance image thereof, and adjusting the back cover appearance region image to be an image having an equal length and width; inputting the adjusted back cover appearance region image into a trained model combining an FPN network and a backbone network; and receiving a defect detection result of a back cover appearance region of the electronic apparatus from the model combining the FPN network and the backbone network, wherein the defect detection result comprises a defect type of a back cover of the electronic apparatus, a position of a defect in the back cover of the electronic apparatus, and a confidence level of the defect detection result. The invention accurately identifies appearance defect differences of electronic apparatus back covers of second-handed electronic apparatuses, such as mobile phones.

Description

Method and equipment for detecting appearance defect of electronic equipment backplane

Technical field

The invention relates to the field of computers, and in particular to a method and equipment for detecting defects in the appearance of an electronic device backplane.

Background technique

At present, the defect detection of the appearance of the back panel of second-hand electronic equipment such as mobile phones is mainly based on traditional image algorithms. The detection is performed through color space conversion, filtering, feature point extraction, and pattern matching. The traditional detection methods can only A defect is detected in a certain area, but the definition of the defect cannot be distinguished.

Summary of the invention

An object of the present invention is to provide a method and equipment for detecting the appearance defect of the backplane of an electronic device.

According to one aspect of the present invention, there is provided a method for detecting appearance defects of an electronic device backplane, the method including:

Obtain the appearance image of the electronic device;

Extracting an image of the appearance area of the back panel of the electronic device from the appearance image of the electronic device, and adjusting the image of the appearance area of the back panel to an image with the same length and width;

Input the adjusted backplane appearance area image into the model of the FPN network after training and the backbone network;

The defect detection result of the backplane appearance area of the electronic device is received and output from the model of the FPN network combined with the backbone network. The defect detection result includes: the type of the defect of the backplane of the electronic device, and the number of defects in the backplane of the electronic device Confidence of location and defect detection results.

Further, in the above method, extracting the appearance area image of the backplane of the electronic device from the appearance image of the electronic device includes:

The Unet instance segmentation method is used to extract the appearance area image of the backplane of the electronic device from the appearance image of the electronic device.

Further, in the above method, the first two layers of the backbone network adopt a res structure, and the last two layers of the network adopt an inception structure.

Further, in the above method, after receiving the output defect detection result of the backplane appearance area of the electronic device from the model of the FPN network combined with the backbone network, the method further includes:

Identifying whether the confidence level of the defect detection result is greater than a first preset threshold,

If it is greater than the first preset threshold, output result information including the type of the defect of the backplane of the electronic device and the position of the defect on the backplane of the electronic device.

Further, in the above method, before inputting the backplane appearance area image into the FPN network combined with the backbone network model, the method further includes:

Step one, preset the FPN network combined with the backbone network model and its initial model parameters;

Step 2: Input the image of the backplane appearance area of the sample electronic device into the FPN network with the current model parameters combined with the backbone network model to obtain the defect prediction result of the backplane of the sample electronic device. The defect prediction result includes: sample electronics The type of defect on the backplane of the device, the position of the defect in the backplane of the sample electronic device, and the confidence level of the defect detection result;

Step 3: Calculate the difference between the defect prediction result and the real defect result of the sample electronic device based on a preset objective function, and identify whether the difference is greater than a second preset threshold,

If the difference is greater than the second preset threshold, step 4, after updating the model parameters of the FPN network combined with the backbone network based on the difference, restart execution from step 2;

If the difference is less than or equal to the second preset threshold, step 5, the FPN network combined with the backbone network model with the current model parameters is used as the model of the FPN network combined with the backbone network after the training.

Further, in the above method, adjusting the image of the appearance area of the backplane to an image with the same length and width includes:

The length direction of the image of the appearance area of the backplane is scaled.

According to another aspect of the present invention, there is also provided a device for detecting the appearance defect of the backplane of an electronic device, the device including:

The first device is used to obtain the appearance image of the electronic device;

The second device is used to extract the appearance area image of the backboard of the electronic device from the appearance image of the electronic equipment, and adjust the appearance area image of the backboard to an image with the same length and width;

The third device is used to input the adjusted backplane appearance area image into the model of the FPN network combined with the backbone network after the training;

The fourth device is used to receive and output the defect detection result of the backplane appearance area of the electronic device from the model of the FPN network combined with the backbone network. The defect detection result includes: the type of the defect of the backplane of the electronic device and the defect in the electronic device. The position in the backplane of the device and the confidence level of the defect detection result.

Further, in the above-mentioned device, the second device is used to extract the appearance area image of the backboard of the electronic device from the appearance image of the electronic device by using Unet instance segmentation.

Further, in the above device, the first two layers of the backbone network adopt a res structure, and the last two layers of the network adopt an inception structure.

Further, in the above-mentioned device, the fourth device is also used to identify whether the confidence of the defect detection result is greater than a first preset threshold, and if it is greater than the first preset threshold, the output includes the back of the electronic device. The result information of the type of the board defect and the position of the defect in the backplane of the electronic device.

Further, the above-mentioned equipment further includes a fifth device, including:

The fifth device is used to preset the FPN network combined with the backbone network model and its initial model parameters;

The fifth and second device is used to input the image of the backplane appearance area of the sample electronic device into the FPN network with the current model parameters combined with the backbone network model to obtain the defect prediction result of the backplane of the sample electronic device, and the defect prediction result Including: the type of defect on the backplane of the sample electronic device, the position of the defect in the backplane of the sample electronic device, and the confidence level of the defect detection result;

The fifty-third device is configured to calculate the difference between the defect prediction result and the actual defect result of the sample electronic device based on a preset objective function, and to identify whether the difference is greater than a second preset threshold, and if the difference is Is greater than the second preset threshold, execute the fifth and fourth device, which is used to update the model parameters of the FPN network combined with the backbone network based on the difference based on the fifth and fourth device, and then restart from the fifth and second device carried out;

If the difference is less than or equal to the second preset threshold, the fifth and fifth device is executed, and the model of the FPN network combined with the backbone network with the current model parameters is used as the model of the FPN network combined with the backbone network after the training is completed.

Further, in the above-mentioned equipment, the second device is used to scale the longitudinal direction of the image of the appearance area of the backplane.

The present invention also provides a computing-based device, which includes:

Processor; and

A memory arranged to store computer-executable instructions which, when executed, cause the processor to:

Obtain the appearance image of the electronic device;

The defect detection result of the backplane appearance area of the electronic device is received and output from the model of the FPN network combined with the backbone network. The defect detection result includes: the type of the defect of the backplane of the electronic device and the number of defects in the backplane of the electronic device Confidence of location and defect detection results.

The present invention also provides a computer-readable storage medium on which computer-executable instructions are stored, wherein, when the computer-executable instructions are executed by a processor, the processor:

Obtain the appearance image of the electronic device;

Compared with the prior art, the present invention obtains the appearance image of the electronic device; extracts the appearance area image of the backboard of the electronic device from the appearance image of the electronic device, and adjusts the appearance area image of the backboard to the length and width. The same image; input the adjusted backplane appearance area image into the model of the FPN network combined with the backbone network after training; receive the output defect detection result of the backplane appearance area of the electronic device from the model of the FPN network combined with the backbone network The defect detection result includes: the type of the defect of the backplane of the electronic device, the position of the defect in the backplane of the electronic device, and the confidence level of the defect detection result, which can accurately identify the appearance of the backplane of the electronic device of the second-hand electronic device such as a mobile phone The difference in blemishes.

Description of the drawings

By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more apparent:

FIG. 1 shows a flowchart of a method for detecting appearance defects of an electronic device backplane according to an embodiment of the present invention;

Fig. 2 shows a schematic diagram of a defect detection result according to an embodiment of the present invention;

FIG. 3 shows a schematic diagram of a model of an FPN network combined with a backbone network according to an embodiment of the present invention.

The same or similar reference signs in the drawings represent the same or similar components.

Detailed ways

The present invention will be described in further detail below in conjunction with the accompanying drawings.

In a typical configuration of this application, the terminal, the equipment of the service network, and the trusted party all include one or more processors (CPU), input/output interfaces, network interfaces, and memory.

The memory may include non-permanent memory in a computer readable medium, random access memory (RAM) and/or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer readable media.

Computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be realized by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical storage, Magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media can be used to store information that can be accessed by computing devices. According to the definition in this article, computer-readable media does not include non-transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

The present invention provides a method for detecting appearance defects of an electronic device backplane, the method including:

Step S1, obtaining an appearance image of the electronic device;

Step S2, extracting the appearance area image of the backplane of the electronic device from the appearance image of the electronic device, and adjusting the appearance area image of the backplane to an image with the same length and width;

Here, the appearance area of the back panel of the electronic device includes the back area other than the front screen area and the side area where the electronics are installed. The back area is generally equipped with components such as a camera.

The aspect ratio of the image of the appearance area of the backplane is abnormal, which is convenient for subsequent model recognition and avoids image loss. The aspect ratio of the image of the appearance area of the backplane needs to be adjusted to 1:1; step S3, the adjusted appearance area of the backplane The model of FPN network combined with backbone network after image input training;

Step S4, receiving and outputting the defect detection result of the backplane appearance area of the electronic device from the model of the FPN network combined with the backbone network. The defect detection result includes: the defect type of the backplane of the electronic device and the defect on the back of the electronic device. The position in the board and the confidence level of the defect detection result.

The model of the FPN network combined with the backbone network can be shown in FIG. 3.

Here, the defect detection results of the backplane appearance area of the electronic device received and output from the FPN network combined with the backbone network model, as shown in Figure 2, each defect detection result includes cls, x1, y1, x2, y2, score, where cls is the defect type, x1, y1, x2, y2 are the 4 coordinates of the position of the defect in the image of the backplane appearance area, and score is the confidence level of this defect.

The present invention mainly uses the improved feature pyramid (FPN) network combined with the deep learning model of the backbone network to accurately identify the difference in the appearance of the backplane of the electronic device of the second-hand electronic equipment such as the mobile phone, and can accurately distinguish the types of the defects.

In an embodiment of the method for detecting the appearance defect of the backplane of an electronic device of the present invention, step S2, extracting the appearance area image of the backplane of the electronic device from the appearance image of the electronic device includes:

Here, through Unet instance segmentation, the appearance area image of the backplane can be obtained quickly and efficiently.

In an embodiment of the method for detecting the appearance defect of the backplane of an electronic device of the present invention, the first two layers of the backbone network adopt a res structure, and the last two layers of the network adopt an inception structure.

In an embodiment of the method for detecting the appearance defect of the backplane of an electronic device of the present invention, step S4, after receiving the output defect detection result of the appearance area of the backplane of the electronic device from the model of the FPN network combined with the backbone network, further includes:

Here, the types of defects on the back panel of the electronic device may include: cracks, bracket screen separation, deformation, missing chipping, large-area paint drop, small-area paint drop (deformation, depression to reveal color), depression and no change in color, deep scratches And the color is different from the surroundings, small dots and the color is different from the surroundings, chipping, etc.

In this embodiment, by identifying the confidence level of the defect detection result, reliable results can be screened out from the defect detection results for output.

In an embodiment of the method for detecting the appearance defect of the backplane of an electronic device of the present invention, in step S3, before inputting the appearance area image of the backplane into the model of the FPN network combined with the backbone network, the method further includes:

Here, by identifying whether the difference is greater than the second preset threshold, the model of the FPN network combined with the backbone network is cyclically trained to obtain a reliable model.

In an embodiment of the method for detecting the appearance defect of the backplane of an electronic device of the present invention, adjusting the appearance area image of the backplane to an image with the same length and width includes:

Scale the length direction of the image of the appearance area of the backplane.

Here, the image of the appearance area of the backplane may be scaled to a size of 2048*2048 pixels, so as to obtain the image of the appearance area of the backplane adjusted to an image with the same length and width.

The present invention provides a device for detecting defects in the appearance of a backplane of an electronic device, and the device includes:

Here, the appearance area of the back panel of the electronic device includes side areas other than the front screen area and the back area where the electronics are arranged. The side areas are generally installed with earphone holes, speakers, charging holes and other components.

The image of the appearance area of the backplane has an abnormal aspect ratio, which is convenient for subsequent model recognition and avoids image loss. It is necessary to adjust the aspect ratio of the image of the appearance area of the backplane to 1:1;

Here, the defect detection result of the backplane appearance area of the electronic device received and output from the FPN network combined with the backbone network model, each defect detection result includes cls, x1, y1, x2, y2, score, where cls is Defect type, x1, y1, x2, y2 are the 4 coordinates of the position of the defect in the image of the back panel appearance area, and score is the confidence level of this defect.

The present invention mainly utilizes the improved feature pyramid (FPN) network combined with the deep learning model of the backbone network to accurately identify the difference in appearance of the electronic device backplanes of second-hand electronic devices such as mobile phones.

In an embodiment of the method for detecting the appearance defect of the backplane of an electronic device of the present invention, the second device is used to extract the appearance area image of the backplane of the electronic device from the appearance image of the electronic device by using the Unet instance segmentation method .

In an embodiment of the method for detecting the appearance defect of an electronic device backplane of the present invention, the fourth device is also used to identify whether the confidence of the defect detection result is greater than a first preset threshold, and if it is greater than the first preset Threshold, then output the result information including the defect type of the backplane of the electronic device and the position of the defect on the backplane of the electronic device.

Here, the types of defects on the backplane of the electronic device may sequentially include the types of shallow scratches, hard scratches, and chipping with increasing levels.

In an embodiment of the method for detecting appearance defects of an electronic device backplane of the present invention, it further includes a fifth device, including:

The fifty-third device is configured to calculate the difference between the defect prediction result and the actual defect result of the sample electronic device based on a preset objective function, and to identify whether the difference is greater than a second preset threshold, and if the difference is Greater than the second preset threshold, execute the fifth and fourth device, which is used to update the model parameters of the FPN network combined with the backbone network based on the difference based on the fifth and fourth device, and then restart from the fifth and second device carried out;

In an embodiment of the method for detecting the appearance defect of the backplane of an electronic device of the present invention, the second device is used to scale the longitudinal direction of the image of the appearance area of the backplane.

The present invention also provides a computing-based device, which includes:

Processor; and

Step S1, obtaining an appearance image of the electronic device;

Step S2, extracting the appearance area image of the backboard of the electronic device from the appearance image of the electronic device, and adjusting the appearance area image of the backboard to an image with the same length and width;

Step S3, input the adjusted backplane appearance area image into the model of the FPN network after the training and the backbone network;

Step S1, obtaining an appearance image of the electronic device;

Step S3, input the adjusted backplane appearance area image into the model of the FPN network combined with the backbone network after the training;

For details of the device and storage medium embodiments of the present invention, please refer to the corresponding parts of the method embodiments for details, which will not be repeated here.

Obviously, those skilled in the art can make various changes and modifications to the application without departing from the spirit and scope of the application. In this way, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these modifications and variations.

It should be noted that the present invention can be implemented in software and/or a combination of software and hardware. For example, it can be implemented by an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In an embodiment, the software program of the present invention may be executed by a processor to realize the steps or functions described above. Similarly, the software program (including related data structure) of the present invention can be stored in a computer-readable recording medium, such as a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present invention may be implemented by hardware, for example, as a circuit that cooperates with a processor to execute each step or function.

In addition, a part of the present invention can be applied as a computer program product, such as a computer program instruction, when it is executed by a computer, through the operation of the computer, the method and/or technical solution according to the present invention can be invoked or provided. The program instructions for invoking the method of the present invention may be stored in a fixed or removable recording medium, and/or transmitted through a data stream in a broadcast or other signal-bearing medium, and/or stored in accordance with the Said program instructions run in the working memory of the computer equipment. Here, an embodiment according to the present invention includes a device including a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, trigger The operation of the device is based on the aforementioned methods and/or technical solutions according to multiple embodiments of the present invention.

For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description, and therefore it is intended to fall into the claims. All changes within the meaning and scope of the equivalent elements of are included in the present invention. Any reference signs in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "including" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claims can also be implemented by one unit or device through software or hardware. Words such as first and second are used to denote names, but do not denote any specific order.

Claims

A method for detecting appearance defects of an electronic device backplane, wherein the method includes:

Obtain the appearance image of the electronic device;

Extracting an image of the appearance area of the back panel of the electronic device from the appearance image of the electronic device, and adjusting the image of the appearance area of the back panel to an image with the same length and width;

Input the adjusted backplane appearance area image into the model of the FPN network after training and the backbone network;

The defect detection result of the backplane appearance area of the electronic device is received and output from the model of the FPN network combined with the backbone network. The defect detection result includes: the type of the defect of the backplane of the electronic device and the number of defects in the backplane of the electronic device Confidence of location and defect detection results.
The method according to claim 1, wherein extracting the appearance area image of the backplane of the electronic device from the appearance image of the electronic device comprises:

The Unet instance segmentation method is used to extract the appearance area image of the backplane of the electronic device from the appearance image of the electronic device.
The method according to claim 1, wherein the first two layers of the backbone network adopt a res structure, and the last two layers of the network adopt an inception structure.
The method according to claim 1, wherein after receiving the output of the defect detection result of the backplane appearance area of the electronic device from the model of the FPN network combined with the backbone network, the method further comprises:

Identifying whether the confidence level of the defect detection result is greater than a first preset threshold,

If it is greater than the first preset threshold, output result information including the type of the defect of the backplane of the electronic device and the position of the defect on the backplane of the electronic device.
The method according to claim 1, wherein before inputting the image of the appearance area of the backplane into the FPN network combined with the backbone network model, the method further comprises:

Step one: preset the FPN network combined with the backbone network model and its initial model parameters;

Step 2: Input the image of the backplane appearance area of the sample electronic device into the FPN network with the current model parameters combined with the backbone network model to obtain the defect prediction result of the backplane of the sample electronic device. The defect prediction result includes: sample electronics The type of defect on the backplane of the device, the position of the defect in the backplane of the sample electronic device, and the confidence level of the defect detection result;

Step 3: Calculate the difference between the defect prediction result and the real defect result of the sample electronic device based on a preset objective function, and identify whether the difference is greater than a second preset threshold,

If the difference is greater than the second preset threshold, step 4, after updating the model parameters of the FPN network combined with the backbone network based on the difference, restart execution from step 2;

If the difference is less than or equal to the second preset threshold, step 5, the FPN network combined with the backbone network model with the current model parameters is used as the model of the FPN network combined with the backbone network after the training.
The method according to claim 1, wherein adjusting the image of the appearance area of the backplane to an image with the same length and width comprises:

Scale the length direction of the image of the appearance area of the backplane.
An electronic device backplane appearance defect detection device, wherein the device includes:

The first device is used to obtain the appearance image of the electronic device;

The second device is used to extract the appearance area image of the backboard of the electronic device from the appearance image of the electronic equipment, and adjust the appearance area image of the backboard to an image with the same length and width;

The third device is used to input the adjusted backplane appearance area image into the model of the FPN network combined with the backbone network after the training;

The fourth device is used to receive and output the defect detection result of the backplane appearance area of the electronic device from the model of the FPN network combined with the backbone network, and the defect detection result includes: the defect type of the backplane of the electronic device and the defect in the electronic device. The position in the backplane of the device and the confidence level of the defect detection result.
7. The device according to claim 7, wherein the second device is configured to extract the appearance area image of the backboard of the electronic device from the appearance image of the electronic device by using Unet instance segmentation.
7. The device according to claim 7, wherein the first two layers of the backbone network adopt a res structure, and the last two layers of the network adopt an inception structure.
7. The device according to claim 7, wherein the fourth device is further configured to identify whether the confidence level of the defect detection result is greater than a first preset threshold, and if it is greater than the first preset threshold, the output includes The result information of the defect type of the backplane of the electronic device and the position of the defect in the backplane of the electronic device.
The device according to claim 7, further comprising a fifth device, comprising:

The fifth device is used to preset the FPN network combined with the backbone network model and its initial model parameters;

The fifth and second device is used to input the image of the backplane appearance area of the sample electronic device into the FPN network with the current model parameters combined with the backbone network model to obtain the defect prediction result of the backplane of the sample electronic device, and the defect prediction result Including: the type of defect on the backplane of the sample electronic device, the position of the defect in the backplane of the sample electronic device, and the confidence level of the defect detection result;

The fifty-third device is configured to calculate the difference between the defect prediction result and the actual defect result of the sample electronic device based on a preset objective function, and to identify whether the difference is greater than a second preset threshold, and if the difference is Is greater than the second preset threshold, execute the fifth and fourth device, which is used to update the model parameters of the FPN network combined with the backbone network based on the difference based on the fifth and fourth device, and then restart from the fifth and second device carried out;

If the difference is less than or equal to the second preset threshold, the fifth and fifth device is executed, and the model of the FPN network combined with the backbone network with the current model parameters is used as the model of the FPN network combined with the backbone network after the training is completed.
8. The device according to claim 7, wherein the second device is used to scale the length direction of the image of the appearance area of the backplane.
A computing-based device, which includes:

Processor; and

A memory arranged to store computer-executable instructions which, when executed, cause the processor to:

Obtain the appearance image of the electronic device;

Extracting an image of the appearance area of the back panel of the electronic device from the appearance image of the electronic device, and adjusting the image of the appearance area of the back panel to an image with the same length and width;

Input the adjusted backplane appearance area image into the model of the FPN network after training and the backbone network;

The defect detection result of the backplane appearance area of the electronic device is received and output from the model of the FPN network combined with the backbone network. The defect detection result includes: the type of the defect of the backplane of the electronic device and the number of defects in the backplane of the electronic device Confidence of location and defect detection results.
A computer-readable storage medium having computer-executable instructions stored thereon, wherein, when the computer-executable instructions are executed by a processor, the processor:

Obtain the appearance image of the electronic device;

Extracting an image of the appearance area of the back panel of the electronic device from the appearance image of the electronic device, and adjusting the image of the appearance area of the back panel to an image with the same length and width;

Input the adjusted backplane appearance area image into the model of the FPN network after training and the backbone network;

The defect detection result of the backplane appearance area of the electronic device is received and output from the model of the FPN network combined with the backbone network. The defect detection result includes: the type of the defect of the backplane of the electronic device, and the defect in the backplane of the electronic device Confidence of location and defect detection results.