CN112989864A

CN112989864A - Method, device, storage medium and program product for identifying graphic code damage

Info

Publication number: CN112989864A
Application number: CN202110267584.9A
Authority: CN
Inventors: 张锦涛
Original assignee: Beijing Qisheng Technology Co Ltd
Current assignee: Beijing Qisheng Technology Co Ltd
Priority date: 2021-03-11
Filing date: 2021-03-11
Publication date: 2021-06-18

Abstract

The embodiment of the disclosure provides a method, a device, a storage medium and a program product for identifying graphic code damage, wherein a first image acquired by a terminal device is acquired after the terminal device fails to scan a target graphic code, and the first image comprises the target graphic code; acquiring position information of a target graphic code in a first image according to the first image and a preset graphic code recognition model; acquiring an image of the target graphic code from the first image according to the position information; and acquiring a recognition result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model. According to the image code recognition method and device, the position of the target image code in the first image can be determined through the image code recognition model, the image of the target image code can be accurately obtained, the recognition result of whether the target image code is damaged or not is determined through the image code classification model, the recognition accuracy and efficiency are high, the recognition of the image codes in batches can be achieved, and the labor cost is reduced.

Description

Method, device, storage medium and program product for identifying graphic code damage

Technical Field

The embodiment of the disclosure relates to the technical field of internet and artificial intelligence, in particular to a method, equipment, a storage medium and a program product for identifying damage of a graphic code.

Background

Graphic codes such as bar codes and two-dimensional codes are ubiquitous in daily scenes of the current society, such as code scanning payment in a consumption scene, code scanning pickup in an express scene, code scanning unlocking in a shared bicycle scene and the like.

However, the bar code, the two-dimensional code and other graphic codes may be damaged due to various factors, such as altering, breaking, tearing and the like, so that the graphic codes may not be recognized. In the prior art, after a user scans a graphic code and fails to recognize the graphic code, an operation and maintenance worker usually needs to judge whether the graphic code is damaged through a manual checking mode and then maintain the damaged graphic code.

In the prior art, whether the graphic codes are damaged or not is judged manually, so that the labor cost is increased, the judgment efficiency is low, and the judgment on whether the massive graphic codes are damaged or not cannot be met.

Disclosure of Invention

The embodiment of the disclosure provides a method, a device, a storage medium and a program product for identifying graphic code damage, so as to reduce the cost of the graphic code damage identification process, improve the identification efficiency and enable a user to judge whether a large amount of graphic codes are damaged.

In a first aspect, an embodiment of the present disclosure provides a method for identifying a damaged graphic code, including:

acquiring a first image acquired by terminal equipment after the terminal equipment fails to scan a target graphic code; the first image comprises a target graphic code;

acquiring position information of the target graphic code in the first image according to the first image and a preset graphic code identification model;

acquiring an image of the target graphic code from the first image according to the position information;

and acquiring a recognition result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model.

In a second aspect, an embodiment of the present disclosure provides an apparatus for identifying a damaged graphic code, including:

the acquisition module is used for acquiring a first image acquired by the terminal equipment after the terminal equipment fails to scan the target graphic code; the first image comprises a target graphic code;

the identification module is used for acquiring the position information of the target graphic code in the first image according to the first image and a preset graphic code identification model;

the matting module is used for acquiring an image of the target graphic code from the first image according to the position information;

and the classification module is used for acquiring the identification result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model.

In a third aspect, an embodiment of the present disclosure provides an apparatus for identifying a damaged graphic code, including: a memory and a processor;

the memory is to store program instructions;

the processor is used for calling the program instructions in the memory to execute the identification method of the graphic code damage according to the first aspect.

In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon; when executed, the computer program implements the method for identifying damaged graphic codes according to the first aspect.

In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, where the computer program is executed by a processor to implement the method for identifying damaged graphic code according to the first aspect.

According to the method, the device, the storage medium and the program product for identifying the graphic code damage, the first image collected by the terminal device is obtained after the terminal device fails to scan the target graphic code; the first image comprises a target graphic code; acquiring position information of the target graphic code in the first image according to the first image and a preset graphic code identification model; acquiring an image of the target graphic code from the first image according to the position information; and acquiring a recognition result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model. According to the image code recognition method and device, the position of the target image code in the first image can be determined through the image code recognition model, the image of the target image code can be accurately obtained, the recognition result of whether the target image code is damaged or not is determined through the image code classification model, the recognition accuracy and efficiency are high, the recognition of the image codes in batches can be achieved, and the labor cost is reduced.

Various possible embodiments of the present disclosure and technical advantages thereof will be described in detail below.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.

FIG. 1 is a schematic diagram of a shared vehicle system provided by an embodiment of the present disclosure;

fig. 2 is a schematic flow chart of a method for identifying damaged graphic codes according to an embodiment of the present disclosure;

fig. 3 is a schematic flow chart of a method for identifying damaged graphic codes according to another embodiment of the present disclosure;

fig. 4 is a schematic flow chart of a method for identifying damaged graphic codes according to another embodiment of the present disclosure;

fig. 5 is a schematic flow chart of a method for identifying damaged graphic codes according to another embodiment of the present disclosure;

fig. 6 is a schematic flow chart of a method for identifying damaged graphic codes according to another embodiment of the present disclosure;

fig. 7 is a schematic flowchart of a method for identifying a damaged graphic code according to another embodiment of the present disclosure;

fig. 8 is a schematic flowchart of a method for identifying a damaged graphic code according to another embodiment of the present disclosure;

fig. 9a is a schematic diagram of a connected component binary image set according to an embodiment of the present disclosure;

FIG. 9b is a schematic view of an image set of earth spoiled material provided by an embodiment of the present disclosure;

FIG. 9c is a schematic view of an image set of rust damage material according to an embodiment of the present disclosure;

FIG. 9d is a schematic diagram of an undamaged graphical code image set according to an embodiment of the present disclosure;

fig. 9e is a schematic diagram of a synthesized damage graph code image according to an embodiment of the disclosure;

FIG. 10 is a graph illustrating the loss curves from the verification set according to an embodiment of the present disclosure;

fig. 11 is a schematic structural diagram of an apparatus for identifying a damaged graphic code according to an embodiment of the present disclosure;

fig. 12 is a schematic structural diagram of a device for identifying a damaged graphic code according to another embodiment of the present disclosure.

With the foregoing drawings in mind, certain embodiments of the disclosure have been shown and described in more detail below. These drawings and written description are not intended to limit the scope of the disclosed concepts in any way, but rather to illustrate the concepts of the disclosure to those skilled in the art by reference to specific embodiments.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

In the prior art, image acquisition equipment or photoelectric scanning equipment is generally used for detecting graphic codes such as bar codes and two-dimensional codes and then identifying content information contained in the graphic codes. However, if the bar code, the two-dimensional code and other graphic codes are altered, damaged, torn and damaged, the scanning failure probability is greatly improved, so that the graphic codes may not be identified, and the user experience is directly influenced. In the prior art, it is generally necessary for an operation and maintenance person to determine whether a damaged graphic code, such as a correction, a damage, or a tear-out, is damaged by a manual review mode, for example, after a user fails to recognize the graphic code after scanning the graphic code, the operation and maintenance person can confirm whether the graphic code is damaged or cannot be recognized due to shooting related reasons, such as incomplete shooting, by observing whether the graphic code is corrected, damaged, or torn-out with naked eyes, and after determining that the graphic code is damaged, the operation and maintenance person maintains the graphic code, such as cleaning the graphic code or replacing a new graphic code.

Whether the graphic codes are damaged or not is judged manually, so that the labor cost is increased, the operation and maintenance cost is increased, the judgment efficiency is low, and the labor cost is increased sharply when a large number of graphic codes are faced, so that the judgment of whether the graphic codes are damaged or not can not be met by manually judging.

With the successful landing of a deep neural network represented by a convolutional neural network in computer vision and other application fields, the method for detecting the target and key point detection model in the neural network can be applied to locate the graphic code, the method for classifying the classification model in the neural network is also applied to classify whether the graphic code is damaged, and other learning models can be adopted. The method comprises the steps that whether a graphic code which fails to be scanned is damaged or not can be identified based on a trained model, and specifically, a first image which is collected by a terminal device and comprises a target graphic code is obtained after the terminal device fails to scan the target graphic code; acquiring position information of a target graphic code in a first image according to the first image and a preset graphic code recognition model; acquiring an image of the target graphic code from the first image according to the position information; and acquiring a recognition result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model. By applying the neural network and the computer vision technology, the method can realize the identification of whether the graphic code is damaged or not, has higher accuracy and efficiency of the identification result, can realize the maintenance of the graphic codes in batches, and reduces the labor cost.

The method for identifying the damaged graphic code provided by the embodiment of the disclosure is suitable for application scenarios of any bar code, two-dimensional code and other graphic codes, such as code scanning payment in a consumption scenario, code scanning pickup in an express delivery scenario, code scanning unlocking in a shared bicycle scenario and the like. The application scenes at least include a server and a terminal device, and graphic codes such as bar codes and two-dimensional codes are arranged on specific objects in the application scenes, taking a shared bicycle system as an example, as shown in fig. 1, the shared bicycle system includes: server 110, terminal device 120, and shared vehicle 130. The application scenarios of other bar codes, two-dimensional codes and other graphic codes are similar to those of a shared bicycle system, and no examples are given here.

The server 110 provides a service point for processes, databases, and communications facilities. The server 110 may be a unitary server or a distributed server across multiple computers or computer data centers. The server 110 may be of various types, such as, but not limited to, a web server, a news server, a mail server, a message server, an advertisement server, a file server, an application server, an interaction server, a database server, or a proxy server. In some embodiments, each server may include hardware, software, or embedded logic components or a combination of two or more such components for performing the appropriate functions supported or implemented by the server. For example, a server, such as a blade server, a cloud server, etc., or may be a server group consisting of a plurality of servers, which may include one or more of the above types of servers, etc.

The terminal device 120 is, for example, a smart phone, a laptop, a tablet, a palmtop, a wearable device, a virtual reality device, an augmented reality device, or the like, or any combination thereof, and is not limited herein. Fig. 1 illustrates the terminal device 120 as a smart phone.

The shared vehicle 130 is, for example, a shared electric bicycle, a shared tricycle, a shared electric scooter, a shared motorcycle, a shared four-wheel passenger vehicle, and the like, and is not limited herein. Fig. 1 shows the sharing vehicle 130 as a sharing bicycle as an example. Wherein, sharing vehicle 130 is provided with lock mechanism 140, and lock mechanism 140 is used for locking or unlocking sharing vehicle 130. The lock mechanism 140 may be a mechanical lock or an electronic lock. The shared vehicle 130 and the lock mechanism 140 may be mechanically connected components to each other. For example, the shared vehicle 130 and the lock mechanism 140 may be separate components, and the lock mechanism 140 may be mounted on the shared vehicle 130. Alternatively, the shared vehicle 130 and the lock mechanism 140 may be integrated into a single unit.

The terminal device 120 may communicate with the server 110 over a wireless or wired network. The wireless network may be a 2G, 3G, 4G, or 5G communication network, or may be a wireless local area network, which is not limited herein. The terminal device 120 and the shared vehicle 130 and/or the lock mechanism 140 may communicate with each other in a short-range communication manner, which may be bluetooth or wifi. Communication between the shared vehicle 130 and/or the lock mechanism 140 and the server 110 may be via a wireless network.

Terminal device 120 may send a request associated with shared vehicle 130 to server 130, such as a service request for a car use request, an end car use request, etc. of shared vehicle 130. A use request for the shared vehicle 130, the use request carrying a unique identification of the shared vehicle or the lock mechanism. The unique identification may include a barcode, a Quick Response (QR) code, a serial number including letters and/or numbers, or the like, or any combination thereof. For example, the car use request may be triggered by the user scanning the QR code of the shared vehicle 130 or the lock mechanism 140 through the camera of the terminal device, or triggered by the terminal device 120 inputting a unique identifier.

The service request may include riding related information including one or a combination of shared vehicle type, departure location, destination, riding time, riding mileage, route, etc. The service request may also include information related to the user (e.g., user account information) and/or the terminal device 120 (e.g., location of the terminal device 120).

The server 110 may also send information to the end device 120, the shared vehicle 130, and/or the lock mechanism 140. For example, the server 110 may send at least one of the following to the shared vehicle 140 and/or the lock mechanism 140: instructions to lock and unlock shared vehicle 130, information about shared vehicle 130 (e.g., information that prompts whether shared vehicle 130 is locked). For example, the server 110 may send at least one of the following to the terminal device 120: information indicating whether sharing vehicle 130 is available, information indicating whether sharing vehicle 130 is allowed to be locked at the current location).

The sharing vehicle 130 and/or the lock mechanism 140 may transmit status information to the server 110, the terminal device 120, respectively, which may include one or more of a location of the sharing vehicle 130, a locked/unlocked state of the sharing vehicle 130, battery power of the sharing vehicle 130, operational information, and the like. In some embodiments, the sharing vehicle 130 and/or the lock mechanism 140 may receive instructions (e.g., instructions to lock/unlock the sharing vehicle 130) from the end device 120 and/or the server 110.

It should be understood that although fig. 1 shows one server 110, one terminal device 120, one shared vehicle 130, it is not meant to limit the respective numbers, and multiple servers 110, multiple terminal devices 120, multiple shared vehicles 130, etc. may be included in the shared vehicle system.

The execution subject of the method for identifying the damaged graphic code provided by the embodiment of the disclosure may be the terminal device 120 or the service 110.

In an optional embodiment, an execution main body of the method for identifying the damage of the graphic code provided by the embodiment of the present disclosure is the terminal device 120, and then a graphic code identification model and a graphic code classification model are deployed in advance on the terminal device 120, and when a user fails to scan a target graphic code (QR code) of the shared vehicle 130 or the lock mechanism 140 through the terminal device 120, the terminal device 120 may obtain position information of the target graphic code in a first image according to a first image which includes the target graphic code and a preset graphic code identification model and is acquired by the terminal device 120; then acquiring an image of the target graphic code from the first image according to the position information; and acquiring a recognition result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model. Further, the terminal device 120 may report the identification result to the server 110, and may also report positioning information (for example, GPS positioning information) when the terminal device 120 scans the target graphic code and an image of a surrounding environment of the device where the target graphic code is located, and the server 110 may send a maintenance instruction to the maintenance staff terminal, where the maintenance instruction includes the positioning information and the image of the surrounding environment, so that the maintenance staff may position the target graphic code according to the positioning information and the image of the surrounding environment to maintain the target graphic code.

In another optional embodiment, an execution subject of the method for identifying the damage of the graphic code provided by the embodiment of the present disclosure is a server 110, a graphic code identification model and a graphic code classification model are deployed in advance on the server 110, when a user fails to scan a target graphic code (QR code) of the shared vehicle 130 or the lock mechanism 140 through the terminal device 120, the terminal device 120 sends a first image including the target graphic code acquired by the user to the server 110, and the server 110 may obtain position information of the target graphic code in the first image according to the first image including the target graphic code and the preset graphic code identification model; then acquiring an image of the target graphic code from the first image according to the position information; and acquiring a recognition result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model. Further, if the server 110 determines that the identification result is that the target pattern code is damaged, the terminal device 120 may be requested to obtain the positioning information when the terminal device 120 scans the target pattern code and the image of the surrounding environment of the device where the target pattern code is located, or the terminal device 120 sends the positioning information and the image of the surrounding environment of the device where the target pattern code is located to the server 110 while sending the first image; further, the server 110 may send a maintenance instruction to a maintenance person terminal, where the maintenance instruction includes the positioning information and the surrounding environment image, so that the maintenance person positions the target graphic code according to the positioning information and the surrounding environment image to maintain the target graphic code.

The following describes technical solutions of the embodiments of the present disclosure and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present disclosure will be described below with reference to the accompanying drawings.

Referring to fig. 2, fig. 2 is a schematic flow chart of a method for identifying a damaged graphic code according to an embodiment of the present disclosure. The executed main body of the method for identifying damaged graphic codes in this embodiment may be a terminal device, a server, or other electronic devices, and the method for identifying damaged graphic codes in this embodiment includes:

s201, after the terminal equipment fails to scan the target graphic code, acquiring a first image acquired by the terminal equipment; the first image includes a target graphics code.

In the embodiment, the terminal device opens the camera, collects a first image including a target graphic code, and scans the first image, and if the target graphic code fails to be scanned, the terminal device can continue to execute subsequent S202-S204 based on the first graphic; or the terminal equipment can also re-acquire a first image comprising the target graphic code and then continue to execute the subsequent S202-S204; or the terminal device may send the first image to the server, and after the server acquires the first image, the server continues to execute subsequent S202-S204.

In the present embodiment, the processes of the subsequent S202-S204 performed by the terminal device or the server are substantially similar.

S202, acquiring position information of the target graphic code in the first image according to the first image and a preset graphic code identification model.

In this embodiment, the preset pattern code recognition model may be a pattern code recognition model constructed based on a target detection model of a convolutional neural network. Optionally, in consideration of user experience and transmission traffic, the embodiment may select a Thundernet model as the baseline version of the graph barcode recognition model, where the Thundernet model is the first Real-time Object Detection model (Thundernet: directions Real-time Generic Object Detection) for embedded device design published by the open-air science and technology corporation (megvii) in ICCV2009 conference. Without loss of generality, other models can be adopted, for example, a mobile-end real-time target detection model such as mobilene-ssd (lite), pelee and the like can be selected as a terminal graphic code detection model, or a high-precision version target detection model such as ssd, yolo, eficientde and the like can be selected as a server-end graphic code detection model.

After the preset pattern code recognition model is trained, in a model application stage, the position information of the target pattern code in the first image can be acquired according to the first image and the preset pattern code recognition model. If the input data of the graphic code recognition model during training is an image comprising a two-dimensional code, the first image can be directly input into the graphic code recognition model in the application stage, and the position information of the target graphic code in the first image is output; if the graphic code recognition model preprocesses the image including the two-dimensional code during training, the same preprocessing process can be performed on the first image in the application stage, and then the image is input into the graphic code recognition model, and the position information of the target graphic code in the first image is output.

As an alternative embodiment, as shown in fig. 3, the step S202 of acquiring the position information of the target graphic code in the first image according to the first image and a preset graphic code recognition model may include:

s2021, preprocessing the first image to obtain first pixel matrix data corresponding to the first image.

S2022, inputting the first pixel matrix data into the graphic code identification model, and obtaining a predicted position of the target graphic code in the first image and a corresponding confidence coefficient.

S2023, finally determining the position information of the target graphic code in the first image according to the predicted position in the first image and the corresponding confidence coefficient.

In this embodiment, the first image is preprocessed, for example, values of RGB channels of the first image are converted into pixel matrix data; optionally, the pixel matrix data may be normalized, and the normalized value is between 0 and 1, and accordingly, the same normalization processing is required during model training.

Further, inputting the first pixel matrix data into a pattern code recognition model, so as to obtain a predicted position of the target pattern code in the first image and a corresponding confidence coefficient, where it should be noted that there may be more than one predicted position of the target pattern code in the first image, and the predicted positions may be screened or fused through the corresponding confidence coefficients, for example, a first confidence coefficient threshold may be set, for example, 50%, or an appropriate value between 0 and 1.0 is set according to the performance of the model on the test set, so as to screen out a predicted position with a confidence coefficient meeting the first confidence coefficient threshold from the plurality of predicted positions, and determine the predicted position as position information of the target pattern code in the first image; if there are multiple predicted positions with confidence levels satisfying the first confidence level threshold, a final predicted position may be determined according to the multiple predicted positions, for example, the multiple predicted positions may be averaged, or other approaches may be used, which is not limited herein.

S203, acquiring the image of the target graphic code from the first image according to the position information.

In this embodiment, after determining the position information of the target graphic code in the first image, the target graphic code may be subjected to matting or cropping based on the position information, and an image including only the target graphic code is acquired. Of course, in this embodiment, the image of the target graphic code obtained by matting may also be further processed, such as scaling, rotating, flipping, affine transformation, and the like, which may not be limited herein.

S204, acquiring a recognition result of whether the target graphic code is damaged according to the image of the target graphic code and a preset graphic code classification model.

In this embodiment, after the image of the target graphic code is obtained, whether the target graphic code is damaged or not may be classified based on the image of the target graphic code and a preset graphic code classification model, that is, the output category includes that the target graphic code is damaged or not.

The preset graphic code classification model in this embodiment may be a graphic code classification model constructed based on an image classification model of a convolutional neural network. Optionally, in consideration of user experience and transmission traffic, in this embodiment, a shufflenetv2 model may be used as a baseline version of the graphical code classification model, where the shufflenetv2 model is an image classification model (shuffle V2: Practical Guidelines for Efficient CNN Architecture Design) published by the spacious technology corporation (megvii) in the ECCV2018 conference, and is a lightweight neural network, which may be used as a terminal graphical code classification model. Without loss of generality, other models can be adopted, for example, high-precision domain image classification models such as snet and eficientnet series can also be selected as server-side graph code classification models.

After the preset graphic code classification model is trained, in a model application stage, a recognition result of whether the target graphic code is damaged or not can be obtained according to the image of the target graphic code and the preset graphic code classification model. If the input data of the graphic code classification model is an image of a two-dimensional code during training, the image of the target graphic code can be directly input into the graphic code classification model in the application stage, and the recognition result of whether the target graphic code is damaged or not is output; if the graphic code classification model preprocesses the image including the two-dimensional code during training, the same preprocessing process can be performed on the image of the target graphic code in the application stage, and then the image of the target graphic code is input into the graphic code classification model, and the recognition result of whether the target graphic code is damaged or not is output.

As an alternative embodiment, as shown in fig. 4, the step S203 of obtaining a recognition result of whether the target graphic code is damaged according to the image of the target graphic code and a preset graphic code classification model includes:

s2031, preprocessing the image of the target graphic code to obtain second pixel matrix data corresponding to the image of the target graphic code;

s2032, inputting the second pixel matrix data into the graphic code classification model, and acquiring a classification result and a corresponding confidence coefficient of the target graphic code;

s2033, acquiring an identification result of whether the target graphic code is damaged or not according to the classification result of the target graphic code and the corresponding confidence coefficient.

In this embodiment, the image of the target graphic code is preprocessed, for example, values of RGB channels of the image of the target graphic code are converted into pixel matrix data; optionally, the pixel matrix data may be normalized, and the normalized value is between 0 and 1, and accordingly, the same normalization processing is required during model training.

Further, the second pixel matrix data is input into the graphic code classification model, a classification result and a corresponding confidence coefficient of the target graphic code may be obtained, for example, the confidence coefficient corresponding to the damage of the target graphic code is 60%, the confidence coefficient corresponding to the undamaged target graphic code is 40%, and then, according to the classification result and the corresponding confidence coefficient of the target graphic code, an identification result of whether the target graphic code is damaged or not is obtained, for example, a second confidence coefficient threshold value may be set, for example, 50%, or an appropriate value between 0 and 1.0 is set according to the performance of the model on the test set, and then, if the confidence coefficient corresponding to the damage of the target graphic code is higher than the second confidence coefficient threshold value, the target graphic code is determined to be damaged.

In the method for identifying the graphic code damage provided by the embodiment, after the terminal device fails to scan the target graphic code, a first image acquired by the terminal device is acquired; the first image comprises a target graphic code; acquiring position information of the target graphic code in the first image according to the first image and a preset graphic code identification model; acquiring an image of the target graphic code from the first image according to the position information; and acquiring a recognition result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model. According to the embodiment, the position of the target graphic code in the first image can be determined through the graphic code recognition model, the image of the target graphic code can be accurately obtained, the recognition result of whether the target graphic code is damaged or not is determined through the graphic code classification model, the recognition accuracy and the recognition efficiency are high, the recognition of the graphic codes in batches can be realized, and the labor cost is reduced.

In an optional embodiment, as shown in fig. 5, on the basis of any one of the foregoing embodiments, if the method is applied to the terminal device, the method further includes:

s301, if the identification result is that the target graphic code is damaged, acquiring positioning information when the terminal device scans the target graphic code and a surrounding environment image of the device where the target graphic code is located;

s302, sending the positioning information, the surrounding environment image and the identification result to a server so that the server sends a maintenance instruction to a maintenance personnel terminal, wherein the maintenance instruction comprises the positioning information and the surrounding environment image so that the maintenance personnel can position the target graphic code according to the positioning information and the surrounding environment image.

In this embodiment, when the method execution subject of the above embodiment is a terminal device, if the terminal device determines that the target graphic code is damaged, the terminal device may acquire positioning information when scanning the target graphic code, and may be used as positioning information of a device where the target graphic code is located, and may further acquire an image of a surrounding environment of the device where the target graphic code is located (certainly, the image of the surrounding environment may not be acquired), and further send the positioning information, the image of the surrounding environment, and the recognition result to a server, the server may send a maintenance instruction to a maintenance staff terminal according to the information, carry the positioning information and the image of the surrounding environment in the maintenance instruction, and the maintenance staff may reach the positioning location according to the positioning information, and then find the target graphic code by combining with the image of the surrounding environment, and further maintain the damaged target graphic code, for example, a correction mark, the method has the advantages that stains are cleaned, damaged and torn graphic codes are replaced, accuracy and efficiency of graphic code maintenance can be improved, and maintenance cost is reduced.

In another alternative embodiment, as shown in fig. 6, on the basis of any one of the above embodiments, if the method is applied to the server, the method further includes:

s401, if the identification result is that the target graphic code is damaged, acquiring positioning information when the terminal device scans the target graphic code and a surrounding environment image of the device where the target graphic code is located;

s402, sending a maintenance instruction to a maintenance personnel terminal, wherein the maintenance instruction comprises the positioning information and the surrounding environment image, so that the maintenance personnel can position the target graphic code according to the positioning information and the surrounding environment image.

In this embodiment, when the method execution subject of the above embodiment is a server, if it is determined that the target graphic code in the first image acquired by the terminal device is damaged, the server may request, from the terminal device, the positioning information when the terminal device scans the target graphic code and the image of the surrounding environment of the device where the target graphic code is located, and the terminal device returns the positioning information and the image of the surrounding environment after receiving the request (of course, the image of the surrounding environment may not be necessary); or after the terminal device fails to scan the target graphic code, the first image is sent to the server, and meanwhile, the positioning information when the terminal device scans the target graphic code and the surrounding environment image of the device where the target graphic code is located can also be sent (of course, the surrounding environment image is not necessary).

Furthermore, the server can send a maintenance instruction to a maintenance personnel terminal according to the information, the positioning information and the surrounding environment image are carried in the maintenance instruction, the maintenance personnel can reach the positioning position according to the positioning information, and then find the target graphic code by combining the surrounding environment image, so that the damaged target graphic code is maintained, for example, correction marks and stains are cleaned, damaged and torn graphic codes are replaced, and the like, so that the accuracy and efficiency of graphic code maintenance can be improved, and the maintenance cost is reduced.

On the basis of any of the above embodiments, as shown in fig. 7, the method further includes a training process for the pattern code recognition model, specifically including:

s501, obtaining a first image set, wherein each image in the first image set comprises a graphic code and is marked with a graphic code position;

s502, determining a training set, a verification set and a test set according to the first image set;

s503, training, verifying and testing the initial pattern code recognition model according to the training set, the verifying set and the testing set respectively to finally obtain the pattern code recognition model.

In this embodiment, a first image set is obtained as a basis of a training set, a verification set and a test set of a pattern code recognition model, where each image in the first image set is an image including a pattern code and labeled with a position of the pattern code, and the labeled position of the pattern code may be labeled manually or by other means. And then the first image set can be divided into a training set, a verification set and a test set according to a certain proportion.

In this embodiment, model parameter initialization may be performed for the pattern code recognition model based on a thundernet-snet146 sub-model pre-trained on an MS-COCO (target detection authority data set), then migration training may be performed using the training set, and the model and the training parameters may be optimized by observing the change of the accuracy and the loss function of the pattern code recognition model in the verification set and the test set. Wherein the specific training, validation and testing may not be limiting herein.

On the basis of any of the above embodiments, as shown in fig. 8, the method further includes a training process of the graphic code classification model, which specifically includes:

s601, acquiring a second image set, wherein the second image set comprises graphic code images marked whether to be damaged or not;

s602, determining a training set, a verification set and a test set according to the second image set;

and S603, training, verifying and testing the initial graphic code classification model according to the training set, the verifying set and the testing set respectively to finally obtain the graphic code classification model.

In this embodiment, a second image set is obtained as a basis of a training set, a verification set and a test set of a graphic code classification model, where each image in the second image set is a graphic code image marked whether to be damaged or not, and whether to be damaged or not can be marked manually or by other ways. And then the second image set can be divided into a training set, a verification set and a test set according to a certain proportion.

In this embodiment, model parameters can be initialized for the graphic code classification model based on the shufflenetv2-1.0 sub-model pre-trained on the Imagenet-1k image classification data set, then migration training is performed by using the training set, and the model and the training parameters are optimized by observing the accuracy and loss function changes of the graphic code classification model on the verification set and the test set. Wherein the specific training, validation and testing may not be limiting herein.

In consideration of the fact that the graphic code image samples are seriously unbalanced in a real application scene, damaged graphic code image samples in all graphic code images are far lower than undamaged graphic code image samples, the number of the damaged graphic code image samples is difficult to meet the requirement of ensuring the robustness of a graphic code classification model, and when the real samples are used for training the graphic code classification model, heavy sample cleaning and labeling work is introduced, and the model has overfitting risks. Therefore, in this embodiment, virtual graphic code images may be generated in batch based on a synthesis technology, so that the graphic code images marked as damaged in the second image set include: a real damaged graphical code image and a composite damaged graphical code image.

The acquisition process of the synthesized damaged graphic code image is as follows:

acquiring a damaged material image; and carrying out image synthesis on the damaged material image and the undamaged graphic code image to obtain the synthesized damaged graphic code image.

In this embodiment, some images of the alteration, damage, and tear traces may be obtained as damaged material images, and then the damaged material images are used for image synthesis of undamaged graphic code images to obtain the synthesized damaged graphic code images. The position of the damaged material image in the undamaged graphic code image may be any position, which is not limited herein.

Further, in order to ensure the authenticity of the synthesized damaged graphic code image, the image synthesis result of the damaged material image and the undamaged graphic code image can be subjected to image processing according to a preset algorithm to obtain the synthesized damaged graphic code image, wherein the preset algorithm comprises at least one of the following: a motion blur algorithm, a brightness adjustment algorithm, a noise algorithm. Through the process, the synthesized sample can be fitted with the graphic code form which can appear in a real scene as much as possible.

Certainly, the undamaged graphic code image in the above embodiment may not be the actually acquired graphic code image, or some graphic code images may be generated by performing image processing through at least one preset algorithm such as a motion blur algorithm, a brightness adjustment algorithm, a noise algorithm, and the like, so that the cleaning and labeling work on the actually acquired graphic code image can be effectively reduced.

In an alternative embodiment, the specific process of synthesizing the damaged graphical code image may be as follows:

acquiring any connected domain binary image from a preset connected domain binary image set;

acquiring any damaged material image from a preset damaged material image set;

multiplying the connected domain binary image with the damaged material image to obtain a damaged noise image;

and carrying out image fusion on the damaged noise image and the undamaged noise image to obtain the synthesized damaged graphic code image.

In this embodiment, a connected component binary map may be generated in advance. Specifically, a connected domain binary graph can be randomly generated, an implementation manner is not limited, and the purpose is to generate a connected domain in any shape, and in order to simulate various damaged graph code regions in a real scene, parameters such as the number of vertices and an area threshold need to be set to ensure the rationality of the damaged graph code regions.

The generated connected domain binary images are all normalized to be the same size as the undamaged graphic code image (normal graphic code image), and the number of the top points can be set to be a positive integer larger than 8 so as to ensure the diversity of the generated connected domain binary images; the area of the connected region can be set with two thresholds to control the proportion of the damaged region occupying the whole graphic code region: the low threshold may be set to 0.1 and the high threshold may be set to 0.8. The set of connected component binary images can be as follows in FIG. 9 a.

In addition, the damaged material images similar to the damaged graphic code area in shape can be collected and sorted in advance.

Through statistical analysis of the collected damage graph code data, the damage graph code of real damage mainly includes but is not limited to the following: rust, dirt, dust, abrasion, correction, etc. Therefore, corresponding damaged material images are classified and collected, and the damaged material images are all normalized to be the same size as the undamaged graphic code image (normal graphic code image), such that an earth damaged material image set is shown in the following fig. 9b, and a rust damaged material image set is shown in fig. 9 c.

And further, extracting the undamaged graphic code image data. Since the undamaged graphic code image (normal graphic code image) is relatively large in data and easy to collect, the undamaged graphic code image (normal graphic code image) can be normalized to a fixed preset size (including but not limited to 224 × 224), while the connected domain binary image and the damaged material image need to be normalized to the preset size, and the undamaged graphic code image set is shown in fig. 9 d.

Finally, when synthesizing the damaged graphic code image, the damaged graphic code image is synthesized in batch based on the connected domain binary image, the damaged material image and the undamaged graphic code image (normal graphic code image) by using a preset image synthesis algorithm, wherein the preset image synthesis algorithm can be Alpha-Blending, the Alpha-Blending is an image processing technology for mixing source pixels and target pixels according to the value of an ' Alpha ' mixing vector, for each connected domain binary image, the damaged material image is randomly extracted from various damaged material images, the connected domain binary image is multiplied by the selected damaged material image to obtain a damaged noise image (damaged graphic code noise), and the damaged noise image and the undamaged graphic code image (normal graphic code image) are subjected to Alpha-Blending image fusion operation, namely, the value of any pixel in the target damaged graphic code image is the value of the pixel in the undamaged graphic code image + (1-Alpha) ' damaged graphic code image + The value of this pixel in the acoustic image, resulting in a corrupted graphic code image, as shown in fig. 9 e.

Based on the above embodiment, the pattern code image samples (including damaged pattern code images and/or undamaged pattern code images) synthesized in batches can be combined with the pattern code image samples (including damaged pattern code images and/or undamaged pattern code images) collected in the real scene according to a certain proportion to construct a training set, and the remaining pattern code image samples (including damaged pattern code images and undamaged pattern code images) collected in the real samples can be used as a verification set and a test set. Of course, other strategies may be used to partition the training set, validation set, and test set. The graph code classification model obtained based on the training of the synthetic data and the real data has the advantages that the cross test result on the test set constructed based on the real data is superior to that of the graph code classification model obtained based on the training of the real data, and the overfitting risk of the model is reduced. Meanwhile, as the label of the synthetic data can be automatically generated during synthesis, the introduction of the synthetic data does not bring any manual labeling cost. The loss curve on the verification set of the graphic code classification model can be seen in fig. 10.

The embodiment supplementarily constructs the training sample set for model training based on the image batch synthesis technology, so that the overfitting risk of the model can be reduced, the convergence speed of the model can be increased, heavy manual labeling is not needed for synthesizing the sample label, the problems of unbalanced sample and too few damaged graphic code image samples caused by too low damaged graphic code image proportion in a real sample are solved, the convergence speed of the model is increased by displaying the loss curve of the model on the verification set, the loss function oscillation existing only in the training of the real sample is relieved, and meanwhile, the accuracy rate and the robustness of the model are increased. The synthesized samples can be used only for sample supplementation of the training set, and the test set can still be derived from the real samples, so as to ensure the reliability of the model cross-validation result in the real application scene.

It should be noted that any of the above model training processes may be executed on a terminal device or a server, or may also be executed on any other electronic device, and the trained model is deployed on the terminal device or the server.

Corresponding to the method for identifying damaged graphic codes in the foregoing embodiments, fig. 11 is a block diagram of a structure of an apparatus for identifying damaged graphic codes according to an embodiment of the present disclosure. For ease of illustration, only portions that are relevant to embodiments of the present disclosure are shown. Referring to fig. 11, the apparatus 700 for identifying a damage of a graphic code includes: an acquisition module 701, an identification module 702, a matting module 703, and a classification module 704.

The acquisition module 701 is used for acquiring a first image acquired by a terminal device after the terminal device fails to scan a target graphic code; the first image comprises a target graphic code;

the identification module 702 is configured to obtain position information of the target pattern code in the first image according to the first image and a preset pattern code identification model;

a matting module 703 configured to obtain an image of the target graphics code from the first image according to the location information;

and the classification module 704 is configured to obtain an identification result of whether the target graphical code is damaged according to the image of the target graphical code and a preset graphical code classification model.

On the basis of any of the above embodiments, when the identification module 702 obtains the position information of the target pattern code in the first image according to the first image and a preset pattern code identification model, it is configured to:

preprocessing the first image to acquire first pixel matrix data corresponding to the first image;

inputting the first pixel matrix data into the graphic code identification model, and acquiring a predicted position and a corresponding confidence coefficient of the target graphic code in the first image;

and finally determining the position information of the target graphic code in the first image according to the predicted position in the first image and the corresponding confidence coefficient.

On the basis of any of the above embodiments, when obtaining the recognition result of whether the target graphic code is damaged according to the image of the target graphic code and a preset graphic code classification model, the classification module 704 is configured to:

preprocessing the image of the target graphic code to acquire second pixel matrix data corresponding to the image of the target graphic code;

inputting the second pixel matrix data into the graphic code classification model, and acquiring a classification result and a corresponding confidence coefficient of the target graphic code;

and acquiring an identification result of whether the target graphic code is damaged or not according to the classification result of the target graphic code and the corresponding confidence coefficient.

On the basis of any of the above embodiments, optionally, the device is applied to the terminal device;

the obtaining module 701 is further configured to, if it is determined that the target graphic code is damaged according to the identification result, obtain positioning information obtained when the terminal device scans the target graphic code and an image of a surrounding environment of a device where the target graphic code is located;

the device further comprises a sending module configured to:

and sending the positioning information, the surrounding environment image and the identification result to a server so as to enable the server to send a maintenance instruction to a maintenance personnel terminal, wherein the maintenance instruction comprises the positioning information and the surrounding environment image so as to enable maintenance personnel to position the target graphic code according to the positioning information and the surrounding environment image.

On the basis of any of the above embodiments, optionally, the apparatus is applied to a server;

the device further comprises a sending module configured to:

and sending a maintenance instruction to a maintenance personnel terminal, wherein the maintenance instruction comprises the positioning information and the surrounding environment image, so that the maintenance personnel can position the target graphic code according to the positioning information and the surrounding environment image.

On the basis of any of the above embodiments, the apparatus further comprises a first training module configured to:

acquiring a first image set, wherein each image in the first image set comprises a graphic code and is marked with a graphic code position;

determining a training set, a verification set and a test set according to the first image set;

and respectively training, verifying and testing the initial pattern code recognition model according to the training set, the verifying set and the testing set to finally obtain the pattern code recognition model.

On the basis of any of the above embodiments, the apparatus further comprises a second training module configured to:

acquiring a second image set, wherein the second image set comprises a graphic code image marked whether to be damaged or not;

determining a training set, a verification set and a test set according to the second image set;

and respectively training, verifying and testing the initial graphic code classification model according to the training set, the verifying set and the testing set to finally obtain the graphic code classification model.

On the basis of any of the above embodiments, the graphic code image marked as damaged in the second image set includes: real damaged graphic code images and synthesized damaged graphic code images; the second training module is further to:

and acquiring a damaged material image, and carrying out image synthesis on the damaged material image and the undamaged graphic code image to obtain the synthesized damaged graphic code image.

On the basis of any of the above embodiments, the second training module, when acquiring a damaged material image, and performing image synthesis on the damaged material image and an undamaged graphic code image to obtain the synthesized damaged graphic code image, is configured to:

acquiring any damaged material image from a preset damaged material image set;

On the basis of any of the above embodiments, when the second training module performs image synthesis on the damaged material image and the undamaged graphic code image to obtain the synthesized damaged graphic code image, the second training module is further configured to:

and carrying out image processing on the result of image synthesis on the damaged material image and the undamaged graphic code image according to a preset algorithm to obtain the synthesized damaged graphic code image, wherein the preset algorithm comprises at least one of the following steps: a motion blur algorithm, a brightness adjustment algorithm, a noise algorithm.

The device for identifying a damaged graphic code provided in this embodiment may be used to implement the technical solution of the above method embodiment, and the implementation principle and technical effect of the device are similar, which is not described herein again.

In the identification device for the damage of the graphic code, after the terminal device fails to scan the target graphic code, a first image acquired by the terminal device is acquired; the first image comprises a target graphic code; acquiring position information of the target graphic code in the first image according to the first image and a preset graphic code identification model; acquiring an image of the target graphic code from the first image according to the position information; and acquiring a recognition result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model. According to the embodiment, the position of the target graphic code in the first image can be determined through the graphic code recognition model, the image of the target graphic code can be accurately obtained, the recognition result of whether the target graphic code is damaged or not is determined through the graphic code classification model, the recognition accuracy and the recognition efficiency are high, the recognition of the graphic codes in batches can be realized, and the labor cost is reduced.

Fig. 12 is a schematic structural diagram of a device for identifying a damaged graphic code according to an embodiment of the present invention. The identifying device for damaged graphic codes provided by the embodiment of the present invention may execute the processing flow provided by the identifying method for damaged graphic codes, as shown in fig. 12, the identifying device 80 for damaged graphic codes includes a memory 81 and a processor 82; wherein the computer program is stored in the memory 81 and configured to execute the method for identifying a damage of a graphic code described in the above embodiments by the processor 82. The device 80 for identifying graphic code corruption may also have a communication interface 83 for receiving and transmitting data and/or instructions.

The device for identifying damaged graphic codes in the embodiment shown in fig. 12 may be used to implement the technical solution of the above-mentioned method for identifying damaged graphic codes, and the implementation principle and technical effect are similar, which are not described herein again.

In an exemplary embodiment, a non-transitory computer readable storage medium including instructions, such as a memory including instructions, executable by a processor to perform the above method of identifying graphic code corruption is also provided. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

In an exemplary embodiment, a computer program product is also provided, which comprises a computer program, and the computer program is executed by a processor to perform the above method for identifying a damaged graphic code.

The present application further provides the following embodiments:

the embodiment 1, a method for identifying graphic code damage, comprising:

Embodiment 2 and the method according to embodiment 1, where the obtaining of the position information of the target pattern code in the first image according to the first image and a preset pattern code recognition model includes:

Embodiment 3 and the method according to embodiment 1, wherein the obtaining of the recognition result of whether the target graphic code is damaged according to the image of the target graphic code and a preset graphic code classification model includes:

Embodiment 4, the method according to any of embodiments 1-3, the method is applied to the terminal device; the method further comprises the following steps:

if the identification result is that the target graphic code is damaged, acquiring positioning information when the terminal equipment scans the target graphic code and a peripheral environment image of equipment where the target graphic code is located;

Embodiment 5, the method according to any of embodiments 1-3, applied to a server; the method further comprises the following steps:

Embodiment 6, the method of any of embodiments 1-5, further comprising:

Embodiment 7, the method of any of embodiments 1-6, further comprising:

Embodiment 8, according to the method of embodiment 7, the graphic code image marked as damaged in the second image set includes: real damaged graphic code images and synthesized damaged graphic code images; wherein the method further comprises:

acquiring a damaged material image;

and carrying out image synthesis on the damaged material image and the undamaged graphic code image to obtain the synthesized damaged graphic code image.

Embodiment 9 and according to the method of embodiment 8, the obtaining a damaged material image, and performing image synthesis on the damaged material image and an undamaged graphic code image to obtain the synthesized damaged graphic code image includes:

acquiring any damaged material image from a preset damaged material image set;

Embodiment 10, according to the method of

embodiment

8 or 9, the image synthesizing the damaged material image and the undamaged graphic code image to obtain the synthesized damaged graphic code image, further includes:

Embodiment 11, an apparatus for identifying a damage to a graphic code, comprising:

Embodiment 12 and the apparatus according to embodiment 11, wherein the identification module, when acquiring the position information of the target pattern code in the first image according to the first image and a preset pattern code identification model, is configured to:

Embodiment 13 and the apparatus according to embodiment 11, wherein the classification module, when obtaining a recognition result of whether the target graphic code is damaged according to the image of the target graphic code and a preset graphic code classification model, is configured to:

Embodiment 14, the apparatus according to any of embodiments 11-13, the apparatus is applied to the terminal device;

the acquisition module is further configured to acquire, if it is determined that the target graphic code is damaged as a result of the identification, positioning information obtained when the terminal device scans the target graphic code and an image of a surrounding environment of a device in which the target graphic code is located;

the device further comprises a sending module configured to:

Embodiment 15, the apparatus of any of embodiments 11-13, applied to a server;

the device further comprises a sending module configured to:

Embodiment 16, the apparatus of any of embodiments 11-15, further comprising a first training module to:

Embodiment 17, the apparatus of any of embodiments 11-16, further comprising a second training module to:

Embodiment 18, according to the apparatus of embodiment 17, the graphic code image marked as damaged in the second image set includes: real damaged graphic code images and synthesized damaged graphic code images; the second training module is further to:

Embodiment 19 and according to the apparatus of embodiment 18, when the second training module obtains a damaged material image, and performs image synthesis on the damaged material image and an undamaged graphics code image to obtain the synthesized damaged graphics code image, the second training module is configured to:

acquiring any damaged material image from a preset damaged material image set;

Embodiment 20, according to the apparatus of

embodiment

18 or 19, when the second training module performs image synthesis on the damaged material image and the undamaged graphic code image to obtain the synthesized damaged graphic code image, the second training module is further configured to:

Embodiment 21, an apparatus for recognizing damage to a graphic code, comprising: a memory and a processor;

the memory is to store program instructions;

the processor is configured to invoke program instructions in the memory to perform the method of any of embodiments 1-10.

Embodiment 22, a computer-readable storage medium having a computer program stored thereon; when executed, implement the method of any of embodiments 1-10.

Embodiment 23, a computer program product comprising a computer program which, when executed by a processor, performs the steps of the method of any of embodiments 1-10.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. The embodiments of the disclosure are intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for identifying damage of a graphic code is characterized by comprising the following steps:

2. The method according to claim 1, wherein the obtaining the position information of the target graphic code in the first image according to the first image and a preset graphic code recognition model comprises:

3. The method according to claim 1, wherein the obtaining of the recognition result of whether the target graphic code is damaged according to the image of the target graphic code and a preset graphic code classification model comprises:

4. The method according to any of claims 1-3, wherein the method is applied to the terminal device; the method further comprises the following steps:

5. The method according to any one of claims 1-3, wherein the method is applied to a server; the method further comprises the following steps:

6. The method according to any one of claims 1 to 3, wherein the pattern code recognition model is trained by:

7. An apparatus for recognizing damage of a graphic code, comprising:

8. An apparatus for recognizing damage of a graphic code, comprising: a memory and a processor;

the memory is to store program instructions;

the processor is configured to invoke program instructions in the memory to perform the method of any of claims 1-6.

9. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a computer program; the computer program, when executed, implementing the method of any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the method of any one of claims 1-6 when executed by a processor.