CN112989864B

CN112989864B - Method, device, storage medium, and program product for identifying graphic code corruption

Info

Publication number: CN112989864B
Application number: CN202110267584.9A
Authority: CN
Inventors: 张锦涛
Original assignee: Beijing Qisheng Technology Co Ltd
Current assignee: Beijing Qisheng Technology Co Ltd
Priority date: 2021-03-11
Filing date: 2021-03-11
Publication date: 2024-06-07
Anticipated expiration: 2041-03-11
Also published as: CN112989864A

Abstract

The embodiment of the disclosure provides a method, equipment, a storage medium and a program product for identifying graphic code damage, wherein after a terminal device fails to scan a target graphic code, a first image acquired by the terminal device is acquired, and the first image comprises the target graphic code; acquiring the position information of a target graphic code in a first image according to the first image and a preset graphic code identification model; acquiring an image of a target graphic code from the first image according to the position information; and acquiring the recognition result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model. According to the embodiment of the disclosure, the position of the target graphic code in the first image can be determined through the graphic code identification model, so that the image of the target graphic code can be accurately obtained, the identification result of whether the target graphic code is damaged or not is determined through the graphic code classification model, the identification accuracy and efficiency are high, the identification of batch graphic codes can be realized, and the labor cost is reduced.

Description

Method, device, storage medium, and program product for identifying graphic code corruption

Technical Field

Embodiments of the present disclosure relate to the field of internet and artificial intelligence technologies, and in particular, to a method, a device, a storage medium, and a program product for identifying graphic code damage.

Background

Graphic codes such as bar codes and two-dimensional codes are commonly existing in various daily scenes of the current society, such as code scanning payment in a consumption scene, code scanning and pickup in an express scene, code scanning and unlocking in a sharing bicycle scene, and the like.

However, the bar code, the two-dimensional code, and the like may be damaged due to various factors, such as correction, breakage, tearing, and the like, so that the graphic code may not be recognized. In the prior art, after the user fails to identify the graphic code after scanning the graphic code, an operation and maintenance person is usually required to judge whether the graphic code is damaged in a manual checking mode, and then the damaged graphic code is maintained.

In the prior art, whether the graphic codes are damaged is judged manually, so that the labor cost is increased, the judging efficiency is low, and the judgment of whether the massive graphic codes are damaged cannot be dealt with.

Disclosure of Invention

The embodiment of the disclosure provides a method, equipment, a storage medium and a program product for identifying graphic code damage, which are used for reducing the cost of a graphic code damage identification process, improving the identification efficiency and judging whether massive graphic codes are damaged or not by a user.

In a first aspect, an embodiment of the present disclosure provides a method for identifying damage to a graphic code, including:

after the terminal equipment fails to scan the target graphic code, acquiring a first image acquired by the terminal equipment; the first image includes a target graphic code;

acquiring the position information of the target graphic code in the first image according to the first image and a preset graphic code identification model;

acquiring an image of the target graphic code from the first image according to the position information;

And acquiring a recognition result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model.

In a second aspect, embodiments of the present disclosure provide an identification device of graphic code corruption, comprising:

the acquisition module is used for acquiring a first image acquired by the terminal equipment after the terminal equipment fails to scan the target graphic code; the first image includes a target graphic code;

The identification module is used for acquiring the position information of the target graphic code in the first image according to the first image and a preset graphic code identification model;

the matting module is used for acquiring the image of the target graphic code from the first image according to the position information;

and the classification module is used for acquiring the identification result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model.

In a third aspect, embodiments of the present disclosure provide an identification apparatus of graphic code corruption, comprising: a memory and a processor;

The memory is used for storing program instructions;

the processor is configured to invoke program instructions in the memory to perform the method of identifying a corruption of a graphic code as described in the first aspect.

In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon; the computer program, when executed, implements the method for identifying graphic code corruption as described in the first aspect.

In a fifth aspect, embodiments of the present disclosure provide a computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the method for identifying graphic code corruption according to the first aspect.

The method, the device, the storage medium and the program product for identifying the damage of the graphic code, provided by the embodiment of the disclosure, acquire a first image acquired by the terminal device after the terminal device fails to scan the target graphic code; the first image includes a target graphic code; acquiring the position information of the target graphic code in the first image according to the first image and a preset graphic code identification model; acquiring an image of the target graphic code from the first image according to the position information; and acquiring a recognition result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model. According to the embodiment of the disclosure, the position of the target graphic code in the first image can be determined through the graphic code identification model, so that the image of the target graphic code can be accurately obtained, the identification result of whether the target graphic code is damaged or not is determined through the graphic code classification model, the identification accuracy and efficiency are high, the identification of batch graphic codes can be realized, and the labor cost is reduced.

Various possible embodiments of the present disclosure and their technical advantages are described in detail below.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure.

FIG. 1 is a schematic diagram of a shared vehicle system provided in an embodiment of the present disclosure;

FIG. 2 is a flow chart illustrating a method for identifying graphic code corruption according to an embodiment of the disclosure;

FIG. 3 is a flowchart illustrating a method for identifying graphic code corruption according to another embodiment of the present disclosure;

FIG. 4 is a flowchart illustrating a method for identifying graphic code corruption according to another embodiment of the present disclosure;

FIG. 5 is a flowchart illustrating a method for identifying graphic code corruption according to another embodiment of the present disclosure;

FIG. 6 is a flowchart illustrating a method for identifying graphic code corruption according to another embodiment of the present disclosure;

FIG. 7 is a flowchart illustrating a method for identifying graphic code corruption according to another embodiment of the present disclosure;

FIG. 8 is a flowchart illustrating a method for identifying graphic code corruption according to another embodiment of the present disclosure;

FIG. 9a is a schematic diagram of connected domain binary image set according to an embodiment of the disclosure;

FIG. 9b is a schematic view of a set of images of soil damage material provided in an embodiment of the present disclosure;

FIG. 9c is a schematic view of a rust damaged material image set provided by an embodiment of the disclosure;

FIG. 9d is a schematic diagram of a synthesized defect graphical code image provided by an embodiment of the present disclosure;

FIG. 10 is a graph illustrating the loss curve from the verification set provided by an embodiment of the present disclosure;

FIG. 11 is a schematic diagram of a graphic code damage identification device according to an embodiment of the present disclosure;

Fig. 12 is a schematic structural diagram of an identifying device for graphic code damage according to another embodiment of the present disclosure.

Specific embodiments of the present disclosure have been shown by way of the above drawings and will be described in more detail below. These drawings and the written description are not intended to limit the scope of the disclosed concepts in any way, but rather to illustrate the disclosed concepts to those skilled in the art by reference to specific embodiments.

Detailed Description

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.

In the prior art, an image acquisition device or a photoelectric scanning device is generally used to detect a graphic code such as a bar code or a two-dimensional code, and then identify content information contained in the graphic code. However, if the bar code, the two-dimensional code and other graphic codes are damaged by correction, breakage, tearing and the like, the scanning failure probability is greatly improved, so that the graphic codes can not be identified, and the user experience is directly affected. In the prior art, for damaged graphic codes such as correction, breakage, tearing and the like, an operation and maintenance person is usually required to judge whether the graphic code is damaged in a manual checking mode, for example, after a user scans the graphic code and fails to identify the graphic code, the operation and maintenance person can confirm whether the graphic code is damaged or not because of failure in shooting due to the fact that the graphic code cannot be identified caused by the fact that the related reasons such as incomplete shooting are eliminated by naked eyes after the user scans the graphic code, and after the graphic code is confirmed to be damaged, the graphic code is maintained, for example, the graphic code is cleaned or a new graphic code is replaced.

The method has the advantages that whether the graphic codes are damaged is judged manually, the labor cost is increased, the operation and maintenance cost is increased, the judging efficiency is low, and the labor cost is increased sharply when the massive graphic codes are faced, so that the judgment of whether the graphic codes are damaged or not can not be carried out by the manual judgment.

With the successful landing of the deep neural network represented by the convolutional neural network in the fields of computer vision and other applications, in the embodiment of the disclosure, the detection method of the target and key point detection model in the neural network can be applied to locate the graphic code, and the classification method of the classification model in the neural network is also adopted to classify whether the graphic code is damaged or not, which can also adopt other learning models. Whether the graphic code which fails to be scanned is damaged or not can be identified based on the trained model, and specifically, after the terminal equipment fails to scan the target graphic code, a first image which is acquired by the terminal equipment and comprises the target graphic code is acquired; acquiring the position information of a target graphic code in a first image according to the first image and a preset graphic code identification model; acquiring an image of a target graphic code from the first image according to the position information; and acquiring the recognition result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model. By applying the neural network and the computer vision technology, whether the graphic codes are damaged or not can be identified, the accuracy and the efficiency of identification results are high, the maintenance of the batch graphic codes can be realized, and the labor cost is reduced.

The identification method for the damage of the graphic code is suitable for application scenes of graphic codes such as any bar code, two-dimensional code and the like, for example, scanning code payment in a consumption scene, scanning code picking in an express scene, scanning code unlocking in a shared bicycle scene and the like. The application scenes at least include a server and a terminal device, and the graphic codes such as bar codes and two-dimensional codes are arranged on specific objects in the application scenes, and as shown in fig. 1, the shared vehicle system includes: server 110, terminal device 120, and shared vehicle 130. Other application scenarios of bar codes, two-dimensional codes and other graphic codes are similar to the shared bicycle system, and are not exemplified here.

The server 110 provides a service point for processing, database, communication facilities. Server 110 may be a monolithic server or a distributed server across multiple computers or computer data centers. The server 110 may be of various types such as, but not limited to, a web server, news server, mail server, message server, advertisement server, file server, application server, interaction server, database server, or proxy server. In some embodiments, each server may include hardware, software, or embedded logic components or a combination of two or more such components for performing the appropriate functions supported by or implemented by the server. For example, a server, such as a blade server, cloud server, etc., or may be a server group consisting of multiple servers, may include one or more of the types of servers described above, etc.

The terminal device 120 is, for example, a smart phone, a portable computer, a tablet computer, a palm computer, a wearable device, a virtual reality device, an augmented reality device, etc., or any combination thereof, and is not limited herein. Fig. 1 illustrates a terminal device 120 as a smart phone.

The shared vehicle 130 is not limited herein, and is, for example, a shared electric bicycle, a shared tricycle, a shared electric power assisted vehicle, a shared motorcycle, a shared four-wheel passenger vehicle, or the like. Fig. 1 illustrates a shared bicycle as the shared vehicle 130. The shared vehicle 130 is provided with a lock mechanism 140, and the lock mechanism 140 is used for locking or unlocking the shared vehicle 130. The lock mechanism 140 may be a mechanical lock or an electronic lock. The shared vehicle 130 and the lock mechanism 140 may be components that are mechanically coupled to each other. For example, the shared vehicle 130 and the lock mechanism 140 may be separate components from each other, and the lock mechanism 140 may be mounted on the shared vehicle 130. Alternatively, the shared vehicle 130 and the lock mechanism 140 may be integrated as a single piece.

Terminal device 120 may communicate with server 110 over a wireless or wired network. The wireless network may be a communication network such as 2G, 3G, 4G, or 5G, or a wireless lan, which is not limited herein. The terminal device 120 and the shared vehicle 130 and/or the lock mechanism 140 may communicate with each other through a short-range communication method, which may be bluetooth or wifi. Communication between the sharing vehicle 130 and/or the lock mechanism 140 and the server 110 may be via a wireless network.

The terminal device 120 may send a request associated with the shared vehicle 130, such as a service request of the shared vehicle 130, a vehicle-using request, a vehicle-ending request, etc., to the server 130. For the shared vehicle 130, the vehicle request carries a unique identification of the shared vehicle or the lock mechanism. The unique identification may include a bar code, a Quick Response (QR) code, a serial number including letters and/or numbers, etc., or any combination thereof. For example, the vehicle-using request may be triggered by the user scanning the QR code of the shared vehicle 130 or the lock mechanism 140 through the camera of the terminal device, or may be triggered by the terminal device 120 inputting a unique identifier.

The service request may include information related to riding, including one or a combination of several of a shared vehicle type, a departure location, a destination, a riding time, a riding mileage, a route, and the like. The service request may also include information related to the user (e.g., user account information) and/or the terminal device 120 (e.g., the location of the terminal device 120).

Server 110 may also send information to terminal device 120, shared vehicle 130, and/or lock mechanism 140. For example, the server 110 may send at least one of the following to the sharing vehicle 140 and/or the lock mechanism 140: instructions to lock and unlock the shared vehicle 130, information about the shared vehicle 130 (e.g., information prompting whether the shared vehicle 130 is locked). For example, the server 110 may send at least one of the following to the terminal device 120: information indicating whether the shared vehicle 130 is available, information indicating whether the shared vehicle 130 is allowed to lock at the current location).

The shared vehicle 130 and/or the lock mechanism 140 may transmit status information to the server 110, the terminal device 120, respectively, which may include one or more of a location of the shared vehicle 130, a locked/unlocked status of the shared vehicle 130, battery power of the shared vehicle 130, operation information, and the like. In some embodiments, the shared vehicle 130 and/or the lock mechanism 140 may receive instructions (e.g., instructions to lock/unlock the shared vehicle 130) from the terminal device 120 and/or the server 110.

It should be understood that although fig. 1 shows one server 110, one terminal device 120, one shared vehicle 130, it is not meant to limit the respective numbers, and that a plurality of servers 110, a plurality of terminal devices 120, a plurality of shared vehicles 130, and the like may be included in the shared vehicle system.

The method for identifying graphic code damage provided in the embodiments of the present disclosure may be executed by the terminal device 120 or the service 110.

In an alternative embodiment, the execution main body of the method for identifying graphic code damage provided in the embodiment of the present disclosure is a terminal device 120, and then the terminal device 120 is pre-deployed with a graphic code identification model and a graphic code classification model, when a user fails to scan a target graphic code (QR code) of a shared vehicle 130 or a lock mechanism 140 through the terminal device 120, the terminal device 120 may acquire, according to a first image including the target graphic code and a preset graphic code identification model acquired by the user, position information of the target graphic code in the first image; then, acquiring an image of the target graphic code from the first image according to the position information; and acquiring the recognition result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model. Further, the terminal device 120 may report the identification result to the server 110, and may report positioning information (such as GPS positioning information) and a surrounding image of the device where the target graphic code is located when the terminal device 120 scans the target graphic code, where the server 110 may send a maintenance instruction to a maintainer terminal, where the maintenance instruction includes the positioning information and the surrounding image, so that the maintainer positions the target graphic code according to the positioning information and the surrounding image, so as to maintain the target graphic code.

In another optional embodiment, the execution subject of the method for identifying a damaged graphic code provided in the embodiment of the present disclosure is a server 110, where a graphic code identification model and a graphic code classification model are pre-deployed on the server 110, and when a user fails to scan a target graphic code (QR code) of a shared vehicle 130 or a lock mechanism 140 through a terminal device 120, the terminal device 120 sends a first image including the target graphic code acquired by the user to the server 110, and the server 110 may obtain, according to the first image including the target graphic code and a preset graphic code identification model, location information of the target graphic code in the first image; then, acquiring an image of the target graphic code from the first image according to the position information; and acquiring the recognition result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model. Further, if the server 110 determines that the identification result is that the target graphic code is damaged, the terminal device 120 may request the terminal device 120 for the positioning information and the surrounding image of the device where the target graphic code is located when the terminal device 120 scans the target graphic code, or the terminal device 120 sends the positioning information and the surrounding image of the device where the target graphic code is located to the server 110 while sending the first image; further, the server 110 may send a maintenance instruction to a maintenance personnel terminal, where the maintenance instruction includes the positioning information and the surrounding environment image, so that the maintenance personnel positions the target graphic code according to the positioning information and the surrounding environment image to maintain the target graphic code.

The following describes in detail, with specific embodiments, a technical solution of an embodiment of the present disclosure and how the technical solution of the present disclosure solves the foregoing technical problems. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present disclosure will be described below with reference to the accompanying drawings.

Referring to fig. 2, fig. 2 is a flowchart illustrating a method for identifying graphic code corruption provided in an embodiment of the disclosure. The main execution body of the method for identifying the damage to the graphic code in this embodiment may be a terminal device, a server, or other electronic devices, and the method for identifying the damage to the graphic code in this embodiment includes:

S201, after a terminal device fails to scan a target graphic code, acquiring a first image acquired by the terminal device; the first image includes a target graphic code.

In this embodiment, the terminal device opens the camera, collects a first image including the target graphic code, scans the first image, and if the scanning of the target graphic code fails, the terminal device may continue to execute subsequent S202-S204 based on the first graphic; or the terminal equipment can also re-acquire a first image comprising the target graphic code and then continue to execute the following S202-S204; or the terminal device may send the first image to the server, and after the server acquires the first image, the subsequent S202-S204 may be continuously performed.

In this embodiment, the subsequent processes of S202-S204 are substantially similar to each other by the terminal device or server.

S202, acquiring the position information of the target graphic code in the first image according to the first image and a preset graphic code identification model.

In this embodiment, the preset pattern recognition model may be a pattern recognition model constructed based on a target detection model of a convolutional neural network. Optionally, in view of user experience and transmission traffic, the present embodiment may choose Thundernet model as a baseline version of the barcode recognition model, where Thundernet model is the first real-time Object Detection model (ThunderNet: towards Real-TIME GENERIC Object Detection) designed for embedded devices in the industry published by open-eye science and technology company (megvii) at ICCV conference 2009. Other models can be adopted without loss of generality, for example, a mobile terminal real-time target detection model such as mobilenet-ssd (lite) and pelee can be selected as a terminal graphic code detection model, or a high-precision target detection model such as ssd, yolo, efficientdet can be selected as a server-side graphic code detection model.

After training the preset graphic code recognition model, in a model application stage, the position information of the target graphic code in the first image can be obtained according to the first image and the preset graphic code recognition model. If the input data of the pattern code recognition model is an image comprising a two-dimensional code during training, the application stage can directly input a first image into the pattern code recognition model and output the position information of the target pattern code in the first image; if the pattern code recognition model performs preprocessing on the image including the two-dimensional code during training, the application stage may perform the same preprocessing process on the first image, input the same into the pattern code recognition model, and output the position information of the target pattern code in the first image.

As an optional embodiment, as shown in fig. 3, the step S202 of obtaining the position information of the target graphic code in the first image according to the first image and the preset graphic code recognition model may include:

s2021, preprocessing the first image to obtain first pixel matrix data corresponding to the first image.

S2022, inputting the first pixel matrix data into the graphic code identification model, and acquiring a predicted position and a corresponding confidence of the target graphic code in the first image.

S2023, according to the predicted position in the first image and the corresponding confidence, finally determining the position information of the target graphic code in the first image.

In this embodiment, the first image is preprocessed, for example, values of RGB channels of the first image are converted into pixel matrix data; optionally, the pixel matrix data can be normalized, the normalized value is between 0 and 1, and correspondingly, the same normalization process is needed during model training.

Further, the first pixel matrix data is input into the graphic code recognition model, so that a predicted position of the target graphic code in the first image and a corresponding confidence coefficient can be obtained, it is to be noted that more than one predicted position of the target graphic code in the first image may be obtained, screening or fusion may be performed through the corresponding confidence coefficient, for example, a first confidence coefficient threshold value, for example, 50% may be set, or a proper value between 0 and 1.0 may be set according to the performance of the model on the test set, and further, a predicted position with the confidence coefficient satisfying the first confidence coefficient threshold value is selected from a plurality of predicted positions, and is determined as the position information of the target graphic code in the first image; if there are multiple predicted locations for which the confidence meets the first confidence threshold, a final predicted location may be determined based on the predicted locations, for example, by averaging the predicted locations, or by other approaches, which may not be limiting herein.

S203, acquiring the image of the target graphic code from the first image according to the position information.

In this embodiment, after determining the position information of the target graphic code in the first image, the target graphic code may be scratched or cut based on the position information, to obtain an image including only the target graphic code. Of course, in this embodiment, further processing, such as scaling, rotation, flipping, affine transformation, etc., may be performed on the image of the target graphic code obtained by matting, which is not limited herein.

S204, acquiring an identification result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model.

In this embodiment, after the image of the target graphic code is obtained, whether the target graphic code is damaged or not may be classified based on the image of the target graphic code and a preset graphic code classification model, that is, the output class includes damage or non-damage of the target graphic code.

The preset graphic code classification model in the embodiment may be a graphic code classification model constructed based on an image classification model of a convolutional neural network. Optionally, in consideration of user experience and transmission traffic, the embodiment may select shufflenetv model as a graphic code classification model of a baseline version, where shufflenetv model is an image classification model (ShuffleNet V2: PRACTICAL GUIDELINES FOR EFFICIENT CNN Architecture Design) published by open-eye science and technology company (megvii) at ECCV2018 conference, which is a lightweight neural network and can be used as a terminal graphic code classification model. Other models can be adopted without loss of generality, for example, snet, efficientnet series of high-precision layout image classification models can be selected as the server-side graphic code classification model.

After training the preset graphic code classification model, in the model application stage, the recognition result of whether the target graphic code is damaged can be obtained according to the image of the target graphic code and the preset graphic code classification model. If the input data of the graphic code classification model is an image of a two-dimensional code during training, the application stage can directly input the image of the target graphic code into the graphic code classification model and output an identification result of whether the target graphic code is damaged or not; if the graphic code classification model performs preprocessing on the image comprising the two-dimensional code during training, the application stage can perform the same preprocessing process on the image of the target graphic code, input the same into the graphic code classification model and output the recognition result of whether the target graphic code is damaged.

As an optional embodiment, as shown in fig. 4, S203, according to the image of the target graphic code and a preset graphic code classification model, obtains a recognition result of whether the target graphic code is damaged, including:

S2031, preprocessing an image of the target graphic code to obtain second pixel matrix data corresponding to the image of the target graphic code;

s2032, inputting the second pixel matrix data into the graphic code classification model to acquire a classification result and a corresponding confidence coefficient of the target graphic code;

s2033, acquiring the identification result of whether the target graphic code is damaged or not according to the classification result and the corresponding confidence coefficient of the target graphic code.

In this embodiment, preprocessing is performed on the image of the target graphic code, for example, converting the values of RGB channels of the image of the target graphic code into pixel matrix data; optionally, the pixel matrix data can be normalized, the normalized value is between 0 and 1, and correspondingly, the same normalization process is needed during model training.

Further, the second pixel matrix data is input to the graphic code classification model, so that a classification result and corresponding confidence coefficient of the target graphic code can be obtained, for example, the classification result of the target graphic code is 60% of the confidence coefficient corresponding to damage, the classification result of the target graphic code is 40% of the confidence coefficient corresponding to uncorrupt, further, whether the identification result of the target graphic code is damaged or not is obtained according to the classification result and the corresponding confidence coefficient of the target graphic code, for example, a second confidence coefficient threshold value, for example, 50% can be set, or a proper value between 0 and 1.0 can be set according to the performance of the model on the test set, and further, if the classification result of the target graphic code is that the confidence coefficient corresponding to damage is higher than the second confidence coefficient threshold value, the damage of the target graphic code is determined.

According to the identification method for graphic code damage provided by the embodiment, after the terminal equipment fails to scan the target graphic code, a first image acquired by the terminal equipment is acquired; the first image includes a target graphic code; acquiring the position information of the target graphic code in the first image according to the first image and a preset graphic code identification model; acquiring an image of the target graphic code from the first image according to the position information; and acquiring a recognition result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model. According to the embodiment, the position of the target graphic code in the first image can be determined through the graphic code identification model, the image of the target graphic code can be accurately obtained, whether the target graphic code is damaged or not is determined through the graphic code classification model, the identification accuracy and the identification efficiency are high, the identification of batch graphic codes can be realized, and the labor cost is reduced.

In an alternative embodiment, as shown in fig. 5, on the basis of any one of the foregoing embodiments, if the method is applied to the terminal device, the method further includes:

S301, if the identification result is determined to be that the target graphic code is damaged, positioning information when the terminal equipment scans the target graphic code and surrounding environment images of equipment where the target graphic code is located are obtained;

S302, the positioning information, the surrounding environment image and the identification result are sent to a server, so that the server sends maintenance instructions to a maintenance personnel terminal, wherein the maintenance instructions comprise the positioning information and the surrounding environment image, and the maintenance personnel can position the target graphic code according to the positioning information and the surrounding environment image.

In this embodiment, when the executing body of the method in the foregoing embodiment is a terminal device, if the terminal device determines that the target graphic code is damaged, the terminal device may acquire positioning information when scanning the target graphic code, and may be used as positioning information of a device where the target graphic code is located, in addition, may also acquire a surrounding environment image of the device where the target graphic code is located (or may not, of course, acquire the surrounding environment image), and further send the positioning information, the surrounding environment image, and the identification result to a server, and the server may send a maintenance instruction to a maintenance personnel terminal according to these information, and carry the positioning information and the surrounding environment image in the maintenance instruction, and the maintenance personnel may arrive at the positioning location according to the positioning information, and find the target graphic code in combination with the surrounding environment image, so as to maintain the damaged target graphic code, for example, clean a correction trace and a stain, replace the damaged and torn graphic code, and so on, so as to improve accuracy and efficiency of maintenance of the graphic code, and reduce maintenance cost.

In another alternative embodiment, as shown in fig. 6, if the method is applied to the server, on the basis of any one of the foregoing embodiments, the method further includes:

S401, if the identification result is determined to be that the target graphic code is damaged, positioning information when the terminal equipment scans the target graphic code and surrounding environment images of equipment where the target graphic code is located are obtained;

And S402, sending a maintenance instruction to a maintenance personnel terminal, wherein the maintenance instruction comprises the positioning information and the surrounding environment image, so that maintenance personnel can position the target graphic code according to the positioning information and the surrounding environment image.

In this embodiment, when the method execution body of the above embodiment is a server, if the server determines that the target graphic code in the first image acquired by the terminal device is damaged, the server may request, from the terminal device, positioning information when the terminal device scans the target graphic code and an image of a surrounding environment of the device in which the target graphic code is located, and the terminal device returns the positioning information and the image of the surrounding environment after receiving the request (although the image of the surrounding environment is not necessary); or after the terminal device fails to scan the target graphic code, the positioning information and the surrounding environment image of the device where the target graphic code is located when the terminal device scans the target graphic code can be sent to the server while the first image is sent to the server (of course, the surrounding environment image is not necessary).

Further, the server can send maintenance instructions to the maintenance personnel terminal according to the information, carry the positioning information and the surrounding environment images in the maintenance instructions, the maintenance personnel can reach the positioning position according to the positioning information, and then find the target graphic code by combining the surrounding environment images, so that the damaged target graphic code is maintained, for example, correction marks and stains are cleaned, damaged and torn graphic codes are replaced, and the accuracy and efficiency of graphic code maintenance can be improved, and the maintenance cost is reduced.

On the basis of any one of the above embodiments, as shown in fig. 7, the method further includes a training process for the graphic code recognition model, which specifically includes:

S501, acquiring a first image set, wherein each image in the first image set comprises a graphic code and is marked with the position of the graphic code;

s502, determining a training set, a verification set and a test set according to the first image set;

s503, training, verifying and testing the initial pattern code recognition model according to the training set, the verification set and the test set respectively, and finally obtaining the pattern code recognition model.

In this embodiment, a first image set is obtained as a basis for a training set, a verification set and a test set of a pattern recognition model, where each image in the first image set includes a pattern, and the positions of the pattern are marked by the images, and the positions of the marked pattern can be marked manually or by other means. The first image set can be further divided into a training set, a verification set and a test set according to a certain proportion.

In this embodiment, model parameters may be initialized for the pattern recognition model based on the pre-trained thundernet-snet sub-model on the MS-COCO (target detection authority data set), then migration training may be performed by using the training set, and the model and training parameters may be optimized by observing the change of accuracy and loss function of the pattern recognition model on the verification set and the test set. Wherein specific training, validation and testing may not be limiting herein.

On the basis of any one of the above embodiments, as shown in fig. 8, the method further includes a training process for the graphic code classification model, which specifically includes:

S601, acquiring a second image set, wherein the second image set comprises a graphic code image marked with whether damage exists or not;

s602, determining a training set, a verification set and a test set according to the second image set;

And S603, training, verifying and testing the initial graphic code classification model according to the training set, the verification set and the test set respectively, and finally obtaining the graphic code classification model.

In this embodiment, the second image set is obtained as a basis of the training set, the verification set and the test set of the graphic code classification model, where each image in the second image set is a graphic code image marked with whether damage occurs, and marking whether damage occurs may be manual marking or marking by other means. The second image set can be further divided into a training set, a verification set and a test set according to a certain proportion.

In the embodiment, model parameters can be initialized for the graphic code classification model based on the shufflenetv-1.0 sub-model pre-trained on the Imagenet-1k image classification dataset, then the training set is utilized for migration training, and the model and the training parameters are optimized by observing the change of accuracy and loss functions of the graphic code classification model on the verification set and the test set. Wherein specific training, validation and testing may not be limiting herein.

Considering that graphic code image samples are seriously unbalanced in a real application scene, the number of damaged graphic code image samples in all graphic code images is far lower than that of undamaged graphic code image samples, the number of the damaged graphic code image samples is difficult to meet the requirement of ensuring the robustness of a graphic code classification model, and the training of the graphic code classification model based on the real samples not only can introduce heavy sample cleaning and labeling work, but also has the risk of fitting the model. Therefore, in this embodiment, virtual graphic code images may be generated in batches based on the synthesis technology, so that the graphic code images marked as damaged in the second image set include: a true damage graphic code image and a composite damage graphic code image.

The process of obtaining the synthesized damaged graphic code image is as follows:

Obtaining a damaged material image; and carrying out image synthesis on the damaged material image and the undamaged graphic code image to obtain the synthesized damaged graphic code image.

In this embodiment, some correction, damage and tear trace images may be obtained as damage material images, and then these damage material images are used for image synthesis of undamaged graphic code images, so as to obtain the synthesized damage graphic code images. The position of the damaged material image in the undamaged graphic code image may be any position, and is not limited herein.

Further, in order to ensure the authenticity of the synthesized damaged graphic code image, the image processing may be further performed on the result of image synthesis of the damaged material image and the undamaged graphic code image according to a preset algorithm, so as to obtain the synthesized damaged graphic code image, where the preset algorithm includes at least one of the following: motion blur algorithm, brightness adjustment algorithm, noise algorithm. Through the process, the synthesized sample can be fitted with the pattern code form which can appear in the real scene as much as possible.

Of course, the undamaged graphic code image in the above embodiment may also be not a graphic code image actually collected, or may be generated by performing image processing on some graphic code images by at least one preset algorithm such as a motion blur algorithm, a brightness adjustment algorithm, a noise algorithm, etc., so as to effectively reduce the cleaning and labeling work on the graphic code image actually collected.

In an alternative embodiment, the specific process of synthesizing the corrupted graphic code image may be as follows:

acquiring any connected domain binary image from a preset connected domain binary image set;

any damaged material image is obtained from a preset damaged material image set;

multiplying the connected domain binary image with the damaged material image to obtain a damaged noise image;

and carrying out image fusion on the damaged noise image and the undamaged noise image to obtain the synthesized damaged graphic code image.

In this embodiment, a connected domain binary image may be generated in advance. Specifically, the connected domain binary image can be randomly generated, and the connected domain binary image is not limited to an implementation manner, so that the connected domain with any shape is generated, and parameters such as the number of top points, the area threshold value and the like are required to be set to ensure the rationality of the damaged graphic code region in order to simulate various damaged graphic code regions in a real scene.

All the generated connected domain binary images are normalized to be the same size as the undamaged graphic code images (normal graphic code images), and the number of top points can be set to be a positive integer greater than 8 so as to ensure the diversity of the generated connected domain binary images; the connected domain area can be set with two thresholds to control the proportion of the damaged area occupying the whole graphic code area: the low threshold may be set to 0.1 and the high threshold may be set to 0.8. The connected domain binary pattern set can be as shown in fig. 9a below.

In addition, the damaged material images similar to the damaged graphic code area in form can be collected and sorted in advance.

By statistical analysis of the collected damage pattern data, the damage pattern of a real damage mainly includes, but is not limited to, the following: rust, soil, dust, abrasion, correction, and the like. Accordingly, the corresponding damaged material images are collected in a classified manner, wherein the damaged material images are all normalized to be the same size as the undamaged graphic code images (normal graphic code images), and the damaged material images are shown as a soil damaged material image set in fig. 9b and a rust damaged material image set in fig. 9 c.

Further, uncorrupted graphic code image data is extracted. Since the undamaged graphic code image (normal graphic code image) data is relatively large and easy to collect, the undamaged graphic code image (normal graphic code image) can be normalized to a fixed preset size (including but not limited to 224224 The connected domain binary image and the damage material image described above also need to be normalized to the preset size.

Finally, when synthesizing the damaged graphic code image, synthesizing the damaged graphic code image in batches based on the connected domain binary image, the damaged material image and the undamaged graphic code image (normal graphic code image) by using a preset image synthesis algorithm, wherein the preset image synthesis algorithm can be Alpha-Blending, which is an image processing technology for mixing source pixels and target pixels according to the value of an Alpha mixing vector, randomly extracting the damaged material image from various damaged material images for each connected domain binary image, multiplying the connected domain binary image with the selected damaged material image to obtain a damaged noise image (damaged graphic code noise), and performing Alpha-Blending image fusion operation on the damaged noise image and the undamaged graphic code image (normal graphic code image), namely, the value=alpha of any pixel in the target damaged graphic code imageThe value of this pixel in the uncorrupted graphic code image + (1-alpha)The value of this pixel in the noisy image is corrupted, resulting in a corrupted graphics code image, as shown in figure 9 d.

Based on the above embodiments, the batch of synthesized graphic code image samples (including the damaged graphic code image and/or the undamaged graphic code image) may be combined with the graphic code image samples (including the damaged graphic code image and/or the undamaged graphic code image) collected in the real scene in a proportion to construct a training set, and the remaining graphic code image samples (including the damaged graphic code image and the undamaged graphic code image) collected in the real sample are used as a verification set and a test set. Of course, other strategies may be employed to divide the training set, validation set, and test set. The graphic code classification model obtained based on the synthetic data and the real data training has the advantages that the cross test result on the test set constructed based on the real data is superior to the graphic code classification model obtained based on the real data training, and the risk of model overfitting is reduced. Meanwhile, as the label of the synthesized data can be automatically generated during synthesis, the introduction of the synthesized data does not bring any manual labeling cost. The loss curve on the graphic code classification model validation set can be seen in fig. 10.

According to the embodiment, the training sample set is constructed based on the image batch synthesis technology in a supplementing manner for model training, so that the risk of model overfitting can be reduced, the convergence rate of the model is improved, heavy manual labeling is not needed for synthesizing sample labels, the problems of sample imbalance and too few damaged graphic code image samples caused by too low proportion of damaged graphic code images in real samples are solved, the convergence rate of the model is improved by displaying a loss curve of the model on the verification set, the shock of a loss function existing in the training of the real samples is relieved, and meanwhile, the accuracy and the robustness of the model are improved. The composite sample may be used only for sample replenishment of the training set, and the test set may still all come from the real sample to ensure reliability of the model cross-validation results in a real application scenario.

It should be noted that any of the model training processes described above may be performed on the terminal device or the server, or may be performed on any other electronic device, and the trained model is deployed on the terminal device or the server.

Corresponding to the method for identifying graphic code damage of the above embodiment, fig. 11 is a block diagram of a structure of an apparatus for identifying graphic code damage provided by an embodiment of the present disclosure. For ease of illustration, only portions relevant to embodiments of the present disclosure are shown. Referring to fig. 11, the graphic code damage recognition apparatus 700 includes: an acquisition module 701, an identification module 702, a matting module 703 and a classification module 704.

An acquiring module 701, configured to acquire a first image acquired by a terminal device after a failure in scanning a target graphic code by the terminal device; the first image includes a target graphic code;

The identifying module 702 is configured to obtain, according to the first image and a preset graphic code identifying model, location information of the target graphic code in the first image;

A matting module 703, configured to obtain an image of the target graphic code from the first image according to the location information;

And the classification module 704 is configured to obtain a recognition result of whether the target graphic code is damaged according to the image of the target graphic code and a preset graphic code classification model.

On the basis of any one of the foregoing embodiments, when the recognition module 702 obtains the position information of the target graphic code in the first image according to the first image and a preset graphic code recognition model, the recognition module is configured to:

preprocessing the first image to obtain first pixel matrix data corresponding to the first image;

Inputting the first pixel matrix data into the graphic code identification model, and acquiring a predicted position and a corresponding confidence coefficient of the target graphic code in the first image;

and finally determining the position information of the target graphic code in the first image according to the predicted position in the first image and the corresponding confidence level.

On the basis of any of the foregoing embodiments, the classification module 704 is configured to, when acquiring the recognition result of whether the target graphic code is damaged according to the image of the target graphic code and a preset graphic code classification model:

Preprocessing the image of the target graphic code to obtain second pixel matrix data corresponding to the image of the target graphic code;

inputting the second pixel matrix data into the graphic code classification model to obtain a classification result and a corresponding confidence coefficient of the target graphic code;

and acquiring a recognition result of whether the target graphic code is damaged or not according to the classification result of the target graphic code and the corresponding confidence coefficient.

On the basis of any one of the above embodiments, optionally, the device is applied to the terminal device;

The obtaining module 701 is further configured to obtain positioning information and a surrounding image of a device where the target graphic code is located when the terminal device scans the target graphic code if the identification result is determined to be that the target graphic code is damaged;

the device further comprises a sending module for:

And sending the positioning information, the surrounding environment image and the identification result to a server so that the server sends a maintenance instruction to a maintainer terminal, wherein the maintenance instruction comprises the positioning information and the surrounding environment image so that maintainers can position the target graphic code according to the positioning information and the surrounding environment image.

On the basis of any one of the above embodiments, optionally, the device is applied to a server;

the device further comprises a sending module for:

And sending a maintenance instruction to a maintenance personnel terminal, wherein the maintenance instruction comprises the positioning information and the surrounding environment image, so that maintenance personnel can position the target graphic code according to the positioning information and the surrounding environment image.

On the basis of any one of the foregoing embodiments, the apparatus further includes a first training module configured to:

Acquiring a first image set, wherein each image in the first image set comprises a graphic code and is marked with the position of the graphic code;

Determining a training set, a verification set and a test set according to the first image set;

and training, verifying and testing the initial pattern code recognition model according to the training set, the verification set and the test set respectively, and finally obtaining the pattern code recognition model.

On the basis of any one of the foregoing embodiments, the apparatus further includes a second training module configured to:

Acquiring a second image set, wherein the second image set comprises a graphic code image marked with whether the graphic code image is damaged or not;

determining a training set, a verification set and a test set according to the second image set;

And training, verifying and testing the initial graphic code classification model according to the training set, the verifying set and the testing set respectively, and finally obtaining the graphic code classification model.

In accordance with any of the above embodiments, the graphic code image in the second image set that has been marked as damaged includes: a true damage graphic code image and a synthesized damage graphic code image; the second training module is further configured to:

and obtaining a damaged material image, and performing image synthesis on the damaged material image and an undamaged graphic code image to obtain the synthesized damaged graphic code image.

On the basis of any one of the above embodiments, when the second training module obtains a damaged material image, and performs image synthesis on the damaged material image and an undamaged graphic code image to obtain the synthesized damaged graphic code image, the second training module is configured to:

On the basis of any one of the above embodiments, when the second training module performs image synthesis on the damaged material image and the undamaged graphic code image to obtain the synthesized damaged graphic code image, the second training module is further configured to:

Image processing is carried out on the result of image synthesis of the damaged material image and the undamaged graphic code image according to a preset algorithm to obtain the synthesized damaged graphic code image, wherein the preset algorithm comprises at least one of the following steps: motion blur algorithm, brightness adjustment algorithm, noise algorithm.

The identifying device for graphic code damage provided in this embodiment may be used to execute the technical scheme of the above method embodiment, and its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

The identification equipment for the damage of the graphic code provided by the embodiment acquires a first image acquired by terminal equipment after the terminal equipment fails to scan a target graphic code; the first image includes a target graphic code; acquiring the position information of the target graphic code in the first image according to the first image and a preset graphic code identification model; acquiring an image of the target graphic code from the first image according to the position information; and acquiring a recognition result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model. According to the embodiment, the position of the target graphic code in the first image can be determined through the graphic code identification model, the image of the target graphic code can be accurately obtained, whether the target graphic code is damaged or not is determined through the graphic code classification model, the identification accuracy and the identification efficiency are high, the identification of batch graphic codes can be realized, and the labor cost is reduced.

Fig. 12 is a schematic structural diagram of a graphic code damage identification device according to an embodiment of the present invention. The identifying device for graphic code damage provided in the embodiment of the present invention may execute the processing flow provided in the identifying method embodiment for graphic code damage, as shown in fig. 12, where the identifying device 80 for graphic code damage includes a memory 81 and a processor 82; wherein the computer program is stored in the memory 81 and configured to perform the method of identifying graphic code corruption described in the above embodiment by the processor 82. The identification device 80 of the graphic code corruption may furthermore have a communication interface 83 for receiving and transmitting data and/or instructions.

The identifying device for graphic code damage in the embodiment shown in fig. 12 may be used to implement the technical solution of the above-mentioned identifying method embodiment for graphic code damage, and its implementation principle and technical effects are similar, and will not be repeated here.

In an exemplary embodiment, a non-transitory computer readable storage medium is also provided, such as a memory, comprising instructions executable by a processor to perform the method of identifying a graphic code corruption as described above. For example, the non-transitory computer readable storage medium may be ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

In an exemplary embodiment, a computer program product is also provided, comprising a computer program which, when executed by a processor, performs the above-mentioned method of identifying graphic code corruption.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. The embodiments of the present disclosure are intended to cover any adaptations, uses, or adaptations of the disclosure following the general principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for identifying graphic code corruption, comprising:

According to the first image and a preset graphic code identification model, acquiring the position information of the target graphic code in the first image comprises the following steps:

According to the predicted position in the first image and the corresponding confidence, finally determining the position information of the target graphic code in the first image;

According to the image of the target graphic code and a preset graphic code classification model, acquiring the identification result of whether the target graphic code is damaged or not, wherein the identification result comprises the following steps:

2. The method according to claim 1, characterized in that the method is applied to the terminal device; the method further comprises the steps of:

if the identification result is determined to be that the target graphic code is damaged, positioning information when the terminal equipment scans the target graphic code and a surrounding environment image of the equipment where the target graphic code is located are obtained;

3. The method of claim 1, wherein the method is applied to a server; the method further comprises the steps of:

4. The method of claim 1, wherein the pattern recognition model is trained by:

5. The method of claim 1, wherein the graphic code classification model is trained by:

6. The method of claim 5, wherein the graphic code images in the second set of images that have been marked as corrupted comprise: a true damage graphic code image and a synthesized damage graphic code image; wherein the method further comprises:

7. The method of claim 6, wherein the obtaining the damaged material image, and the image synthesizing the damaged material image and the undamaged graphic code image to obtain the synthesized damaged graphic code image, comprises:

and carrying out image fusion on the damaged noise image and the undamaged graphic code image to obtain the synthesized damaged graphic code image.

8. The method of claim 6, wherein the image synthesizing the damaged material image and the undamaged graphic code image to obtain the synthesized damaged graphic code image, further comprises:

9. An identification device for graphic code corruption, comprising:

the identification module is used for acquiring the position information of the target graphic code in the first image according to the first image and a preset graphic code identification model, and comprises the following steps:

the classification module is used for acquiring the identification result of whether the target graphic code is damaged or not according to the image of the target graphic code and a preset graphic code classification model, and comprises the following steps:

10. The device according to claim 9, characterized in that the device is applied to the terminal device;

The acquisition module is further used for acquiring positioning information when the terminal equipment scans the target graphic code and a surrounding environment image of the equipment where the target graphic code is located if the identification result is determined to be that the target graphic code is damaged;

the device further comprises a sending module for:

11. The device of claim 9, wherein the device is applied to a server;

the device further comprises a sending module for:

12. The apparatus of claim 9, further comprising a first training module to:

13. The apparatus of claim 9, further comprising a second training module for:

14. The apparatus of claim 13, wherein the graphic code images in the second set of images that have been marked as corrupted comprise: a true damage graphic code image and a synthesized damage graphic code image; the second training module is further configured to:

15. The apparatus of claim 14, wherein the second training module, when acquiring the damaged material image, performs image synthesis on the damaged material image and an undamaged graphic code image to obtain the synthesized damaged graphic code image, is configured to:

16. The apparatus of claim 14, wherein the second training module, when performing image synthesis on the damaged material image and the undamaged graphic code image to obtain the synthesized damaged graphic code image, is further configured to:

17. An identification device for graphic code corruption, comprising: a memory and a processor;

The memory is used for storing program instructions;

The processor is configured to invoke program instructions in the memory to perform the method of any of claims 1-8.

18. A computer readable storage medium having a computer program stored thereon; the computer program, when executed, implements the method of any of claims 1-8.

19. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any one of claims 1-8.