TWI729331B - Image annotation information processing method, device, server and system - Google Patents

Abstract

The embodiment of this specification discloses an image labeling information processing method, device, server and system, which can provide monitoring and judgment processing logic for different processing results of multiple nodes. When the image labeling information is wrong, the result can be automatically returned to make the operation Personnel conduct re-examination, modification and other processing. In this way, the business ability of the operators can be improved in the continuous review and feedback interaction, and the efficiency of image labeling can be gradually improved, and the accuracy of image labeling of the training set can be greatly improved. The implementation of this description can effectively guarantee the quality of labeling, provide timely and effective information feedback in the workflow, and improve the efficiency of sample image labeling information.

Description

Image annotation information processing method, device, server and system

The present invention relates to the technical field of computer data processing, in particular to a method, device, server and system for processing annotation information in images.

The method of detecting objects in images in existing applications mainly relies on the target detection/recognition model obtained by training, such as the component recognition model in the vehicle damage assessment business, and the model for determining the purchased item by taking pictures. These target detection/recognition models usually need to rely on a large number of marked sample images for training. Therefore, the accuracy of the annotation information in the sample images has a significant impact on the output of the model. In the early stage of the processing of sample image annotation information, the sample image often includes multiple target subjects. Multi-target tagging tasks are more complicated than conventional similarly judging whether a picture is a dog or a cat, and the classification of a single subject is more complicated, and it is often required to mark the area of all targets in an image and give the corresponding target. Classification results. Especially in some more professional business fields, some tasks not only need to be completed by conventional knowledge/understanding, but usually also require a certain degree of professionalism of the annotator, and a certain professional training and skill strengthening process. Some professional multi-target annotation processing requirements for preparing training samples for image multi-target detection algorithms in a professional field are very strict, such as identifying and labeling human organs and bone structures after a period of professional training by medical professionals or non-professionals. Types of complex tasks that can be labeled. When the number of sample images that need to be processed is large and the target subjects in most images are dense (at least two), manual annotation processing is due to limited attention on the one hand, and excessive classification on the other hand, which requires professional At the same time, it is easy to be confused, which often results in low accuracy of the output sample image labeling information, and it is difficult to guarantee the labeling quality, which in turn leads to poor image recognition algorithms and low target prediction accuracy in the image. Therefore, how to effectively guarantee the accuracy of the sample image labeling information is a technical problem that needs to be solved urgently.

The embodiments of this specification aim to provide an image tagging information processing method, device, server and system, which can effectively improve the processing accuracy of multi-target sample image tagging information, thereby improving the accuracy of the image target detection algorithm. The image annotation information processing method, device, server and system provided by the embodiments of this specification are implemented in the following ways: An image annotation information processing method, the method includes: The first node receives the task image, and the task image includes at least the following label information: the category corresponding to the target in the identified task image, and the label frame of the target; The first node receives the inspection result of the task image, and sends the task image processed by the first inspection to the second node. The inspection result includes: performing a first inspection process on the annotation information of the task image; When it is determined that there is an error in the annotation information, the annotation result obtained after correcting the annotation information; The second node receives the review result, and if the review result includes an error in the annotation information, the review result is sent to the first node for a first re-inspection process, and the review result includes: Perform a second check process, when there is an error in the annotation information, the determined check result has an error type of error; The first node receives the first re-inspection result, and sends the first re-inspection result to the second node for the second inspection process, and the first re-inspection result includes information based on the re-inspection result The error type is the annotation result obtained by correcting the annotation information of the task image. An image annotation information processing method, the method includes: Receiving a task image, the task image including at least the following label information: the category corresponding to the target in the identified task image, and the label frame of the target; Receive a check result of the task image, the check result includes: performing a first check process on the label information of the task image, and when it is determined that the label information has errors, the result is obtained after correcting the label information Mark the result; Receive a review result of the task image, and if the review result includes an error in the annotation information, feedback the error type. The review result includes: performing a second inspection process on the task image, and if the annotation information exists In the event of an error, the type of error in which the result of the check is determined to be wrong; A first re-inspection result is received, and the second re-inspection process is performed on the first re-inspection result, where the first re-inspection result includes the annotation obtained by correcting the annotation information of the task image based on the error type result. A sample image labeling information processing device, the device comprising: The image receiving module is configured to receive a task image, the task image at least including the following label information: the category corresponding to the target in the identified task image, and the label frame of the target; The annotation inspection interaction module is configured to receive the inspection result of the task image, and send the task image processed by the first inspection to the annotation review interaction module, and the inspection result includes: an annotation of the task image The information undergoes the first inspection process, and when it is determined that the annotation information has errors, the annotation result obtained after correcting the annotation information; is also used to receive the first re-inspection result, and send the first re-inspection result to the The annotation review interaction module performs a second inspection process, and the first re-inspection result includes an annotation result obtained by correcting the annotation information of the task image based on the error type in the review result; The annotation review interaction module is used to receive the review result, and when the review result includes an error in the annotation information, send the review result to the annotation inspection interaction module for the first re-inspection process, and the review result includes: A second inspection process is performed on the task image, and when there is an error in the annotation information, it is determined that the inspection result has an error type of error. A server includes a processor and a memory for storing executable instructions of the processor. When the processor executes the instructions, the following is achieved: Receiving a task image, the task image including at least the following label information: the category corresponding to the target in the identified task image and the label frame of the target; Receive a check result of the task image, the check result includes: performing a first check process on the label information of the task image, and when it is determined that the label information has errors, the result is obtained after correcting the label information Mark the result; Receive a review result of the task image, and if the review result includes an error in the annotation information, feedback the error type. The review result includes: performing a second inspection process on the task image, and if the annotation information exists In the event of an error, the type of error identified in the inspection result that is incorrect; A first re-inspection result is received, and the second re-inspection process is performed on the first re-inspection result, where the first re-inspection result includes the annotation obtained by correcting the annotation information of the task image based on the error type result. A sample image annotation information processing system, including: The first processing terminal is used to receive the task image and the inspection result of the task image, and send the task image processed after the inspection to the second processing terminal; it is also used to receive the first re-inspection result, and the second processing terminal The first check result is sent to the second terminal for second check processing. The task image includes at least the following annotation information: the category corresponding to the target in the identified task image and the marking frame of the target, the The inspection result includes: performing a first inspection process on the annotation information of the task image, and when it is determined that there is an error in the annotation information, the annotation result obtained after correcting the annotation information, the first re-inspection result includes The type of error in the review result is an annotation result obtained by correcting the annotation information of the task image; The second processing terminal is configured to receive the review result, and when the review result includes an error in the labeled information, send the review result to the first processing terminal for the first re-inspection process; and also to receive the second re-inspection As a result, the second re-check result is sent to a third processing terminal, and the re-check result includes: performing a second check process on the task image, and determining the check result when there is an error in the annotation information The error type of the error, the second recheck result includes the labeling result obtained by correcting the labeling information of the corresponding task image based on the sampling information; The third processing terminal is used to receive the task image with correct annotation information sent by the second processing terminal, and is also used to receive the random inspection result, and when the random inspection result includes the error in the annotation information, the corresponding random inspection image The sampling information of is sent to the second terminal for a second re-inspection process, and the sampling result includes: selecting a sampling image from the received task images according to a preset rule, and verifying whether the label information of the sampling image is correct And the processing result obtained. The sample image annotation information processing method, device, server and system provided by the embodiments of this specification can provide the supervision of multiple nodes and different processing results in the complex and professional multi-target sample image annotation information processing operation And judgment processing logic, when there is an error in the image annotation information, the result can be automatically returned, allowing the operator to re-examine, modify and other processing, to achieve good interactive feedback between the system and the operator, quality monitoring, ability detection, and prevent collusion between operators at different nodes Wait. In this way, the business ability of the operators can be improved in the continuous review and feedback interaction, and the efficiency of image labeling can be gradually improved, and the accuracy of the training sample image labeling information can be effectively improved.

In order to enable those skilled in the art to better understand the technical solutions in this specification, the following will clearly and completely describe the technical solutions in the embodiments of this specification in conjunction with the drawings in the embodiments of this specification. Obviously, the described The embodiments are only a part of the embodiments in this specification, rather than all the embodiments. Based on one or more embodiments in this specification, all other embodiments obtained by a person of ordinary skill in the art without making progressive work shall fall within the protection scope of the embodiments of this specification. In some business scenarios, it is often necessary to deal with the situation of annotated information of training sample images with a wide range of target classifications or strong professionalism. In the processing of sample image labeling information, it is usually required to identify the target subject in the image, mark the target category, and the location area of the target in the image. The categories can include different categories of targets or selected attribute values of a certain dimension (such as categories based on location, connection relationship, material, color, purpose, etc.), and can be the name or code of the category, number, etc., specific The classification of the target can be defined in advance according to the rules. The target described in this specification usually includes the main object in the image, such as various organs or tissues in the human anatomical image, various vehicle parts in the vehicle image, and so on. In the image processing of training samples, after recognizing the target in the image, it is usually necessary to mark the location area of the target. Specifically, the location area of the subject can be displayed in the image by marking the frame. The label box can be rectangular or other regular or irregular graphic representations, can be closed graphics, such as a rectangular frame, and can also include a way to indicate the location area in a manner similar to a line segment and an arrow, and there is no specific limitation on this. . For unified description, in one or more embodiments of this specification, the label information of the target location area in the identification image may be collectively referred to as a label frame. When there are many types of objects in the image, it is usually required to label all the objects in the image, as well as the types and block diagrams. However, in actual applications, due to the limitations of the annotator's own attention, work status, and memory, various types of Types of errors, such as false detections, missed detections, incorrect classification and labeling, and labeling boxes that are too large or too small. The occurrence of any of the above types of errors will affect the accuracy of the label information of the sample images that finally enter the training set, which will result in poor algorithm training and reduced prediction accuracy. An example of an application scenario is shown in Figure 1. Figure 1 is a schematic diagram of a job scenario for labeling multiple targets in a sample image. The operator is required to label each part of the car and the corresponding part classification label, for example, the label of the tire is 31. The size of the corresponding labeling frame of 31 should be reasonable to frame the area where the tire is located. However, in actual operations, the labelers will always make mistakes due to the limited attention and lack of professionalism of the people themselves. For example, the label box 31 is too small, or the tire label is incorrectly marked as 30, or the “front” in the image is omitted. Mudguard" components. At present, a single business-level training is commonly used to improve the accuracy of the training sample image labeling information, such as professional domain knowledge training, auto parts classification and label memory. In the embodiment of one or more sample image annotation information processing methods provided in this specification, the fast and efficient interactive feedback of annotation information between the machine and the operator is used to realize standardized operation flow, effectively discover and feedback operation problems, supervise and Improve the processing quality of sample image annotation information. It is possible to provide timely feedback and correct incorrect annotation information by setting up multiple control node annotation information inspection processing, so that the operator can gradually improve the annotation ability during the continuous operation of the sample image annotation information, and effectively guarantee the annotation quality, so that The efficiency of the entire annotation processing system is improved. In some embodiments provided in this specification, multiple control nodes can be divided according to different processing stages of the annotation information, such as the first node used to check the initial annotation information in the sample image, and the graph after the first node is detected and processed. Like the second node for re-examination of labeling information, even in some embodiments, it is possible to add the first node randomly selected or the third node of the second node labeling processing result. When a node finds an error in the label information of the previous node, it can give feedback in time, and continue to check after correction. The images that pass the check of the annotation information can be used as the sample images for model training and added to the training sample set. In this way, through interactive processing and information feedback between different nodes, the quality of labeling information can be monitored, and the business ability of labelers can be gradually improved, which greatly improves the accuracy of labeling samples that fall into the training sample set. For example, in a simple application example, the operator A at the second node finds that the annotator B at the first node has made a mistake, he can select the option of annotating the wrong information in the system, and can give an error remark or error classification, the system can Automatically return to the first node, so that the annotator B of the first node can make timely modifications. After B is modified, it can be returned to the operator A of the second node to continue the inspection, if it is correct, it will pass, if it is wrong, it can be returned again. Through the image annotation information processing method of this solution, the annotator B can gradually reduce the error rate of the sample image annotation information. An implementation solution provided in this specification can be applied to a multi-terminal system architecture (including different terminals belonging to the same system), a distributed system, or a dedicated image annotation information processing application. The system may include a single computer device, a server cluster composed of multiple servers, or a distributed system structure. In some application scenarios, operators can interact with the system. In an embodiment of this specification, the system can be divided into corresponding processing nodes according to different operators or different work phases (processes), such as the above The first node of the initial annotation information in the inspection sample image described in the above, the second node that reviews the annotation information of the image after the first node detection processing, and so on. It should be noted that the first node, the second node, and the third node involved in other embodiments may be different terminals of the label information processing system, for example, the label automatically generated by the outsourcing operator on the server The computer terminal of the first node where the information is checked for the first time. Similarly, there can also be a computer terminal of the second node that is specially provided for operators to review labeled information, and a computer terminal of the third node that internal personnel conduct random inspections of the labeled information. . Of course, this specification does not exclude that in other implementation manners, one or more of the first node, second node, third node, etc. may be the same terminal, or some of the nodes may be the same terminal. These same or different terminals can be logically divided into different processing nodes from the business processing logic when the method is implemented in a specific application system. These nodes can be physically separated different terminals or the same terminal device. The following takes a specific application scenario of a car damage sample image as an example to describe the implementation of this specification. Specifically, FIG. 2 is a schematic flowchart of an embodiment of the method for processing sample image annotation information provided in this specification. Although this specification provides method operation steps or device structures as shown in the following embodiments or drawings, the method or device may include more or less operations after partial merging based on conventional or no progressive labor. Step or module unit. In steps or structures where there is no necessary causal relationship logically, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure shown in the embodiments or drawings of this specification. When the described method or module structure is applied to an actual device, server or terminal product, it can be executed sequentially or in parallel according to the method or module structure shown in the embodiments or drawings (for example, parallel processors or Multi-threaded processing environment, even including distributed processing, server cluster implementation environment). A specific embodiment is shown in Fig. 2, and the method may include: S0: The first node receives a task image, and the task image includes at least the following label information: the category corresponding to the target in the identified task image and the label frame of the target. In an application scenario for processing car damage sample image annotation information in this embodiment, the original car damage image collected on site can be preprocessed with the annotation information to obtain the task image that needs to be processed in this embodiment. The preprocessing may include labeling the target of the car damage image to obtain labeling information. Generally, in the preprocessing, the unobstructed acquisition of labeling information can use a preset algorithm to quickly identify the components in the image, mark the category numbers of the components, and at the same time frame the labeling frame where the components are located. The labeling information in the task image can be implemented by using various icon target detection algorithms, which are not limited in the embodiment of this specification. In some application scenarios, the task image can be sent to the first node for processing in a dispatch mode according to the case level of the job. For example, all pictures of a car damage case can be given at one time, and the operator can perform AI (Artificial Intelligence, artificial intelligence) on the terminal device of the first node. Here it can refer to the preprocessing of the car damage image using a preset algorithm to obtain the annotation information. Server system) to check and modify the pre-labeled label information. The labeling interface of the terminal device can give the AI prediction label box and allow the operator to modify it. The operator can perform the first detection process on the annotation information in the task image received by the first node, and detect whether there is an error in the annotation information in the task image. If there is an error, it can be corrected, and the corrected detection result can be saved; For a task image, if the operator finds that the target is detected correctly and all the labeled frames and classifications are also correct after detection, it can be determined that the task image labeled information has no wrong detection results. The operator detects the label information in the task image at the first node, corrects the incorrect label information pre-labeled by AI, and then transfers it to the second node. If there is no error, it can directly transfer to the second node. Therefore, the method may further include: S2: The first node receives the inspection result of the task image, and sends the inspected task image to the second node. The inspection result includes: performing a first inspection process on the annotation information of the task image; When it is determined that there is an error in the annotation information, the annotation result obtained after correcting the annotation information. The worker at the second node can review the task image after the first node performs the first detection processing. The operator of this node mainly judges whether the identification of the job task of the first node is right or wrong. In specific applications, the labeling processing interface can be similar to the processing interface of the first node operator, but the difference is that it can be separate Give the error type for the first node operator to mark the job processing error. As shown in Figure 3, when the operator at the second node reviews the label information of the task image P1, it is judged whether there is an error in the label information. If so, the first node operator can select the first node obtained during the first detection process. An error type of the test result, if not, it can be passed directly. For example, in Figure 3, when there is no error, you can check "A: Correct" on the label processing interface of the terminal device of the second node. If there is an error, you can check "B1: Missing mark box" according to the corresponding error type. , "B2: Classification error", "B3 false detection (no damage is judged as damaged)", etc. one or more error types. At the second node, if the operator rechecks that there is an error in the annotation information of the image, he can return the review result of the image to the first node, so that the operator at the first node can recheck and correct the annotation information (in This can be referred to as the first recheck process). Therefore, the method may further include: S4: The second node receives the result of the review, and if the result of the review includes an error in the annotation information, the result of the review is sent to the first node for a first re-inspection process, and the result of the review includes: The image undergoes a second inspection process, and when there is an error in the annotation information, it is determined that the inspection result has an error type of error; S6: The first node receives the first re-check result, and sends the first re-check result to the second node for the second check process, where the first re-check result includes a result based on the re-check result The error type in is the annotation result obtained by correcting the annotation information of the task image. When the operator of the second node performs the second detection and finds that the detection result of the first node is wrong, the error information such as the error type shown in this embodiment can be fed back to the first node. The first node can display the review result on the display interface of the terminal device of the first node, or display it in the form of notifications and reminders, so that the operator of the first node can check the wrong task image based on the feedback of the review result. Mark the information for re-check processing. In one embodiment, the review result returned by the second node to the first node may include the error type of the error in the first detection result, but does not include the task image corresponding to the error. In this case, the first node may use the cache or backup The task images are re-checked, which can reduce the amount of data sent by the second node to the first node and save network overhead. Of course, in other embodiments of this specification, the review results may also include task images with incorrect annotation information in the review, and the error type is connected to the corresponding task image and fed back to the first node, so that the first node The operator of the node quickly locates the re-checked image and processes it in time to improve the efficiency of error correction processing. It should be noted that the first inspection process, the second inspection process, and the first re-inspection process described above, and even the second re-inspection process described in the following embodiments, can be the same inspection of image annotation information Processing methods, such as the same marked information inspection items or the same inspection procedures, requirements, etc. Of course, the second inspection process can also be set to be different from the second inspection process. Since the second inspection mainly implements the review of the detection results of the second node operators, the targeted second inspection process can be set according to the actual operation scenario. . The operator at the first node re-corrects the wrong task image based on the re-check result and determines the first re-inspection result. Then the first re-inspection result of the corresponding task image can be sent to the second node again. The operator at the second node performs the second inspection process again. After re-correction processing, if there is still an error in the annotation information, in the application scenario of this embodiment, the review result can be fed back to the first node for processing; if no errors are found in the second inspection process after the correction, the annotation information is re-examined at the second node Pass, you can flow to the next processing node. The processing of sample image annotation information is a basic and important business process, which has a significant impact on subsequent online product target detection and recognition, as well as related businesses such as product positioning, search, and push. In one or more embodiments of the present specification, through the interaction of the operators of at least two of the nodes and the feedback processing of the operation results, it is possible to effectively and timely find and feedback the incorrectly marked information. Generally, the interactive feedback of the information between the operator and each node in the solution of this embodiment can make each node operator continuously realize their own weaknesses, and then assist them in gradually improving their operational capabilities and improving the ability of the entire sample image to label information. Processing quality. In the above embodiment, the task image after the second node review process can be used as a sample image and put into the corresponding sample image collection, or the task image processed by the second node can be marked as Sample image. In another embodiment of the method provided in this specification, the interactive feedback processing with the operator in Section 3 may also be added. The third node can perform random inspection processing on the review results of the second node to verify whether the label information of the task image processed by the second node operator is correct. If an error in the labeled information is found in the randomly checked images, in one embodiment, the task image with the wrong labeled information can be sent to the second node for recheck, and in another embodiment, the task image with the wrong labeled information can be located. The task images of that batch are sent to the second node. The operators of the third node can include internal quality inspectors, or specially set up operators who regularly or irregularly perform random inspection processing on the label information of the task images. They can be internal operators or entrusted third parties. mechanism. Therefore, as shown in FIG. 4, FIG. 4 is another embodiment of the method provided in this specification, and the method may further include: S80: Send the task image with the correct labeled information in the review result to the third node; S82: The third node receives the result of the random inspection, and if the result of the random inspection includes an error in the annotation information, the corresponding random inspection information of the random image is sent to the second node for a second re-inspection process. The result of the random inspection is Including: the processing result obtained by selecting random images from the received task images according to preset rules, and verifying whether the label information of the random images is correct; S84: The second node receives a second re-check result, and sends the second re-check result to the third node, where the second re-check result includes the corresponding task image based on the sampling information The annotation result obtained by modifying the annotation information. Operators at the third node can randomly extract task images, or they can extract task images based on one or more of "inspector", "re-inspector", and "inspection date". For ease of description, we will follow The images used for fundraising processing selected by the preset rules from the received task images are called sampling images. Therefore, in an embodiment of the method, the selection of random images may include: selecting task images based on at least one of the user identification and the execution date of the annotation information processing. The execution date may include the re-examination date of the second node in the above-mentioned re-examination. Of course, in other embodiments, if it is an implementation scenario of reviewing the task image of the first node, the execution date may include the inspection date of the inspection performed by the operator at the first node. The second node sends the task image with the correct labeled information in the review to the third node, and the third node can persist the task image. The operator of the third node can obtain random inspection images from the third node to verify the annotation information. If the marking information of the random inspection image is correct, the corresponding task image or the batch corresponding to the task image can be marked as a training sample. If the random inspection result includes an error in the annotation information, the corresponding random inspection information of the random inspection image may be sent to the second node for second re-inspection processing. The second inspection process is the same as the second inspection process described in the above embodiment, or the same as the first re-inspection process. Of course, it is also possible to separately set the way of labeling information processing for the sampling results. In the above-mentioned sampling process, it is found that the error in the annotation information is found to be sent to the sampling information of the second node. In one embodiment, the sampling image with the error in the annotation information can be sent to the second node, and specific error information and task images may not be sent; In one embodiment, only the identification error information may be sent to the second image, and the identification error information may include specific information about which image is sampled and what error occurred, and the task image may not be sent; other embodiments If there is an error in the annotation information, all the images corresponding to the batch can be sent to the second node for re-checking. Therefore, in another embodiment of the method described in this specification, the sending the sampling information of the corresponding sampling image to the second node includes at least one of the following methods: Sending the spot-checked images with errors in the labeled information to the second node; If there is an error in the annotation information, send the task image set corresponding to the sampled image to the second node; Send the marking error information of the randomly checked image to the second node. Of course, in some embodiments, processing can also be performed in combination with multiple methods described above. For example, all task images of batches with incorrect labeling information are returned to the second node, and the labeling error information is returned at the same time. The batch may include a collection of task images divided according to predetermined dimensions. For example, a car damage case is a batch, and all images in the case are in a task image collection, such as a single-vehicle collision accident. The case contains 100 car damage images. After AI pre-labeling, first node inspection and correction, and second node review, 10 images of the case are randomly selected at the third node to verify whether the labeled information is correct. If the labeling information of the 10 images is correct, the 100 car damage images of the case are all added to the sample image training set. If the labeling information of at least one of the 10 images sampled is incorrect, you can All the 100 car damage images of this case are returned to the second node (in some embodiments, it may also be directly returned to the first node), and the second node's operator will recheck and process. Alternatively, the batches can be divided according to the identity of the first node inspector or the inspection date. For example, when the random inspection reviewer A finds errors in the marked information in the review results dated December 20, 2017, the reviewer A can be changed On December 20, 2017, all the task images processed by the review process were returned to the second node for re-inspection. Of course, according to actual operation needs or quality monitoring standards, the specific content and method of the returned sampling information can be set accordingly. The specific implementation of the sampling processing of the third node in this embodiment may be to perform sampling processing on each batch of task images, which can be used as training sample images after the sampling passes. It can also be performed on some images in the task images transferred from the second node regularly and irregularly. The task images or task image collections (batch) that have not been sampled and processed can follow the correct business process. As a training sample image, if there is no internal sampling processing after 3 days of persistence, it will pass by default and can be added to the corresponding sample image training set. Therefore, in another embodiment of the method described in this specification, it may further include: The task image set corresponding to the sampled images with the correct labeled information as the result of the random inspection is marked as the training sample image. In this way, the accuracy of the labeling information in the training sample image obtained by the sample image labeling information processing method of this embodiment is higher, and the labeling quality and the level of the operator are gradually and steadily improved, thereby improving the accuracy of the algorithm based on the training sample image. Accuracy rate. The task image set corresponding to the above-mentioned sampling image may include the image set of the batch where the task image is located, usually including multiple task images, but this specification does not exclude that the task image set includes The implementation of a task image. In some other embodiments of the method provided in this specification, it is also possible to embed points in the task image to detect the correct rate of the processing of annotated information by the operator in the workflow. The task image with embedded point information can be referred to as a surveillance image here. The surveillance image usually includes all the targets in the image that have been identified and determined in advance, the category to which the target belongs, and the size of the mark frame of the target. Location etc. A certain proportion of monitoring images can be added to the task picture in advance. In one embodiment, the accuracy of the detection result processed by the first node operator can be determined at the first node based on the recognition result of the annotation information of the monitoring image. In a specific embodiment, the method further includes: S102: Add a surveillance image of a predetermined proportion to the task image, and the known label information of the surveillance image includes the identified target and the corresponding category and label frame information; S104: Acquire a recognition result of the annotation information of the surveillance image in the inspection result; S106: Compare the recognition result with the known label information, and determine the check accuracy rate of the check result. For example, a specific implementation example is shown in Fig. 5, which can compare the area ratio of the marked frame of the identified target in the surveillance image with the area of the marked frame in the inspection result obtained by the operator's detection processing (the block diagram can be used in this example) Crossing area/combined area of two boxes, shown in solid and dashed boxes in Figure 5) and corresponding categories. If the type of the marked target is correct, all are the front door of the vehicle, and the area ratio of the marked frame is within the error range, it can be considered that the marked frame in the inspection result processed by the operator is correct. For an image, if all the boxes are marked correctly, it can be determined that the marking information of the image is correct. In practical applications, a certain percentage of the correct block diagram is added to the picture to be labeled as a buried point, which can monitor the check accuracy rate in the annotator's workflow, and then set the corresponding message or action to be triggered based on the check accuracy rate for effective implementation Label information supervision and quality control. Therefore, in some embodiments, the method may further include: When the check accuracy rate reaches the first threshold within the first preset period, a corresponding notification message is sent. In some of the foregoing embodiments, it is described that a monitoring image can be added to the task image, and the inspection accuracy of the operator can be determined according to the recognition result of the annotation information of the monitoring image in the inspection result of the first node. In another embodiment of the method provided in this specification, the monitoring image can also be used to detect the accuracy of the second node reviewer's review result processing, so as to realize the quality supervision and feedback of the review process. Specifically, the method may further include: S122: Add a surveillance image of a predetermined ratio to the task image, and the known label information of the surveillance image includes the identified target and the corresponding category and label frame information; S124: Obtain a recognition result of the annotation information of the surveillance image in the review result; S126: Determine a review accuracy rate of the review result based on the recognition result. Through the above scheme, the accuracy of re-examination can be monitored in real time. Based on the inspection results of the monitoring images with standard monitoring information, the inspection results of the monitoring images marked by the reviewers can be compared. When the errors between the two are in line with expectations, the review processing of the task reviewers can be normal and meet the job requirements. With reference to the processing of checking accuracy, another embodiment of the method may further include: When the recheck accuracy rate reaches the second threshold within the second preset period, a corresponding notification message is sent. The first preset period, the second preset period, and the corresponding first threshold and second threshold for notification can be set according to actual needs. For example, if the review accuracy rate is lower than the second threshold within the second preset period, a post transfer suggestion message is sent to the designated recipient, and the second threshold may be set to be less than the first threshold. For example, the annotators with low accuracy in certain time periods can be reminded of suggestions, and the annotators with low accuracy in the long-term can send a post transfer suggestion message to the designated recipient, such as the manager terminal or the personnel management terminal. The solution of this embodiment can also be used to obtain the accuracy and time relationship of the labeling personnel's labeling, which can be used as data support for optimized management and improve the processing efficiency of labeling information. It should be noted that the above-mentioned processing of obtaining inspection accuracy and review accuracy by burying points can be implemented at designated nodes. For example, the inspection accuracy rate is calculated at the first node and the review accuracy rate is calculated at the second node. Set up a separate logic processing unit to achieve. In another embodiment of the method provided in this specification, in the sampling process of the third node, the accuracy of the review and the sampling result can be combined to determine whether the task image can be used as the training sample image. Specifically, in another embodiment of the method, it may further include: S140: If the recheck accuracy rate is within the error range and the sampling result passes, then the task image set corresponding to the sampling image is added to the training sample set. The coincidence rate between the reviewer's review result and the buried spot detection result can be compared. If the internal random inspection passes and the agreement rate meets expectations, the task images processed by the reviewer can be passed in batches and added to the corresponding training sample set. The calculation of the different nodes and the check accuracy rate and the recheck accuracy rate of the above embodiment can be arranged in different processing links in actual applications. For example, in an application example, the first node and the second node can be processed by external operators. The internal spot checks in Section 3 can be handled by internal operators. In this way, the external and internal task quality inspections can prevent external operators from colluding to cause a large number of task images to be released. At the same time, it can also detect the accuracy of the operator's processing through the buried point, find the problem in time, and feed it back to the operator or remind the operator to replace it. Through the node interaction and feedback processing of one or more embodiments provided in this specification, the accuracy of the final image annotation information in the training sample set can be effectively improved. Compared with the conventional continuous emphasis on annotation rules and content training, From another perspective, the entire sample image annotation information processing is realized while doing learning while progressing, which effectively guarantees the annotation quality, and provides timely and effective information feedback in the workflow, and improves the efficiency of the sample image annotation information operation. The foregoing embodiments describe the implementation of the sample image annotation information processing method that can be implemented with different logical processing nodes. Specifically, different processing nodes can be implemented by different terminal devices in the system, such as the first server of the first node, the first server of the first node, and the first server of the first node. The second server of the two nodes, the third server of the third node, or the implementation steps of the first node and the second node are implemented by the same server or server system (application). The method provided in this manual can also be implemented by the same physical terminal device, or a complete set of multiple terminal devices, such as a sample image tagging information system server, in which different operators can interact with it. For example, the operator A can check the label information of the task image on the system server, correct the AI label information, and confirm and persist the inspection result. Operator B can review the inspection result of operator A on the system server. If an error in the labeling information is found, it can mark and feedback the error type of the labeling information error to operator A through the system server. Correspondingly, the operator C can also perform spot checks on the review results of the operator B through the system server. Of course, spot checks on the inspection results of the operator A can also be implemented in some implementation scenarios. Therefore, in another embodiment of a method for processing sample image annotation information provided in this specification, it may include: S100: Receive a task image, where the task image includes at least the following label information: the category corresponding to the target in the identified task image, and the label frame of the target; S200: Receive an inspection result of the task image, the inspection result includes: performing a first inspection process on the annotation information of the task image, and when it is determined that there is an error in the annotation information, after correcting the annotation information The labeling result obtained; S300: Receive a review result of the task image, and if the review result includes an error in the annotation information, feedback the type of error, and the review result includes: performing a second inspection process on the task image, and performing a second inspection process on the task image. When there is an error in the information, the type of error identified in the check result is wrong; S400: Receive a first re-inspection result, and perform the second inspection process on the first re-inspection result, where the first re-inspection result includes correcting the annotation information of the task image based on the error type. The annotation result. Fig. 6 is a schematic diagram of a method flow of another embodiment of the method provided in this specification. Of course, as described in the foregoing embodiment, in another embodiment of the method, sampling of task images can also be implemented to further ensure the accuracy of the annotation information of the training sample images. Therefore, the method may further include: S500: Receive a random inspection result of the task image, where the random inspection result includes: selecting a random inspection image from the task images whose annotation information is correct as the re-examination result according to a preset rule, and verifying the annotation information of the random inspection image Whether the processing result is correct; S502: If the random inspection result includes an error in the labeling information, feedback the random inspection information of the corresponding random inspection image; S504: Receive a second re-inspection result, where the second re-inspection result includes an annotation result obtained by correcting annotation information of the corresponding task image based on the sampling information. It should be noted that the sample image annotation information processing method described in the foregoing embodiment of this specification that can be applied to the same system server, according to the foregoing method of dividing multiple nodes and setting corresponding node terminal equipment, the description of the embodiment may also include Other implementations, such as setting up buried points to monitor the correct rate of processing the information marked by the operators of different processing nodes, and so on. For specific implementation manners, reference may be made to the description of the related method embodiments, which will not be repeated here. The various embodiments of the above method in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For related details, please refer to the part of the description of the method embodiment. The method embodiments provided in the embodiments of this specification can be executed in a mobile terminal, a computer terminal, a server, or a similar computing device. Taking running on a server as an example, FIG. 7 is a block diagram of the hardware structure of a training sample image annotation information processing server according to an embodiment of the present invention. As shown in FIG. 7, the server 10 may include one or more (only one is shown in the figure) processor 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA. ), a memory 104 for storing data, and a transmission module 106 for communication. Those of ordinary skill in the art can understand that the structure shown in FIG. 7 is only for illustration, and does not limit the structure of the above electronic device. For example, the server 10 may also include more or fewer components than those shown in FIG. 7, for example, may also include other processing hardware, such as a GPU (Graphics Processing Unit, image processor), or have the same components as those shown in FIG. Different configurations are shown. The storage 104 can be used to store software programs and modules of application software, such as program instructions/modules corresponding to the search method in the embodiment of the present invention. The processor 102 runs the software programs and modules stored in the storage 104, In this way, various functional applications and data processing are executed, that is, the processing method for displaying the contents of the navigation interactive interface described above is realized. The storage 104 may include a high-speed random storage device, and may also include a non-volatile storage, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state storage. In some examples, the storage 104 may further include storage remotely disposed relative to the processor 102, and these remote storages may be connected to the computer terminal 10 via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranet, local area network, mobile communication network, and combinations thereof.

The transmission module 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station to communicate with the Internet. In an example, the transmission module 106 may be a radio frequency (RF) module, which is used to communicate with the Internet in a wireless manner.

Based on the above-mentioned sample image labeling information processing method, this specification also provides a sample image labeling information processing device. The described devices may include systems (including distributed systems), software (applications), modules, components, servers, clients, etc. that use the methods described in the embodiments of this specification, combined with necessary hardware-implemented equipment and devices . Based on the same innovative concept, the processing device in an embodiment provided in this specification is as described in the following embodiment. Since the implementation scheme of the device to solve the problem is similar to the method, the implementation of the specific processing device in the embodiment of this specification can refer to the implementation of the foregoing method, and the repetition will not be repeated. Although the devices described in the following embodiments are preferably implemented by software, the implementation of hardware or a combination of software and hardware is also possible and conceived. Specifically, as shown in FIG. 8, FIG. 8 is a schematic diagram of the module structure of an embodiment of a sample image tagging information processing apparatus provided in this specification. Specifically, it may include: an image receiving module 201, which can be used to receive task diagrams. For example, the task image includes at least the following label information: the category corresponding to the target in the identified task image and the label frame of the target; The annotation inspection interaction module 202 may be used to receive the inspection result of the task image, and send the inspection processed task image to the annotation review interaction module 203. The inspection result includes: The annotation information undergoes the first inspection process, and when it is determined that there is an error in the annotation information, the annotation result obtained after correcting the annotation information; can also be used to receive the first re-inspection result and send the first re-inspection result The annotation review interaction module 203 performs a second inspection process, and the first re-inspection result includes an annotation result obtained by correcting the annotation information of the task image based on the error type in the review result; an annotation review interaction The module 203 may be used to receive the recheck result, and when the recheck result includes an error in the labeling information, send the recheck result to the label check interaction module 202 for the first recheck process, and the recheck result includes: A second inspection process is performed on the task image, and when there is an error in the annotation information, it is determined that the inspection result has an error type of error.

As shown in FIG. 9, FIG. 9 is a schematic diagram of the module structure of another embodiment of a sample image annotation information processing apparatus provided in this specification. The apparatus may further include: an annotation spot check interaction module 204, which can be used for the The task image with correct annotation information sent by the annotation review interaction module 203 can also be used to receive the random inspection result, and when the random inspection result includes an error in the annotation information, the random inspection information of the corresponding random inspection image is sent to the The marking review interaction module 203 performs the second inspection process, and the sampling result includes: selecting a sampling image from the received task images according to a preset rule, and verifying the The processing result obtained from whether the marking information of the random inspection image is correct; correspondingly, the marking review interaction module 203 can also be used to receive the second re-inspection result, and send the second re-inspection result to the marking random inspection In the interaction module 204, the second recheck result includes an annotation result obtained by correcting the annotation information of the corresponding task image based on the sampling information.

In another embodiment of the device, the selection of the randomized images by the marking and random check interaction module 204 may specifically include: selecting a task image based on at least one of the user identification and the execution date of the marked information processing.

In another embodiment of the device, the tagging and random checking interaction module 204 sending the sampling information of the corresponding sampling image to the tagging review interaction module 203 may include at least one of the following methods: Sampling images with errors are sent to the annotation review interaction module 203; if there are errors in the annotation information, the task image set corresponding to the sampling images is sent to the annotation review interaction module 203; The marking error information is sent to the marking review interaction module 203.

In another embodiment of the device, the task image or the image set corresponding to the task image that passed the random inspection can be added to the corresponding training sample set, so that the annotation information of the images in the training sample set is more improved. Accurate, making subsequent algorithm processing based on training sample images more accurate. Specifically, in another embodiment, the device may further include: The output module 205 may be used to mark the task image set corresponding to the sampled image whose labeling information is correct as the sampled image as a training sample image, and store it in the corresponding training sample set.

Another embodiment of the device provided in this specification may further include:

The first buried point processing module 206 can be used to identify a predetermined ratio of surveillance images added to the task image, and the known annotation information of the surveillance image includes the identified target and the corresponding category and annotation Frame information; can also be used to obtain the recognition result of the annotation information of the monitoring image in the inspection result; it can also be used to compare the recognition result with the known annotation information to determine the inspection standard of the inspection result Accuracy rate.

Figure 10 is a schematic diagram of the module structure of another embodiment of the device provided in this specification. As shown in Figure 10, in another embodiment of the device, the results of the review of the second node operator can also be embedded. monitor. The monitoring image used for buried point monitoring here can be the same as the monitoring image used by the first node or the first buried point processing module 206, that is, the same batch of monitoring images can be used to calculate the correct rate of labeling information processing by different operators. Of course, different surveillance images can also be used. Specifically, in another embodiment of the device, it may further include:

The second buried point processing module 207 can be used to identify a predetermined ratio of surveillance images added to the task image, and the known annotation information of the surveillance image includes the identified target and the corresponding category and annotation Frame information; can also be used to obtain the recognition result of the annotation information of the monitoring image in the inspection result; it can also be used to compare the recognition result with the known annotation information to determine the inspection standard of the inspection result Accuracy rate.

In another embodiment of the device, a notification message can be sent correspondingly based on the detection accuracy rate or the recheck accuracy rate calculated by the buried point. For example, when the inspection accuracy rate of operator A is within the first threshold interval within a week, operator A can send "the accuracy rate of labeling information is low, please check the reason and handle it carefully". If the inspection accuracy rate of the operator A in 10 working days is lower than the minimum assessment threshold on average, he can send a notification message to the designated supervision terminal, for example, send a message to the marked spot check interaction module 104 or a dedicated terminal device, or even attach It is recommended that information such as job transfer or statistical error types be fed back to the designated receiver. Therefore, in another embodiment of the device provided in this specification, it may further include: The first notification module 2082 may be used to send a corresponding notification message when the check accuracy rate reaches a first threshold within a first preset period. Correspondingly, in another embodiment of the device, it may further include: The second notification module 2084 may be used to send a corresponding notification message when the review accuracy rate reaches a second threshold within a second preset period. FIG. 11 is a schematic diagram of the module structure of another embodiment of the device provided in this specification. In another embodiment of the device, the output module 105 can determine whether the sampled task image or task image set passes by combining the sampling inspection result and the re-examination accuracy rate calculated by the buried point. Specifically, in another embodiment of the device, the output module 105 adds the corresponding task image set to the training sample set when the recheck accuracy rate is within the error range and the sampling result passes. . The sample image annotation information processing method provided by the embodiment of this specification can be implemented by the processor in the computer by executing the corresponding program instructions, such as using the application programming language of the windows and Linux operating systems on the PC side, or other such as android, The necessary hardware implementation of the application design language corresponding to the iOS system, or the implementation of the processing logic based on a quantum computer, etc. Specifically, in an embodiment in which a server provided in this specification implements the above method, the server may include a processor and a storage for storing executable instructions of the processor, and the processor implements the following when executing the instructions: Receiving a task image, the task image including at least the following label information: the category corresponding to the target in the identified task image and the label frame of the target; Receive a check result of the task image, the check result includes: performing a first check process on the label information of the task image, and when it is determined that the label information has errors, the result is obtained after correcting the label information Mark the result; Receive a review result of the task image, and if the review result includes an error in the annotation information, feedback the error type. The review result includes: performing a second inspection process on the task image, and if the annotation information exists In the event of an error, the type of error identified in the inspection result that is incorrect; A first re-inspection result is received, and the second re-inspection process is performed on the first re-inspection result, where the first re-inspection result includes the annotation obtained by correcting the annotation information of the task image based on the error type result. The server may be a single server, a server cluster, or a server in a distributed system, and server terminals distributed on different processing nodes may be regarded as the same server. Based on the above-mentioned method, device or server, this specification also provides a sample image annotation information processing system. FIG. 12 is a schematic diagram of the framework structure of an embodiment of the system provided in this specification. As shown in FIG. 11, include: The first processing terminal may be used to receive the task image and the inspection result of the task image, and send the task image after the inspection process to the second processing terminal, and the task image includes at least the following annotation information: The category corresponding to the target in the identified task image and the label frame of the target, the check result includes: performing a first check process on the label information of the task image, and when it is determined that there is an error in the label information , The labeling result obtained after correcting the labeling information; and also for receiving the first re-inspection result, and sending the first re-inspection result to the first terminal for second inspection processing, the first re-inspection The result includes the annotation result obtained by correcting the annotation information of the task image based on the error type in the review result; The second processing terminal may be used to receive the review result, and send the review result to the first processing terminal to perform the first re-inspection process when the re-inspection result includes an error in the labeled information, and the review result includes: The task image is subjected to a second inspection process. When there is an error in the annotation information, it is determined that the inspection result has an error type; it can also be used to receive a second inspection result and perform the second inspection. The result is sent to the third processing terminal, and the second recheck result includes the annotation result obtained by correcting the annotation information of the corresponding task image based on the sampling information The third processing terminal can be used to receive the task image with the correct labeling information sent by the second processing terminal, and also to receive the random inspection result, and when the random inspection result includes an error in the labeling information, the corresponding random inspection map The sampling information of the image is sent to the second terminal for second re-inspection processing, and the sampling result includes: selecting a sampling image from the received task images according to a preset rule, and verifying whether the annotation information of the sampling image is Correct result of processing. The above-mentioned instructions can be stored in a variety of computer-readable storage media. The computer-readable storage medium may include a physical device for storing information, and the information may be digitized and then stored in a medium using electric, magnetic, or optical methods. The computer-readable storage medium described in this embodiment may include: devices that use electrical energy to store information, such as various types of storage, such as RAM, ROM, etc.; devices that use magnetic energy to store information, such as hard disks, floppy disks, Magnetic tapes, magnetic core storage, bubble storage, flash drives; devices that use optical methods to store information, such as CDs or DVDs. Of course, there are other ways of readable storage media, such as quantum storage, graphene storage, and so on. The instructions in the device or server or client or system described below are the same as those described above. It should be noted that the above-mentioned devices, servers, and systems in the embodiments of this specification may also include other implementation manners according to the description of the relevant method embodiments. For specific implementation manners, reference may be made to the description of the method embodiments, which will not be repeated here. The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for the relevant parts, please refer to the part of the description of the method embodiment. The foregoing describes specific embodiments of this specification. Other embodiments are within the scope of the attached patent application. In some cases, the actions or steps described in the scope of the patent application may be performed in a different order than in the embodiments and still achieve desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown in order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. This manual provides interactive judgment processing logic for different processing results of multiple operation nodes. When the image annotation information is wrong, it can automatically return to allow the operator to re-examine, modify, and other processing. In this way, the business ability of the operators can be improved in the continuous feedback interaction, the labeling efficiency can be gradually improved, and the accuracy of the training sample image labeling information can be effectively improved. Although this application provides method operation steps as described in the embodiments or flowcharts, conventional or non-progressive labor may include more or fewer operation steps. The sequence of steps listed in the embodiment is only one way of the execution sequence of many steps, and does not represent the only execution sequence. When the actual device or client product is executed, it can be executed sequentially or in parallel according to the method shown in the embodiment or the drawings (for example, a parallel processor or a multi-threaded environment). Although the content of the embodiments of this specification mentions operations and descriptions of data acquisition, interaction, calculation, judgment, etc., such as the calculation of inspection accuracy/re-examination accuracy by marking the area of the frame, the random inspection of the second node by the third node, etc., but The embodiments of this specification are not limited to the situations described in the embodiments of this specification that must comply with industry communication standards, standard image data processing protocols, communication protocols, and standard data models/templates. Certain industry standards or implementations described in custom methods or examples with slight modifications can also achieve the same, equivalent or similar implementation effects of the foregoing examples, or predictable implementation effects after modification. The examples obtained by applying these modified or deformed data acquisition, storage, judgment, processing methods, etc., can still fall within the scope of the optional implementation schemes of this specification. In the 1990s, the improvement of a technology can be clearly distinguished from the improvement of the hardware (for example, the improvement of the circuit structure of diodes, transistors, switches, etc.) or the improvement of the software (for the process of the method). Improve). However, with the development of technology, the improvement of many methods and processes of today can be regarded as a direct improvement of the hardware circuit structure. Designers almost always get the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be realized by the hardware entity module. For example, Programmable Logic Device (PLD) (such as Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device . It is programmed by the designer to "integrate" a digital system on a single PLD, without requiring the chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this programming is mostly realized by using "logic compiler" software, which is similar to the software compiler used in program development and writing. The original code before compilation must also be written in a specific programming language, which is called Hardware Description Language (HDL), and HDL is not only one, but there are many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., Currently the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. It should also be clear to those skilled in the art that only need to logically program the method flow in the above-mentioned hardware description languages and program it into an integrated circuit, the hardware circuit that implements the logic method flow can be easily obtained. The controller can be implemented in any suitable manner. For example, the controller can be a microprocessor or a processor and a computer readable program code (such as software or firmware) that can be executed by the (micro) processor. Media, logic gates, switches, application specific integrated circuits (ASICs), programmable logic controllers and embedded microcontrollers. Examples of controllers include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicon Labs C8051F320, the memory controller can also be implemented as part of the memory control logic. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, it is entirely possible to program the method steps in logic to enable the controller to be controlled by logic gates, switches, dedicated integrated circuits, and programmable logic. The same function can be realized in the form of an embedded micro-controller, etc. Therefore, such a controller can be regarded as a hardware component, and the devices included in it for realizing various functions can also be regarded as a structure within the hardware component. Or even, the device for realizing various functions can be regarded as both a software module for realizing the method and a structure within a hardware component. The systems, devices, modules, or units explained in the above embodiments may be implemented by computer chips or entities, or implemented by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, and a tablet. Computers, wearable devices, or any combination of these devices. Although the embodiments of this specification provide method operation steps as described in the embodiments or flowcharts, conventional or non-progressive means may include more or fewer operation steps. The sequence of steps listed in the embodiment is only one way of the execution sequence of many steps, and does not represent the only execution sequence. When the actual device or terminal product is executed, it can be executed sequentially or in parallel according to the method shown in the embodiment or the drawings (for example, a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, product, or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed. Elements, or also include elements inherent to such processes, methods, products, or equipment. If there are no more restrictions, it does not exclude that there are other identical or equivalent elements in the process, method, product, or device including the elements. For the convenience of description, when describing the above device, the functions are divided into various modules and described separately. Of course, when implementing the embodiments of this specification, the functions of each module can be implemented in the same or multiple software and/or hardware, or the modules that implement the same function can be composed of multiple sub-modules or sub-units. Realization and so on. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components can be combined or integrated into Another system, or some features can be ignored, or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, it is entirely possible to program the method steps in logic to enable the controller to be controlled by logic gates, switches, dedicated integrated circuits, and programmable logic. The same function can be realized in the form of an embedded micro-controller, etc. Therefore, such a controller can be regarded as a hardware component, and the devices included in the controller for realizing various functions can also be regarded as a structure within the hardware component. Or even, the device for realizing various functions can be regarded as both a software module for realizing the method and a structure within a hardware component. The present invention is described with reference to flowcharts and/or block diagrams of methods, equipment (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and/or block in the flowchart and/or block diagram, and the combination of processes and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, dedicated computers, embedded processors, or other programmable data processing equipment to generate a machine that can be executed by the processors of the computer or other programmable data processing equipment A device for realizing the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram is generated. These computer program instructions can also be stored in a computer-readable storage that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable storage produce a manufactured product including the instruction device , The instruction device realizes the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram. These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operating steps are performed on the computer or other programmable equipment to generate computer-implemented processing, so that the computer or other programmable equipment The instructions executed above provide steps for implementing functions specified in a flow or multiple flows in the flowchart and/or a block or multiple blocks in the block diagram. In a typical configuration, the computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory. Memory may include non-permanent storage in computer-readable media, random access storage (RAM) and/or non-volatile storage, such as read-only storage (ROM) or flash memory ( flash RAM). Memory is an example of computer readable media. Computer-readable media include permanent and non-permanent, movable and non-movable media, and information storage can be realized by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), and other types of random access memory (RAM) , Read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital multi-function Optical discs (DVD) or other optical storage, cassette tapes, magnetic tape storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by computing devices. According to the definition in this article, computer-readable media does not include transitory media, such as modulated data signals and carrier waves. Those skilled in the art should understand that the embodiments of this specification can be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may adopt the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of this specification may adopt computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. form. The embodiments of this specification can be described in the general context of computer-executable instructions executed by a computer, such as a program module. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The embodiments of this specification can also be practiced in distributed computing environments. In these distributed computing environments, remote processing devices connected through a communication network perform tasks. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices. The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, as for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the part of the description of the method embodiment. In the description of this specification, descriptions with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. mean specific features described in conjunction with the embodiment or example , Structure, materials or features are included in at least one embodiment or example of the embodiments of this specification. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the described specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples and the features of the different embodiments or examples described in this specification without contradicting each other. The above descriptions are only examples of the embodiments of the present specification, and are not used to limit the embodiments of the present specification. For those skilled in the art, various modifications and changes can be made to the embodiments of this specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiment of this specification should be included in the scope of the claims of the embodiment of this specification.

S0‧‧‧Step S2‧‧‧Step S4‧‧‧Step S6‧‧‧Step S80‧‧‧Step S82‧‧‧Step S84‧‧‧Step S86‧‧‧Step S100‧‧‧Step S200‧‧‧Step S300‧‧‧Step S400‧‧‧Step 10‧‧‧Server 102‧‧‧Processor 104‧‧‧Non-volatile memory 106‧‧‧Transmission Module 201‧‧‧Image receiving module 202‧‧‧Annotation check interactive module 203‧‧‧Marking review interactive module 204‧‧‧Marking and Spot Checking Interactive Module 206‧‧‧The first buried point processing module 207‧‧‧Second buried point processing module 2082‧‧‧First Notification Module 2084‧‧‧Second Notification Module

In order to more clearly explain the technical solutions in the embodiments of this specification or the prior art, the following will briefly introduce the drawings that need to be used in the embodiments or the description of the prior art. Obviously, the drawings in the following description are merely the present For some of the embodiments described in the specification, for those of ordinary skill in the art, other schemes can be obtained based on these schemes without making progressive labor. Fig. 1 is a schematic diagram of a job scene for labeling multiple targets in a sample image according to the method described in this specification; 2 is a schematic flowchart of an embodiment of the method for processing sample image annotation information provided in this specification; Fig. 3 is a schematic diagram of a process of reviewing task images at a second node in an implementation scenario of this specification; FIG. 4 is a schematic diagram of the method flow of another embodiment of the method provided in this specification; Figure 5 is a schematic diagram of a processing scenario for determining the inspection accuracy in this manual; FIG. 6 is a schematic diagram of a process flow diagram of a method for processing sample image annotation information for a server provided in this specification; 7 is a block diagram of the hardware structure of a training sample image annotation information processing server according to an embodiment of the present invention; 8 is a schematic diagram of the module structure of an embodiment of a sample image annotation information processing device provided in this specification; 9 is a schematic diagram of the module structure of another embodiment of the device provided in this specification; Figure 10 is a schematic diagram of the module structure of another embodiment of the device provided in this specification 11 is a schematic diagram of the module structure of another embodiment of the device provided in this specification; Fig. 12 is a schematic diagram of the frame structure of an embodiment of the system provided in this specification.

Claims (24)

An image annotation information processing method, the method comprising: a first node receives a task image, the task image includes at least the following annotation information: the category corresponding to the target in the identified task image, and the The marking frame of the target; the first node receives the inspection result of the task image, and sends the task image processed by the first inspection to the second node, the inspection result includes: marking information on the task image Perform the first inspection process, and when it is determined that the annotation information has errors, the annotation result obtained after correcting the annotation information; the second node receives the review result, and if the review result includes an error in the annotation information, it will The recheck result is sent to the first node for a first recheck process, and the recheck result includes: performing a second check process on the task image, and when there is an error in the annotation information, the determined check result appears The error type of the error; the first node receives the first re-check result, and sends the first re-check result to the second node for the second check process, and the first re-check result includes The annotation result obtained by correcting the annotation information of the task image by describing the error type in the review result; adding a predetermined ratio of surveillance images to the task image, and the known annotation information of the surveillance image includes recognition The target and the corresponding category and label box information. According to the method of claim 1, the method further includes: sending the task image with the correct annotation information in the review result to a third node; the third node receives the result of the random inspection, and the result of the random inspection includes : The processing result obtained by selecting random images from the received task images according to preset rules and verifying whether the annotation information of the random images is correct; if the random inspection results include errors in the annotation information, the corresponding random inspection The sampling information of the image is sent to the second node for second re-inspection processing; correspondingly, the second node receives the second re-inspection result, and sends the second re-inspection result to the third node, The second recheck result includes an annotation result obtained by correcting the annotation information of the corresponding task image based on the sampling information. According to the method of claim 2, the selecting the random images includes: selecting the task image based on at least one of the user identification and the execution date of the annotation information processing. According to the method of claim 2, the sending the sampling information of the corresponding sampling image to the second node includes at least one of the following methods: sending the sampling image with the error in the annotation information to the first node Two nodes; if there is an error in the annotation information, send the task image set corresponding to the sampled image to the second node; Send the marking error information of the randomly checked image to the second node. According to the method described in claim 2, the method further includes: marking the task image set corresponding to the sampled image whose labeling information is correct as the sampled image as the training sample image. According to the method of claim 1, the method further includes: obtaining a recognition result of the annotation information of the monitoring image in the inspection result; comparing the recognition result with the known annotation information to determine the inspection Check accuracy of results. According to the method of claim 6, the method further includes: sending a corresponding notification message when the check accuracy rate reaches a first threshold within a first preset period. According to the method of claim 2, the method further includes: obtaining a recognition result of the annotation information of the monitoring image in the review result; comparing the recognition result with the known annotation information to determine the review The accuracy of the results of the review. According to the method according to claim 8, the method further includes: when the recheck accuracy rate reaches a second threshold within a second preset period When the time, the corresponding notification message is sent out. The method according to claim 8, the method further includes: if the recheck accuracy rate is within the error range and the sampling check result passes, adding the task image set corresponding to the sampling image to the training sample set. An image annotation information processing method, the method comprising: receiving a task image, the task image includes at least the following annotation information: the category corresponding to the target in the identified task image, and the target's Annotation frame; receiving the inspection result of the task image, the inspection result includes: performing a first inspection process on the annotation information of the task image, and correcting the annotation information when it is determined that there is an error in the annotation information The annotation result obtained later; receiving the review result of the task image, and if the review result includes an error in the annotation information, feedback the error type, and the review result includes: performing a second inspection process on the task image, When there is an error in the annotation information, it is determined that the check result has an error type; the first recheck result is received, the second check process is performed on the first recheck result, and the first recheck is The result includes the annotation result obtained by correcting the annotation information of the task image based on the error type; adding a predetermined ratio of monitoring images to the task image, and the monitoring The known label information of the control image includes the identified target and the corresponding category and label frame information. According to the method of claim 11, the method further includes: receiving a sampling result of the task image, the sampling result including: selecting, according to a preset rule, a task image whose re-examination result is the correct annotation information Sampling images, verifying whether the annotation information of the sampling images is correct and obtaining the processing results; if the sampling results include errors in the annotation information, then feeding back the sampling information of the corresponding sampling images; receiving the second inspection result, The second recheck result includes an annotation result obtained by correcting the annotation information of the corresponding task image based on the sampling information. A sample image tagging information processing device, the device comprising: an image receiving module for receiving task images, the task images including at least the following tagging information: the identified target location in the task image The corresponding category, and the marking frame of the target; the marking inspection interactive module, which is used to receive the inspection result of the task image, and send the task image processed by the first inspection to the marking review interactive module, the The inspection result includes: performing a first inspection process on the annotation information of the task image, and when it is determined that the annotation information has errors, the annotation result obtained by correcting the annotation information; and is also used to receive the first re-inspection result , Sending the first re-check result to the annotation re-check interaction model The group performs a second inspection process. The first re-inspection result includes the annotation result obtained by correcting the annotation information of the task image based on the error type in the review result; the annotation review interaction module is used to receive the review result, And when the review result includes an error in the annotation information, sending the review result to the annotation inspection interaction module for a first re-inspection process, the review result includes: performing a second inspection process on the task image, When there is an error in the annotation information, the determined check result has an error type. According to the device of claim 13, the device further includes: an annotation random check interaction module for receiving task images with correct annotation information sent by the annotation review interaction module, and also for receiving random inspection results, and When the spot check result includes the error in the labeling information, the spot check information of the corresponding spot check image is sent to the labeling review interaction module for second recheck processing, and the spot check result includes: the task received from the preset rule Select a sampling image from the image to verify whether the annotation information of the sampling image is correct and obtain the processing result; correspondingly, the annotation review interaction module is also used to receive the second re-inspection result, and the second The re-inspection result is sent to the annotation random-check interaction module, and the second re-inspection result includes the annotation result obtained by correcting the annotation information of the corresponding task image based on the random inspection information. In the device according to claim 14, the mark-check interaction module selects The random inspection images include at least one selected task image based on the user identification and the execution date of the annotation information processing. According to the device according to claim 14, the marking and spot check interaction module sending the corresponding spot check information of the spot check image to the mark review interaction module includes at least one of the following ways: the marking information has an error in the spot check The image is sent to the annotation review interaction module; if there is an error in the annotation information, the task image set corresponding to the sampling image is sent to the annotation review interaction module; the annotation error information of the sampling image is sent to the office Describe the annotation review interactive module. According to the device of claim 14, the device further includes: an output module for marking the task image set corresponding to the sampling image with the correct labeling information as the training sample image as the training sample image, and storing it in the corresponding In the training sample collection. The device according to claim 13, wherein the device further includes: a first buried point processing module for identifying a predetermined ratio of surveillance images added to the task image, and the known surveillance image The label information includes the identified target and the corresponding category and label frame information; it is also used to obtain the recognition result of the label information of the monitoring image in the inspection result; It is also used to compare the recognition result with the known label information to determine the inspection accuracy rate of the inspection result. The device according to claim 14, the device further includes: a second buried point processing module, configured to identify a predetermined ratio of surveillance images added to the task image, the known surveillance image The label information includes the identified target and the corresponding category and label frame information; it is also used to obtain the recognition result of the label information of the monitoring image in the review result; and it is also used to compare the recognition result with the known Mark the information to determine the check accuracy rate of the recheck result. The device according to claim 18, further comprising: a first notification module, configured to send a corresponding notification message when the check accuracy rate reaches the first threshold within the first preset period. The device according to claim 19, further comprising: a second notification module, configured to send a corresponding notification message when the review accuracy rate reaches a second threshold within a second preset period. The device according to claim 21, wherein the device further includes an output module for adding the corresponding task image set to when the recheck accuracy rate is within the error range and the sampling result passes Training sample collection. A server includes a processor and a memory for storing executable instructions of the processor. When the processor executes the instructions, the processor receives a task image, and the task image includes at least the following annotation information: recognition The category corresponding to the target in the task image and the label frame of the target; receiving a check result of the task image, the check result including: performing a first check process on the label information of the task image , When it is determined that there is an error in the annotation information, the annotation result obtained after the annotation information is corrected; the review result of the task image is received, and if the review result includes an error in the annotation information, the error type is fed back, so The description of the review result includes: performing a second inspection process on the task image, and when there is an error in the annotation information, determining the error type of the error in the inspection result; receiving the first re-inspection result, and verifying the first inspection result. The second inspection process is performed on the re-inspection result, and the first re-inspection result includes the annotation result obtained by correcting the annotation information of the task image based on the error type. An image tagging information processing system, including: a first processing terminal for receiving task images and inspection results of the task images, and sending the task images after inspection processing to the second processing terminal; and also for receiving The first re-inspection result, the first re-inspection result is sent to the second terminal for second inspection processing, the task image includes at least the following annotation information: the category corresponding to the target in the identified task image In addition to the marking frame of the target, the inspection result includes: performing a first inspection process on the marking information of the task image, and when it is determined that the marking information has errors, the marking obtained by correcting the marking information As a result, the first re-inspection result includes the annotation result obtained by correcting the annotation information of the task image based on the error type in the re-inspection result; the second processing terminal is used to receive the re-inspection result, and When the recheck result includes the error in the labeled information, the recheck result is sent to the first processing terminal for the first recheck process; it is also used to receive the second recheck result, and send the second recheck result to the third process In the terminal, the review result includes: performing a second inspection process on the task image, and when there is an error in the annotation information, the determined error type of the inspection result is wrong, and the second inspection result includes an error type based on The sampling information is the marking result obtained by modifying the marking information of the corresponding task image; the third processing terminal is used to receive the task image with the correct marking information sent by the second processing terminal, and is also used to receive the sampling result , And when the spot check result includes an error in the labeling information, the spot check information of the corresponding spot check image is sent to the second terminal for second recheck processing, and the spot check result includes: The processing result obtained by selecting a random inspection image from the task image and verifying whether the annotation information of the random inspection image is correct.

Images