CN115272742A - Medical image labeling method and system - Google Patents

Abstract

The disclosure relates to a medical image labeling method and system. The method comprises the following steps: acquiring a medical image labeling task; pre-labeling the medical image through a preset pre-labeling model to obtain a pre-labeling result; obtaining at least one annotator marking result corresponding to at least one annotator according to modification confirmation of the pre-marking result by the at least one annotator; automatically verifying at least one marker marking result according to a preset verification rule, determining whether a marker marking result which fails in verification exists, and returning the marker marking result which fails in verification to a corresponding marker for confirmation when the marker marking result which fails in verification exists; and when the labeling result of the labeling personnel which fails to pass the verification does not exist, determining a final labeling result according to at least one labeling personnel labeling result. Through the scheme disclosed by the invention, the time consumed by manual marking can be reduced, the marking efficiency is improved, and the marking quality is ensured.

Description

Medical image labeling method and system

Technical Field

The present disclosure relates generally to the field of medical image annotation technology. More particularly, the present disclosure relates to a medical image annotation method and system.

Background

With the vigorous development of medical technology, artificial intelligence is applied more and more in the medical field, and behind the application of the artificial intelligence and relevant algorithm models, a large amount of labeled data is required to be continuously fed so as to be capable of continuously iterating the algorithm models and generating the models.

In the conventional method based on medical image annotation, an annotator needs to read and identify an image and make final diagnosis and judgment on the image for annotation. The marking mode is very inefficient, the individual difference is large, a marker is easy to miss marks and mistake marks by the experience of a person, the marker is tired due to long-time marking, and the film reading accuracy rate is reduced.

In addition, for some tasks with complicated labeling, such as lesion area segmentation, a labeling person needs to perform complete edge delineation with dense and numb dots or a painting brush, which is time-consuming, not only causes visual fatigue, but also is tired due to tedious and repeated work, is difficult to maintain a stable labeling level and is slow in labeling progress.

Disclosure of Invention

In order to at least partially solve the technical problems mentioned in the background, the present disclosure provides a medical image annotation method and system.

According to a first aspect of the present disclosure, the present disclosure provides a medical image annotation method, wherein the method comprises: acquiring a medical image annotation task, wherein the medical image annotation task comprises a medical image to be annotated, a preset pre-annotation model and a preset verification rule; pre-labeling the medical image to be labeled through the preset pre-labeling model to obtain a pre-labeling result; obtaining at least one annotator marking result corresponding to at least one annotator according to modification confirmation of the pre-marking result by the at least one annotator; automatically verifying the at least one marker marking result corresponding to the at least one marker according to the preset verification rule, determining whether a marker marking result which fails in verification exists, and returning the marker marking result which fails in verification to the corresponding marker for confirmation when the marker marking result which fails in verification exists; and when the labeling result of the labeling personnel which fails to pass the verification does not exist, determining a final labeling result according to the labeling result of the at least one labeling personnel corresponding to the at least one labeling personnel.

Optionally, the pre-labeling the medical image to be labeled through the preset pre-labeling model, and obtaining a pre-labeling result includes: respectively pre-labeling the medical image to be labeled through a plurality of preset pre-labeling models to obtain a plurality of model labeling results; and obtaining the pre-labeling result according to the plurality of model labeling results.

Optionally, the preset pre-labeling model includes a disease recognition model, a lesion detection model, and a lesion segmentation model.

Optionally, obtaining at least one annotator tagging result corresponding to each of the at least one annotator according to the modification confirmation performed on the pre-tagging result by the at least one annotator includes: acquiring modification confirmation information provided by the annotator, and when the modification confirmation information indicates that the pre-annotation result needs to be modified, acquiring the pre-annotation result modified by the annotator as the annotation result of the annotator; and when the modification confirmation information indicates that the pre-labeling result does not need to be modified, taking the pre-labeling result as a labeling result of a labeling operator.

Optionally, automatically verifying the at least one annotator marking result corresponding to the at least one annotator according to the preset verification rule, and determining whether the annotator marking result which fails to pass the verification exists includes: aiming at the labeling result of the label maker, selecting a corresponding verification option according to a preset pre-labeling model for verification, wherein the verification option comprises image label omission, label omission, area label omission, label error label, area error label and label contradiction; and when any one of the selected verification options is verified to exist, determining that the marking result of the marking person fails to be verified, otherwise, determining that the marking result of the marking person passes to be verified.

Optionally, the method further includes: and when the marking result of the marking member which fails in the verification exists, returning verification information corresponding to the marking result of the marking member which fails in the verification to the corresponding marking member, so that the corresponding marking member modifies the marking result of the marking member which fails in the verification according to the verification information.

Optionally, when there is no annotator marking result that fails in the verification, determining a final marking result according to the at least one annotator marking result corresponding to the at least one annotator includes: when the labeling result of the labeling personnel which fails to pass the verification does not exist, taking the labeling result of one labeling personnel corresponding to one labeling personnel as the final labeling result aiming at the condition that the number of the labeling personnel is one; when the labeling result of the labeling personnel which fails to pass the verification does not exist, comparing a plurality of labeling personnel labeling results respectively corresponding to a plurality of labeling personnel aiming at the condition that the number of the labeling personnel is multiple, and determining whether the labeling results of the labeling personnel meet the preset consistency requirement or not; if the labeling results of the plurality of markers meet the preset consistency requirement, taking any one of the labeling results of the plurality of markers as a final labeling result; and if the labeling results of the plurality of annotators do not meet the preset consistency requirement, enabling an auditor to audit the labeling results of the plurality of annotators, and obtaining a final labeling result according to confirmation made by one labeling result selected by the auditor.

Optionally, the method further includes: and forming training data based on the final labeling result so as to train the preset pre-labeling model by using the training data.

Optionally, the method further comprises creating the medical image annotation task.

According to a second aspect of the present disclosure, the present disclosure provides a medical image annotation system, wherein the system comprises: the medical image annotation system comprises an acquisition module, a verification module and a verification module, wherein the acquisition module is used for acquiring a medical image annotation task, and the medical image annotation task comprises a medical image to be annotated, a preset pre-annotation model and a preset verification rule; the pre-labeling module is used for pre-labeling the medical image to be labeled through the preset pre-labeling model to obtain a pre-labeling result; the modification confirmation module is used for obtaining at least one annotator marking result corresponding to at least one annotator according to modification confirmation of the pre-marking result by the at least one annotator; the verification module is used for automatically verifying the at least one marker marking result corresponding to the at least one marker according to the preset verification rule, determining whether a marker marking result which fails in verification exists or not, and returning the marker marking result which fails in verification to the corresponding marker for confirmation when the marker marking result which fails in verification exists; and the marking result confirming module is used for determining a final marking result according to the at least one marking member marking result corresponding to the at least one marking member when the marking member marking result which is failed in verification does not exist.

Optionally, the pre-labeling module pre-labels the medical image to be labeled through the preset pre-labeling model in the following manner to obtain a pre-labeling result: respectively pre-labeling the medical image to be labeled through a plurality of preset pre-labeling models to obtain a plurality of model labeling results; and obtaining the pre-labeling result according to the plurality of model labeling results.

Optionally, the preset pre-labeling model includes a disease recognition model, a lesion detection model and a lesion segmentation model.

Optionally, the modification confirming module obtains at least one annotator tagging result corresponding to each of the at least one annotator according to modification confirmation performed on the pre-tagging result by the at least one annotator in the following manner: acquiring modification confirmation information provided by the annotator, and when the modification confirmation information is that the pre-annotation result needs to be modified, acquiring the pre-annotation result modified by the annotator as an annotation result of the annotator; and when the modification confirmation information is that the pre-labeling result does not need to be modified, taking the pre-labeling result as a labeling result of a labeling person.

Optionally, the verification module automatically verifies the at least one annotator tagging result corresponding to the at least one annotator according to the preset verification rule, and determines whether an annotator tagging result which fails to be verified exists: aiming at the labeling result of the label maker, selecting a corresponding verification option according to a preset pre-labeling model for verification, wherein the verification option comprises image label omission, label omission, area label omission, label error label, area error label and label contradiction; and when any one of the selected verification options is verified to exist, determining that the annotation result verification of the annotator fails, otherwise, determining that the annotation result verification of the annotator passes.

Optionally, the verification module is further configured to, when a marker marking result that fails to be verified exists, return verification information corresponding to the marker marking result that fails to be verified to the corresponding marker, so that the corresponding marker modifies the marker marking result that fails to be verified according to the verification information.

The labeling result confirming module comprises: the comparison unit is used for comparing a plurality of label maker labeling results respectively corresponding to a plurality of label makers when the number of label makers is multiple and no label maker labeling result which fails to pass the verification exists, and determining whether the plurality of label maker labeling results meet the preset consistency requirement; and the auditing unit is used for enabling an auditor to audit the plurality of annotator marking results if the plurality of annotator marking results do not accord with the preset consistency requirement, and obtaining a final marking result according to the confirmation made by one annotator marking result selected by the auditor.

Optionally, the system further includes: and the training module is used for forming training data based on the final labeling result so as to train the preset pre-labeling model by utilizing the training data.

Optionally, the system further includes: and the task creating module is used for creating the medical image labeling task.

According to a third aspect of the present disclosure, there is provided an electronic device, wherein the electronic device comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the method of the first aspect of the present disclosure.

According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium, wherein the storage medium stores a computer program which, when executed, implements the method of the first aspect of the present disclosure as described above.

Through the technical scheme, the algorithm model can be utilized to label the medical image data in advance, the target area and the label are labeled, and then the labeling personnel confirms and modifies the target area and the label, and the labeling result of the labeling personnel is automatically checked, so that the workload of the labeling personnel can be greatly reduced, the labeling efficiency is improved, and the labeling result is not easy to omit and has stable labeling level. In addition, the production speed of the labeling result can be accelerated by the technical scheme, so that training data formed by the labeling result is used for training the artificial intelligent algorithm model, the iteration of the algorithm model is facilitated, the algorithm model can be trained by a large amount of data, and the trained algorithm model can feed a labeling system for certain pre-labeling, so that the cost of the artificial feeding algorithm model can be reduced while the performance of the algorithm model is improved, and the closed-loop efficiency is improved.

Drawings

The above and other objects, features and advantages of exemplary embodiments of the present disclosure will become readily apparent from the following detailed description, which proceeds with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are illustrated by way of example and not by way of limitation, and like or corresponding reference numerals indicate like or corresponding parts, in which:

FIG. 1 is a flow diagram illustrating a medical image annotation method according to one embodiment of the present disclosure;

FIG. 2 is a flow diagram illustrating a medical image annotation method according to another embodiment of the present disclosure;

FIG. 3 is a flow diagram illustrating a medical image annotation method according to yet another embodiment of the present disclosure;

FIG. 4 is a schematic block diagram illustrating a medical annotation system in accordance with one embodiment of the present disclosure;

FIG. 5 is a schematic block diagram illustrating a medical annotation system in accordance with another embodiment of the present disclosure;

FIG. 6 is a schematic block diagram illustrating a medical annotation system in accordance with yet another embodiment of the present disclosure.

Detailed Description

The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are some, but not all embodiments of the present disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.

Specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

The present disclosure provides a medical image annotation method. Referring to fig. 1, fig. 1 is a flowchart illustrating a medical image annotation method according to an embodiment of the present disclosure. As shown in fig. 1, the medical image annotation method includes the following steps S101 to S105. Step S101: and acquiring a medical image labeling task, wherein the medical image labeling task comprises a medical image to be labeled, a preset pre-labeling model and a preset check rule. Step S102: and pre-labeling the medical image to be labeled through the preset pre-labeling model to obtain a pre-labeling result. Step S103: and obtaining at least one annotator marking result corresponding to each of the at least one annotator according to the modification confirmation of the pre-marking result by the at least one annotator. Step S104: and automatically verifying the at least one marker marking result corresponding to the at least one marker according to the preset verification rule, determining whether a marker marking result which fails in verification exists, and returning the marker marking result which fails in verification to the corresponding marker for confirmation when the marker marking result which fails in verification exists. Step S105: and when the labeling result of the labeling personnel which fails to pass the verification does not exist, determining a final labeling result according to the labeling result of the at least one labeling personnel corresponding to the at least one labeling personnel.

By the medical image labeling method, the medical image data can be labeled in advance by the aid of the algorithm model, the target area and the label are labeled, the label is submitted to a labeling person for confirmation and modification, the labeling result of the labeling person is automatically verified, workload of the labeling person can be greatly reduced, labeling efficiency is improved, and the labeling result is not easy to omit and is stable in labeling level.

In step S101, a medical image annotation task may be obtained, where the medical image annotation task includes a medical image to be annotated, a preset pre-annotation model, and a preset verification rule.

According to the embodiment of the disclosure, the medical image can be labeled based on the medical image labeling task, before the medical image is labeled, the medical image to be labeled is firstly obtained, and the medical image labeling task can comprise the medical image to be labeled, a preset check rule used in the labeling process and a preset pre-labeling model matched with the medical image to be labeled. Wherein the medical image to be annotated may be one or more.

In step S102, the medical image to be labeled may be pre-labeled through the preset pre-labeling model, so as to obtain a pre-labeling result.

According to the embodiment of the disclosure, after the medical image to be labeled and the corresponding preset pre-labeling model are obtained, the medical image to be labeled can be pre-labeled through the preset pre-labeling model, so that a pre-labeling result is obtained. The preset pre-labeling model may include a disease recognition model, a lesion detection model, and a lesion segmentation model, but is not limited thereto, and any suitable algorithm model may be used as the pre-labeling model of the present disclosure.

In an embodiment, the pre-labeling the medical image to be labeled through the preset pre-labeling model, and obtaining a pre-labeling result may include: respectively pre-labeling the medical image to be labeled through a plurality of preset pre-labeling models to obtain a plurality of model labeling results; and obtaining the pre-labeling result according to the plurality of model labeling results.

According to the embodiment, according to the medical image annotation task, the medical image can be individually pre-annotated through each pre-annotation model of the multiple pre-annotation models to obtain the corresponding annotation result of the model, and then the pre-annotation result is obtained by integrating the annotation results obtained by all the pre-annotation models. For example, the medical image may be subjected to lesion segmentation and disease identification through different pre-labeling models, so as to mark a lesion region and mark a specific disease label, and then the lesion region and the disease label are superimposed to obtain a pre-labeling result.

In another embodiment, the medical image to be labeled can be pre-labeled through a preset pre-labeling model, and a pre-labeling result is directly obtained. In this embodiment, for a simple medical image labeling task, the task may only need to label the lesion region (segmentation task), or the image to be labeled is specific enough (sufficient localization) and only needs to label the disease (classification task), so that a pre-labeling result can be directly obtained by using a given pre-labeling model.

In step S103, at least one annotator tagging result corresponding to each of the at least one annotator may be obtained according to the modification confirmation performed on the pre-tagging result by the at least one annotator.

According to the embodiment of the disclosure, after the medical image is pre-labeled, the pre-labeling result can be modified and confirmed by the annotator, and the annotating result of the annotator can be obtained. The annotators are the annotators appointed in the medical image annotation task, the number of the annotators can be one or more, and for the condition of a plurality of annotators, each annotator modifies and confirms the pre-annotation result to generate the annotation result of each annotator, so that the annotation results of the annotators can be obtained.

Further, according to the modification confirmation performed on the pre-annotation result by at least one annotator, obtaining at least one annotator annotation result corresponding to each of the at least one annotator comprises: acquiring modification confirmation information provided by the annotator, and when the modification confirmation information is that the pre-annotation result needs to be modified, acquiring the pre-annotation result modified by the annotator as an annotation result of the annotator; and when the modification confirmation information indicates that the pre-labeling result does not need to be modified, taking the pre-labeling result as a labeling result of a labeling operator.

In this embodiment, each annotator provides modification confirmation information, and if the pre-annotation result needs to be modified, provides the modified pre-annotation result, and if the pre-annotation result does not need to be modified, provides the unmodified pre-annotation result, so that the pre-annotation result returned by the annotator can be obtained according to the modification confirmation information.

In addition, the modification operation of the annotator for the output results of different types of preset pre-annotation models is as follows. When the preset pre-labeling model comprises a detection and segmentation type model, a labeling person can modify and confirm the following two dimensions according to actual conditions: on one hand, a region (which can be a point, a line segment, a rectangle, a polygon, an ellipse, a painting brush and the like) with a specific shape and composed of a plurality of points can be moved, deleted, newly added and the like, and the label corresponding to the current region can be only modified; on the other hand, operations such as moving, deleting, adding and the like can be carried out on a specific certain point. When the preset pre-labeled model comprises the classification model, the labels given by the classification model can be deleted and modified, and labels which are not predicted by the classification model can be added.

In step S104, the at least one annotator marking result corresponding to the at least one annotator may be automatically verified according to the preset verification rule, so as to determine whether there is an annotator marking result that fails to be verified, and when there is an annotator marking result that fails to be verified, the annotator marking result that fails to be verified is returned to the corresponding annotator for confirmation.

According to the embodiment of the disclosure, verification is required for each annotator marking result, and when the annotator marking result which fails in verification exists, the annotator marking result which fails in verification is returned to the corresponding annotator for confirmation and modification, that is, the method of the disclosure is executed from step 103 again.

According to an embodiment of the present disclosure, the method may further include: and when the marking result of the marking member which fails in the verification exists, returning verification information corresponding to the marking result of the marking member which fails in the verification to the corresponding marking member, so that the corresponding marking member modifies the marking result of the marking member which fails in the verification according to the verification information. Therefore, the annotator can know the problem of the annotation more clearly, and the modification efficiency is improved.

Further, automatically verifying the at least one annotator marking result corresponding to the at least one annotator according to the preset verification rule, and determining whether the annotator marking result which fails to be verified exists may include: selecting a corresponding verification option according to a preset pre-labeling model for verifying according to the labeling result of the labeling personnel, wherein the verification option comprises image label omission, label omission, area label omission, label error label, area error label and label contradiction; and when any one of the selected verification options is verified to exist, determining that the annotation result verification of the annotator fails, otherwise, determining that the annotation result verification of the annotator passes.

In this embodiment, the annotation result of the annotator needs to be verified according to the preset pre-annotation model, because the content to be verified for different pre-annotation models is different, that is, the verification options are different, for example, for a detection and segmentation type model, the output of the model is originally not labeled, and therefore, the verification option that the label is missed to be labeled does not need to be selected. The verification options can include image missing labeling, label missing labeling, region missing labeling, label error labeling, region error labeling and label contradiction. Of course, the verification option may also include other verification contents, which is not limited herein.

The verification option will be described in detail below.

The image missing label is that: in the labeling result of the labeling personnel, the whole image has no label, and neither area label nor label appears. This check can be performed, for example, by comparing the image in the annotation result with the original image to be annotated by a known detection method. The preset verification rule comprises a detection method for verifying.

Label missing is as follows: in the labeling result of the labeling personnel, only the region labeling appears in the whole image, but the label labeling does not appear. This check can be performed, for example, by comparing the image in the annotation result with the original image to be annotated by known detection methods. The preset check rule includes a detection method for performing the check.

The region missing label means: in the labeling result of the labeling personnel, only label labeling appears in the whole image, but no region labeling appears. This check can be performed, for example, by comparing the image in the annotation result with the original image to be annotated by known detection methods.

Label mislabeling means: in the annotator annotation result, the labels in the image are not the labels specified, because the image can be annotated with labels that are not professional or do not conform to the specified name, for example, due to the error of the annotator. The preset check rule comprises a specified label.

The region mislabeling means: in the annotation result of the annotator, the annotated region in the image has a shape other than the specified shape, because after the image is annotated by the annotator, for example, due to the mistake of the annotator, the region that does not meet the requirement of the region shape may be annotated. For example, the segmentation labeling of the region is performed based on a preset region shape, for example, if the rectangle is set for image detection rather than for image segmentation, the region is regarded as a region mislabel when the rectangular labeled region appears in the image. The preset check rule includes the designated shape of the region.

The label contradiction means: and the preset check rule comprises tags which cannot appear simultaneously.

Through the verification for the annotation result of the annotator, the accuracy of the annotation result and the matching with the medical image annotation task can be further ensured.

In step S105, when there is no annotator marking result that fails to pass the verification, a final marking result may be determined according to the at least one annotator marking result corresponding to the at least one annotator.

According to the embodiment of the disclosure, when no annotator marking result which fails in verification exists, the final marking result is determined according to the number of the marking results of the annotator.

Specifically, when there is no annotator marking result which fails to pass the verification, determining a final marking result according to the at least one annotator marking result corresponding to the at least one annotator may include: when the labeling result of the labeling personnel which fails to pass the verification does not exist, taking the labeling result of one labeling personnel corresponding to one labeling personnel as the final labeling result aiming at the condition that the number of the labeling personnel is one; when the labeling result of the labeling personnel which fails to pass the verification does not exist, comparing a plurality of labeling personnel labeling results respectively corresponding to a plurality of labeling personnel aiming at the condition that the number of the labeling personnel is multiple, and determining whether the labeling results of the labeling personnel meet the preset consistency requirement or not; if the labeling results of the plurality of annotators meet the preset consistency requirement, taking any one of the labeling results of the plurality of annotators as a final labeling result; and if the plurality of label member marking results do not accord with the preset consistency requirement, enabling an auditor to audit the plurality of label member marking results, and obtaining a final labeling result according to confirmation made by one label member marking result selected by the auditor.

In this embodiment, if only one annotator modifies and confirms the pre-annotation result, only one annotator annotation result is generated, and when the one annotator annotation result passes verification, the annotator annotation result is directly used as the final annotation result. If a plurality of annotators respectively modify and confirm the pre-annotation results, a plurality of annotator annotation results are generated, and when the plurality of annotator annotation results are verified, the plurality of annotator annotation results need to be compared to determine whether the annotator annotation results meet the preset consistency requirements.

And judging whether the labeling results of the plurality of labels meet the preset consistency requirement or not according to the types of the pre-labeling models aiming at the labeling results of the plurality of labels. For label labeling, each labeling member labeling result comprises at least one type of focus label, if any type of focus label in different labeling results is different, the labeling results of the labeling members do not accord with the preset consistency requirement, otherwise, the labeling results of the labeling members accord with the preset consistency requirement. For region labeling, each marker labeling result comprises at least one type of lesion region, and for a plurality of marker labeling results passing the verification, the intersection region area and the union region area between the same type of lesion regions in different marker labeling results are obtained, so that the area ratio between the intersection region area and the union region area corresponding to the type of lesion regions, namely the lesion region overlap ratio, can be obtained. Therefore, for a plurality of annotator marking results, each type of lesion area has a corresponding lesion area coincidence degree, when the lesion area coincidence degree corresponding to any type of lesion area is smaller than a preset threshold (for example, 0.6), the annotator marking results do not meet the preset consistency requirement, otherwise, the annotator marking results do not meet the preset consistency requirement. Wherein the lesion region overlap ratio of each category may have a respective predetermined threshold.

And aiming at the condition that the labeling results of the plurality of annotators meet the preset consistency requirement, any one of the labeling results of the plurality of annotators can be used as a final labeling result. And aiming at the condition that the labeling results of the plurality of annotators do not meet the preset consistency requirement, the labeling results of the plurality of annotators can be submitted to an auditor appointed in the medical image labeling task for auditing, the auditor can select one labeling result which is most consistent with the expected labeling result from the labeling results of the plurality of annotators for modification and confirmation, and then the labeling result modified by the auditor is obtained as the final labeling result.

In addition, when there are a plurality of auditors, each auditor can audit the labeling results of a plurality of annotators and select one labeling result for modification confirmation, so that the labeling result modified by each auditor can be obtained and all the labeling results can be used as the final labeling result, and certainly, one of the labeling results can be randomly selected as the final labeling result.

In addition, for the output results of different types of preset pre-labeled models, the modification operation of the auditor is as follows. When the preset pre-labeled model contains a detection and segmentation type model, an auditor can modify and confirm the model in the following two dimensions according to actual conditions: on one hand, a region (which can be a point, a line segment, a rectangle, a polygon, an ellipse, a painting brush and the like) with a specific shape and composed of a plurality of points can be moved, deleted, newly added and the like, and the label corresponding to the current region can be only modified; on the other hand, operations such as moving, deleting, adding and the like can be carried out on a specific certain point. When the preset pre-labeling model comprises the classification model, the labels given by the classification model can be deleted and modified, and labels which are not predicted by the classification model can be newly added.

It is noted that, when there are a plurality of medical images to be annotated, steps S102 to S105 in the method of the present disclosure above are performed for each medical image.

Referring to fig. 2, fig. 2 is a flowchart illustrating a medical image annotation method according to another embodiment of the present disclosure. As shown in fig. 2, the method may include the following step S206 after step S105.

In step S206, training data may be formed based on the final labeling result, so as to train the preset pre-labeling model with the training data.

According to embodiments of the present disclosure, after the final annotation result is obtained, the medical image and the annotations (region annotations and labels) formed thereon may be derived to form annotation data. When the amount of the images to be labeled in the medical image labeling task is enough, the amount of the labeled data is large, so that the labeled data can be used as a training data set to train the used preset pre-labeled model. When the amount of the image to be labeled in the medical image labeling task is small, the amount of the labeled data is not large enough, so that the labeled data can be added into a training data set given in the medical image labeling task to serve as a new training data set to train the used preset pre-labeled model through a matching training method given in the medical image labeling task.

The production speed of the labeling result can be accelerated by the technical scheme disclosed by the invention, so that training data formed by the labeling result is used for training the artificial intelligent algorithm model, and the iteration of the algorithm model is facilitated, so that the algorithm model can be trained by a large amount of data, and on the other hand, the trained algorithm model can feed a labeling system back to perform certain pre-labeling, thereby improving the performance of the algorithm model, reducing the cost of the artificial feeding algorithm model and improving the closed-loop efficiency.

Referring to fig. 3, fig. 3 is a flowchart illustrating a medical image annotation method according to yet another embodiment of the present disclosure. As shown in fig. 3, the method may comprise the following step S301 before step S101.

In step S301, the medical image annotation task may be created.

According to an embodiment of the present disclosure, the medical image annotation task may also be created before it is acquired. The medical image annotation task can be created by using the following elements: the method comprises the steps of task name, marking data, pre-marking models, model selection, marking personnel, marking modes, preset check rules, auditors, audit modes and model training.

Wherein the task name can be defined according to the task characteristics and the target. The annotation data refers to a folder or a compressed package including the medical image to be annotated, or an address (URL) of the medical image to be annotated, which may be a single or a plurality of medical images. The pre-labeling model refers to all models which can be selected by medical image labeling. The model selection refers to selecting a model related to the task as a preset pre-labeled model. The annotator refers to a person with annotation authority in an annotation platform user system, and the annotator at least has one annotation authority. The annotation mode refers to the number of people of a designated annotator who selects to perform an annotation task, such as single annotation and double annotation, and the number of people of the designated annotator is less than or equal to the total number of people of the annotator. The preset verification rule is a rule used when the result marked by the marker is verified, and the preset verification rule is matched with the preset pre-marked model so as to select different verification options for verifying the marking result output by different pre-marked models. The auditors refer to persons with auditing authorities in the annotation platform user system, and at least one auditor is provided. The auditing mode refers to the number of designated auditors for selecting the auditing task, such as single audit, double audit and the like, wherein the number of designated auditors is less than or equal to the total number of auditors. The model training refers to selecting a training method and a training data set corresponding to a preset pre-labeled model.

Therefore, after the medical image annotation task is created, the medical image annotation task may include a task name and the specified annotator, the specified auditor, the medical image to be annotated, the preset verification rule, and the preset pre-annotation model, the training method thereof, and the training dataset thereof described in steps S101 to S105.

Referring to fig. 4, fig. 4 is a schematic block diagram illustrating a medical image annotation system 100 according to one embodiment of the present disclosure. As shown in FIG. 4, the medical image annotation system 100 comprises an acquisition module 101, a pre-annotation module 102, a modification confirmation module 103, a verification module 104 and an annotation result confirmation module 105. The obtaining module 101 is configured to obtain a medical image annotation task, where the medical image annotation task includes a medical image to be annotated, a preset pre-annotation model, and a preset verification rule. The pre-labeling module 102 is configured to pre-label the medical image to be labeled through the preset pre-labeling model, so as to obtain a pre-labeling result. The modification confirmation module 103 is configured to obtain at least one annotator tagging result corresponding to at least one annotator according to modification confirmation performed on the pre-tagging result by the at least one annotator. The checking module 104 is configured to automatically check the at least one annotator marking result corresponding to the at least one annotator according to the preset checking rule, determine whether an annotator marking result failing to be checked exists, and return the annotator marking result failing to be checked to the corresponding annotator for confirmation when the annotator marking result failing to be checked exists. The annotation result confirmation module 105 is configured to, when there is no annotator annotation result that fails to pass the verification, determine a final annotation result according to the at least one annotator annotation result corresponding to the at least one annotator.

According to the embodiment of the present disclosure, the pre-labeling module 102 pre-labels the medical image to be labeled through the preset pre-labeling model in the following manner to obtain a pre-labeling result: respectively pre-labeling the medical image to be labeled through a plurality of preset pre-labeling models to obtain a plurality of model labeling results; and obtaining the pre-labeling result according to the plurality of model labeling results.

According to an embodiment of the present disclosure, the preset pre-labeling model includes a disease recognition model, a lesion detection model, and a lesion segmentation model.

According to an embodiment of the present disclosure, the modification confirmation module 103 obtains at least one annotator tagging result corresponding to each of the at least one annotator according to the modification confirmation performed on the pre-tagging result by the at least one annotator by the following means: acquiring modification confirmation information provided by the annotator, and when the modification confirmation information indicates that the pre-annotation result needs to be modified, acquiring the pre-annotation result modified by the annotator as the annotation result of the annotator; and when the modification confirmation information is that the pre-labeling result does not need to be modified, taking the pre-labeling result as a labeling result of a labeling person.

According to the embodiment of the present disclosure, the verification module 104 automatically verifies the at least one annotator tagging result corresponding to the at least one annotator according to the preset verification rule, and determines whether there is an annotator tagging result that fails to be verified: selecting a corresponding verification option according to a preset pre-labeling model for verifying according to the labeling result of the labeling personnel, wherein the verification option comprises image label omission, label omission, area label omission, label error label, area error label and label contradiction; and when any one of the selected verification options is verified to exist, determining that the marking result of the marking person fails to be verified, otherwise, determining that the marking result of the marking person passes to be verified.

According to the embodiment of the present disclosure, the verification module 104 is further configured to, when a labeling member labeling result that fails to pass the verification exists, return verification information corresponding to the labeling member labeling result that fails to pass the verification to a corresponding labeling member, so that the corresponding labeling member modifies the labeling member labeling result that fails to pass the verification according to the verification information.

According to an embodiment of the present disclosure, the annotation result confirmation 105 module includes: a comparing unit 1051, configured to compare, when there is no annotator marking result that fails to pass the verification, multiple annotator marking results respectively corresponding to multiple annotators for the case that the number of the annotators is multiple, and determine whether the multiple annotator marking results meet a preset consistency requirement; an auditing unit 1052, configured to enable an auditor to audit the plurality of annotator tagging results if the plurality of annotator tagging results do not meet a preset consistency requirement, and obtain a final tagging result according to a confirmation made by one annotator tagging result selected by the auditor.

It is to be understood that, with respect to the medical image annotation system in the embodiment described above with reference to fig. 4, the specific manner in which the respective modules perform the operations has been described in detail in the embodiment of the medical image annotation method described in conjunction with fig. 1, and will not be elaborated herein.

Referring to fig. 5, fig. 5 is a schematic block diagram illustrating a medical annotation system 200 according to another embodiment of the present disclosure. The medical annotation system 200 shown in FIG. 5 differs from the medical annotation system 100 shown in FIG. 4 only in that the medical annotation system 200 further comprises a training module 206.

The training module 206 is configured to form training data based on the final labeling result, so as to train the preset pre-labeling model by using the training data.

It is to be understood that, regarding the medical image annotation system in the embodiment described above with reference to fig. 5, the specific manner in which the respective modules perform the operations has been described in detail in the embodiment of the medical image annotation method described in conjunction with fig. 2, and will not be elaborated herein.

Referring to fig. 6, fig. 6 is a schematic block diagram illustrating a medical annotation system 300 according to yet another embodiment of the present disclosure. The medical annotation system 300 shown in FIG. 6 differs from the medical annotation system 100 shown in FIG. 4 only in that the medical annotation system 300 further comprises a task creation module 301.

The task creation module 301 is configured to create the medical image annotation task.

It is to be understood that, regarding the medical image annotation system in the embodiment described above with reference to fig. 6, the specific manner in which the respective modules perform the operations has been described in detail in the embodiment of the medical image annotation method described in conjunction with fig. 3, and will not be elaborated herein.

The present disclosure also provides an electronic device, wherein the electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the medical image annotation method as described in conjunction with fig. 1 when executing the computer program.

It is understood that the steps implemented when the computer program is executed by the processor are substantially the same as the implementation of the steps in the above method, and the specific manner is described in detail in the embodiment of the method for labeling medical images, and will not be described in detail herein.

The present disclosure also provides a computer-readable storage medium, wherein the storage medium stores a computer program which, when executed, implements a medical image annotation method as described in connection with fig. 1.

It is understood that the steps implemented when the computer program is executed by the processor are substantially the same as the implementation of the steps in the above method, and the specific manner is described in detail in the embodiment of the method for labeling medical images, and will not be described in detail herein.

The embodiments of the present disclosure are described in detail above, and the principles and embodiments of the present disclosure are explained herein by applying specific embodiments, and the descriptions of the embodiments are only used to help understanding the method and the core ideas of the present disclosure; meanwhile, for a person skilled in the art, according to the idea of the present disclosure, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present disclosure.

It should be understood that the terms "first" and "second," etc. in the claims, description, and drawings of the present disclosure are used for distinguishing between different objects and not for describing a particular order. The terms "comprises" and "comprising," when used in the specification and claims of this disclosure, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It is also to be understood that the terminology used in the description of the disclosure herein is for the purpose of describing particular embodiments only, and is not intended to be limiting of the disclosure. As used in the specification and claims of this disclosure, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should be further understood that the term "and/or" as used in the specification and claims of this disclosure refers to any and all possible combinations of one or more of the associated listed items and includes such combinations.

The embodiments of the present disclosure have been described in detail, and the principles and embodiments of the present disclosure are explained herein using specific examples, which are provided only to help understand the method and the core idea of the present disclosure. Meanwhile, a person skilled in the art should, based on the idea of the present disclosure, change or modify the specific embodiments and application scope of the present disclosure. In view of the above, the description is not intended to limit the present disclosure.

Claims (15)

1. A medical image annotation method, wherein the method comprises:

acquiring a medical image annotation task, wherein the medical image annotation task comprises a medical image to be annotated, a preset pre-annotation model and a preset verification rule;

pre-labeling the medical image to be labeled through the preset pre-labeling model to obtain a pre-labeling result;

obtaining at least one annotator marking result corresponding to at least one annotator according to modification confirmation of the pre-marking result by the at least one annotator;

automatically verifying the at least one marker marking result corresponding to the at least one marker according to the preset verification rule, determining whether a marker marking result which fails in verification exists, and returning the marker marking result which fails in verification to the corresponding marker for confirmation when the marker marking result which fails in verification exists;

and when the labeling result of the labeling personnel which fails to pass the verification does not exist, determining a final labeling result according to the labeling result of the at least one labeling personnel corresponding to the at least one labeling personnel.

2. The medical image annotation method according to claim 1, wherein the pre-annotation of the medical image to be annotated by the preset pre-annotation model comprises:

respectively pre-labeling the medical image to be labeled through a plurality of preset pre-labeling models to obtain a plurality of model labeling results;

and obtaining the pre-labeling result according to the plurality of model labeling results.

3. The medical image annotation method according to claim 1 or 2, wherein the preset pre-annotation model comprises a disease recognition model, a lesion detection model, and a lesion segmentation model.

4. The medical image annotation method of claim 1, wherein obtaining at least one annotator annotation result corresponding to each of the at least one annotator, in accordance with the confirmation of the modification of the pre-annotation result by the at least one annotator, comprises:

acquiring modification confirmation information provided by the annotator, and when the modification confirmation information indicates that the pre-annotation result needs to be modified, acquiring the pre-annotation result modified by the annotator as the annotation result of the annotator;

and when the modification confirmation information indicates that the pre-labeling result does not need to be modified, taking the pre-labeling result as a labeling result of a labeling operator.

5. The medical image annotation method according to claim 1, wherein the automatic verification of the at least one annotator annotation result corresponding to the at least one annotator according to the preset verification rule comprises:

aiming at the labeling result of the label maker, selecting a corresponding verification option according to a preset pre-labeling model for verification, wherein the verification option comprises image label omission, label omission, area label omission, label error label, area error label and label contradiction;

and when any one of the selected verification options is verified to exist, determining that the marking result of the marking person fails to be verified, otherwise, determining that the marking result of the marking person passes to be verified.

6. The medical image annotation method of claim 5, wherein the method further comprises:

and when the marking result of the marking member which fails in the verification exists, returning verification information corresponding to the marking result of the marking member which fails in the verification to the corresponding marking member, so that the corresponding marking member modifies the marking result of the marking member which fails in the verification according to the verification information.

7. The medical image annotation method of claim 1, wherein when there is no annotator annotation result which fails in verification, determining a final annotation result according to the at least one annotator annotation result corresponding to the at least one annotator comprises:

when the labeling result of the labeling personnel which fails to pass the verification does not exist, taking the labeling result of one labeling personnel corresponding to one labeling personnel as the final labeling result aiming at the condition that the number of the labeling personnel is one;

when the labeling result of the labeling personnel which fails to pass the verification does not exist, comparing a plurality of labeling personnel labeling results respectively corresponding to a plurality of labeling personnel aiming at the condition that the number of the labeling personnel is multiple, and determining whether the labeling results of the labeling personnel meet the preset consistency requirement or not; if the labeling results of the plurality of markers meet the preset consistency requirement, taking any one of the labeling results of the plurality of markers as a final labeling result; and if the labeling results of the plurality of annotators do not meet the preset consistency requirement, enabling an auditor to audit the labeling results of the plurality of annotators, and obtaining a final labeling result according to confirmation made by one labeling result selected by the auditor.

8. The medical image annotation method of claim 1, wherein the method further comprises:

and forming training data based on the final labeling result so as to train the preset pre-labeling model by utilizing the training data.

9. The medical image annotation method of claim 1, wherein the method further comprises:

and creating the medical image annotation task.

10. A medical image annotation system, wherein the system comprises:

the system comprises an acquisition module, a verification module and a verification module, wherein the acquisition module is used for acquiring a medical image annotation task, and the medical image annotation task comprises a medical image to be annotated, a preset pre-annotation model and a preset verification rule;

the pre-labeling module is used for pre-labeling the medical image to be labeled through the preset pre-labeling model to obtain a pre-labeling result;

the modification confirmation module is used for obtaining at least one annotator marking result corresponding to at least one annotator according to modification confirmation of the pre-marking result by the at least one annotator;

the verification module is used for automatically verifying the at least one marker marking result corresponding to the at least one marker according to the preset verification rule, determining whether a marker marking result which fails to pass the verification exists or not, and returning the marker marking result which fails to pass the verification to the corresponding marker for confirmation when the marker marking result which fails to pass the verification exists;

and the marking result confirming module is used for determining a final marking result according to the at least one marking member marking result corresponding to the at least one marking member when the marking member marking result which is failed in verification does not exist.

11. The medical image annotation system of claim 10, wherein the annotation result confirmation module comprises:

the comparison unit is used for comparing a plurality of label operator labeling results respectively corresponding to a plurality of label operators when the number of the label operators is multiple and determining whether the plurality of label operator labeling results meet the preset consistency requirement or not when the number of the label operators does not pass the verification;

and the auditing unit is used for enabling an auditor to audit the plurality of annotator marking results if the plurality of annotator marking results do not meet the preset consistency requirement, and obtaining a final marking result according to the confirmation made by one annotator marking result selected by the auditor.

12. The medical image annotation system of claim 10, wherein the system further comprises:

and the training module is used for forming training data based on the final labeling result so as to train the preset pre-labeling model by utilizing the training data.

13. The medical image annotation system of claim 10, wherein the system further comprises:

and the task creating module is used for creating the medical image labeling task.

14. An electronic device, wherein the electronic device comprises a memory in which a computer program is stored and a processor, which when executed implements the medical image annotation method according to any one of claims 1 to 9.

15. A computer-readable storage medium, wherein the storage medium stores a computer program which, when executed, implements the medical image annotation method according to any one of claims 1 to 9.

Images