WO2020238256A1 - 基于弱分割的损伤检测方法及装置 - Google Patents

基于弱分割的损伤检测方法及装置 Download PDF

Info

Publication number
WO2020238256A1
WO2020238256A1 PCT/CN2020/072058 CN2020072058W WO2020238256A1 WO 2020238256 A1 WO2020238256 A1 WO 2020238256A1 CN 2020072058 W CN2020072058 W CN 2020072058W WO 2020238256 A1 WO2020238256 A1 WO 2020238256A1
Authority
WO
WIPO (PCT)
Prior art keywords
damage
frame
prediction
segmentation
category
Prior art date
Application number
PCT/CN2020/072058
Other languages
English (en)
French (fr)
Inventor
张伟
褚崴
Original Assignee
创新先进技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 创新先进技术有限公司 filed Critical 创新先进技术有限公司
Priority to US16/808,208 priority Critical patent/US10783643B1/en
Priority to US16/945,774 priority patent/US11004204B2/en
Publication of WO2020238256A1 publication Critical patent/WO2020238256A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/12—Edge-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • One or more embodiments of this specification relate to the field of machine learning, and in particular to a method and device for damage detection based on weak segmentation using machine learning.
  • the insurance company needs to send professional damage assessment personnel to the accident site to conduct on-site investigation and assessment of the damage, give the vehicle's maintenance plan and compensation amount, and take photos of the scene, and keep the damage assessment photos on file for use Backstage inspectors verify damage and price.
  • Due to the need for manual damage inspection insurance companies need to invest a lot of labor costs and professional knowledge training costs.
  • the claim settlement process waits for the manual surveyor to take photos on site, the damage assessor assesses the damage at the repair site, and the damage inspector checks the damage in the background.
  • the claim settlement cycle is as long as 1-3 days, and users have a long waiting time. , The experience is poor.
  • One or more embodiments of this specification describe a damage detection method and device based on weak segmentation, in which a weak segmentation damage detection model is trained based on automatically generated weak segmentation annotation data, and the damage frame and the damage segmentation contour predicted by the model can be mutually verified , Improve the accuracy of damage prediction.
  • a computer-executed method for training a weak segmentation damage detection model including:
  • the sample picture corresponding to the frame label data indicates at least one damage label frame, each damage label frame is marked by the annotator, the frame selects the smallest damaged object in the sample picture Rectangle;
  • each damage labeling frame as a contour of a corresponding damage object, and labeling a segmentation category for each pixel in the sample image based on the contour, thereby generating segmentation labeling data
  • the sample picture is input into a weak segmentation damage detection model
  • the weak segmentation damage detection model includes a frame prediction branch and a segmentation prediction branch
  • the frame prediction branch outputs frame prediction data for indicating a damage prediction frame
  • the segmentation prediction branch Outputting segmentation prediction data for predicting the segmentation category of each pixel in the sample picture
  • the weak segmentation damage detection model is updated.
  • the segmentation category is labeled for each pixel in the following manner:
  • the pixels located in the damage labeling frame are marked as the first segmentation category, and the pixels located outside the damage labeling frame are marked as the second segmentation category.
  • the at least one damage labeling frame includes a first damage labeling frame; the frame labeling data further includes that the labeling staff selects the first damage labeling frame to be labelled from the predetermined N damage categories.
  • the at least one damage labeling frame includes a first damage labeling frame and a second damage labeling frame, and the first damage labeling frame and the second damage labeling frame have overlapping areas; the frame labeling data It also includes that the annotator selects the first damage category labeled for the first damage labeling frame and the second damage category labeled for the second damage labeling frame from the predetermined N damage categories, where the damage of the second damage category The severity is greater than the first damage category; in this case, labeling the segmentation category for each pixel includes: for the pixel located within the overlap area, labeling its segmentation category as the second damage category The corresponding category.
  • the weak segmentation damage detection model is implemented based on a convolutional neural network CNN
  • the convolutional neural network CNN includes a basic convolutional layer, which is used to perform convolution processing on the sample image to obtain corresponding convolutional features Figure
  • the frame prediction branch is used to predict the frame prediction data based on the convolution feature map
  • the segment prediction branch is used to predict the segment prediction data based on the convolution feature map.
  • the division prediction branch may include:
  • An up-sampling layer for up-sampling the convolution processed features into a first feature map with the same size as the sample picture
  • the prediction processing layer is used to predict the probability that each pixel belongs to each segmentation category based on the first feature map.
  • the segmentation prediction data includes the probability that each pixel belongs to each segmentation category; in this case, the segmentation prediction loss item can be determined as follows: Determine the probability of each pixel point belonging to each segmentation category Predict the segmentation category; compare the predicted segmentation category of each pixel with its labeled segmentation category, and determine the segmentation prediction loss item according to the comparison result.
  • the segmentation prediction loss item can be determined as follows: determine the prediction probability of each pixel belonging to its corresponding labeled segmentation category; determine the segmentation prediction loss item according to the prediction probability.
  • a computer-executed method for detecting damage from a picture including:
  • the picture to be tested is input to the weak segmentation damage detection model, the weak segmentation damage detection model includes a frame prediction branch and a segmentation prediction branch, and the frame prediction branch outputs frame prediction data for indicating at least one damage prediction frame.
  • the segmentation prediction branch outputs segmentation prediction data predicted for the segmentation category of each pixel in the picture to be tested;
  • a damage detection result for the picture to be tested is determined.
  • the frame prediction data may not indicate the damage prediction frame, or the segmentation prediction data does not indicate the damage target area, where the damage target area is a set of pixels whose predicted segmentation category is the same category and the area is greater than a certain threshold Connected areas.
  • the damage detection result can be determined as the picture to be tested does not contain the damaged object.
  • the frame prediction data indicates at least one damage prediction frame
  • the segmentation prediction data indicates at least one damage object area; in this case, it may be based on the at least one damage prediction frame and the at least one damage object Area, determine the damage detection result for the picture to be tested.
  • determining the damage detection result for the picture to be tested may include: a union of the area set corresponding to the at least one damage prediction frame and the area set corresponding to the at least one damage object area, As a result of damage detection.
  • determining the damage detection result for the picture to be tested may include: if the intersection ratio between any first damage prediction frame in the at least one damage prediction frame and each damage object area is less than a preset Threshold, remove the first damage prediction frame from the damage detection result.
  • determining the damage detection result for the picture to be tested may include: if the overlap area between any first damage prediction frame and each damage object area in the at least one damage prediction frame is If the ratio of the frame area of a damage prediction frame is less than a preset threshold, the first damage prediction frame is removed from the damage detection result.
  • the at least one damage prediction frame includes a first damage prediction frame
  • the frame prediction data further includes a first damage category predicted for the first damage prediction frame
  • the at least one damage object area includes a first damage prediction frame.
  • a damage object area, where the pixels in the first damage object area correspond to the first segmentation category; in this case, determining the damage detection result for the picture to be tested may include:
  • the first damage detection The frame is determined as an abnormality detection frame, or the first damage target area is determined as an abnormal area.
  • an apparatus for training a weak segmentation damage detection model including:
  • the sample acquisition unit is configured to acquire a sample picture, the sample picture corresponding to frame labeling data, the frame labeling data indicates at least one damage labeling frame, each damage labeling frame is marked by an annotator, and the sample is selected by the frame The smallest rectangular frame of the damaged object in the picture;
  • An annotation generating unit configured to use each damage annotation frame as the contour of the corresponding damage object, and based on the contour, mark a segmentation category for each pixel in the sample image, thereby generating segmentation annotation data;
  • a model input unit configured to input the sample picture into a weak segmentation damage detection model, the weak segmentation damage detection model including a frame prediction branch and a segmentation prediction branch, and the frame prediction branch outputs frame prediction data for indicating a damage prediction frame ,
  • the segmentation prediction branch outputs segmentation prediction data predicted for the segmentation category of each pixel in the sample picture;
  • the first determining unit is configured to determine a frame prediction loss item based on the comparison of the frame prediction data and the frame annotation data, and determine a segmentation prediction loss item based on the comparison of the segmentation prediction data and the segmentation annotation data ;
  • the second determining unit is configured to determine the loss function of this prediction according to the frame prediction loss item and the segmentation prediction loss item;
  • the model update unit is configured to update the weak segmentation damage detection model in the direction in which the loss function decreases.
  • a device for detecting damage from a picture including:
  • the model acquisition unit is configured to acquire a weak segmentation damage detection model trained by the device of claim 15;
  • a model input unit configured to input a picture to be tested into the weak segmentation damage detection model, the weak segmentation damage detection model including a frame prediction branch and a segmentation prediction branch, and the frame prediction branch output is used to indicate the damage prediction frame Frame prediction data, the segmentation prediction branch outputs segmentation prediction data predicted for the segmentation category of each pixel in the picture to be tested;
  • the result determining unit is configured to determine a damage detection result for the picture to be tested according to the frame prediction data and the segmentation prediction data.
  • a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed in a computer, the computer is caused to execute the methods of the first aspect and the second aspect.
  • a computing device including a memory and a processor, characterized in that executable code is stored in the memory, and when the processor executes the executable code, the first aspect and the first aspect are implemented. Two-sided approach.
  • weak segmentation annotation data is generated based on the artificially annotated damage frame, and such weak segmentation annotation data is used to train a weak segmentation damage detection model with two branches of frame detection and segmentation prediction.
  • the picture to be tested is input into the weak segmentation damage detection model, and the damage prediction frame and the damage object area are obtained through the above two branches.
  • the damage prediction frame and the damage object area can verify and complement each other, thereby improving the accuracy of damage detection.
  • Fig. 1 shows a schematic diagram of an implementation scenario according to an embodiment
  • Figure 2 shows a flowchart of a method for training a weakly segmented damage detection model according to an embodiment
  • Figure 3 shows a specific example of a sample picture that has been manually annotated
  • FIG. 4 shows a schematic diagram of labeling segmentation categories for pixels in an embodiment
  • Fig. 5 shows a schematic structural diagram of a weak segmentation damage detection model according to an embodiment
  • Fig. 6 shows a flowchart of steps of a method for identifying damage from a picture according to an embodiment
  • Fig. 7 is the damage prediction frame and the damage object area output by the weak segmentation damage detection model in an example
  • Fig. 8 shows a schematic block diagram of an apparatus for training a weak segmentation damage detection model according to an embodiment
  • Fig. 9 shows a schematic block diagram of a damage detection device according to an embodiment.
  • a damage detection model that takes the damaged object as the detection target can be trained, and damage identification can be performed through the damage detection model.
  • the training of the model requires a lot of labeled data.
  • the labeling staff can mark the damaged object as a specific target object, that is, mark the damaged object with the smallest rectangular frame containing the damaged object in the picture.
  • Such a rectangular frame is also called the damage labeling frame.
  • the original image of the sample and the damage labeling frame together constitute the image training sample. Using such training samples for model training, the damage detection model can be obtained.
  • the damage detection model can be used to identify the damage in the image to be tested.
  • the output result of the damage detection model is a number of damage prediction frames predicted in the picture to be tested, and each damage prediction frame uses the smallest rectangular frame to select the predicted damage object.
  • the detection accuracy of the current damage detection model needs to be improved. Especially in the scene of vehicle damage recognition, because the shooting environment of car damage pictures is more complicated, it is often affected by reflections, stains, etc., making the accuracy of damage detection unsatisfactory, such as misdetecting reflections, stains, etc. as damage.
  • the inventor proposes to perform image segmentation on the picture to be tested on the basis of conventional damage detection, and use the results of the segmentation to verify or supplement the conventional damage detection results.
  • Image segmentation is also called image semantic segmentation, which is used to segment or divide an image into areas that belong to/not belong to a specific target object, and its output can be expressed as a mask covering the area of a specific target object.
  • image segmentation model is trained based on segmentation and annotation data that annotates the contour of the target object.
  • image segmentation is used to divide the picture into areas that belong to/not belong to the damaged object.
  • model training requires annotation data for marking the contour of the damaged object.
  • the labeling of contours requires the labeling personnel to draw the boundary of the target object with several labeling points. This process is time-consuming and laborious, and the labor cost is high. Especially for target objects with irregular shapes and unclear boundaries such as damaged objects, the labeling cost is even higher.
  • the damage labeling frame is used as the rough outline labeling of the damaged object, and image segmentation training is performed to obtain a two-branch damage detection model, one of which performs conventional damaged object detection.
  • the other branch performs image segmentation. Since the segmentation labeling data is automatically generated by directly using the damage labeling frame as the contour of the damage object, the accuracy is limited, and the model trained in this way can be called a weak segmentation damage detection model.
  • the weak segmentation damage detection model can output the conventional damage prediction frame and the weak segmentation result. The results of these two aspects can be merged with each other to optimize the final damage detection result.
  • Fig. 1 shows a schematic diagram of an implementation scenario according to an embodiment.
  • the weak segmentation damage detection model is trained in advance using training samples.
  • the training sample is composed of sample pictures and annotation data.
  • the annotation data used for segmentation training is automatically generated by directly using the damage labeling frame as the contour of the damaged object.
  • the two-branch damage detection model shown in Figure 1 can be trained.
  • the first branch is the detection branch, used for conventional damage detection
  • the other branch is the segmentation branch, used for weak image segmentation. .
  • the model can be used for damage detection.
  • the picture to be tested can be input into the damage detection model.
  • first feature extraction is performed on the picture to be tested, and then the damage prediction frame for the picture to be tested is output through the detection branch, and the damage segmentation area for the picture to be tested is output through the segmentation branch.
  • the final damage detection result can be obtained.
  • Fig. 2 shows a flowchart of a method for training a weakly segmented damage detection model according to an embodiment. It can be understood that the method can be executed by any device, device, platform, or device cluster with computing and processing capabilities. As shown in Fig.
  • the training process includes at least the following steps: Step 21: Obtain a sample picture corresponding to the frame label data indicating the damage label frame; Step 22: Use each damage label frame as the contour of the corresponding damage object , Based on the contour, mark the segmentation category for each pixel in the sample image, thereby generating segmentation annotation data; step 23, input the sample image into the weak segmentation damage detection model, the model includes a frame prediction branch and a segmentation prediction branch, the frame prediction The branch output indicates the damage prediction data of the damage prediction frame, and the segmentation prediction branch outputs the segmentation prediction data by predicting the segmentation category of each pixel; step 24, based on the comparison of the damage prediction data and the frame annotation data, determine the frame prediction loss item, And, based on the comparison of the segmentation prediction data and the segmentation annotation data, determine the segmentation prediction loss item; step 25, determine the loss function of this prediction according to the frame prediction loss item and the segmentation prediction loss item; step 26, in The direction in which the loss function decreases, and the damage prediction model is updated.
  • the sample picture is generally a picture containing a damaged object, such as a photo of a damaged vehicle taken at the scene of a car damage.
  • a damaged object such as a photo of a damaged vehicle taken at the scene of a car damage.
  • Such sample pictures are annotated by an annotator, and have corresponding annotation data.
  • the labeling staff can use a minimum rectangular box surrounding the damage to select the damaged object.
  • a damage labeling frame is called frame labeling data, which can indicate each damage labeling frame.
  • Figure 3 shows a specific example of a sample picture that has been manually annotated.
  • the sample picture is a picture of a damaged vehicle.
  • the labeler selects a number of damaged objects in the picture with a rectangular frame, that is, marks a number of damage labeling frames.
  • the labeling personnel may only label the location of the damaged object, that is, use the damage labeling frame to select the damaged object without distinguishing the type of damage.
  • the labeling personnel may also label the category of the damaged object from the preset N types of damage categories for each damaged object.
  • N 10 damage categories can be preset, including, for example, scratching, deformation, tearing, (glass) cracking, and so on.
  • the labeling staff selects the category corresponding to the damage from the above 10 damage categories on the basis of marking the damage labeling frame for labeling.
  • the number in the upper right corner of the damage labeling box shows the damage category of the damaged object in the damage labeling box.
  • 12 corresponds to scratching and 10 corresponds to deformation.
  • Figure 3 is just one Example.
  • segmentation label data is automatically generated based on the frame label data.
  • the segmentation and labeling data is data for labeling the contour of the target object for image segmentation training.
  • segmentation and labeling data are obtained based on the outline of the target object drawn by the labeler using a number of labeling points.
  • the segmentation annotation data is pixel-level annotation, that is, whether each pixel in the picture belongs to a specific target object.
  • each damage labeling frame is directly used as the contour of the corresponding damage object, and the segmentation category is labelled for each pixel in the sample image based on the contour, thereby automatically generating segmentation and labeling data.
  • the damage labeling frame is used as the contour of the damage object, and according to whether each pixel falls into the damage labeling frame, it is determined whether the pixel belongs to the damage object, and then the segmentation category is marked. For example, the pixels located in the damage labeling frame are marked as the first segmentation category, and the pixels located outside the damage labeling frame are marked as the second segmentation category. In this way, the pixels in the sample picture are divided into two categories, namely, the foreground part belonging to the damaged object (corresponding to the first segmentation category), and the background part not belonging to the damaged object (corresponding to the second segmentation category).
  • the frame labeling data also includes the damage category marked by the labeler for each damage labeling frame, for example, the damage category 12 corresponding to scratching in FIG. 3, and the damage category 10 corresponding to deformation.
  • the segmentation category is labeled for the pixel based on the damage category of the damage labeling frame in which the pixel falls. Specifically, for any damage marking frame, for simplicity of description, it is called the first damage marking frame, and it is assumed that the first damage marking frame is marked with the first damage category. Then, in step 22, for the pixel points falling within the first damage labeling frame, the segmentation category may be labeled as the category corresponding to the first damage category.
  • the pixel can be divided into N+1 segmentation categories in the segmentation labeling, where the first N segmentation categories correspond to the aforementioned N damage categories one-to-one.
  • the other segmentation category corresponds to the case that does not belong to any damage labeling frame. More specifically, for vehicle damage, if 10 damage categories are set for the damage object in advance, then when marking the segmentation category of a certain pixel, you can mark the 10 types of damage corresponding to the frame according to the pixel that falls into One of the damage categories is labeled with a segmentation category; if the pixel does not fall into any damage labeling frame, its segmentation category is labeled as the 11th category.
  • segmentation categories can be labeled based on a preset ranking of the severity of damage categories corresponding to the multiple damage labeling frames .
  • the order of severity of damage corresponding to each damage type can also be preset.
  • the severity of the damage category can be set from light to heavy, including scratching ⁇ deformation ⁇ tear..., etc.
  • the damage category with the heavier damage in the multiple damage labeling frames can be determined based on the ranking of the severity of the damage category.
  • the corresponding category of the damage category is used as the segmentation category of the pixel.
  • Fig. 4 shows a schematic diagram of labeling segmentation categories for pixels in an embodiment.
  • three damage marking boxes A, B and C are shown, respectively marked with damage categories a, b and c.
  • the pixels in the overlapping area can be marked as the category corresponding to damage category b.
  • the pixels outside the overlap area in the damage labeling boxes B and C mark them as the corresponding categories of the damage categories b and c, respectively.
  • the pixels in the diagonal area are marked as category b, and the pixels in the square area are marked as category c.
  • the damage labeling frame is used as the contour of the corresponding damage object, and the segmentation category is labeled for each pixel, thereby automatically generating the segmentation labeling data.
  • the damage marking frame is the smallest rectangular frame for selecting the damaged object, and it is usually not equal to the true outline of the damaged object. Taking the damage labeling frame as the contour of the damage object for segmentation labeling is only a rough approximate labeling, so it can also be called weak segmentation labeling data. Such weak segmentation labeled data can be used to train weak segmentation damage detection models.
  • the sample picture is input to the weak segmentation damage detection model, and the damaged object in the sample picture is predicted by the weak segmentation damage detection model.
  • the weak segmentation damage detection model here can be the initial model, or a model that needs to be further updated during the training process.
  • the weak segmentation damage detection model has two branches, the frame prediction branch and the segmentation prediction branch.
  • the frame prediction branch outputs the damage prediction frame predicted for the sample picture, and the segmentation prediction branch performs the segmentation category of each pixel. Forecast and generate segmented forecast data.
  • the weak segmentation damage detection model can be implemented based on various algorithms and model structures. Typically, the weak segmentation damage detection model can be implemented based on the convolutional neural network CNN.
  • Fig. 5 shows a schematic structural diagram of a weak segmentation damage detection model according to an embodiment.
  • the weak segmentation damage detection model is implemented as a convolutional neural network CNN.
  • the neural network includes a basic convolutional layer 510, a border prediction branch 520, and a segmentation prediction branch 530.
  • the basic convolution layer 510 is used to perform convolution processing on the input sample pictures to obtain a convolution feature map (feature map).
  • the basic convolution layer 510 may include multiple sub-layers, and in each sub-layer, a corresponding convolution kernel is used to perform convolution processing on the picture.
  • pooling processing can also be performed. Through multiple convolution and pooling, the obtained feature map can reflect the more abstract and high-level features in the original sample image.
  • the feature map after convolution processing has a smaller dimension than the original sample picture.
  • the frame prediction branch 520 performs frame prediction based on the convolution feature map, and outputs frame prediction data.
  • the frame prediction data indicates the predicted damage prediction frame.
  • the frame prediction branch also predicts the damage category corresponding to each damage prediction frame.
  • the frame prediction branch can be implemented by various known target detection algorithms. In terms of structure, various known neural network structures for target detection can also be used. For example, it can include further convolution processing layers and frame regression. Layer, fully connected processing layer, etc.
  • the segmentation prediction branch 530 performs segmentation category prediction of pixels based on the above-mentioned convolution feature map, thereby obtaining segmentation prediction data.
  • the segmentation prediction branch 530 may include a segmentation convolution layer 531 for performing further convolution processing on the convolution feature map.
  • the convolution kernel in the segmentation convolution layer 531 may be designed for the characteristics of segmentation prediction, which is different from the convolution kernel in the basic convolution layer, so that the features after further convolution processing are more conducive to the subsequent segmentation prediction processing .
  • the convolutional layer 531 is an optional network processing layer, and in some cases, the network processing layer may also be omitted.
  • the segmentation prediction branch 530 also includes an up-sampling layer 532 for up-sampling the convolution processed features into a first feature map with the same size as the sample picture.
  • the feature map after the convolution process has a smaller dimension than the original sample image. Therefore, in the segmentation prediction branch, the up-sampling layer is used to restore the feature map of the original image size.
  • the up-sampling layer 532 may adopt an up-sampling method such as interpolation processing to restore the feature with a smaller dimension after the convolution processing to the first feature map with the same dimension as the original sample picture. Therefore, each pixel in the first feature map corresponds to each pixel in the original sample picture.
  • the segmentation processing layer 533 predicts the probability that each pixel in the first feature map belongs to each segmentation category. Since the pixels in the first feature map correspond to the pixels in the original sample picture, the segmentation processing layer 533 predicts the segmentation category of the pixels in the first feature map, that is, corresponds to each pixel in the predicted sample picture The probability of belonging to each segmentation category.
  • the segmentation processing layer 533 can predict the probability that each pixel belongs to the damaged object (corresponding to the first segmentation category).
  • a pixel with a probability value greater than a predetermined threshold may be determined as belonging to the damaged object.
  • the segmentation processing layer 533 can predict the probability P ij that each pixel i belongs to the j-th segmentation category in the N+1 segmentation categories. Therefore, for each pixel i, a probability vector (P i0 , P i1 , P i2 ,..., P iN ) belonging to each segmentation category can be formed. Or, for each segmentation category j, a probability set of each pixel belonging to the segmentation category j can be formed. Since the pixels in the picture are generally arranged in a matrix form, the probability set for each segmentation category j can be formed as a probability matrix corresponding to the pixel.
  • the segmentation prediction branch 530 can output segmentation prediction data that predicts the segmentation category of each pixel.
  • the segmentation prediction data is the probability that the pixel points belong to each segmentation category predicted by the segmentation processing layer 533.
  • the segmentation prediction branch may further determine the prediction segmentation category of each pixel based on the foregoing probability as the segmentation prediction data. For example, for each pixel, the segmentation category with the highest predicted probability is used as its predicted segmentation category.
  • the model structure of the weak segmentation damage detection model is also different from the conventional damage detection model.
  • the conventional damage detection model does not have a segmentation branch, and therefore does not perform image segmentation; the conventional image segmentation model does not have a frame prediction branch, and does not perform frame prediction.
  • neural network models that perform frame prediction and image segmentation at the same time, such as Mask-RCNN, such models are usually divided into two branches after the candidate damage frame is selected.
  • One branch performs further regression and category prediction on the candidate damage frame, and the other branch performs image segmentation based on the candidate damage frame, that is, the image segmentation is performed within the candidate damage frame.
  • image segmentation is performed on the basis of frame prediction, rather than an independent branch.
  • the model is split into two independent branches.
  • the border prediction branch and the segmentation prediction branch are further processed and predicted based on the feature map after convolution processing, and the prediction processes are independent of each other.
  • the two branches are independently predicted, so that the predicted results of each output are more conducive to mutual verification.
  • a variety of specific CNN network structures can be modified to obtain the above-mentioned weak segmentation damage detection model.
  • a cyclic full convolutional network RFCN can be used as the basic network structure, and segmentation prediction branches can be added after the feature map and before position-sensitive convolution.
  • a two-branch weak segmentation damage detection model is realized.
  • the model generates frame prediction data related to the damage prediction frame through the frame prediction branch, and generates the segmentation category for each pixel through the segmentation prediction branch. Relevant segmentation prediction data.
  • the frame prediction loss item is determined based on the comparison of the frame prediction data and the frame annotation data.
  • the frame prediction data indicates the position of each damage prediction frame predicted.
  • the frame prediction loss item can be determined by comparing the position difference between the damage prediction frame and the damage label frame.
  • the frame prediction data also indicates the predicted damage category corresponding to each damage prediction frame.
  • the determination of the frame prediction loss item can be performed by various conventional algorithms, for example, the form of L2 error is used as the frame prediction loss item.
  • step 24 the segmentation prediction loss item is determined based on the comparison of segmentation prediction data and segmentation annotation data.
  • the segmentation prediction loss item is determined based on the comparison of segmentation categories. Specifically, in an embodiment, the segmentation prediction branch outputs the probability that each pixel belongs to each segmentation category as segmentation prediction data. At this time, the predicted segmentation category of each pixel can be determined according to the probability, for example, the segmentation category with the highest probability is used as its corresponding predicted segmentation category. In another embodiment, the segmentation prediction branch directly outputs the prediction segmentation category of each pixel. In this case, the prediction segmentation category can be directly obtained from the segmentation prediction data.
  • the predicted segmentation category of each pixel is compared with its labeled segmentation category, and the pixels whose predicted segmentation category is consistent with the labeled segmentation category are counted, that is, the number of correctly predicted pixels, or the ratio of the number to the total number of pixels. Based on the above number or ratio, the segmentation prediction loss item is determined, so that the larger the number or ratio, the smaller the segmentation prediction loss item.
  • the segmentation prediction loss item is determined based on the probability that each pixel in the segmentation prediction data belongs to each segmentation category and the labeled segmentation category of each pixel. Specifically, for each pixel i, the probability vector (P i0 , P i1 , P i2 ,..., P iN ) that the pixel i belongs to each segmentation category can be obtained from the segmentation prediction data. On the other hand, the label segmentation category k of the pixel i is obtained. Therefore, the segmentation loss item corresponding to the pixel i can be determined as:
  • P ij represents the probability that the pixel i belongs to the segmentation category j
  • the segmentation category k is the labeled segmentation category. Then, the segmentation loss item of each pixel i is summed to obtain the segmentation loss item of this prediction.
  • the softmax function can be used to determine the segmentation loss item L according to the predicted probability of each pixel belonging to each segmentation category and its labeled segmentation category:
  • the segmentation prediction loss item is determined based on the prediction probability of each pixel corresponding to the labeled segmentation category. Specifically, for any pixel i, the label segmentation category k of the pixel is obtained from the segmentation label data. On the other hand, the probability vector (P i0 , P i1 , P i2 ,..., P iN ) of the pixel i belonging to each segmentation category is obtained from the segmentation prediction data, and the prediction probability corresponding to the labeled segmentation category k is extracted from it P ik .
  • the segmentation prediction loss item can be determined based on the prediction probability of each pixel corresponding to the labeled segmentation category. For example, in an example, the segmentation prediction loss term can be expressed as:
  • Pi represents the predicted probability that the i-th pixel belongs to its labeled segmentation category.
  • the loss function of this prediction is determined according to the frame prediction loss item and the segmentation prediction loss item determined in step 24.
  • the sum of the frame prediction loss item and the segmentation prediction loss item may be determined as the loss function of this prediction.
  • the weighted summation of the frame prediction loss item and the segmentation prediction loss item may also be performed to obtain the loss function of this prediction.
  • step 26 the weak segmentation damage prediction model is updated in the direction where the loss function decreases.
  • the model parameters of the weak segmentation damage detection model are adjusted so that the loss function is continuously reduced.
  • the process of parameter adjustment can use conventional methods such as back propagation and gradient descent.
  • the model parameters are continuously updated until the predetermined convergence condition is reached, or the model test result reaches a certain accuracy condition.
  • the training of the model is completed, and the trained weak segmentation damage detection model is obtained.
  • the model is trained based on automatically generated weak segmentation and annotation data, and achieves strong prediction of the damage frame and weak prediction of the damage contour through two independent branches.
  • the following describes the process of damage prediction using the weak segmentation damage detection model obtained from the above training.
  • Fig. 6 shows a flowchart of a method for identifying damage from a picture according to an embodiment. As shown in Figure 6, the process includes the following steps.
  • a weak segmentation damage detection model trained by the method in FIG. 2 is obtained, and the model includes a frame prediction branch and a segmentation prediction branch.
  • the weak segmentation damage detection model has, for example, the structure shown in FIG. 5.
  • step 62 the image to be tested is input to the weak segmentation damage detection model. Therefore, the frame prediction branch in the model outputs frame prediction data for indicating the damage prediction frame, and the segmentation prediction branch outputs segmentation prediction data for predicting the segmentation category of each pixel in the picture to be tested.
  • the process of respectively outputting the frame prediction data and the segmentation prediction data through the two branches is similar to step 23 in the foregoing training process, and will not be repeated.
  • step 63 a damage detection result for the picture to be tested is determined according to the frame prediction data and the segmentation prediction data.
  • an attempt is made to determine the damaged object area based on the segmentation prediction data, that is, a connected area whose area is greater than a certain threshold formed by a set of pixel points whose prediction segmentation category is the same damage category.
  • the segmentation prediction data includes the prediction segmentation category corresponding to each pixel in the picture to be tested.
  • the segmentation prediction data contains the probability that each pixel in the picture to be tested belongs to each segmentation category. In this case, the predicted segmentation category of each pixel can be determined based on the above probability.
  • the damaged target area can be obtained. Specifically, a set of pixel points whose predicted segmentation category is the same damage category can be obtained first, and it is determined whether the set constitutes a connected region and whether the area of the connected region is greater than a certain threshold. If a set of pixels of the same damage category can constitute a connected area with an area greater than a certain threshold, then the connected area is regarded as a damage target area.
  • the frame prediction data indicates at least one damage prediction frame, but the aforementioned damage object area cannot be obtained from the segmentation prediction data; or, at least one damage object area is obtained from the segmentation prediction data, but the frame prediction data does not indicate damage Forecast box.
  • step 63 it is determined that the picture to be tested does not contain a damaged object.
  • the frame prediction data indicates at least one damage prediction frame, and at least one damage object area is obtained from the segmentation prediction data.
  • step 63 according to the above at least one damage prediction frame and at least one damage object Area, determine the damage detection result for the picture to be tested.
  • the union of the area set corresponding to the at least one damage prediction frame and the area set corresponding to the at least one damage object area is used as the damage detection result.
  • the damage detection results obtained in this way will more comprehensively include possible damages, and miss detection can be avoided to the greatest extent.
  • the abnormal damage prediction frame is excluded from the damage detection result based on the overlap between the damage prediction frame and the damage target area.
  • the following takes any damage prediction frame, called the first damage prediction frame, as an example.
  • the intersection ratio (IoU) of the first damage prediction frame and each damage target area can be calculated separately, that is, the ratio of the overlap area to the union area. If the calculated intersection ratios are all smaller than the preset threshold, it means that the first damage prediction frame overlaps each damage target area too little. In other words, there is no damage target area to verify the first damage prediction frame. , Then, remove the first damage prediction frame from the damage detection result.
  • the ratio of the overlapping area of the first damage prediction frame and each damage target area to the frame area of the first damage prediction frame itself is calculated. If the calculated ratios are all less than the preset threshold, the first damage prediction frame is removed from the damage detection result.
  • the frame prediction data is required to also include predictions of damage categories for each damage prediction frame. Assume that in the frame prediction data, the first damage prediction frame corresponds to the predicted first damage category. On the other hand, for any first damage object area in at least one damage object area, it is assumed that each pixel therein corresponds to the first segmentation category.
  • the intersection ratio IoU of the first damage prediction frame and the first damage target area is first determined. If the intersection ratio is greater than the preset threshold, that is, the overlap ratio between the two is sufficiently high, then it is determined whether the first damage category corresponds to the first segmentation category. If they correspond, the first damage prediction frame and the first damage target area mutually verify each other; if they do not correspond, for example, the first damage category indicates scratching, and the first segmentation category indicates glass breakage, it means one of the two At least one is abnormal, so the first damage detection frame can be determined as the abnormal detection frame, or the first damage object area can be determined as the abnormal area, and it is left for further detection and confirmation. In this way, the abnormal result of false detection is further excluded from the damage detection result.
  • Fig. 7 shows the damage prediction frame and the damage target area output by the weak segmentation damage detection model in an example.
  • the picture to be tested is a picture of vehicle damage.
  • a plurality of damage prediction frames are predicted in the picture to be tested, and the two numbers marked in the upper left corner of each damage prediction frame respectively indicate the predicted damage category and the confidence of the prediction.
  • the damage target area is generated according to the predicted segmentation type of each pixel, and is shown by a mask. In practice, in the case of multiple damage categories, different color masks can be used to show the damage object regions corresponding to different segmentation categories.
  • the damage prediction frame and damage target area shown in FIG. 7 the above-mentioned various methods can be used to comprehensively analyze the damage prediction frame and damage target area to obtain the damage detection result.
  • a two-branch weak segmentation damage detection model is trained, in which the frame prediction branch and the segmentation prediction branch independently perform frame prediction and segmentation prediction. . Since the segmentation and annotation data is weakly annotated, the accuracy is not high. Accordingly, the prediction result of the segmentation prediction branch is usually not accurate, and it is difficult to be an independent damage object contour result. However, the prediction result of the weak segmentation can be used in combination with the prediction result of the frame to verify each other with the damage prediction frame, and find the missed and false detections, thereby optimizing and perfecting the damage detection result and improving the accuracy of the detection.
  • a device for training a weak segmentation damage detection model is provided.
  • the device can be deployed in any device, platform or device cluster with computing and processing capabilities.
  • Fig. 8 shows a schematic block diagram of an apparatus for training a weak segmentation damage detection model according to an embodiment.
  • the training device 800 includes:
  • the sample obtaining unit 81 is configured to obtain sample pictures, the sample pictures corresponding to have frame marking data, the frame marking data indicates at least one damage marking frame, and each damage marking frame is marked by the marking staff and selected by the frame.
  • the label generating unit 82 is configured to use each damage labeling frame as the contour of the corresponding damage object, and label the segmentation category for each pixel in the sample image based on the contour, thereby generating segmentation label data;
  • the model input unit 83 is configured to input the sample picture into a weak segmentation damage detection model, the weak segmentation damage detection model including a frame prediction branch and a segmentation prediction branch, the frame prediction branch output is used to indicate the frame prediction of the damage prediction frame Data, the segmentation prediction branch outputs segmentation prediction data predicted for the segmentation category of each pixel in the sample picture;
  • the first determining unit 84 is configured to determine a frame prediction loss item based on the comparison of the frame prediction data and the frame annotation data, and determine the segmentation prediction loss based on the comparison of the segmentation prediction data and the segmentation annotation data item;
  • the second determining unit 85 is configured to determine the loss function of this prediction according to the frame prediction loss item and the segmentation prediction loss item;
  • the model updating unit 86 is configured to update the weak segmentation damage detection model in the direction in which the loss function decreases.
  • the annotation generating unit 82 is configured to:
  • the pixels located in the damage labeling frame are marked as the first segmentation category, and the pixels located outside the damage labeling frame are marked as the second segmentation category.
  • the at least one damage labeling frame includes a first damage labeling frame; the frame labeling data further includes that the labeling staff selects the first labelled first damage labeling frame from the predetermined N damage categories. Damage category.
  • the label generating unit 82 may be configured to label the segmentation category of the pixel points located within the first damage labeling frame as the category corresponding to the first damage category.
  • the at least one damage labeling frame includes a first damage labeling frame and a second damage labeling frame, and the first damage labeling frame and the second damage labeling frame have an overlapping area;
  • the frame labeling data further includes ,
  • the annotator selects the first injury category labeled for the first damage labeling frame and the second damage category labeled for the second damage labeling frame from the predetermined N damage categories, where the damage severity of the second damage category Greater than the first damage category.
  • the label generating unit 82 may be configured to label the segmentation category of the pixel points located within the overlapping area as the category corresponding to the second damage category.
  • the weak segmentation damage detection model can be implemented based on a convolutional neural network CNN, which includes a basic convolutional layer, which is used to perform convolution processing on the sample image to obtain the corresponding convolution Feature map; correspondingly, the frame prediction branch may predict the frame prediction data based on the convolution feature map; the segmentation prediction branch may predict the partition prediction data based on the convolution feature map.
  • CNN convolutional neural network
  • the frame prediction branch may predict the frame prediction data based on the convolution feature map
  • the segmentation prediction branch may predict the partition prediction data based on the convolution feature map.
  • the division prediction branch may include:
  • An up-sampling layer for up-sampling the convolution processed features into a first feature map with the same size as the sample picture
  • the prediction processing layer is used to predict the probability that each pixel belongs to each segmentation category based on the first feature map.
  • the segmentation prediction data includes the probability that each pixel belongs to each segmentation category; the first determining unit 84 is configured to:
  • the predicted segmentation category of each pixel is compared with its labeled segmentation category, and the segmentation prediction loss item is determined according to the comparison result.
  • the first determining unit 84 is configured to:
  • the segmentation prediction loss item is determined.
  • a device for detecting damage from a picture is provided.
  • the device can be deployed in any device, platform or device cluster with computing and processing capabilities.
  • Fig. 9 shows a schematic block diagram of a damage detection device according to an embodiment. As shown in FIG. 9, the detection device 900 includes:
  • the model acquisition unit 91 is configured to acquire a weak segmentation damage detection model trained by the device in FIG. 8;
  • the model input unit 92 is configured to input a picture to be tested into the weak segmentation damage detection model, the weak segmentation damage detection model including a frame prediction branch and a segmentation prediction branch, and the frame prediction branch output is used to indicate at least one damage prediction frame
  • the frame prediction data of the segmentation prediction branch outputs segmentation prediction data predicted for the segmentation category of each pixel in the picture to be tested;
  • the result determining unit 93 is configured to determine a damage detection result for the picture to be tested according to the frame prediction data and the segmentation prediction data.
  • the result determining unit 93 is configured to:
  • the damage detection result is determined as the picture to be tested does not contain a damaged object, where the damaged object area is
  • the predicted segmentation category is a connected region whose area is larger than a certain threshold formed by a set of pixel points of the same damage category.
  • the frame prediction data indicates at least one damage prediction frame
  • the segmentation prediction data indicates at least one damage object area
  • the result determination unit 93 is configured to, according to the at least one damage prediction frame and the At least one damage object area determines the damage detection result for the picture to be tested.
  • the result determining unit 93 may be configured to:
  • the union of the area set corresponding to the at least one damage prediction frame and the area set corresponding to the at least one damage target area is used as a damage detection result.
  • the at least one damage prediction frame includes a first damage prediction frame; the result determination unit 94 is configured to: if the intersection ratio of the first damage prediction frame and each damage object area is less than a preset threshold , Remove the first damage prediction frame from the damage detection result.
  • the result determining unit 94 may be further configured to: if the ratio of the overlapping area of the first damage prediction frame and each damage target area to the frame area of the first damage prediction frame is less than the predicted If the threshold is set, the first damage prediction frame is removed from the damage detection result.
  • the at least one damage prediction frame includes a first damage prediction frame, and the frame prediction data further includes a first damage category predicted for the first damage prediction frame; the at least one damage object area includes a first damage prediction frame.
  • a damage object area, and the pixels in the first damage object area correspond to the first segmentation category; in this case, the result determination unit 94 may be configured as:
  • the first damage detection The frame is determined as an abnormality detection frame, or the first damage target area is determined as an abnormal area.
  • the weak segmentation damage detection model is trained based on the weakly labeled data, and the damaged object is identified from the picture using the model.
  • a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed in a computer, the computer is caused to execute the method described in conjunction with FIG. 2 and FIG. 6.
  • a computing device including a memory and a processor, the memory stores executable code, and when the processor executes the executable code, a combination of FIGS. 2 and 6 is implemented. The method described.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)

Abstract

本说明书实施例提供一种计算机执行的基于弱分割进行损伤检测的方法和装置。在实施例中，获取对样本图片进行人工标注的边框标注数据，将标注框作为损伤对象的轮廓，自动生成分割弱标注数据。基于这样的边框标注数据和分割弱标注数据，训练两分支的弱分割损伤检测模型。在使用时，将待测图片输入该弱分割损伤检测模型，通过上述两个分支分别得到损伤预测边框和损伤对象区域。损伤预测边框和损伤对象区域可以互相验证补充，从而提高损伤检测的准确性。

Description

基于弱分割的损伤检测方法及装置 技术领域
本说明书一个或多个实施例涉及机器学习领域，尤其涉及利用机器学习基于弱分割进行损伤检测的方法和装置。
背景技术
随着机器学习的快速发展，各种人工智能技术已经应用于多种场景，帮助人们解决相应场景下的技术问题。其中，计算机视觉图像识别技术在多种领域多种场景下，都有强烈的应用需求，例如，应用于医疗影像分析，车损智能识别，等等。
例如，在传统车险理赔场景中，保险公司需要派出专业的查勘定损人员到事故现场进行现场查勘定损，给出车辆的维修方案和赔偿金额，并拍摄现场照片，定损照片留档以供后台核查人员核损核价。由于需要人工查勘定损，保险公司需要投入大量的人力成本，和专业知识的培训成本。从普通用户的体验来说，理赔流程由于等待人工查勘员现场拍照、定损员在维修地点定损、核损人员在后台核损，理赔周期长达1-3天，用户的等待时间较长，体验较差。针对这样的行业痛点，希望能够利用图像识别技术，根据普通用户拍摄的现场损失图片，自动识别图片中反映的车损状况，并自动给出维修方案。如此，无需人工查勘定损核损，大大减少保险公司的成本，提升普通用户的车险理赔体验。
在医疗影像分析中，也希望能够借助于图像识别技术，基于医疗影像智能地给出图像特点的分析，帮助医师进行诊断。
以上场景中，都需要从图片(车损图片或医疗图像)中识别出损伤对象。然而，目前的图像识别方案中，对于损伤对象识别的准确度以及可用性还有待进一步提高。因此，希望能有改进的方案，能够更精准地从图片中识别出损伤对象。
发明内容
本说明书一个或多个实施例描述了基于弱分割的损伤检测方法和装置，其中基于自动生成的弱分割标注数据训练弱分割损伤检测模型，该模型预测输出的损伤边框和损伤分割轮廓可以互相验证，提升损伤预测的准确性。
根据第一方面，提供了一种计算机执行的、训练弱分割损伤检测模型的方法，包括：
获取样本图片，所述样本图片对应具有边框标注数据，所述边框标注数据指示出至少一个损伤标注框，每个损伤标注框是标注人员标注的、框选出所述样本图片中损伤对象的最小矩形框；
将各个损伤标注框作为对应损伤对象的轮廓，基于所述轮廓为所述样本图片中各个像素点标注分割类别，从而生成分割标注数据；
将所述样本图片输入弱分割损伤检测模型，所述弱分割损伤检测模型包括边框预测分支和分割预测分支，所述边框预测分支输出用于指示损伤预测框的边框预测数据，所述分割预测分支输出针对所述样本图片中各个像素点的分割类别进行预测的分割预测数据；
基于所述边框预测数据与所述边框标注数据的比对，确定边框预测损失项，以及，基于所述分割预测数据和分割标注数据的比对，确定分割预测损失项；
根据所述边框预测损失项和所述分割预测损失项，确定本次预测的损失函数；
在所述损失函数减小的方向，更新所述弱分割损伤检测模型。
在一个实施例中，通过以下方式为各个像素点标注分割类别：
将位于损伤标注框之内的像素点标注为第一分割类别，将位于损伤标注框之外的像素点标注为第二分割类别。
在一种实施方式中，上述至少一个损伤标注框包括第一损伤标注框；所述边框标注数据还包括，标注人员从预定的N个损伤类别中，针对该第一损伤标注框选择标注的第一损伤类别；在这样的情况下，为各个像素点标注分割类别包括：对于位于所述第一损伤标注框之内的像素点，将其分割类别标注为所述第一损伤类别对应的类别。
在一种实施方式中，所述至少一个损伤标注框包括第一损伤标注框和第二损伤标注框，所述第一损伤标注框和第二损伤标注框存在交叠区域；所述边框标注数据还包括，标注人员从预定的N个损伤类别中，针对该第一损伤标注框选择标注的第一损伤类别和针对第二损伤标注框选择标注的第二损伤类别，其中第二损伤类别的损伤严重程度大于所述第一损伤类别；在这样的情况下，为各个像素点标注分割类别包括：对于位于所述交叠区域之内的像素点，将其分割类别标注为所述第二损伤类别对应的类别。
根据一种实施方式，弱分割损伤检测模型基于卷积神经网络CNN实现，所述卷积 神经网络CNN包括基础卷积层，用于对所述样本图片进行卷积处理，得到对应的卷积特征图；所述边框预测分支用于基于所述卷积特征图预测得到所述边框预测数据；所述分割预测分支用于基于所述卷积特征图预测得到所述分割预测数据。
进一步的，在一个实施例中，所述分割预测分支可以包括：
上采样层，用于将经卷积处理的特征上采样为与所述样本图片大小相同的第一特征图；
预测处理层，用于基于所述第一特征图预测各个像素点属于各个分割类别的概率。
根据一种实现方式，分割预测数据包括，各个像素点属于各个分割类别的概率；在这样的情况下，可以如下确定分割预测损失项：根据各个像素点属于各个分割类别的概率确定各个像素点的预测分割类别；将各个像素点的预测分割类别与其标注分割类别进行比对，根据比对结果确定所述分割预测损失项。
根据另一种实现方式，可以如下确定分割预测损失项：确定各个像素点属于其对应的标注分割类别的预测概率；根据所述预测概率，确定所述分割预测损失项。
根据第二方面，提供一种计算机执行的从图片中检测损伤的方法，包括：
获取根据第一方面的方法训练得到的弱分割损伤检测模型；
将待测图片输入所述弱分割损伤检测模型，所述弱分割损伤检测模型包括边框预测分支和分割预测分支，所述边框预测分支输出用于指示至少一个损伤预测框的边框预测数据，所述分割预测分支输出针对所述待测图片中各个像素点的分割类别进行预测的分割预测数据；
根据所述边框预测数据和所述分割预测数据，确定针对所述待测图片的损伤检测结果。
在一种情况下，边框预测数据可能未指示出损伤预测框，或者分割预测数据未指示出损伤对象区域，其中损伤对象区域为预测分割类别为同一类别的像素点集合构成的面积大于一定阈值的连通区域，此时，根据一种实施方式，可以将损伤检测结果确定为待测图片不包含损伤对象。
在另一种情况下，边框预测数据指示出至少一个损伤预测框，分割预测数据指示出至少一个损伤对象区域；在此情况下，可以根据所述至少一个损伤预测框和所述至少一个损伤对象区域，确定针对所述待测图片的损伤检测结果。
根据一种具体实施方式，确定针对所述待测图片的损伤检测结果可以包括：将所述至少一个损伤预测框对应的区域集合，与所述至少一个损伤对象区域对应的区域集合的并集，作为损伤检测结果。
根据另一种实施方式，确定针对所述待测图片的损伤检测结果可以包括：如果所述至少一个损伤预测框中任意的第一损伤预测框与各个损伤对象区域的交并比均小于预设阈值，则从所述损伤检测结果中移除该第一损伤预测框。
根据又一种实施方式，确定针对所述待测图片的损伤检测结果可以包括：如果所述至少一个损伤预测框中任意的第一损伤预测框与各个损伤对象区域的交叠面积，与该第一损伤预测框的框面积的比例均小于预设阈值，则从所述损伤检测结果中移除该第一损伤预测框。
在一个实施例中，所述至少一个损伤预测框包括第一损伤预测框，所述边框预测数据还包括针对该第一损伤预测框预测的第一损伤类别；所述至少一个损伤对象区域包括第一损伤对象区域，所述第一损伤对象区域中的像素点对应于第一分割类别；在这样的情况下，确定针对所述待测图片的损伤检测结果可以包括：
如果所述第一损伤预测框与所述第一损伤对象区域的交并比大于预设阈值，但所述第一损伤类别与所述第一分割类别不对应，则将所述第一损伤检测框确定为异常检测框，或将所述第一损伤对象区域确定为异常区域。
根据第三方面，提供一种训练弱分割损伤检测模型的装置，包括：
样本获取单元，配置为获取样本图片，所述样本图片对应具有边框标注数据，所述边框标注数据指示出至少一个损伤标注框，每个损伤标注框是标注人员标注的、框选出所述样本图片中损伤对象的最小矩形框；
标注生成单元，配置为将各个损伤标注框作为对应损伤对象的轮廓，基于所述轮廓为所述样本图片中各个像素点标注分割类别，从而生成分割标注数据；
模型输入单元，配置为将所述样本图片输入弱分割损伤检测模型，所述弱分割损伤检测模型包括边框预测分支和分割预测分支，所述边框预测分支输出用于指示损伤预测框的边框预测数据，所述分割预测分支输出针对所述样本图片中各个像素点的分割类别进行预测的分割预测数据；
第一确定单元，配置为基于所述边框预测数据与所述边框标注数据的比对，确定边框预测损失项，以及，基于所述分割预测数据和分割标注数据的比对，确定分割预测损 失项；
第二确定单元，配置为根据所述边框预测损失项和所述分割预测损失项，确定本次预测的损失函数；
模型更新单元，配置为在所述损失函数减小的方向，更新所述弱分割损伤检测模型。
根据第四方面，提供一种从图片中检测损伤的装置，包括：
模型获取单元，配置为获取通过权利要求15的装置训练得到的弱分割损伤检测模型；
模型输入单元，配置为将待测图片输入所述弱分割损伤检测模型，所述弱分割损伤检测模型包括边框预测分支和分割预测分支，所述边框预测分支输出用于指示至少一个损伤预测框的边框预测数据，所述分割预测分支输出针对所述待测图片中各个像素点的分割类别进行预测的分割预测数据；
结果确定单元，配置为根据所述边框预测数据和所述分割预测数据，确定针对所述待测图片的损伤检测结果。
根据第五方面，提供了一种计算机可读存储介质，其上存储有计算机程序，当所述计算机程序在计算机中执行时，令计算机执行第一方面和第二方面的方法。
根据第六方面，提供了一种计算设备，包括存储器和处理器，其特征在于，所述存储器中存储有可执行代码，所述处理器执行所述可执行代码时，实现第一方面和第二方面的方法。
根据本说明书实施例提供的方法和装置，基于人工标注的损伤框生成弱分割标注数据，利用这样的弱分割标注数据训练具有边框检测和分割预测两个分支的弱分割损伤检测模型。在使用时，将待测图片输入该弱分割损伤检测模型，通过上述两个分支分别得到损伤预测边框和损伤对象区域。损伤预测边框和损伤对象区域可以互相验证补充，从而提高损伤检测的准确性。
附图说明
为了更清楚地说明本发明实施例的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其它的附图。
图1示出根据一个实施例的实施场景示意图；
图2示出根据一个实施例的训练弱分割的损伤检测模型的方法流程图；
图3示出经过人工标注的样本图片的一个具体示例；
图4示出在一个实施例中为像素点标注分割类别的示意图；
图5示出根据一个实施例的弱分割损伤检测模型的结构示意图；
图6示出根据一个实施例从图片中识别损伤的方法的步骤流程图；
图7是在一个例子中弱分割损伤检测模型输出的损伤预测框和损伤对象区域；
图8示出根据一个实施例的训练弱分割损伤检测模型的装置的示意性框图；
图9示出根据一个实施例的损伤检测装置的示意性框图。
具体实施方式
下面结合附图，对本说明书提供的方案进行描述。
如前所述，在车辆智能定损、医疗影像分析等多种场景中，需要从图片中识别出损伤对象，例如车辆损伤，器官病变损伤。为此，可以训练以损伤对象为检测目标的损伤检测模型，通过损伤检测模型来进行损伤识别。一般的，模型的训练需要大量的标注数据。对于损伤检测来说，可以由标注人员将损伤对象作为特定的目标对象进行标注，也就是，在图片中用包含损伤对象的最小矩形框标注出损伤对象，这样的矩形框又称为损伤标注框。样本原图和损伤标注框共同构成图片训练样本。利用这样的训练样本进行模型训练，即可得到损伤检测模型。在训练得到损伤检测模型后，即可利用损伤检测模型对待测图片进行损伤识别。一般地，损伤检测模型的输出结果为，在待测图片中预测得到的若干损伤预测框，每个损伤预测框用最小矩形框框选出预测的损伤对象。
然而，如前所述，目前损伤检测模型的检测准确率还有待提高。特别是在车辆损伤识别的场景下，由于车损图片的拍摄环境比较复杂，常常受到反光、污渍等影响，使得损伤检测的准确性不够理想，例如将反光、污渍等情况误检测为损伤。
为此，发明人提出，在进行常规损伤检测的基础上，还对待测图片进行图像分割，利用分割的结果来验证或补充常规损伤检测结果。
图像分割又称为图像语义分割，用于将图像分割或者划分为属于/不属于特定目标对象的区域，其输出可以表现为覆盖特定目标对象区域的蒙层(Mask)。一般地，图像分 割模型基于对目标对象的轮廓进行标注的分割标注数据而训练。在损伤识别的场景下，图像分割用于将图片划分为属于/不属于损伤对象的区域。相应的，可以理解，模型训练则要求对损伤对象的轮廓进行标注的标注数据。
一般地，轮廓的标注要求标注人员用若干标注点描画出目标对象的边界，这个过程费时费力，人工成本很高。特别是对于损伤对象这样的形状不规则、边界不清晰的目标对象来说，标注成本更是高昂。
综合考虑到以上因素，根据本发明的实施例，利用损伤标注框作为损伤对象的粗略轮廓标注，进行图像分割的训练，得到一个两分支的损伤检测模型，其中一个分支进行常规的损伤对象检测，另一分支进行图像分割。由于分割标注数据是将损伤标注框直接作为损伤对象的轮廓而自动生成的，精度有限，可以将如此训练的模型称为弱分割损伤检测模型。弱分割损伤检测模型可以输出常规损伤预测框和弱分割结果，这两方面的结果可以互相融合，来优化最终的损伤检测结果。
图1示出根据一个实施例的实施场景示意图。如图1所示，预先利用训练样本训练弱分割损伤检测模型。具体而言，训练样本由样本图片和标注数据构成，其中用于分割训练的标注数据，是通过直接将损伤标注框作为损伤对象的轮廓而自动生成的。利用这样的训练样本可以训练得到如图1所示的两分支的损伤检测模型，其中第一分支为检测分支，用于进行常规的损伤检测，另一分支为分割分支，用于进行图像弱分割。
在训练得到弱分割损伤检测模型后，就可以利用该模型进行损伤检测。如图1所示，可以将待测图片输入该损伤检测模型。在该模型中，首先对待测图片进行特征提取，之后，经由检测分支，输出针对待测图片的损伤预测框，经由分割分支，输出针对待测图片的损伤分割区域。通过对损伤预测框和损伤分割区域的融合分析，可以得到最终的损伤检测结果。
下面分别描述以上弱分割的损伤检测模型的训练过程，以及使用该模型进行损伤检测的预测过程。
图2示出根据一个实施例的训练弱分割的损伤检测模型的方法流程图。可以理解，该方法可以通过任何具有计算、处理能力的装置、设备、平台、设备集群来执行。如图2所示，训练过程至少包括以下步骤：步骤21，获取样本图片，所述样本图片对应具有指示出损伤标注框的边框标注数据；步骤22，将各个损伤标注框作为对应损伤对象的轮廓，基于所述轮廓为样本图片中各个像素点标注分割类别，从而生成分割标注数据；步 骤23，将样本图片输入弱分割损伤检测模型，该模型包括边框预测分支和分割预测分支，所述边框预测分支输出指示损伤预测框的损伤预测数据，所述分割预测分支通过预测各个像素点的分割类别输出分割预测数据；步骤24，基于损伤预测数据与边框标注数据的比对，确定边框预测损失项，以及，基于分割预测数据和分割标注数据的比对，确定分割预测损失项；步骤25，根据所述边框预测损失项和所述分割预测损失项，确定本次预测的损失函数；步骤26，在损失函数减小的方向，更新所述损伤预测模型。下面描述以上各个步骤的具体执行方式。
首先，在步骤21，获取经过人工标注的样本图片。该样本图片一般是包含有损伤对象的图片，例如车损现场拍摄的受损车辆的照片。这样的样本图片经由标注人员标注，而具有对应的标注数据。对于损伤检测任务的标注来说，针对图片中的一处损伤，标注人员可以采用一个包围该处损伤的最小矩形框来框选出该损伤对象，这样的最小矩形框称为损伤标注框。因此，上述样本图片的人工标注数据又称为边框标注数据，可以指示出各个损伤标注框。
图3示出经过人工标注的样本图片的一个具体示例。在图3的示例中，样本图片为受损车辆的图片。经过人工标注，标注人员在该图片中用矩形框框选出若干损伤对象，即标注出若干损伤标注框。
在一个实施例中，标注人员可以仅对损伤对象所在位置进行标注，也就是用损伤标注框框选出损伤对象，而不区分损伤种类。
在另一实施例中，标注人员还可以针对每个损伤对象，从预先设定的N种损伤类别中标注出该损伤对象的类别。例如，对于车辆损伤而言，可以预先设定N＝10种损伤类别，例如包括，刮擦，变形，撕裂，(玻璃)碎裂等等。如此，对于每处车辆损伤，标注人员在标出损伤标注框的基础上，从以上10种损伤类别中选择该处损伤对应的类别进行标注。
在图3中，损伤标注框右上角的数字，示出该损伤标注框中损伤对象的损伤类别，例如12对应于刮擦，10对应于变形。可以理解，损伤类别的标注方式可以有多种，除了图3所示的用不同数字表示不同损伤类别，还可以例如用不同颜色的标注框表示不同损伤类别，等等，图3仅仅是一种示例。
在获取到样本图片和该图片对应的边框标注数据的基础上，在步骤22，基于边框标注数据自动生成分割标注数据。
如前所述，分割标注数据是为进行图像分割训练，而对目标对象的轮廓进行标注的数据。常规技术中，基于标注人员用若干标注点描画出的目标对象的轮廓，得到分割标注数据。典型的，分割标注数据为像素级的标注，即标注出图片中各个像素点是否属于特定目标对象。
为了降低标注成本，在步骤22中，直接将各个损伤标注框作为对应损伤对象的轮廓，基于该轮廓为样本图片中各个像素点标注分割类别，从而自动生成分割标注数据。
更具体的，在一个实施例中，将损伤标注框作为损伤对象的轮廓，根据各个像素点是否落入损伤标注框，确定该像素点是否属于损伤对象，进而为其标注分割类别。例如，将位于损伤标注框之内的像素点标注为第一分割类别，将位于损伤标注框之外的像素点标注为第二分割类别。如此，样本图片中的像素点被划分为2个类别，即属于损伤对象的前景部分(对应于第一分割类别)，和不属于损伤对象的背景部分(对应于第二分割类别)。
在一种实施方式下，如前所述，边框标注数据中还包括标注人员针对各个损伤标注框标注的损伤类别，例如图3中对应于刮擦的损伤类别12，对应于变形的损伤类别10。在这样的情况下，在一个实施例中，还基于像素点落入的损伤标注框的损伤类别，为像素点标注分割类别。具体的，对于任意的一个损伤标注框，为了描述的简单，将其称为第一损伤标注框，假定该第一损伤标注框被标注具有第一损伤类别。那么，在步骤22，对于落入该第一损伤标注框之内的像素点，可以将其分割类别标注为第一损伤类别对应的类别。
在这样的情况下，如果预先设定了N种损伤类别，那么在分割标注中可以将像素点划分为N+1个分割类别，其中前N个分割类别与前述N种损伤类别一一对应，另外的1个分割类别对应于不属于任何损伤标注框的情况。更具体的，对于车辆损伤而言，如果预先为损伤对象设定了10种损伤类别，那么在标注某个像素点的分割类别时，可以根据该像素点落入的损伤标注框对应的10种损伤类别之一，为其标注分割类别；如果该像素点不落入任何损伤标注框，则将其分割类别标注为第11种类别。
在一种实施方式中，允许损伤标注框之间存在重叠。在这样的情况下，对于落入多个损伤标注框的交叠区域的像素点，可以基于预先设定的、该多个损伤标注框对应的损伤类别的严重程度的排序，为其标注分割类别。
一般而言，在设定N种损伤类别的基础上，还可以预先设定各个损伤类别对应的损 伤严重程度的排序。例如，在车辆损伤的情况下，可以设定损伤类别的严重程度从轻到重的排序包括，刮擦<变形<撕裂…，等等。在一个实施例，当某个像素点落入多个损伤标注框的交叠区域时，可以基于损伤类别的严重程度的排序，确定出该多个损伤标注框中损伤程度较重的损伤类别，将该损伤类别的对应类别作为该像素点的分割类别。
图4示出在一个实施例中为像素点标注分割类别的示意图。在图4中示出3个损伤标注框A，B和C，分别被标注有损伤类别a，b和c。对于损伤标注框A中包含的各个像素点(阴影区域的像素点)，可以将其分割类别标注为损伤类别a对应的类别。
损伤标注框B和C之间存在交叠区域。假定损伤类别b的严重程度大于损伤类别c，那么可以将交叠区域中的像素点标注为损伤类别b对应的类别。对于损伤标注框B和C中交叠区域之外的像素点，分别将其标注为损伤类别b和c对应的类别。如此，斜线区域的像素点被标注为类别b，方格区域的像素点被标注为类别c。
对于未落入任何损伤标注框的其他像素点，可以将其分割类别标注为类别0。
如此，通过以上各种方式，将损伤标注框作为对应损伤对象的轮廓，针对各个像素点标注分割类别，从而自动生成分割标注数据。
可以理解，损伤标注框是框选出损伤对象的最小矩形框，通常并不等于损伤对象的真实轮廓。将损伤标注框作为损伤对象轮廓进行分割标注，只是一种粗略的近似标注，因此也可以称为弱分割标注数据。这样的弱分割标注数据可以用于训练弱分割损伤检测模型。
回到图2，另一方面，在步骤23，将样本图片输入弱分割损伤检测模型，由该弱分割损伤检测模型来预测样本图片中的损伤对象。需要理解，此处的弱分割损伤检测模型可以是初始模型，也可以是训练过程中有待进一步更新的模型。具体而言，该弱分割损伤检测模型具有两个分支，边框预测分支和分割预测分支，其中边框预测分支输出针对所述样本图片预测的损伤预测框，分割预测分支对各个像素点的分割类别进行预测而产生分割预测数据。
弱分割损伤检测模型可以基于各种算法和模型结构实现。典型的，弱分割损伤检测模型可以基于卷积神经网络CNN实现。
图5示出根据一个实施例的弱分割损伤检测模型的结构示意图。在该示例中，弱分割损伤检测模型实现为卷积神经网络CNN。如图5所示，该神经网络包括基础卷积层510，边框预测分支520和分割预测分支530。
基础卷积层510用于对输入的样本图片进行卷积处理，得到卷积特征图(feature map)。一般的，基础卷积层510可以包括多个子层，在每个子层，采用对应的卷积核对图片进行卷积处理。在子层的卷积处理后，可选的，还可以进行池化(pooling)处理。通过多次卷积和池化，得到的特征图可以反映原始样本图片中更抽象和高阶的特征。一般的，取决于卷积核的大小和卷积处理的次数，卷积处理后的特征图比原始样本图片的维度要小。
边框预测分支520基于卷积特征图进行边框预测，输出边框预测数据。边框预测数据指示出所预测的损伤预测框。在训练数据中还包括针对各个损伤标注框的损伤类别标注时，边框预测分支还预测各个损伤预测框对应的损伤类别。边框预测分支可以采用各种已知的目标检测算法实现，在结构上，也可以采用各种已知的用于进行目标检测的神经网络结构，例如可以包括，进一步的卷积处理层，边框回归层，全连接处理层，等等。
分割预测分支530基于上述卷积特征图进行像素点的分割类别预测，从而得到分割预测数据。
具体地，分割预测分支530可以包括分割卷积层531，用于对卷积特征图进行进一步的卷积处理。该分割卷积层531中的卷积核可以是针对分割预测的特点而设计，不同于基础卷积层中的卷积核，从而使得进一步卷积处理后的特征更有利于后续的分割预测处理。不过应理解，该卷积层531为可选的网络处理层，在一些情况下也可以省去该网络处理层。
分割预测分支530还包括上采样层532，用于将经卷积处理的特征上采样为与所述样本图片大小相同的第一特征图。如前所述，经过卷积处理后的特征图比原始样本图片的维度要小，因此，在分割预测分支中，通过上采样层，还原出原图大小的特征图。具体的，上采样层532可以采用插值处理等上采样方式，将卷积处理后的维度较小的特征还原为与原始样本图片维度相同的第一特征图。因此，第一特征图中的各个像素点与原始样本图片中的各个像素点分别对应。
接着，分割处理层533预测所述第一特征图中各个像素点属于各个分割类别的概率。由于第一特征图中的像素点与原始样本图片中的像素点分别对应，因此，分割处理层533对第一特征图的像素点的分割类别进行预测，即对应于预测样本图片中各个像素点属于各个分割类别的概率。
具体的，在二分类的情况下，分割处理层533可以预测各个像素点属于损伤对象(对 应于第一分割类别)的概率。在一个实施例中，可以将概率值大于预定阈值的像素点确定为属于损伤对象。
在多分类的情况下，例如前述的N+1个分割类别的情况下，分割处理层533可以预测各个像素点i分别属于N+1个分割类别中第j个分割类别的概率P ij。于是，对于每个像素点i，可以形成其分别属于各个分割类别的概率向量(P i0,P i1,P i2,…,P iN)。或者，对于每个分割类别j，可以形成各个像素点属于该分割类别j的概率集合。由于图片中的像素点一般排列成矩阵形式，因此针对每个分割类别j的概率集合可以形成为与像素点对应的概率矩阵。
至少通过以上的上采样层532和分割处理层533，分割预测分支530可以输出针对各个像素点的分割类别进行预测的分割预测数据。在一个实施例中，分割预测数据即分割处理层533预测的、像素点属于各个分割类别的概率。在另一实施例中，分割预测分支还可以进一步基于上述概率，确定各个像素点的预测分割类别，作为分割预测数据。例如，对于每个像素点，将预测概率最高的分割类别作为其预测分割类别。
于是，通过图5所示的弱分割损伤检测模型，分别得到边框预测数据和分割预测数据。
需要说明的是，除了用于分割训练的分割标注数据是基于损伤标注框自动生成，因而不同于常规分割标注数据之外，弱分割损伤检测模型的模型结构也与常规的损伤检测模型不同。常规的损伤检测模型并不具有分割分支，因而并不进行图像分割；常规的图像分割模型也不具有边框预测分支，不进行边框预测。尽管也存在一些同时进行边框预测和图像分割的神经网络模型，例如Mask-RCNN，但是，这样的模型通常是在选定备选损伤框后，才划分为两个分支。一个分支对备选损伤框进行进一步的回归和类别预测，另一分支基于该备选损伤框进行图像分割，也就是说，图像分割是在备选损伤框之内进行的。换而言之，在诸如Mask-RCNN之类的模型中，图像分割是在边框预测基础上进行的，而非与其独立的分支。
而在图5所示的弱分割损伤检测模型中，在通过基础卷积层进行基本的卷积处理后，模型就分裂为两个独立的分支。边框预测分支和分割预测分支各自基于卷积处理后的特征图进行进一步处理和预测，预测过程互相独立。两个分支独立进行预测，使得各自输出的预测结果更有利于进行互相验证。
实践中，可以基于多种具体的CNN网络结构进行修改，得到上述弱分割损伤检 测模型。例如，可以采用循环全卷积网络RFCN作为基础网络结构，在特征图之后，位置敏感(position-sensitive)卷积之前，添加分割预测分支。
如此，通过多种具体的神经网络结构，实现两分支的弱分割损伤检测模型，该模型通过边框预测分支产生与损伤预测框相关的边框预测数据，通过分割预测分支产生与各像素点的分割类别相关的分割预测数据。
为了对这样的模型进行训练，接下来，就需要将上述预测数据与对应的标注数据进行比对，以得到本次预测的损失函数值，作为模型中参数更新和调整的依据。
具体的，回到图2，在步骤24，基于边框预测数据与边框标注数据的比对，确定边框预测损失项。在一个实施例中，边框预测数据指示出预测的各个损伤预测框的位置。可以通过比对损伤预测框和损伤标注框的位置差异，确定边框预测损失项。在一个实施例中，边框预测数据还指示出各个损伤预测框对应的预测损伤类别。此时，在上述位置差异的基础上，还比对损伤预测框的预测损伤类别与对应的损伤标注框的标注损伤类别，基于类别比对，确定边框预测损失项。在数学形式上，边框预测损失项的确定可以采用各种常规算法进行，例如采用L2误差的形式作为边框预测损失项。
在步骤24，还基于分割预测数据和分割标注数据的比对，确定分割预测损失项。
根据一种实施方式，基于分割类别的比对，确定分割预测损失项。具体的，在一个实施例中，分割预测分支输出各个像素点属于各个分割类别的概率作为分割预测数据。此时，可以根据该概率确定出各个像素点的预测分割类别，例如将概率最大的分割类别作为其对应的预测分割类别。在另一实施例中，分割预测分支直接输出各个像素点的预测分割类别，此时，可以直接从分割预测数据中获取到该预测分割类别。然后，将各个像素点的预测分割类别与其标注分割类别进行比对，统计预测分割类别与标注分割类别一致的像素点，即预测正确的像素点的数目，或者该数目占像素点总数的比例。基于上述数目或者比例，确定分割预测损失项，使得数目或者比例越大，分割预测损失项越小。
根据另一实施方式，基于分割预测数据中各个像素点属于各个分割类别的概率，与各个像素点的标注分割类别，确定分割预测损失项。具体的，对于每个像素点i，可以从分割预测数据中获取该像素点i分别属于各个分割类别的概率向量(P i0,P i1,P i2,…,P iN)。另一方面，获取像素点i的标注分割类别k。于是，可以将该像素点i对应的分割损失项确定为：
Figure PCTCN2020072058-appb-000001
其中P ij表示像素点i属于分割类别j的概率，分割类别k为标注分割类别。然后对各个像素点i的分割损失项求和，可以得到本次预测的分割损失项。
在另一实施例中，可以采用softmax函数的形式，根据各个像素点属于各个分割类别的预测概率以及其标注分割类别，确定分割损失项L为：
Figure PCTCN2020072058-appb-000002
根据又一实施方式，基于各个像素点对应于标注分割类别的预测概率，确定分割预测损失项。具体的，对于任意像素点i，从分割标注数据中获取该像素点的标注分割类别k。另一方面，从分割预测数据中获取该像素点i分别属于各个分割类别的概率向量(P i0,P i1,P i2,…,P iN)，从中提取出与标注分割类别k对应的预测概率P ik。可以基于各个像素点的与标注分割类别对应的预测概率，确定出分割预测损失项。例如，在一个例子中，分割预测损失项可以表示为：
Figure PCTCN2020072058-appb-000003
其中，Pi表示第i个像素点属于其标注分割类别的预测概率。
在其他实施例中，还可以采用其他函数形式，交叉熵等形式，基于各个像素点属于各个分割类别的概率，确定出分割预测损失项。
接着，在步骤25，根据步骤24确定的边框预测损失项和分割预测损失项，确定本次预测的损失函数。在一个实施例中，可以将边框预测损失项和分割预测损失项的加和，确定为本次预测的损失函数。在另一实施例中，还可以对边框预测损失项和分割预测损失项进行加权求和，得到本次预测的损失函数。
然后在步骤26，在损失函数减小的方向，更新上述弱分割损伤预测模型。换而言之，调整弱分割损伤检测模型的模型参数，使得损失函数不断减小。参数调整的过程可以采用反向传播，梯度下降等常规方式。
通过以上过程，不断更新模型参数，直到达到预定的收敛条件，或者模型测试结果达到一定的准确率条件。此时，完成模型的训练，得到训练的弱分割损伤检测模型。 该模型基于自动生成的弱分割标注数据而训练，通过两个独立分支实现损伤边框的强预测，以及损伤轮廓的弱预测。
下面描述使用以上训练得到的弱分割损伤检测模型进行损伤预测的过程。
图6示出根据一个实施例从图片中识别损伤的方法的步骤流程图。如图6所示，该过程包括以下步骤。
在步骤61，获取通过图2的方法训练得到的弱分割损伤检测模型，该模型包括边框预测分支和分割预测分支。该弱分割损伤检测模型例如具有图5所示的结构。
在步骤62，将待测图片输入所述弱分割损伤检测模型。于是，模型中的边框预测分支输出用于指示损伤预测框的边框预测数据，分割预测分支输出针对待测图片中各个像素点的分割类别进行预测的分割预测数据。通过两个分支分别输出边框预测数据和分割预测数据的过程与前述训练过程中的步骤23相似，不再赘述。
接着，在步骤63，根据所述边框预测数据和所述分割预测数据，确定针对所述待测图片的损伤检测结果。
为此，在一个实施例中，首先尝试根据分割预测数据，确定损伤对象区域，即预测分割类别为同一损伤类别的像素点集合构成的面积大于一定阈值的连通区域。
具体的，在一个例子中，分割预测数据包括待测图片中各个像素点对应的预测分割类别。在另一个例子中，分割预测数据中包含的是，待测图片中各个像素点属于各个分割类别的概率。在这样的情况下，可以基于上述概率，确定出各个像素点的预测分割类别。
基于各个像素点的预测分割类别，可以得到损伤对象区域。具体的，可以首先得到预测分割类别为同一损伤类别的像素点集合，判断该集合是否构成连通区域以及该连通区域面积是否大于一定阈值。如果同一损伤类别的像素点集合可以构成面积大于一定阈值的连通区域，则将该连通区域作为一个损伤对象区域。
在一种情况下，边框预测数据指示出至少一个损伤预测框，但是从分割预测数据无法得到上述损伤对象区域；或者，从分割预测数据得到至少一个损伤对象区域，但是边框预测数据没有指示出损伤预测框。在一个实施例中，在以上两种情况下，在步骤63，确定待测图片中不包含损伤对象。
在另外的情况下，边框预测数据指示出至少一个损伤预测框，从分割预测数据 得到至少一个损伤对象区域，在这样的情况下，在步骤63，根据上述至少一个损伤预测框和至少一个损伤对象区域，确定针对所述待测图片的损伤检测结果。
根据一种实施方式，将所述至少一个损伤预测框对应的区域集合，与所述至少一个损伤对象区域对应的区域集合的并集，作为损伤检测结果。如此得到的损伤检测结果会较为全面地包含可能的损伤，可以最大程度地避免漏检。
在一种实施方式中，根据损伤预测框和损伤对象区域之间的交叠情况，从损伤检测结果中排除掉异常损伤预测框。下面以任意一个损伤预测框，称为第一损伤预测框为例进行说明。
在一个实施例中，可以分别计算第一损伤预测框与各个损伤对象区域的交并比(IoU)，即交叠面积与并集面积的比值。如果计算得到的各个交并比均小于预设阈值，则说明，该第一损伤预测框与各个损伤对象区域重叠太少，换而言之，不存在损伤对象区域可以验证该第一损伤预测框，于是，将该第一损伤预测框从损伤检测结果中移除。
在另一实施例中，计算该第一损伤预测框与各个损伤对象区域的交叠面积与该第一损伤预测框本身的框面积的比例。如果计算得到的比例均小于预设阈值，则从损伤检测结果中移除该第一损伤预测框。
根据一种实施方式，不仅验证损伤预测框和损伤对象区域的重叠大小，还验证类别的异同。在这样的实施方式中，要求边框预测数据还包括针对各个损伤预测框的损伤类别的预测。假定边框预测数据中，第一损伤预测框对应于预测的第一损伤类别。另一方面，对于至少一个损伤对象区域中任意的第一损伤对象区域，假定其中各个像素点对应于第一分割类别。
根据一个实施例，首先判断第一损伤预测框和第一损伤对象区域的交并比IoU。如果交并比大于预设阈值，也就是两者重叠率足够高，那么判断第一损伤类别与第一分割类别是否对应。如果对应，则第一损伤预测框和第一损伤对象区域彼此互相验证；如果不对应，例如第一损伤类别指示出刮擦，而第一分割类别指示出玻璃碎裂，则说明二者之中至少一个存在异常，于是可以将第一损伤检测框确定为异常检测框，或者将第一损伤对象区域确定为异常区域，留待进一步检测确认。如此，更进一步的从损伤检测结果中排除误检的异常结果。
图7是在一个例子中，弱分割损伤检测模型输出的损伤预测框和损伤对象区域。在图7的例子中，待测图片为车辆损伤图片。在待测图片中预测得到多个损伤预测框， 每个损伤预测框左上角标出的两个数字分别示出预测损伤类别和预测的置信度。损伤对象区域根据各个像素点的预测分割类别而生成，用蒙层mask示出。实践中，对于多损伤类别的情况，可以用不同颜色的蒙层示出对应于不同分割类别的损伤对象区域。对于图7所示的损伤预测框和损伤对象区域，可以采用上述各种方式，对损伤预测框和损伤对象区域进行综合分析，得到损伤检测结果。
回顾以上过程，通过人工标注的边框强标注数据和自动生成的分割弱标注数据，训练得到两分支的弱分割损伤检测模型，其中边框预测分支和分割预测分支彼此独立地分别进行边框预测和分割预测。由于分割标注数据为弱标注，精度并不高，相应的，分割预测分支的预测结果通常并不精准，难以作为独立的损伤对象轮廓结果。但是，弱分割的预测结果可以与边框预测结果结合使用，用来与损伤预测框互相验证，从中发现漏检、误检的情况，从而优化、完善损伤检测结果，提高检测的准确性。
根据另一方面的实施例，提供了一种训练弱分割损伤检测模型的装置，该装置可以部署在任何具有计算、处理能力的设备、平台或设备集群中。图8示出根据一个实施例的训练弱分割损伤检测模型的装置的示意性框图。如图8所示，该训练装置800包括：
样本获取单元81，配置为获取样本图片，所述样本图片对应具有边框标注数据，所述边框标注数据指示出至少一个损伤标注框，每个损伤标注框是标注人员标注的、框选出所述样本图片中损伤对象的最小矩形框；
标注生成单元82，配置为将各个损伤标注框作为对应损伤对象的轮廓，基于所述轮廓为所述样本图片中各个像素点标注分割类别，从而生成分割标注数据；
模型输入单元83，配置为将所述样本图片输入弱分割损伤检测模型，所述弱分割损伤检测模型包括边框预测分支和分割预测分支，所述边框预测分支输出用于指示损伤预测框的边框预测数据，所述分割预测分支输出针对所述样本图片中各个像素点的分割类别进行预测的分割预测数据；
第一确定单元84，配置为基于所述边框预测数据与所述边框标注数据的比对，确定边框预测损失项，以及，基于所述分割预测数据和分割标注数据的比对，确定分割预测损失项；
第二确定单元85，配置为根据所述边框预测损失项和所述分割预测损失项，确定本次预测的损失函数；
模型更新单元86，配置为在所述损失函数减小的方向，更新所述弱分割损伤检测模型。
在一个实施例中，所述标注生成单元82配置为：
将位于损伤标注框之内的像素点标注为第一分割类别，将位于损伤标注框之外的像素点标注为第二分割类别。
根据一个实施例，所述至少一个损伤标注框包括第一损伤标注框；所述边框标注数据还包括，标注人员从预定的N个损伤类别中，针对该第一损伤标注框选择标注的第一损伤类别。此时，所述标注生成单元82可以配置为：对于位于所述第一损伤标注框之内的像素点，将其分割类别标注为所述第一损伤类别对应的类别。
根据一个实施例，所述至少一个损伤标注框包括第一损伤标注框和第二损伤标注框，所述第一损伤标注框和第二损伤标注框存在交叠区域；所述边框标注数据还包括，标注人员从预定的N个损伤类别中，针对该第一损伤标注框选择标注的第一损伤类别和针对第二损伤标注框选择标注的第二损伤类别，其中第二损伤类别的损伤严重程度大于所述第一损伤类别。在这样的情况下，所述标注生成单元82可以配置为：对于位于所述交叠区域之内的像素点，将其分割类别标注为所述第二损伤类别对应的类别。
根据一种实施方式，弱分割损伤检测模型可以基于卷积神经网络CNN实现，所述卷积神经网络CNN包括基础卷积层，用于对所述样本图片进行卷积处理，得到对应的卷积特征图；相应的，所述边框预测分支可以基于所述卷积特征图预测得到所述边框预测数据；所述分割预测分支可以基于所述卷积特征图预测得到所述分割预测数据。
进一步的，在一个实施例中，所述分割预测分支可以包括：
上采样层，用于将经卷积处理的特征上采样为与所述样本图片大小相同的第一特征图；
预测处理层，用于基于所述第一特征图预测各个像素点属于各个分割类别的概率。
根据一种实施方式，所述分割预测数据包括，各个像素点属于各个分割类别的概率；所述第一确定单元84配置为：
根据各个像素点属于各个分割类别的概率确定各个像素点的预测分割类别；
将各个像素点的预测分割类别与其标注分割类别进行比对，根据比对结果确定 所述分割预测损失项。
根据另一种实施方式，所述第一确定单元84配置为：
确定各个像素点属于其对应的标注分割类别的预测概率；
根据所述预测概率，确定所述分割预测损失项。
根据又一方面的实施例，提供了一种从图片中检测损伤的装置，该装置可以部署在任何具有计算、处理能力的设备、平台或设备集群中。图9示出根据一个实施例的损伤检测装置的示意性框图。如图9所示，该检测装置900包括：
模型获取单元91，配置为获取通过图8的装置训练得到的弱分割损伤检测模型；
模型输入单元92，配置为将待测图片输入所述弱分割损伤检测模型，所述弱分割损伤检测模型包括边框预测分支和分割预测分支，所述边框预测分支输出用于指示至少一个损伤预测框的边框预测数据，所述分割预测分支输出针对所述待测图片中各个像素点的分割类别进行预测的分割预测数据；
结果确定单元93，配置为根据所述边框预测数据和所述分割预测数据，确定针对所述待测图片的损伤检测结果。
在一个实施例中，所述结果确定单元93配置为：
当所述边框预测数据未指示出损伤预测框，或者所述分割预测数据未指示出损伤对象区域时，将所述损伤检测结果确定为待测图片不包含损伤对象，其中所述损伤对象区域为预测分割类别为同一损伤类别的像素点集合构成的面积大于一定阈值的连通区域。
在一个实施例中，边框预测数据指示出至少一个损伤预测框，分割预测数据指示出至少一个损伤对象区域；此时所述结果确定单元93配置为，根据所述至少一个损伤预测框和所述至少一个损伤对象区域，确定针对所述待测图片的损伤检测结果。
具体地，在一个实施例中，所述结果确定单元93可以配置为：
将所述至少一个损伤预测框对应的区域集合，与所述至少一个损伤对象区域对应的区域集合的并集，作为损伤检测结果。
根据一个实施例，所述至少一个损伤预测框包括第一损伤预测框；所述结果确定单元94配置为：如果所述第一损伤预测框与各个损伤对象区域的交并比均小于预设阈值，则从所述损伤检测结果中移除该第一损伤预测框。
根据另一实施例，所述结果确定单元94还可以配置为：如果所述第一损伤预测框与各个损伤对象区域的交叠面积与所述第一损伤预测框的框面积的比例均小于预设阈值，则从所述损伤检测结果中移除该第一损伤预测框。
根据一种实施方式，所述至少一个损伤预测框包括第一损伤预测框，所述边框预测数据还包括针对该第一损伤预测框预测的第一损伤类别；所述至少一个损伤对象区域包括第一损伤对象区域，所述第一损伤对象区域中的像素点对应于第一分割类别；在这样的情况下，所述结果确定单元94可以配置为：
如果所述第一损伤预测框与所述第一损伤对象区域的交并比大于预设阈值，但所述第一损伤类别与所述第一分割类别不对应，则将所述第一损伤检测框确定为异常检测框，或将所述第一损伤对象区域确定为异常区域。
通过以上图8和图9的装置，基于弱标注数据训练弱分割损伤检测模型，并使用该模型从图片中识别出损伤对象。
根据另一方面的实施例，还提供一种计算机可读存储介质，其上存储有计算机程序，当所述计算机程序在计算机中执行时，令计算机执行结合图2和图6所描述的方法。
根据再一方面的实施例，还提供一种计算设备，包括存储器和处理器，所述存储器中存储有可执行代码，所述处理器执行所述可执行代码时，实现结合图2和图6所述的方法。
本领域技术人员应该可以意识到，在上述一个或多个示例中，本发明所描述的功能可以用硬件、软件、固件或它们的任意组合来实现。当使用软件实现时，可以将这些功能存储在计算机可读介质中或者作为计算机可读介质上的一个或多个指令或代码进行传输。
以上所述的具体实施方式，对本发明的目的、技术方案和有益效果进行了进一步详细说明，所应理解的是，以上所述仅为本发明的具体实施方式而已，并不用于限定本发明的保护范围，凡在本发明的技术方案的基础之上，所做的任何修改、等同替换、改进等，均应包括在本发明的保护范围之内。

Claims (30)

  1. 一种计算机执行的、训练弱分割损伤检测模型的方法，包括：
    获取样本图片，所述样本图片对应具有边框标注数据，所述边框标注数据指示出至少一个损伤标注框，每个损伤标注框是标注人员标注的、框选出所述样本图片中损伤对象的最小矩形框；
    将各个损伤标注框作为对应损伤对象的轮廓，基于所述轮廓为所述样本图片中各个像素点标注分割类别，从而生成分割标注数据；
    将所述样本图片输入弱分割损伤检测模型，所述弱分割损伤检测模型包括边框预测分支和分割预测分支，所述边框预测分支输出用于指示损伤预测框的边框预测数据，所述分割预测分支输出针对所述样本图片中各个像素点的分割类别进行预测的分割预测数据；
    基于所述边框预测数据与所述边框标注数据的比对，确定边框预测损失项，以及，基于所述分割预测数据和分割标注数据的比对，确定分割预测损失项；
    根据所述边框预测损失项和所述分割预测损失项，确定本次预测的损失函数；
    在所述损失函数减小的方向，更新所述弱分割损伤检测模型。
  2. 根据权利要求1所述的方法，其中，基于所述轮廓为所述样本图片中各个像素点标注分割类别包括：
    将位于损伤标注框之内的像素点标注为第一分割类别，将位于损伤标注框之外的像素点标注为第二分割类别。
  3. 根据权利要求1所述的方法，其中，所述至少一个损伤标注框包括第一损伤标注框；所述边框标注数据还包括，标注人员从预定的N个损伤类别中，针对该第一损伤标注框选择标注的第一损伤类别；
    基于所述轮廓为所述样本图片中各个像素点标注分割类别包括：
    对于位于所述第一损伤标注框之内的像素点，将其分割类别标注为所述第一损伤类别对应的类别。
  4. 根据权利要求1所述的方法，其中，所述至少一个损伤标注框包括第一损伤标注框和第二损伤标注框，所述第一损伤标注框和第二损伤标注框存在交叠区域；所述边框标注数据还包括，标注人员从预定的N个损伤类别中，针对该第一损伤标注框选择标注的第一损伤类别和针对第二损伤标注框选择标注的第二损伤类别，其中第二损伤类别的损伤严重程度大于所述第一损伤类别；
    基于所述轮廓为所述样本图片中各个像素点标注分割类别包括：
    对于位于所述交叠区域之内的像素点，将其分割类别标注为所述第二损伤类别对应的类别。
  5. 根据权利要求1所述的方法，其中，所述弱分割损伤检测模型基于卷积神经网络CNN实现，所述卷积神经网络CNN包括基础卷积层，用于对所述样本图片进行卷积处理，得到对应的卷积特征图；
    所述边框预测分支用于基于所述卷积特征图预测得到所述边框预测数据；
    所述分割预测分支用于基于所述卷积特征图预测得到所述分割预测数据。
  6. 根据权利要求5所述的方法，其中，所述分割预测分支包括：
    上采样层，用于将经卷积处理的特征上采样为与所述样本图片大小相同的第一特征图；
    预测处理层，用于基于所述第一特征图预测各个像素点属于各个分割类别的概率。
  7. 根据权利要求1所述的方法，其中，所述基于所述分割预测数据和分割标注数据的比对，确定分割预测损失项包括：
    根据所述分割预测数据，确定各个像素点属于各个分割类别的预测概率；
    根据所述各个像素点属于各个分割类别的预测概率以及其标注分割类别，确定所述分割预测损失项。
  8. 一种计算机执行的从图片中检测损伤的方法，包括：
    获取根据权利要求1的方法训练得到的弱分割损伤检测模型；
    将待测图片输入所述弱分割损伤检测模型，所述弱分割损伤检测模型包括边框预测分支和分割预测分支，所述边框预测分支输出用于指示损伤预测框的边框预测数据，所述分割预测分支输出针对所述待测图片中各个像素点的分割类别进行预测的分割预测数据；
    根据所述边框预测数据和所述分割预测数据，确定针对所述待测图片的损伤检测结果。
  9. 根据权利要求8的方法，其中，根据所述边框预测数据和所述分割预测数据，确定针对所述待测图片的损伤检测结果包括：
    当所述边框预测数据未指示出损伤预测框，或者所述分割预测数据未指示出损伤对象区域时，将所述损伤检测结果确定为待测图片不包含损伤对象，其中所述损伤对象区域为预测分割类别为同一损伤类别的像素点集合构成的面积大于一定阈值的连通区域。
  10. 根据权利要求8所述的方法，其中，所述边框预测数据指示出至少一个损伤预测框，所述分割预测数据指示出至少一个损伤对象区域，其中所述损伤对象区域为预测分 割类别为同一损伤类别的像素点集合构成的面积大于一定阈值的连通区域；
    所述根据所述边框预测数据和所述分割预测数据，确定针对所述待测图片的损伤检测结果包括，根据所述至少一个损伤预测框和所述至少一个损伤对象区域，确定针对所述待测图片的损伤检测结果。
  11. 根据权利要求10所述的方法，其中，根据所述至少一个损伤预测框和所述至少一个损伤对象区域，确定针对所述待测图片的损伤检测结果包括：
    将所述至少一个损伤预测框对应的区域集合，与所述至少一个损伤对象区域对应的区域集合的并集，作为损伤检测结果。
  12. 根据权利要求10所述的方法，其中，所述至少一个损伤预测框包括第一损伤预测框；
    根据所述至少一个损伤预测框和所述至少一个损伤对象区域，确定针对所述待测图片的损伤检测结果，包括：
    如果所述第一损伤预测框与各个损伤对象区域的交并比均小于预设阈值，则从所述损伤检测结果中移除该第一损伤预测框。
  13. 根据权利要求10所述的方法，其中，所述至少一个损伤预测框包括第一损伤预测框；
    根据所述至少一个损伤预测框和所述至少一个损伤对象区域，确定针对所述待测图片的损伤检测结果，包括：
    如果所述第一损伤预测框与各个损伤对象区域的交叠面积与所述第一损伤预测框的框面积的比例均小于预设阈值，则从所述损伤检测结果中移除该第一损伤预测框。
  14. 根据权利要求10所述的方法，其中，所述至少一个损伤预测框包括第一损伤预测框，所述边框预测数据还包括针对该第一损伤预测框预测的第一损伤类别；
    所述至少一个损伤对象区域包括第一损伤对象区域，所述第一损伤对象区域中的像素点对应于第一分割类别；
    根据所述至少一个损伤预测框和所述至少一个损伤对象区域，确定针对所述待测图片的损伤检测结果，包括：
    如果所述第一损伤预测框与所述第一损伤对象区域的交并比大于预设阈值，但所述第一损伤类别与所述第一分割类别不对应，则将所述第一损伤检测框确定为异常检测框，或将所述第一损伤对象区域确定为异常区域。
  15. 一种训练弱分割损伤检测模型的装置，包括：
    样本获取单元，配置为获取样本图片，所述样本图片对应具有边框标注数据，所述 边框标注数据指示出至少一个损伤标注框，每个损伤标注框是标注人员标注的、框选出所述样本图片中损伤对象的最小矩形框；
    标注生成单元，配置为将各个损伤标注框作为对应损伤对象的轮廓，基于所述轮廓为所述样本图片中各个像素点标注分割类别，从而生成分割标注数据；
    模型输入单元，配置为将所述样本图片输入弱分割损伤检测模型，所述弱分割损伤检测模型包括边框预测分支和分割预测分支，所述边框预测分支输出用于指示损伤预测框的边框预测数据，所述分割预测分支输出针对所述样本图片中各个像素点的分割类别进行预测的分割预测数据；
    第一确定单元，配置为基于所述边框预测数据与所述边框标注数据的比对，确定边框预测损失项，以及，基于所述分割预测数据和分割标注数据的比对，确定分割预测损失项；
    第二确定单元，配置为根据所述边框预测损失项和所述分割预测损失项，确定本次预测的损失函数；
    模型更新单元，配置为在所述损失函数减小的方向，更新所述弱分割损伤检测模型。
  16. 根据权利要求15所述的装置，其中，所述标注生成单元配置为：
    将位于损伤标注框之内的像素点标注为第一分割类别，将位于损伤标注框之外的像素点标注为第二分割类别。
  17. 根据权利要求15所述的装置，其中，所述至少一个损伤标注框包括第一损伤标注框；所述边框标注数据还包括，标注人员从预定的N个损伤类别中，针对该第一损伤标注框选择标注的第一损伤类别；
    所述标注生成单元配置为：对于位于所述第一损伤标注框之内的像素点，将其分割类别标注为所述第一损伤类别对应的类别。
  18. 根据权利要求15所述的装置，其中，所述至少一个损伤标注框包括第一损伤标注框和第二损伤标注框，所述第一损伤标注框和第二损伤标注框存在交叠区域；所述边框标注数据还包括，标注人员从预定的N个损伤类别中，针对该第一损伤标注框选择标注的第一损伤类别和针对第二损伤标注框选择标注的第二损伤类别，其中第二损伤类别的损伤严重程度大于所述第一损伤类别；
    所述标注生成单元配置为：对于位于所述交叠区域之内的像素点，将其分割类别标注为所述第二损伤类别对应的类别。
  19. 根据权利要求15所述的装置，其中，所述损伤检测模型基于卷积神经网络CNN实现，所述卷积神经网络CNN包括基础卷积层，用于对所述样本图片进行卷积处理， 得到对应的卷积特征图；
    所述边框预测分支用于基于所述卷积特征图预测得到所述边框预测数据；
    所述分割预测分支用于基于所述卷积特征图预测得到所述分割预测数据。
  20. 根据权利要求19所述的装置，其中，所述分割预测分支包括：
    上采样层，用于将经卷积处理的特征上采样为与所述样本图片大小相同的第一特征图；
    预测处理层，用于基于所述第一特征图预测各个像素点属于各个分割类别的概率。
  21. 根据权利要求15所述的装置，其中，所述第一确定单元配置为：
    根据所述分割预测数据，确定各个像素点属于各个分割类别的预测概率；
    根据所述各个像素点属于各个分割类别的预测概率以及其标注分割类别，确定所述分割预测损失项。
  22. 一种从图片中检测损伤的装置，包括：
    模型获取单元，配置为获取通过权利要求15的装置训练得到的弱分割损伤检测模型；
    模型输入单元，配置为将待测图片输入所述弱分割损伤检测模型，所述弱分割损伤检测模型包括边框预测分支和分割预测分支，所述边框预测分支输出用于指示损伤预测框的边框预测数据，所述分割预测分支输出针对所述待测图片中各个像素点的分割类别进行预测的分割预测数据；
    结果确定单元，配置为根据所述边框预测数据和所述分割预测数据，确定针对所述待测图片的损伤检测结果。
  23. 根据权利要求22的装置，其中，所述结果确定单元配置为：
    当所述边框预测数据未指示出损伤预测框，或者所述分割预测数据未指示出损伤对象区域时，将所述损伤检测结果确定为待测图片不包含损伤对象，其中所述损伤对象区域为预测分割类别为同一类别的像素点集合构成的面积大于一定阈值的连通区域。
  24. 根据权利要求22所述的装置，其中，所述边框预测数据指示出至少一个损伤预测框，所述分割预测数据指示出至少一个损伤对象区域，其中所述损伤对象区域为预测分割类别为同一类别的像素点集合构成的面积大于一定阈值的连通区域；
    所述结果确定单元配置为，根据所述至少一个损伤预测框和所述至少一个损伤对象区域，确定针对所述待测图片的损伤检测结果。
  25. 根据权利要求24所述的装置，其中，所述结果确定单元配置为：
    将所述至少一个损伤预测框对应的区域集合，与所述至少一个损伤对象区域对应的 区域集合的并集，作为损伤检测结果。
  26. 根据权利要求24所述的装置，其中，所述至少一个损伤预测框包括第一损伤预测框；所述结果确定单元配置为：
    如果所述第一损伤预测框与各个损伤对象区域的交并比均小于预设阈值，则从所述损伤检测结果中移除该第一损伤预测框。
  27. 根据权利要求24所述的装置，其中，所述至少一个损伤预测框包括第一损伤预测框；所述结果确定单元配置为：
    如果所述第一损伤预测框与各个损伤对象区域的交叠面积与所述第一损伤预测框的框面积的比例均小于预设阈值，则从所述损伤检测结果中移除该第一损伤预测框。
  28. 根据权利要求24所述的装置，其中，所述至少一个损伤预测框包括第一损伤预测框，所述边框预测数据还包括针对该第一损伤预测框预测的第一损伤类别；所述至少一个损伤对象区域包括第一损伤对象区域，所述第一损伤对象区域中的像素点对应于第一分割类别；
    所述结果确定单元配置为：
    如果所述第一损伤预测框与所述第一损伤对象区域的交并比大于预设阈值，但所述第一损伤类别与所述第一分割类别不对应，则将所述第一损伤检测框确定为异常检测框，或将所述第一损伤对象区域确定为异常区域。
  29. 一种计算机可读存储介质，其上存储有计算机程序，当所述计算机程序在计算机中执行时，令计算机执行权利要求1-14中任一项的所述的方法。
  30. 一种计算设备，包括存储器和处理器，其特征在于，所述存储器中存储有可执行代码，所述处理器执行所述可执行代码时，实现权利要求1-14中任一项所述的方法。
PCT/CN2020/072058 2019-05-27 2020-01-14 基于弱分割的损伤检测方法及装置 WO2020238256A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US16/808,208 US10783643B1 (en) 2019-05-27 2020-03-03 Segmentation-based damage detection
US16/945,774 US11004204B2 (en) 2019-05-27 2020-07-31 Segmentation-based damage detection

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910447785.X 2019-05-27
CN201910447785.XA CN110264444B (zh) 2019-05-27 2019-05-27 基于弱分割的损伤检测方法及装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/808,208 Continuation US10783643B1 (en) 2019-05-27 2020-03-03 Segmentation-based damage detection

Publications (1)

Publication Number Publication Date
WO2020238256A1 true WO2020238256A1 (zh) 2020-12-03

Family

ID=67915525

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/072058 WO2020238256A1 (zh) 2019-05-27 2020-01-14 基于弱分割的损伤检测方法及装置

Country Status (2)

Country Link
CN (1) CN110264444B (zh)
WO (1) WO2020238256A1 (zh)

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110264444B (zh) * 2019-05-27 2020-07-17 阿里巴巴集团控股有限公司 基于弱分割的损伤检测方法及装置
US10783643B1 (en) 2019-05-27 2020-09-22 Alibaba Group Holding Limited Segmentation-based damage detection
CN110992365B (zh) * 2019-11-04 2023-04-18 杭州电子科技大学 一种基于图像语义分割的损失函数及其设计方法
CN111429448B (zh) * 2020-04-09 2023-07-18 常州奥创医疗科技有限公司 一种基于弱分割信息的生物荧光目标计数方法
CN112052883B (zh) * 2020-08-14 2023-10-17 五邑大学 一种衣物检测方法、装置及存储介质
CN112329596B (zh) * 2020-11-02 2021-08-24 中国平安财产保险股份有限公司 目标物定损方法、装置、电子设备及计算机可读存储介质
CN112446870B (zh) * 2020-12-02 2024-07-09 平安科技（深圳）有限公司 管道损伤检测方法、装置、设备及存储介质
CN112927231B (zh) * 2021-05-12 2021-07-23 深圳市安软科技股份有限公司 车身污垢检测模型的训练方法、车身污垢检测方法及装置
CN114782769A (zh) * 2022-03-10 2022-07-22 阿里巴巴（中国）有限公司 训练样本的生成方法、装置、系统及目标对象的检测方法

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106295678A (zh) * 2016-07-27 2017-01-04 北京旷视科技有限公司 神经网络训练与构建方法和装置以及目标检测方法和装置
US20180061091A1 (en) * 2016-08-31 2018-03-01 International Business Machines Corporation Anatomy segmentation through low-resolution multi-atlas label fusion and corrective learning
CN109145903A (zh) * 2018-08-22 2019-01-04 阿里巴巴集团控股有限公司 一种图像处理方法和装置
CN109584248A (zh) * 2018-11-20 2019-04-05 西安电子科技大学 基于特征融合和稠密连接网络的红外面目标实例分割方法
CN109635694A (zh) * 2018-12-03 2019-04-16 广东工业大学 一种行人检测方法、装置、设备及计算机可读存储介质
CN110264444A (zh) * 2019-05-27 2019-09-20 阿里巴巴集团控股有限公司 基于弱分割的损伤检测方法及装置

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104134234B (zh) * 2014-07-16 2017-07-25 中国科学技术大学 一种全自动的基于单幅图像的三维场景构建方法
KR101879207B1 (ko) * 2016-11-22 2018-07-17 주식회사 루닛 약한 지도 학습 방식의 객체 인식 방법 및 장치
CN106855944B (zh) * 2016-12-22 2020-01-14 浙江宇视科技有限公司 行人标志物识别方法及装置
KR102040309B1 (ko) * 2017-09-18 2019-11-04 한국전자통신연구원 멀티미디어 콘텐츠와 연관되는 후각 정보 인식 장치 및 방법, 라벨 정보 생성 장치 및 방법
CN108109152A (zh) * 2018-01-03 2018-06-01 深圳北航新兴产业技术研究院 医学图像分类和分割方法和装置
CN109002834B (zh) * 2018-06-15 2022-02-11 东南大学 基于多模态表征的细粒度图像分类方法
CN109255790A (zh) * 2018-07-27 2019-01-22 北京工业大学 一种弱监督语义分割的自动图像标注方法
CN109242006A (zh) * 2018-08-23 2019-01-18 阿里巴巴集团控股有限公司 基于车型分类的识别车辆损伤的方法及装置
CN109325488A (zh) * 2018-08-31 2019-02-12 阿里巴巴集团控股有限公司 用于辅助车辆定损图像拍摄的方法、装置及设备
CN109615649A (zh) * 2018-10-31 2019-04-12 阿里巴巴集团控股有限公司 一种图像标注方法、装置及系统

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106295678A (zh) * 2016-07-27 2017-01-04 北京旷视科技有限公司 神经网络训练与构建方法和装置以及目标检测方法和装置
US20180061091A1 (en) * 2016-08-31 2018-03-01 International Business Machines Corporation Anatomy segmentation through low-resolution multi-atlas label fusion and corrective learning
CN109145903A (zh) * 2018-08-22 2019-01-04 阿里巴巴集团控股有限公司 一种图像处理方法和装置
CN109584248A (zh) * 2018-11-20 2019-04-05 西安电子科技大学 基于特征融合和稠密连接网络的红外面目标实例分割方法
CN109635694A (zh) * 2018-12-03 2019-04-16 广东工业大学 一种行人检测方法、装置、设备及计算机可读存储介质
CN110264444A (zh) * 2019-05-27 2019-09-20 阿里巴巴集团控股有限公司 基于弱分割的损伤检测方法及装置

Also Published As

Publication number Publication date
CN110264444B (zh) 2020-07-17
CN110264444A (zh) 2019-09-20

Similar Documents

Publication Publication Date Title
WO2020238256A1 (zh) 基于弱分割的损伤检测方法及装置
TWI726364B (zh) 電腦執行的車輛定損方法及裝置
TWI742382B (zh) 透過電腦執行的、用於車輛零件識別的神經網路系統、透過神經網路系統進行車輛零件識別的方法、進行車輛零件識別的裝置和計算設備
CN110569837B (zh) 优化损伤检测结果的方法及装置
US11080839B2 (en) System and method for training a damage identification model
US10853699B2 (en) Method and apparatus for vehicle damage identification
TWI716012B (zh) 樣本標註方法、裝置、儲存媒體和計算設備、損傷類別的識別方法及裝置
US11004204B2 (en) Segmentation-based damage detection
WO2020228370A1 (zh) 计算机执行的从图片中识别损伤的方法及装置
CN114663346A (zh) 一种基于改进YOLOv5网络的带钢表面缺陷检测方法
JP7391504B2 (ja) 情報処理装置、情報処理方法及びプログラム
CN112613569B (zh) 图像识别方法、图像分类模型的训练方法及装置
WO2024002187A1 (zh) 缺陷检测方法、缺陷检测设备及存储介质
CN113469950A (zh) 一种基于深度学习的复合绝缘子异常发热缺陷的诊断方法
CN116682109A (zh) 一种病理显微图像的分析方法、装置、设备及存储介质
CN115019174B (zh) 基于像素重组和注意力的上采样遥感图像目标识别方法
WO2024147006A1 (en) Vehicle inspection system
WO2023112302A1 (ja) 教師データ作成支援装置、教師データ作成支援方法
CN116189230A (zh) 一种人员到位检测方法、装置、设备和存储介质
CN117036961A (zh) 农作物病虫害的智能监测方法及其系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20812680

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20812680

Country of ref document: EP

Kind code of ref document: A1