WO2023040142A1

WO2023040142A1 - Vehicle damage detection method and apparatus, and device and storage medium

Info

Publication number: WO2023040142A1
Application number: PCT/CN2022/071075
Authority: WO
Inventors: 方起明; 刘莉红; 刘玉宇
Original assignee: 平安科技（深圳）有限公司
Priority date: 2021-09-15
Filing date: 2022-01-10
Publication date: 2023-03-23
Also published as: CN113780435B; CN113780435A

Abstract

The embodiments of the present application relate to the technical field of artificial intelligence image processing. Provided are a vehicle damage detection method and apparatus, and a device and a storage medium. The method comprises: acquiring an image of a damaged vehicle; inputting the image of the damaged vehicle into a trained integrated loss assessment model to obtain information of a damaged component; obtaining a damage position corresponding to the image of the damaged vehicle according to a vehicle component segmentation model; and obtaining damage information of the vehicle according to the damage position and the information of the damaged component. Compared with a plurality of damage detection models separated in the related art, an integrated loss assessment model is used in the embodiments of the present application to obtain, by means of one-time forward processing, corresponding information of a damaged component, so that the amount of calculation resources occupied thereby is greatly reduced, the consumed calculation time is greatly reduced, the detection efficiency is effectively improved, and the labor cost during a vehicle loss assessment process is reduced. In addition, the damage position corresponding to the image of the damaged vehicle is obtained by means of combining the vehicle component segmentation models, and the damage information of the vehicle including positioning information can be obtained.

Description

Vehicle damage detection method, device, equipment and storage medium

This application claims the priority of the Chinese patent application with the application number 2021110801171 and the invention title "vehicle damage detection method, device, equipment and storage medium" submitted to the China Patent Office on September 15, 2021, the entire contents of which are incorporated by reference in this application.

technical field

The embodiments of the present application relate to the field of artificial intelligence, and in particular to a vehicle damage detection method, device, equipment, and storage medium.

Background technique

It is a very important work to determine the damaged location and the degree of damage for a damaged vehicle in a traffic accident. The traditional loss assessment method relies on manual judgment, which is inefficient and prone to errors due to personal factors of the assessor. In recent years, with the development of artificial intelligence technology, artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Theories, methods, techniques and application systems for good results. Some institutions have adopted methods based on computer vision (target detection, etc.) to carry out intelligent damage assessment for damaged vehicles. These methods have reduced the dependence of manual damage assessment to a certain extent.

However, due to complex and changeable damage scenarios, many vehicle components, and many damage forms, it is often necessary to develop more damage detection models, such as developing multiple damage detection models for sheet metal parts, glass, tires, etc. different damage detection models. The inventors realized that the above scheme would have obvious disadvantages. First, in the model training and development stage, it is necessary to prepare and label multiple damage data sets, and train and optimize multiple damage detection models separately. The process is cumbersome, the efficiency is relatively low, and more At the same time, because the model obtained by training can only be used for the corresponding component type, its data reuse rate is small and its robustness is poor; second, in the stage of model deployment and online, multiple models need to consume more computing resources. At the same time, it increases the difficulty of scheduling and increases the cost and expenditure.

Contents of the invention

The following is an overview of the topics described in detail in this article. This summary is not intended to limit the scope of the claims.

Embodiments of the present application provide a vehicle damage detection method, device, equipment, and storage medium, which can effectively improve detection efficiency and detection accuracy, and reduce labor costs in the process of vehicle damage determination.

In the first aspect, the embodiment of the present application provides a vehicle damage detection method, including:

Get images of damaged vehicles;

Inputting the image of the damaged vehicle into the trained integrated damage model to obtain damaged component information, the integrated damage model includes:

A shared backbone neural network, where the shared backbone neural network is obtained by integrating damage detection models corresponding to at least one component type, and the shared backbone neural network is used to preprocess the damaged vehicle image to obtain first output data;

A damage detection classification layer corresponding to at least one component type, configured to process the first output data to obtain damaged component information;

inputting the damaged vehicle image into a vehicle parts segmentation model to obtain a damage location corresponding to the damaged vehicle image;

The damage information of the vehicle is obtained by locating from the damaged component information according to the damage location.

In an optional implementation, the shared backbone neural network is a deep residual neural network, and the deep residual neural network includes: Res-Net50 network, Res-Net101 network, Res-Net110 network or Res-Net152 network ;

Said inputting the image of the damaged vehicle into the trained integrated damage model to obtain the information of the damaged parts, including:

Based on the sequentially connected residual blocks in the deep residual neural network, the damaged vehicle image is subjected to residual feature vector extraction processing to obtain the first output data; wherein, any residual block includes a An identity map and at least two convolutional layers, the identity map of any one residual block is directed from the input end of any one residual block to the output end of any one residual block;

The first output data is input to the damage detection classification layer corresponding to the component type to obtain damaged component information.

In the second aspect, the embodiment of the present application provides a vehicle damage detection device, including:

An image acquisition module is used to acquire damaged vehicle images;

The damaged part information determination module is used to input the damaged vehicle image into the trained integrated damage model to obtain the damaged part information, and the integrated damage model includes:

An image segmentation module, configured to input the damaged vehicle image into the vehicle component segmentation model to obtain the damage location corresponding to the damaged vehicle image;

A damage information synthesis module, configured to obtain damage information of the vehicle from the damaged component information according to the damage location.

In a third aspect, a computer device includes a processor and a memory;

The memory is used to store programs;

The processor is configured to execute the vehicle damage detection method according to any one of the first aspect according to the program:

Wherein, the vehicle damage detection method includes:

Get images of damaged vehicles;

In the fourth aspect, the embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to execute the vehicle damage detection method described in any one of the first aspects:

Wherein, the vehicle damage detection method includes:

Get images of damaged vehicles;

The first aspect of the embodiment of the present application provides a vehicle damage detection method. Compared with related technologies, this solution obtains damaged vehicle images and inputs the damaged vehicle images into the trained integrated damage model to obtain damaged parts information , and obtain the damage position corresponding to the damaged vehicle image according to the vehicle part segmentation model, and obtain the damage information of the vehicle according to the damage position and the damaged part information. The integrated damage assessment model includes: a shared backbone neural network and a damage detection classification layer corresponding to at least one component type. The shared backbone neural network is obtained by integrating the damage detection model corresponding to at least one component type. The integration method is convenient for model optimization training, and Labeled data of all damage types can be used during training, effectively improving the robustness of the model and reducing the possibility of overfitting. Compared with the multiple damage detection models separated in the related art, the embodiment of the present application uses the integrated damage determination model to obtain the corresponding damaged component information through one forward processing, which greatly reduces the computing resources and time consumed , so as to effectively improve the detection efficiency and reduce the labor cost in the process of vehicle damage assessment. Moreover, in the embodiment of the present application, the damage information of the vehicle including positioning information can be obtained after merging the damage positions corresponding to the damaged vehicle images obtained from the vehicle part segmentation model; on the other hand, since the damage information of the vehicle includes the damage positions , the embodiment of the present application can filter the false damage detection in the background area, and further improve the detection accuracy of the vehicle damage detection.

It can be understood that the beneficial effects of the above-mentioned second aspect to the fourth aspect compared with the related technology are the same as those of the above-mentioned first aspect compared with the related technology. Please refer to the relevant description in the above-mentioned first aspect. This will not be repeated here.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings that need to be used in the embodiments or related technical descriptions. Obviously, the drawings in the following description are only the embodiments of the present application For some embodiments of the present invention, those skilled in the art can also obtain other drawings based on these drawings without paying creative efforts.

FIG. 1 is a schematic diagram of an exemplary system architecture provided by an embodiment of the present application;

Fig. 2 is a flowchart of a vehicle damage detection method provided by an embodiment of the present application;

Fig. 3 is a structural block diagram of a vehicle detection model in the related art;

Fig. 4 is a structural block diagram of an integrated damage assessment model provided by an embodiment of the present application;

FIG. 5 is a flowchart of a vehicle damage detection method provided by an embodiment of the present application;

FIG. 6 is a flow chart of training an integrated damage assessment model in a vehicle damage detection method provided by an embodiment of the present application;

Fig. 7 is another flowchart of a vehicle damage detection method provided by an embodiment of the present application;

Fig. 8 is a structural block diagram of a vehicle damage detection device provided by an embodiment of the present application.

Detailed ways

In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the embodiments of the present application with unnecessary detail.

It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than in the flowchart. The terms "first", "second" and the like in the specification and claims and the above drawings are used to distinguish similar objects, and not necessarily used to describe a specific sequence or sequence.

It should also be understood that references to "one embodiment" or "some embodiments" described in the description of the embodiments of the present application mean that specific features described in conjunction with the embodiments of the present application are included in one or more embodiments of the embodiments of the present application. , structure or characteristics. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in other embodiments," etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean "one or more but not all embodiments" unless specifically stated otherwise. The terms "including", "comprising", "having" and variations thereof mean "including but not limited to", unless specifically stated otherwise.

Cars are frequently used tools in land transportation. Affected by factors such as weather, road conditions and driver skills, car damage is inevitable, especially in traffic accidents. Therefore, it is very important to determine the damaged location and degree of damage for the damaged vehicle, because it will not only affect the determination of the subsequent vehicle maintenance plan, but also affect the determination of the amount of economic compensation for the parties involved in the accident. The traditional loss assessment method relies on manual judgment, which is inefficient and prone to errors due to personal factors of the assessor. In recent years, with the development of artificial intelligence technology, some institutions have adopted methods based on computer vision (target detection, etc.) to carry out intelligent damage assessment for damaged vehicles. These methods have reduced the dependence of manual damage assessment to a certain extent.

Generally speaking, in the design and development process of this kind of vehicle intelligent damage assessment system, due to the complex damage assessment scenarios, many vehicle components, and many damage forms, it is often necessary to develop more damage detection models, such as Damage detection models for sheet metal parts, glass, tires, etc. are developed according to the material of the parts. The shortcomings of this solution are also obvious. First, in the model training and development stage, multiple damage data sets need to be marked and prepared, and multiple damage detection models should be trained and optimized separately. This process of separately training different damage detection models is cumbersome and inefficient. It is relatively low and requires more manpower and material resources. At the same time, the trained model has poor robustness due to the low data reuse rate. The second is the stage of model deployment and launch. Multiple models need to consume more computing resources, and at the same time increase the difficulty of scheduling and increase costs.

The embodiment of the present application is to improve the above-mentioned defects, and provides a vehicle damage detection method, which relates to the field of artificial intelligence image recognition technology, specifically by acquiring damaged vehicle images, and inputting the damaged vehicle images into the trained integrated damage assessment model The damaged part information is obtained in the vehicle part segmentation model, and the damage position corresponding to the damaged vehicle image is obtained according to the vehicle part segmentation model, and the damage information of the vehicle is obtained according to the damage position and the damaged part information. The integrated damage assessment model includes: a shared backbone neural network and a damage detection classification layer corresponding to at least one component type. The shared backbone neural network is obtained by integrating the damage detection model corresponding to at least one component type. The integration method is convenient for model optimization training, and Labeled data of all damage types can be used during training, effectively improving the robustness of the model and reducing the possibility of overfitting. Compared with the multiple damage detection models separated in related technologies, using the integrated damage determination model can obtain all the damaged component information through one forward reasoning, which greatly reduces the computing resources and time consumed, and effectively improves the detection efficiency. Efficiency, reducing labor costs in the process of vehicle damage assessment. And after merging the damage position corresponding to the damaged vehicle image obtained by the vehicle part segmentation model, the damage information of the vehicle including the positioning information can be obtained; on the other hand, since the damage position of the component is included, the false detection of damage in the background area can be filtered , to further improve the detection accuracy of vehicle damage detection.

The embodiments of the present application will be further described below in conjunction with the accompanying drawings.

Fig. 1 shows a schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of the present application can be applied.

As shown in FIG. 1, the system architecture 100 may include terminal equipment (one or more of a desktop computer 101, a tablet computer 102, and a portable computer 103 as shown in FIG. 1, and of course other terminals with a display screen equipment, etc.), network 104 and server 105. The network 104 is used as a medium for providing a communication link between the terminal device and the server 105 . Network 104 may include various connection types, such as wired communication links, wireless communication links, and so on.

It should be understood that the numbers of terminal devices, networks and servers in Fig. 1 are only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.

Such as server 105 can be an independent server, also can provide cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content distribution network (Content Delivery) Network, CDN), and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms, or server clusters composed of multiple servers.

In one embodiment of the present application, the user can use the terminal device 101 (or the terminal device 102 or 103) to upload a damaged vehicle image to the server 105. The damaged vehicle image can include different parts, different angles, different Images of damaged vehicles collected from a distance, or a single photo. After the server 105 acquires these damaged vehicle images, it inputs the damaged vehicle images into the trained integrated damage model to obtain the damaged parts information, and obtains the damage location corresponding to the damaged vehicle images according to the vehicle part segmentation model, and then according to The damaged location and damaged parts information obtains the damaged information of the vehicle. It can effectively improve the detection efficiency and reduce the labor cost in the process of vehicle damage assessment. And the damage information of the vehicle including the positioning information can be obtained. At the same time, since the damage position of the component is included, the false detection of the damage in the background area can be filtered, and the detection accuracy of the vehicle damage detection can be further improved.

It should be noted that the vehicle damage detection method provided in the embodiment of the present application is generally executed by the server 105 , and correspondingly, the vehicle damage detection device is generally set in the server 105 . However, in other embodiments of the present application, the terminal device may also have a similar function to the server, so as to execute the waiting-for-diagnosis image processing solution provided in the embodiment of the present application.

The system architecture and application scenarios described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute limitations on the technical solutions provided by the embodiments of the present application. Those skilled in the art know that with the system architecture The evolution of the technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems. Those skilled in the art can understand that the system architecture shown in Figure 1 does not constitute a limitation to the embodiment of the present application, and may include more or less components than those shown in the illustration, or combine some components, or different components layout.

The embodiments of the present application may acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. .

Based on the above system architecture, various embodiments of the vehicle damage detection method according to the embodiments of the present application are proposed.

As shown in FIG. 2 , FIG. 2 is a flowchart of a vehicle damage detection method provided by an embodiment of the present application, including but not limited to step S110 and step S140 .

Step S110, acquiring an image of the damaged vehicle.

Step S120, input the image of the damaged vehicle into the trained integrated damage model to obtain the information of the damaged parts.

In step S130, the damaged vehicle image is input into the vehicle component segmentation model to obtain the damage location corresponding to the damaged vehicle image.

Step S140, according to the damage location, locate and obtain the damage information of the vehicle from the damaged part information.

It can be understood that there is no sequence between the above step S120 and step S130, they can be executed at the same time, or step S120 can be executed first and then step S130 can be executed, or step S130 can be executed first and then step S120 can be executed. No restrictions.

In one embodiment, the damaged part information in step S120 includes the damaged part name, damage state and damage degree. The name of damaged parts includes but not limited to: sheet metal parts, glass, tires, etc.; damage information includes but not limited to: scratches, scratches, dents, folds, tears, missing, cracks, etc.; the degree of damage includes but not limited to: minor moderate damage, moderate damage, severe damage, etc.

In one embodiment, when a traffic accident occurs, the user can use his mobile phone to take images of the accident scene, for example, take pictures of damaged vehicles at different positions and angles to obtain images of the damaged vehicles, and then send the images of the damaged vehicles to the background server for further processing. The damage.

The pre-trained integrated damage model is stored in the background server, and the type of damaged parts in the damaged vehicle image is obtained according to the damaged vehicle image uploaded by the user. The damaged part information includes the name of the damaged part, the damage state and the degree of damage, such as the The name of the damaged part of the vehicle is sheet metal, the damage information is scratches, and the damage degree is moderate damage.

And the damage position corresponding to the damaged vehicle in the damaged vehicle image is obtained through the vehicle part segmentation model, for example: the left front door sheet metal part.

Then, the damage information of the damaged vehicle is obtained according to the damage location and the damage component information, that is, the damage information includes: damage component information and damage location. For example, in the above example, the damage information of the damaged vehicle is: the sheet metal part of the left front door is scratched with a moderate degree of damage.

In addition, in an embodiment, the integrated loss assessment model includes:

The shared backbone neural network is obtained by integrating the damage detection model corresponding to at least one component type, and the shared backbone neural network is used to preprocess the damaged vehicle image to obtain the first output data; the damage corresponding to at least one component type The detection and classification layer is configured to process the first output data to obtain damaged component information.

In this embodiment, the integrated damage assessment model is composed of an integrated shared backbone neural network and multiple damage detection classification layers (corresponding to different component types), and each damage detection classification layer corresponds to processing a damage data set of a component type. The damage detection model corresponding to at least one component type is integrated to obtain a shared backbone neural network, and then the damage detection classification layer corresponding to at least one component type is connected to output the damaged component information corresponding to the component type.

It can be said that the damage detection of each different component shares the previous shared backbone neural network except for the damage detection and classification layer of the last layer. layer to predict its damage category. That is, multiple damage detection models are integrated into one model by sharing the backbone neural network. Different from the existing multiple components corresponding to different damage detection models, the shared backbone neural network does not distinguish between component types, and uses a network structure for different Multiple damage detection classification layers correspond to multiple damage data sets to output damage results for all components.

The difference between the structural block diagram of the vehicle detection model in the related art and the structural block diagram of the integrated damage assessment model in this embodiment will be described below.

Referring to FIG. 3 , it is a structural block diagram of a vehicle detection model in the related art. Since there are many vehicle components, this embodiment selects sheet metal parts, glass, and tires for illustration. It is understandable that it does not mean that it is limited to these in three categories.

The structural block diagram of the vehicle detection model includes three functional units, namely:

The first unit: damage data set, including: sheet metal damage detection data set, glass damage detection data set and tire damage detection data set. These damage data sets are all labeled for the damage of a specific component, and do not contain damage label information of other components. The training samples of the damage data set can include: damage images, damage judgment labels, etc.

The second unit: single data set detection model, including: sheet metal damage detection model, glass damage detection model and tire damage detection model, damage detection model includes: damage detection subject and damage detection classifier. Different detection models correspond to different data sets.

The third unit: loss function, including: sheet metal damage loss function, glass damage loss function and tire damage loss function, the loss function corresponds to the damage detection model one by one, and is used to update the parameters of the damage detection model.

During training, input the damage data set of the first unit into the single data set detection model of the second unit to obtain the corresponding damage detection results, and then use the loss function of the third unit to update the parameters of the damage detection model. For example, the sheet metal damage detection data set is input into the sheet metal damage detection model to obtain the sheet metal damage detection result, and then the sheet metal damage loss function is used to update the parameters of the sheet metal damage detection model.

Referring to FIG. 3 , generally speaking, each detection model of the vehicle detection model in the related art works independently without interfering with each other. Therefore, there are the following problems: First, in the model training and development stage, it is necessary to prepare and label multiple damage data sets, and train and optimize multiple damage detection models separately. This process of separately training different damage detection models is cumbersome and inefficient. More manpower and material resources are invested, and at the same time, the trained model has poor robustness due to the low data reuse rate. The second is the stage of model deployment and launch. Multiple models need to consume more computing resources, and at the same time increase the difficulty of scheduling and increase costs. The structural block diagram of the integrated damage assessment model provided by an embodiment of the present application is illustrated below by using FIG. 4 .

Referring to FIG. 4 , it is a structural block diagram of an integrated damage assessment model provided by an embodiment of the present application, including the following functional units.

Due to the large number of vehicle components, three types of sheet metal parts, glass, and tires are selected for illustration in this embodiment. It is understood that it does not mean that they are limited to these three types.

The structural block diagram of the vehicle detection model includes four units, namely:

The first unit: damage data set, including: sheet metal damage detection data set, glass damage detection data set and tire damage detection data set, etc. These damage data sets are all labeled for the damage of a specific component, and do not contain damage label information of other components. The training samples of the damage data set can include: damage images, damage judgment labels, etc., for example:

Sample 1: The damage image is a glass image, and the damage judgment label is: [glass, moderate damage, broken];

Sample 2: The damage image is an image of a sheet metal part, and the damage judgment label is: [sheet metal part, mild damage, scratch].

The second unit: shared backbone neural network, wherein the shared backbone neural network is obtained by integrating damage detection models corresponding to at least one component type, for example, by integrating sheet metal damage detection models, glass damage detection models and tire damage detection models.

The third unit: Different component types correspond to different damage detection classification layers, such as sheet metal damage detection classification layer, glass damage detection classification layer and tire damage detection classification layer. The second unit and the third unit constitute the integrated damage assessment model of this embodiment.

The fourth unit: loss function, including: sheet metal damage loss function, glass damage loss function and tire damage loss function, etc. The loss function corresponds to the damage detection model one by one, and is used to update the parameters of the damage detection model. Since the damage categories of different damage datasets are all for specific parts, there is no overlapping of damage categories between different damage datasets, and there is no need to combine the damage functions into one for training.

During training, after the damage data set of the first unit is input into the shared backbone neural network of the second unit to obtain an output, the corresponding damage detection and classification layer is selected according to the damaged parts of the output. Since the damage data sets are all labeled for the damage of a specific component and do not contain damage label information of other components, it is not possible to simply merge the damage categories and train a detection model to detect the damage of all components. Therefore, without merging multiple In the case of the target category space of the impairment dataset, an integrated shared backbone neural network is trained, that is, multiple impairment detection models for a specific impairment dataset are trained in parallel in the shared backbone network.

For example, the sheet metal damage detection data set is input into the shared backbone neural network and then output to the sheet metal damage detection classification layer, and then the sheet metal damage loss function is used to update the parameters of the sheet metal damage detection model.

In addition, in one embodiment, the shared backbone neural network is a deep residual neural network (Deep Residual Network).

For traditional deep learning network applications, the deeper the network, the more layers, the more things can be learned, and of course the slower the convergence speed, the longer the training time. However, when the depth reaches a certain level, the deeper the learning rate, the lower the learning rate. Even in some scenarios, the deeper the network layer, the lower the accuracy rate, and the phenomenon of gradient disappearance and gradient explosion is easy to occur. This phenomenon is not caused by overfitting. Overfitting is the situation where the model is trained too well in the training set, but the performance in the new data is not satisfactory. Generally speaking, the training error and test error of the deep learning network become larger as the number of layers increases, which shows that when the number of network layers becomes deeper, the deep network becomes difficult to train.

The deep residual neural network introduces the design of the residual block, which overcomes the above-mentioned problems that the learning rate becomes lower and the accuracy rate cannot be effectively improved due to the deepening of the network depth. The principle of the residual block is to directly skip the data output of the previous layers and introduce it into the input part of the subsequent data layer. To put it simply, similar to skip connections, the data that is "clear" in the front and the data that is "lossy compressed" in the back are used as the input of the network data in the back, so that the network can learn richer content, so this implementation The shared backbone neural network of the example uses a deep residual neural network.

In one embodiment, correspondingly, referring to FIG. 5, step S120 includes but is not limited to the following steps:

Step S121 , based on each residual block sequentially connected in the deep residual neural network, perform residual feature vector extraction processing on the damaged vehicle image to obtain first output data.

In an embodiment, any residual block includes an identity map and at least two convolutional layers, and the identity map of any residual block is directed to any residual block from the input end of any residual block The output of , that is, the identity map is the skip connection mentioned above.

Step S122, inputting the first output data to the damage detection classification layer corresponding to the component type to obtain damaged component information.

In an embodiment, the shared backbone neural network includes one of the following: Res-Net50 network, Res-Net101 network, Res-Net110 network or Res-Net152 network. Since the network structure is described in detail in the related art, details are not repeated here.

The training process of the integrated loss assessment model of this embodiment is described below.

In an embodiment, referring to FIG. 6 , it is a flow chart of training an integrated damage assessment model in a vehicle damage detection method provided in an embodiment of the present application, including but not limited to step S610 and step S640 .

In step S610, a damage data set corresponding to at least one component type is obtained as a training data set, and the training data set includes a corresponding damage judgment label.

Step S620, input the training data set into the shared backbone neural network to obtain feature data, which is obtained by extracting residual feature vectors, which is similar to the above-mentioned first output data.

In step S630, the feature data is input into the damage detection classification layer corresponding to the component type, and a detection result of damaged component information is obtained.

Step S640, according to the detection error between the detection result of the damaged component information and the damage judgment label, train to obtain an integrated damage assessment model.

In one embodiment, the damage detection model corresponding to each component type includes its corresponding loss function. Step S640 is specifically described as: adjust the parameters in the integrated damage model according to the detection error until the loss function satisfies the convergence condition, and obtain Integrate the damage model, that is, update the parameters of the damage detection model according to the loss function. In this embodiment, the convergence condition may be: minimizing loss functions, that is, optimizing parameters of each damage detection model by minimizing each loss function.

In this embodiment, all types of damage labeling data are used when training the integrated damage assessment model, which significantly improves the robustness of the integrated damage assessment model and reduces the possibility of overfitting. At the same time, compared with multiple damage detection models, an integrated model can obtain all damage detection results after one forward reasoning, which greatly reduces the computing resources and time consumed.

In addition, in one embodiment, for the problem of unbalanced damage samples in multiple damage data sets, when training the integrated damage assessment model, a uniform sampling strategy and/or an inter-category balanced sampling strategy is used to sample the training data sets.

1) Uniform sampling strategy: the uniform sampling strategy is used to uniformly sample the damage data sets corresponding to each component type to obtain the training data set, so that the number of samples between the loss data sets corresponding to different component types is balanced, and the integrated damage assessment is improved. The overall performance of the model on each damage dataset.

2) Balanced sampling strategy between categories: When the number of samples between the damage data sets corresponding to different component types differs greatly, the balanced sampling strategy between categories is used to oversample the loss data sets corresponding to the component types with a small number of samples to obtain training data set. For example, the number of samples in a damage data set of a certain component category is significantly less than that of other damage data sets, and the amount of information that this category can provide is also less. Use this strategy for balanced sampling, and try to ensure that the amount of training data between categories is consistent.

This strategy aims to alleviate the problem of sample imbalance between different categories in the damage data set and improve the performance on the corresponding damage data set. The main purpose is to ensure as much as possible that in each batch of training samples, the probability of occurrence of each class (samples of different component types) is the same, and to avoid the constant input order of pictures.

In this embodiment, two lists are used to iteratively sample to obtain samples of each batch. In each iteration, first sample a category X (such as glass) in the component category list, and then sample a picture in the image list of this category X (such as the damage data set of glass), when the image list of category X traverses After that, reshuffle the order of the image list and start traversing from the beginning. For example, in this way, "oversampling" can be used to increase the number of samples of the damage data set with a small number of samples, so as to ensure that the number of samples of the damage data set of the component category in the batch is balanced. The part category list is also processed to ensure that the types of part categories in each batch are also balanced. Through the above balanced sampling strategy between categories to solve the problem of unbalanced category distribution in training samples.

In addition, in one embodiment, the vehicle part segmentation model is pre-trained by a neural network model. In the forward reasoning stage, an image is input into the integrated damage model, and the model will simultaneously output the damaged part information for multiple damage data sets. The next step is to determine the specific location of the damaged part information. This embodiment uses the following The two vehicle component segmentation models described above are used to assist localization.

The specific details of these two vehicle part segmentation models are presented below.

1) The vehicle part segmentation model includes a part segmentation network and a global feature extraction network, collects the global feature map of the input damaged vehicle image, and uses the global feature extraction network to obtain multiple local features corresponding to the global feature map, so that it can be used according to the local features The part segmentation network obtains the information of vehicle segmentation parts, and updates the parameters of the model according to the loss function, so that the vehicle part segmentation model can output the damage location corresponding to the damaged vehicle image.

2) Construct the corresponding vehicle part segmentation model according to the idea of semantic segmentation. The training process includes: labeling the categories in the damaged vehicle image samples and then generating a training set. Each category contains corresponding weights, and then the corresponding loss function is used to treat The trained semantic segmentation model is trained, and after the loss value is obtained, the model parameters of the semantic segmentation model to be trained are adjusted according to the loss value, and the corresponding vehicle part segmentation model is constructed according to the semantic segmentation idea.

The above describes the specific structure of the integrated damage model and the vehicle parts segmentation model. Referring to FIG. 7, it is a flow chart of a vehicle damage detection method provided by an embodiment of the present application. It can be seen from the figure that the following steps are included:

Step S710, acquiring the image of the damaged vehicle.

In step S720, the damaged vehicle image is simultaneously input into the integrated damage assessment model and the vehicle part segmentation model, wherein the integrated damage assessment model includes a shared backbone neural network and multiple damage detection and classification layers.

In step S730, the damaged component information output by the integrated damage assessment model and the damaged location output by the vehicle component segmentation model are acquired respectively.

In step S740, the damage information of the damaged vehicle is obtained by locating from the damaged component information according to the damage position, and only the matching damage information under the target component is kept. That is, combined with the vehicle component segmentation model, the detection results of the integrated damage determination model are used for component positioning, so as to obtain the damaged component information and its damaged position, and filter the false detection of damage in the background area.

For example, according to the input damaged vehicle image, the damaged part information output by the integrated damage model includes: for example, the name of the damaged part of the vehicle is sheet metal and glass, the damage information is scratched and broken, and the damage degree is moderate damage .

The damage position corresponding to the damaged vehicle in the damaged vehicle image is obtained through the vehicle part segmentation model, for example: the right side of the front windshield.

Then, the damage location and the damaged parts information are combined to obtain the damage information of the damaged vehicle as follows: moderate damage occurs on the right side of the front windshield, that is, the damage detection results of all the glass under the glass segmentation area are retained, and the damage detection results are removed. Possible damage detection results for tires or sheet metal parts.

The embodiment of the present application provides a vehicle damage detection method, by acquiring damaged vehicle images, inputting the damaged vehicle images into the trained integrated damage model to obtain damaged parts information, and obtaining damaged vehicles according to the vehicle parts segmentation model The damage location corresponding to the image, and the damage information of the vehicle is obtained according to the damage location and damaged component information. The integrated damage assessment model includes: a shared backbone neural network and a damage detection classification layer corresponding to at least one component type. The shared backbone neural network is obtained by integrating the damage detection model corresponding to at least one component type. The integration method is convenient for model optimization training, and Labeled data of all damage types can be used during training, effectively improving the robustness of the model and reducing the possibility of overfitting. Compared with the multiple damage detection models separated in related technologies, using the integrated damage determination model can obtain all the damaged component information through one forward reasoning, which greatly reduces the computing resources and time consumed, and effectively improves the detection efficiency. Efficiency, reducing labor costs in the process of vehicle damage assessment. And after merging the damage position corresponding to the damaged vehicle image obtained by the vehicle part segmentation model, the damage information of the vehicle including the positioning information can be obtained; on the other hand, since the damage position of the component is included, the false detection of damage in the background area can be filtered , to further improve the detection accuracy of vehicle damage detection.

In addition, an embodiment of the embodiment of the present application also provides a vehicle damage detection device, referring to Figure 8, the device includes:

An image acquisition module 810, configured to acquire an image of the damaged vehicle;

The damaged part information determination module 820 is used to input the image of the damaged vehicle into the trained integrated damage model to obtain the damaged part information. The integrated damage model includes:

The shared backbone neural network is obtained by integrating the damage detection model corresponding to at least one component type, and the shared backbone neural network is used to preprocess the damaged vehicle image to obtain the first output data;

The image segmentation module 830 is used to input the damaged vehicle image into the vehicle part segmentation model to obtain the damage position corresponding to the damaged vehicle image;

The damage information synthesis module 840 is configured to locate and obtain the damage information of the vehicle from the damaged component information according to the damage location.

The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

It should be noted that the vehicle damage detection device in this embodiment can implement the vehicle damage detection method in the embodiment shown in Figure 2 . That is, the vehicle damage detection device in this embodiment and the vehicle damage detection method in the embodiment shown in FIG. detail.

In addition, an embodiment of the embodiment of the present application further provides computer equipment, and the computer equipment includes: a memory, a processor, and a computer program stored in the memory and operable on the processor.

The processor and memory can be connected by a bus or other means.

As a non-transitory computer-readable storage medium, memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory optionally includes memory located remotely from the processor, and these remote memories may be connected to the processor via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The non-transient software programs and instructions required to realize the vehicle damage detection method of the above embodiment are stored in the memory, and when executed by the processor, the vehicle damage detection method in the above embodiment is executed, wherein the vehicle damage detection method includes: Obtaining an image of a damaged vehicle; inputting the image of the damaged vehicle into a trained integrated damage model to obtain information on damaged parts, the integrated damage model includes: a shared backbone neural network, the shared backbone neural network consists of at least A damage detection model corresponding to a component type is integrated, and the shared backbone neural network is used to preprocess the damaged vehicle image to obtain first output data; at least one damage detection classification layer corresponding to a component type is used to The first output data is processed to obtain damaged component information; the damaged vehicle image is input into the vehicle component segmentation model to obtain the damaged position corresponding to the damaged vehicle image; The damage information of the vehicle is obtained by locating in the information. For example, method steps S110 and S140 in FIG. 2 , method steps S510 to 530 in FIG. 5 , method steps S610 to S640 in FIG. 6 , etc. described above are performed.

In addition, an embodiment of the embodiment of the present application also provides a computer-readable storage medium, the storage medium is a volatile storage medium or a non-volatile storage medium, and the computer-readable storage medium stores computer-executable Instructions, the computer-executable instructions are executed by a processor or a controller, for example, executed by a processor in the above-mentioned computer device embodiment, so that the above-mentioned processor can execute the native capability expansion method based on the API interface in the above-mentioned embodiment For example, the above-described method steps S110 and S140 in FIG. 2 , method steps S510 to 530 in FIG. 5 , method steps S610 to S640 in FIG. 6 , etc., are executed.

As another example, being executed by a processor in the above-mentioned computer device embodiment, the above-mentioned processor can execute the vehicle damage detection method in the above-mentioned embodiment, for example, execute the method step S110 and step S140 in FIG. 2 described above, and FIG. The method steps S510 to 530 in 5, the method steps S610 to S640 in FIG. 6 and so on.

Those skilled in the art can understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware and an appropriate combination thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit . Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those of ordinary skill in the art, the term computer storage media includes both volatile and nonvolatile media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. permanent, removable and non-removable media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridges, tape, magnetic disk storage or other magnetic storage devices, or can Any other medium used to store desired information and which can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .

The above is a specific description of the preferred implementation of the embodiment of the present application, but the embodiment of the present application is not limited to the above-mentioned implementation, and those skilled in the art can make various modifications without violating the spirit of the embodiment of the present application. These equivalent modifications or replacements are all included within the scope defined by the claims of the embodiments of the present application.

Claims

A vehicle damage detection method, comprising:

Get images of damaged vehicles;

Inputting the image of the damaged vehicle into the trained integrated damage model to obtain damaged component information, the integrated damage model includes:

A shared backbone neural network, where the shared backbone neural network is obtained by integrating damage detection models corresponding to at least one component type, and the shared backbone neural network is used to preprocess the damaged vehicle image to obtain first output data;

A damage detection classification layer corresponding to at least one component type, configured to process the first output data to obtain damaged component information;

inputting the damaged vehicle image into a vehicle parts segmentation model to obtain a damage location corresponding to the damaged vehicle image;

The damage information of the vehicle is obtained by locating from the damaged component information according to the damage location.
The vehicle damage detection method according to claim 1, wherein the shared backbone neural network is a deep residual neural network, and the deep residual neural network comprises: Res-Net50 network, Res-Net101 network, Res-Net110 network or Res-Net152 network;

Said inputting the image of the damaged vehicle into the trained integrated damage model to obtain the information of the damaged parts, including:

Based on the sequentially connected residual blocks in the deep residual neural network, the damaged vehicle image is subjected to residual feature vector extraction processing to obtain the first output data; wherein, any residual block includes a An identity map and at least two convolutional layers, the identity map of any one residual block is directed from the input end of any one residual block to the output end of any one residual block;

The first output data is input to the damage detection classification layer corresponding to the component type to obtain damaged component information.
The vehicle damage detection method according to claim 1, wherein the integrated damage model is trained through the following training process:

Obtaining a damage data set corresponding to at least one component type as a training data set, the training data set including a corresponding damage judgment label;

Inputting the training data set into the shared backbone neural network to obtain feature data;

inputting the characteristic data into the damage detection classification layer corresponding to the component type, and obtaining the damage component information detection result;

The integrated damage assessment model is obtained through training according to a detection error between the detection result of the damaged component information and the damage judgment label.
The vehicle damage detection method according to claim 3, wherein the damage detection model corresponding to each of the component types includes its corresponding loss function, and the relationship between the detection result according to the damaged component information and the damage judgment label The detection error, the training to obtain the integrated damage model also includes:

Adjusting parameters in the integrated loss assessment model according to the detection error until the loss function satisfies a convergence condition to obtain the integrated impairment assessment model.
The vehicle damage detection method according to claim 4, wherein said obtaining a damage data set corresponding to at least one component type as a training data set comprises:

The uniform sampling strategy is used to uniformly sample the damage data sets corresponding to each part type to obtain the training data set, so that the number of samples between the loss data sets corresponding to different part types is balanced.
The vehicle damage detection method according to claim 4, wherein said obtaining a damage data set corresponding to at least one component type as a training data set comprises:

When there is a large difference in the number of samples between the damage datasets corresponding to different part types, a balanced sampling strategy between categories is used to oversample the loss datasets corresponding to the part types with a small number of samples to obtain the training dataset.
The vehicle damage detection method according to any one of claims 1 to 6, wherein the component types include: sheet metal parts, glass and tires;

The shared backbone neural network is obtained by integrating the sheet metal damage detection model, the glass damage detection model and the tire damage detection model;

The damage detection and classification layer includes: sheet metal damage detection and classification layer, glass damage detection and classification layer or tire damage detection and classification layer;

The damaged part information includes: damaged part name, damage state and damage degree;

The name of the damaged part includes: one or more of sheet metal parts, glass or tires;

The damage information includes: one or more of scratches, scratches, dents, wrinkles, tears, missing or ruptures;

The degree of injury includes: one or more of mild injury, moderate injury or severe injury.
A vehicle damage detection device, including:

An image acquisition module is used to acquire damaged vehicle images;

The damaged part information determination module is used to input the damaged vehicle image into the trained integrated damage model to obtain the damaged part information, and the integrated damage model includes:

A shared backbone neural network, where the shared backbone neural network is obtained by integrating damage detection models corresponding to at least one component type, and the shared backbone neural network is used to preprocess the damaged vehicle image to obtain first output data;

A damage detection classification layer corresponding to at least one component type, configured to process the first output data to obtain damaged component information;

An image segmentation module, configured to input the damaged vehicle image into the vehicle component segmentation model to obtain the damage location corresponding to the damaged vehicle image;

A damage information synthesis module, configured to obtain damage information of the vehicle from the damaged component information according to the damage location.
A computer device, including a processor and a memory;

The memory is used to store programs;

The processor is configured to execute the vehicle damage detection method according to the program:

Wherein, the vehicle damage detection method includes:

Get images of damaged vehicles;

Inputting the image of the damaged vehicle into the trained integrated damage model to obtain damaged component information, the integrated damage model includes:

A shared backbone neural network, where the shared backbone neural network is obtained by integrating damage detection models corresponding to at least one component type, and the shared backbone neural network is used to preprocess the damaged vehicle image to obtain first output data;

A damage detection classification layer corresponding to at least one component type, configured to process the first output data to obtain damaged component information;

inputting the damaged vehicle image into a vehicle parts segmentation model to obtain a damage location corresponding to the damaged vehicle image;

The damage information of the vehicle is obtained by locating from the damaged component information according to the damage location.
The computer device according to claim 9, wherein the shared backbone neural network is a deep residual neural network, and the deep residual neural network comprises: Res-Net50 network, Res-Net101 network, Res-Net110 network or Res -Net152 network;

Said inputting the image of the damaged vehicle into the trained integrated damage model to obtain the information of the damaged parts, including:

Based on the sequentially connected residual blocks in the deep residual neural network, the damaged vehicle image is subjected to residual feature vector extraction processing to obtain the first output data; wherein, any residual block includes a An identity map and at least two convolutional layers, the identity map of any one residual block is directed from the input end of any one residual block to the output end of any one residual block;

The first output data is input to the damage detection classification layer corresponding to the component type to obtain damaged component information.
The computer device according to claim 9, wherein the integrated damage model is trained through the following training process:

Obtaining a damage data set corresponding to at least one component type as a training data set, the training data set including a corresponding damage judgment label;

Inputting the training data set into the shared backbone neural network to obtain feature data;

inputting the characteristic data into the damage detection classification layer corresponding to the component type, and obtaining the damage component information detection result;

The integrated damage assessment model is obtained through training according to a detection error between the detection result of the damaged component information and the damage judgment label.
The computer device according to claim 11, wherein the damage detection model corresponding to each of the component types includes its corresponding loss function, and the detection result of the damaged component information and the detection between the damage judgment label Error, training to obtain the integrated loss model also includes:

Adjusting parameters in the integrated loss assessment model according to the detection error until the loss function satisfies a convergence condition to obtain the integrated impairment assessment model.
The computer device according to claim 12, wherein said obtaining the damage data set corresponding to at least one component type as a training data set comprises:

The uniform sampling strategy is used to uniformly sample the damage data sets corresponding to each part type to obtain the training data set, so that the number of samples between the loss data sets corresponding to different part types is balanced.
The computer device according to claim 12, wherein said obtaining the damage data set corresponding to at least one component type as a training data set comprises:

When there is a large difference in the number of samples between the damage datasets corresponding to different part types, a balanced sampling strategy between categories is used to oversample the loss datasets corresponding to the part types with a small number of samples to obtain the training dataset.
The computer device of claim 9, wherein the component types include: sheet metal, glass, and tires;

The shared backbone neural network is obtained by integrating a sheet metal damage detection model, a glass damage detection model and a tire damage detection model;

The damage detection and classification layer includes: sheet metal damage detection and classification layer, glass damage detection and classification layer or tire damage detection and classification layer;

The damaged part information includes: damaged part name, damage state and damage degree;

The name of the damaged part includes: one or more of sheet metal parts, glass or tires;

The damage information includes: one or more of scratches, scratches, dents, wrinkles, tears, missing or ruptures;

The degree of injury includes: one or more of mild injury, moderate injury or severe injury.
A computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to execute a vehicle damage detection method:

Wherein, the vehicle damage detection method includes:

Get images of damaged vehicles;

Inputting the image of the damaged vehicle into the trained integrated damage model to obtain damaged component information, the integrated damage model includes:

A shared backbone neural network, where the shared backbone neural network is obtained by integrating damage detection models corresponding to at least one component type, and the shared backbone neural network is used to preprocess the damaged vehicle image to obtain first output data;

A damage detection classification layer corresponding to at least one component type, configured to process the first output data to obtain damaged component information;

inputting the damaged vehicle image into a vehicle parts segmentation model to obtain a damage location corresponding to the damaged vehicle image;

The damage information of the vehicle is obtained by locating from the damaged component information according to the damage location.
The computer-readable storage medium according to claim 16, wherein the shared backbone neural network is a deep residual neural network, and the deep residual neural network comprises: Res-Net50 network, Res-Net101 network, Res-Net110 network or Res-Net152 network;

Said inputting the image of the damaged vehicle into the trained integrated damage model to obtain the information of the damaged parts, including:

Based on the sequentially connected residual blocks in the deep residual neural network, the damaged vehicle image is subjected to residual feature vector extraction processing to obtain the first output data; wherein, any residual block includes a An identity map and at least two convolutional layers, the identity map of any one residual block is directed from the input end of any one residual block to the output end of any one residual block;

The first output data is input to the damage detection classification layer corresponding to the component type to obtain damaged component information.
The computer-readable storage medium according to claim 16, wherein the integrated loss assessment model is trained through the following training process:

Obtaining a damage data set corresponding to at least one component type as a training data set, the training data set including a corresponding damage judgment label;

Inputting the training data set into the shared backbone neural network to obtain feature data;

inputting the characteristic data into the damage detection classification layer corresponding to the component type, and obtaining the damage component information detection result;

The integrated damage assessment model is obtained through training according to a detection error between the detection result of the damaged part information and the damage judgment label.
The computer-readable storage medium according to claim 18, wherein the damage detection model corresponding to each of the component types includes its corresponding loss function, and the difference between the detection result according to the damaged component information and the damage judgment label Between the detection errors, the training to obtain the integrated damage assessment model also includes:

Adjusting parameters in the integrated loss assessment model according to the detection error until the loss function satisfies a convergence condition to obtain the integrated impairment assessment model.
The computer-readable storage medium according to claim 19, wherein said obtaining the damage data set corresponding to at least one component type as a training data set comprises:

The uniform sampling strategy is used to uniformly sample the damage data sets corresponding to each part type to obtain the training data set, so that the number of samples between the loss data sets corresponding to different part types is balanced.