CN113095288A - Obstacle missing detection repairing method, device, equipment and storage medium - Google Patents
Obstacle missing detection repairing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113095288A CN113095288A CN202110486374.9A CN202110486374A CN113095288A CN 113095288 A CN113095288 A CN 113095288A CN 202110486374 A CN202110486374 A CN 202110486374A CN 113095288 A CN113095288 A CN 113095288A
- Authority
- CN
- China
- Prior art keywords
- feature map
- obstacle
- characteristic diagram
- anchor
- category
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
Abstract
The application provides a method, a device, equipment and a storage medium for repairing missing obstacle detection, which are used for acquiring a plurality of anchor frames and obstacle image characteristics output by a terminal characteristic output layer of a preset model, wherein the preset model comprises a plurality of characteristic output layers. And obtaining a category characteristic diagram and a foreground background characteristic diagram of each anchor frame according to the characteristics of the plurality of anchor frames and the obstacle images. And determining a missed detection target feature map and a feature map to be processed according to the category feature map and the foreground and background feature map of each anchor frame. And strengthening the characteristic diagram to be processed according to the characteristic diagram of the missed detection target. On the premise of not modifying a model frame, not increasing a large amount of missing detection barrier data and not needing to retrain the model, the output of a specific foreground background characteristic diagram is strengthened according to the relationship between the barrier type and an anchor frame, and the missing detection problem is effectively and quickly solved.
Description
Technical Field
The application relates to the technical field of automatic driving, and provides a method, a device, equipment and a storage medium for repairing missing detection of an obstacle.
Background
An L4 unmanned vehicle typically uses multiple cameras to cover 360 ° of view, front, back, left, and right, to detect obstacles around the vehicle. When the existing visual detection model is applied to an unmanned automobile, the condition of missing detection of obstacles can occur.
The missing obstacle problem solving commonly occurs in the sweetgum fruit: firstly, the feature extraction capability of the model is enhanced by modifying a visual inspection model framework; and secondly, acquiring missing detection obstacle data, and adding the missing detection obstacle data into the training data so that the visual detection model has the capability of identifying the obstacle.
However, modifying the model frame and collecting the missing inspection data of the obstacle require a large amount of time cost and labor cost, and the resource consumption is large, so that the problem of missing inspection of the obstacle cannot be solved quickly.
Disclosure of Invention
The application provides a method, a device, equipment and a storage medium for repairing missing obstacle detection, which are used for solving the problems that in the prior art, a large amount of time cost and labor cost need to be invested, the resource consumption is high, and the missing obstacle detection cannot be rapidly solved.
In a first aspect, the present application provides a method for repairing missed obstacle detection, including:
acquiring barrier image characteristics output by a plurality of anchor frames and a terminal characteristic output layer of a preset model, wherein the preset model comprises a plurality of characteristic output layers;
obtaining a category characteristic diagram and a foreground background characteristic diagram of each anchor frame according to the characteristics of the plurality of anchor frames and the obstacle images;
determining a missed detection target feature map and a feature map to be processed according to the category feature map and the foreground background feature map of each anchor frame;
and strengthening the characteristic diagram to be processed according to the characteristic diagram of the missed detection target.
Optionally, according to the missed detection target feature map, the method for enhancing the feature map to be processed includes:
determining a missed detection target characteristic area according to the missed detection target characteristic diagram;
determining a region to be enhanced of the feature map to be processed according to the missing detection target feature region;
the responsivity of the region to be strengthened is strengthened.
Optionally, determining a missed detection target feature map and a feature map to be processed according to the category feature map and the foreground-background feature map of each anchor frame, including:
determining a category characteristic diagram with the responsibility larger than a preset threshold value according to the category characteristic diagram of each anchor frame, and taking the category characteristic diagram as a missed detection target characteristic diagram;
and determining a foreground background characteristic diagram with the same size as the anchor frame of the undetected target characteristic diagram as a characteristic diagram to be processed according to the foreground background characteristic diagram of each anchor frame.
Optionally, obtaining a plurality of anchor frames comprises:
marking the dimension information of all obstacles in the training data;
obtaining clustering centers of the obstacles in the multiple categories by using a clustering algorithm according to the dimension information, wherein the clustering centers are the central values of the sizes of the obstacles in each category;
and obtaining a plurality of anchor frames with different scales and sizes according to the clustering center.
Optionally, the method further comprises:
acquiring the number of obstacle types;
and determining the number of the class characteristic graphs and the number of the foreground and background characteristic graphs according to the number of the obstacle classes and the number of anchor frames under each obstacle class.
Optionally, the method further comprises:
and outputting the type of the obstacle according to the strengthened characteristic diagram to be processed.
In a second aspect, the present application provides an obstacle missed-detection repair device, comprising:
the acquisition module is used for acquiring barrier image characteristics output by a plurality of anchor frames and a tail end characteristic output layer of a preset model, wherein the preset model comprises a plurality of characteristic output layers;
the processing module is used for obtaining a category characteristic diagram and a foreground background characteristic diagram of each anchor frame according to the image characteristics of the plurality of anchor frames and the obstacles;
the processing module is further used for determining a missed detection target feature map and a feature map to be processed according to the category feature map and the foreground background feature map of each anchor frame;
and the processing module is also used for strengthening the characteristic diagram to be processed according to the missed detection target characteristic diagram.
In a third aspect, the present application provides an electronic device, comprising: a memory, a processor;
a memory; a memory for storing processor-executable instructions;
and a processor, configured to implement the obstacle missed detection repairing method according to the first aspect and the alternative schemes, according to executable instructions stored in a memory.
In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and the computer-executable instructions are executed by a processor to implement the method for repairing missing obstacle according to the first aspect and the optional aspects.
In a fifth aspect, the present application provides a computer program product comprising instructions that, when executed by a processor, implement the method for obstacle missed detection repair according to the first aspect and the alternative.
The application provides a method, a device, equipment and a storage medium for repairing missing obstacle detection, which are used for acquiring a plurality of anchor frames and obstacle image characteristics output by a terminal characteristic output layer of a preset model, wherein the preset model comprises a plurality of characteristic output layers. And obtaining a category characteristic diagram and a foreground background characteristic diagram of each anchor frame according to the characteristics of the plurality of anchor frames and the obstacle images. And determining a missed detection target feature map and a feature map to be processed according to the category feature map and the foreground and background feature map of each anchor frame. And strengthening the characteristic diagram to be processed according to the characteristic diagram of the missed detection target. On the premise of not modifying a model frame, not increasing a large amount of missing detection barrier data and not needing to retrain the model, the output of a specific foreground background characteristic diagram is strengthened according to the relationship between the barrier type and an anchor frame, and the missing detection problem is effectively and quickly solved.
Drawings
Fig. 1 is a schematic flow chart diagram illustrating a method for obstacle missed-detection repair according to an exemplary embodiment of the present application;
FIG. 2 is a diagram illustrating the predicted results of multiple anchor frames for an obstacle;
FIG. 3 is a schematic flow chart diagram illustrating a method for obstacle missed-detection remediation according to another exemplary embodiment of the present application;
FIG. 4 is a schematic diagram of a cluster center of a plurality of classes of obstacles obtained using a clustering algorithm;
FIG. 5 is a schematic illustration of two anchor frame sizes;
FIG. 6 is a schematic diagram of a drive test image acquired by a camera;
FIG. 7 is a characteristic diagram of pedestrian classification corresponding to Anchor _ 2;
FIG. 8 is a pedestrian category feature map corresponding to Anchor _ 14;
FIG. 9 is a foreground-background feature map corresponding to anchor _ 2;
FIG. 10 is a foreground-background feature map corresponding to anchor _ 14;
FIG. 11 is a schematic diagram of a model for detecting pedestrians;
FIG. 12 is a schematic structural view of an obstacle missed-detection repair device shown in the present application according to an exemplary embodiment;
fig. 13 is a schematic diagram of a hardware structure of an electronic device according to an exemplary embodiment of the present application.
Detailed Description
To make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be clearly and completely described below with reference to the drawings in the present application, and it is obvious that the described embodiments are some, but not all embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In the automobile industry, the automatic driving technology is the current most fierce and hot research field, and not only can help automobile enterprises to take the lead and the industrial advantages, but also can bring the brand-new, convenient and safe trip mode to the public. In the unmanned system construction at the level of L4, the sensor configuration schemes are typically cameras, lidar, millimeter wave radar. The camera is lower in price compared with a laser radar, is close to the judgment basis of human driving, and provides reliable obstacle information, so that the application range of the camera is wider than that of other sensors. In the camera layout of the L4 unmanned vehicle, multiple cameras are typically used to cover 360 ° of view from front to back, left to right, and to detect obstacles around the vehicle. With the continuous evolution and optimization of visual inspection models, the judgment of the 2D information of the obstacles by the camera has reached higher accuracy, including the types of the obstacles and the sizes of the width and the height on the images, and the judgment of the 3D information including the real length, the width, the height, the distance and the orientation angle of the obstacles also shows stronger and stronger strength. The current visual inspection models are mainly supervised learning methods such as yolo series algorithm, anchernet-free centret, SSD, etc. By using the methods as the backbone network of the visual detection model, the obstacle information in the image can be effectively extracted. When the existing visual detection model is applied to an unmanned automobile, the condition of missing detection of obstacles can occur.
The missing obstacle problem solving commonly occurs in the sweetgum fruit: firstly, the feature extraction capability of the model is enhanced by modifying a visual inspection model framework; and secondly, acquiring missing detection obstacle data, and adding the missing detection obstacle data into the training data so that the visual detection model has the capability of identifying the obstacle.
However, modifying the model frame and collecting the missing inspection data of the obstacle require a large amount of time cost and labor cost, and the resource consumption is large, so that the problem of missing inspection of the obstacle cannot be solved quickly.
The problems of the prior art are described in detail as follows: firstly, the model framework is modified, and the consumption of computing resources for completing forward inference is rapidly increased along with the deepening of the layer number of the neural network. The improvement of the model is a process that requires a lot of basic research and repeated experimental demonstration, and generally requires annual time payment. The designed excellent model still needs a large amount of data training, and does not escape from complicated data cleaning work or expensive data labeling expenditure, which generally needs more than hundreds of thousands of labeling expenditure. And secondly, collecting missing obstacle data, adding the missing obstacle data into training data, retraining the model based on the original network structure, and needing the training data of the original model and a large amount of newly added data marked by the missing obstacles. For open source visual inspection models, it is typically based on training results on an open source data set. For example, large open source data sets such as the KITTI data set, the Nuscenes data set, the Waymo data set, etc. all have redundant or wrong labels and need to be cleaned again. For example, the hundreds degree Apollo open source model is based on a large number of self-labeling data sets, but the training data set is not open source. The premise of the model migration learning is to obtain original labeled data and then to expand the original data set by labeling a certain amount of data. And labeling the data of the unmanned system is a precise and tedious matter which consumes a great deal of manpower. The self-labeling data technology firstly needs accurate calibration of a sensor, including internal and external parameter calibration from a camera to a laser radar, and the laser radar reaches the calibration of an Inertial Measurement Unit (IMU), so that the position of a camera image cataract obstacle relative to a vehicle can be obtained. And collecting data of the undetected obstacles, wherein the collection time of the laser point cloud and the image data must be strictly synchronous, otherwise, the 3D information deviation problem occurs. And marking the boundary frames of the obstacles in the visual field of all cameras on the three-dimensional point cloud by using a marking tool, projecting the three-dimensional data to a picture through sensor parameters, and then storing the three-dimensional data in a KITTI format. The whole process needs to ensure the accuracy of the data. The detection performance of the model on the missed detection obstacles can be influenced by the calibration error of the sensor or the data marking error.
The reason why the visual inspection model fails to detect the obstacle is as follows: firstly, when the features of the input data are extracted, the features of the obstacles are lost. Secondly, through the network structure of the model, the characteristics are reserved, but the characteristic responsivity of the foreground and background classified output is too low, so that after screening through a normal threshold (for example, the responsivity threshold is 0.4), the obstacle information is eliminated, and missing detection occurs. Therefore, the method for repairing the missing detection of the obstacle is provided, the output of the specific foreground and background feature map is strengthened according to the relation between the obstacle category and the anchor frame on the premise that a model frame is not modified, a large amount of missing detection obstacle data is not increased, and the model does not need to be retrained, and the missing detection problem is effectively and quickly solved.
Fig. 1 is a schematic flow chart illustrating a method for repairing missing obstacle according to an exemplary embodiment of the present disclosure. As shown in fig. 1, the method for repairing the missing obstacle provided by this embodiment includes the following steps:
s101, obtaining obstacle image characteristics output by a plurality of anchor frames and a terminal characteristic output layer of a preset model, wherein the preset model comprises a plurality of characteristic output layers.
More specifically, the method for determining the number and size of the anchor frames comprises the steps of predefining the anchor frames, generating the anchor frames according to data clustering, and learning and optimizing the anchor frames through a model. The embodiment generates a plurality of anchor frames with different sizes and sizes according to clustering. The preset model is obtained according to a transfer learning method. The preset model is a deep neural network structure, a characteristic output layer is built layer by layer, and the input is subjected to characteristic extraction layer by layer. Each feature output layer may output an image feature. The preset model includes a plurality of feature output layers. Each feature output layer includes a contribution layer, a Batchnorm layer, a Scale layer, and a Relu layer. And inputting the drive test image collected by the camera of the unmanned automobile into the preset model, and sequentially entering each characteristic output layer to obtain the obstacle image characteristics output by the tail end characteristic output layer of the preset model.
S102, obtaining a category characteristic diagram and a foreground background characteristic diagram of each anchor frame according to the characteristics of the plurality of anchor frames and the obstacle images.
More specifically, the number of class feature maps is equal to the product of the number of anchor frames and the number of obstacle classes. The obstacle category may be a car, a bicycle, a bus, a pedestrian, and the like. The foreground and background characteristic graph is related to the responsivity, the background is determined when the responsivity value is lower than a certain value, the foreground is determined when the responsivity value is higher than a certain value, and the foreground is unrelated to the obstacle category. Therefore, the number of foreground-background feature maps is equal to the number of anchor boxes.
Assume that in the drive test image acquired by the camera, the number of obstacle classes is 7, and the number of anchor frames is 16. The obstacle types are represented in the form of channels, and are channel _0, channel _1, channel _2, channel _3, channel _4, channel _5, and channel _6, respectively. Then 16 anchor frames with different sizes and sizes are used for obtaining 16 category feature maps for each category of obstacles, i.e. a total of 112 category feature maps are obtained. The foreground and background feature maps are 16 in total.
S103, determining a missed detection target feature map and a feature map to be processed according to the category feature map and the foreground and background feature map of each anchor frame.
More specifically, the undetected target feature map is a category feature map with the highest responsiveness selected from the category feature maps of the anchor frames of each type of obstacle. The feature map to be processed is a foreground background feature map under the anchor frame corresponding to the undetected feature map.
Assume that the number of obstacle classes is 7 and the number of anchor boxes is 16. For example, channel _0 indicates an obstacle category of an automobile, and channel _6 indicates an obstacle category of a pedestrian. channel _0 gets 16 car class feature maps. And selecting one with the highest responsivity from the 16 automobile category characteristic maps as an automobile missed detection target characteristic map. And if the anchor frame corresponding to the automobile undetected target feature map is the first anchor frame, acquiring a foreground background feature map corresponding to the first anchor frame as an automobile to-be-processed feature map. channel _6 obtains 16 pedestrian category feature maps. And selecting one with the highest responsivity from the 16 pedestrian category characteristic maps as a pedestrian missing detection target characteristic map. And if the anchor frame corresponding to the pedestrian missing inspection target feature map is the second anchor frame, acquiring a foreground background feature map corresponding to the second anchor frame as a pedestrian feature map to be processed.
And S104, strengthening the characteristic diagram to be processed according to the characteristic diagram of the missed detection target.
More specifically, according to the missed detection target feature map, the target feature region of which the responsivity is greater than a preset value in the missed detection target feature map of the missed detection region is determined. And determining the region corresponding to the target feature region in the feature map to be processed as the region to be strengthened. Increasing the responsivity of the region to be enhanced. So far, the feature enhancement of a specific area in the feature map to be processed is completed to improve the missing detection.
Assume that the number of obstacle classes is 7 and the number of anchor boxes is 16. For example, channel _0 indicates an obstacle category of an automobile, and channel _6 indicates an obstacle category of a pedestrian. And determining that the area where the automobile is located is the target characteristic area if the responsivity of the area where the automobile is located in the automobile omission target characteristic graph is greater than a preset value. The responsivity of the response area of the characteristic diagram to be processed of the automobile is increased, so that the problem of missed detection of the obstacle of the automobile is solved. And determining that the region in which the pedestrian is located is the target characteristic region when the responsivity of the region in which the pedestrian is located in the target characteristic map of the pedestrian omission is greater than a preset value. The responsiveness of the response area of the characteristic diagram to be processed of the pedestrian is increased so as to improve the problem of missing detection of the obstacle, namely the pedestrian.
Fig. 2 is a diagram illustrating the prediction results of a plurality of anchor frames for an obstacle. As shown in fig. 2, after the anchor frame mechanism is applied, two obstacles of a person and an automobile are predicted, and the anchor frame is fitted with the real boundary of the person and the automobile. From this it can be seen that the anchor-box mechanism allows a single window to predict multiple targets, with the predicted targets having multiple scales. The anchor frame mechanism is applied to the deep neural network structure to complete feature extraction and predict the center of the obstacle. The appearance of the anchor frame mechanism changes the situation that one target center can only predict one barrier boundary frame and one target center corresponds to only one barrier boundary frame. The accuracy of the obstacle detection in different scales is effectively improved, and the problem of detection of overlapping of a plurality of obstacles can be solved.
In the method provided by this embodiment, the obstacle image features output by a plurality of anchor frames and a terminal feature output layer of a preset model are obtained, where the preset model includes a plurality of feature output layers. And obtaining a category characteristic diagram and a foreground background characteristic diagram of each anchor frame according to the characteristics of the plurality of anchor frames and the obstacle images. And determining a missed detection target feature map and a feature map to be processed according to the category feature map and the foreground and background feature map of each anchor frame. And strengthening the characteristic diagram to be processed according to the characteristic diagram of the missed detection target. On the premise of not modifying a model frame, not increasing a large amount of missing detection barrier data and not needing to retrain the model, the output of a specific foreground background characteristic diagram is strengthened according to the relationship between the barrier type and an anchor frame, and the missing detection problem is effectively and quickly solved.
Fig. 3 is a schematic flow chart illustrating a method for repairing missing obstacle according to another exemplary embodiment of the present application. As shown in fig. 3, the method for repairing the missing obstacle provided by this embodiment includes the following steps:
s201, acquiring training data, wherein the training data are automatic driving scene data; and training to obtain a preset model according to the training data and the visual detection model, wherein the visual detection model is the existing model.
More specifically, the visual inspection model is an existing model, and the application uses a 3D visual obstacle inspection model that is currently open-source and superior in performance. Since the application scene of the 3D visual obstacle detection model and the automatic driving scene have a certain similarity, the types of the detected targets are approximately consistent. Therefore, the 3D visual obstacle detection model is subjected to transfer learning, and a large amount of model development work is avoided. And inputting the automatic driving scene data serving as training data into the 3D visual obstacle detection model by using a transfer learning method to obtain a preset model.
The transfer learning method is not a new model which is self-trained from zero, can help developers to rapidly own the target detection model which is available in own scene, and is rapidly developed in recent years. The transfer learning method solves the difficulties in the model development process from the following three aspects:
firstly, a target detection model required by an unmanned system needs strong detection capability, various target types are covered, and the shielding conditions of the posture light of each target are varied. This requires massive amounts of data to be continuously trained and updated, and the data needs to be accurately labeled, including obstacle position, category, distance, three-dimensional scale, and orientation angle, to be used for training. Data collection and labeling is quite expensive.
Secondly, model training is performed after massive labeled data are obtained, powerful equipment is needed for data storage and calculation, and the equipment is usually expensive and can not be owned by ordinary people.
Thirdly, the model trained by using the open-source big data faces the problem of data difference between the original training data of the model and the data of the scene, and the generalization performance of the model in a new scene may be difficult to satisfy, or the output of the model is not completely consistent with the current requirement.
S202, marking dimension information of all obstacles in training data; obtaining clustering centers of the obstacles in the multiple categories by using a clustering algorithm according to the dimension information, wherein the clustering centers are the central values of the sizes of the obstacles in each category; and obtaining a plurality of anchor frames with different scales and sizes according to the clustering center.
More specifically, the training data includes an image of an automated driving scene. And marking the dimension characteristics of all the obstacles on the automatic driving scene image. The dimension information of the obstacle is the width and height of the obstacle. And (3) clustering all obstacles in the training data according to the width and the height of the obstacles respectively by using a clustering algorithm to generate a plurality of clustering centers. The cluster center is the center value of the size of each type of obstacle. And reducing the clustering center by a certain multiple according to the size of the window to obtain values of a plurality of anchor frames. The size of the window refers to the size of the image to be detected which is input into the preset model. The image to be detected can be the whole image or the image divided into N × M small windows, and each small window is used as the image to be detected. The number of anchor boxes is determined by the number of obstacle classes in the training data, and the size of the anchor boxes is related to the width and height of the obstacles in the training data.
Fig. 4 is a schematic diagram of a cluster center of obstacles of multiple categories obtained using a clustering algorithm. As shown in fig. 4, the obstacles in the training data are respectively clustered in width and height according to the category, and the generated cluster centers are the center values of the sizes of the obstacles of each category.
It is assumed that the training data includes 7 types of obstacles, which are denoted as channel _0, channel _1, channel _2, channel _3, channel _4, channel _5, and channel _ 6. Each type of barrier corresponds to a plurality of anchor frames, 16 anchor frames are obtained, and the size of each anchor frame is as follows: anchor _0 ═ 4.9434993, 1.516986, anchor _1 ═ 2.1259836, 1.6779645, anchor _2 ═ 1.6779645, anchor _3 ═ 1.6779645, anchor _4 ═ 1.6779645, anchor _5 ═ 1.6779645, anchor _6 ═ 1.6779645, anchor _7 ═ 1.6779645, anchor _8 ═ 1.6779645, anchor _9 ═ 1.6779645, anchor _10 (1.6779645 ), anchor _11 ═ 1.6779645, anchor _12 ═ 1.6779645, anchor _13 ═ 1.6779645, anchor _14 ═ 1.6779645, anchor _15 ═ 1.6779645, anchor _1 ═ 1.6779645, anchor _3, anchor _ 5.
Fig. 5 is a schematic representation of two anchor frame sizes. As shown in FIG. 5, the left hand side is shown as anchor _0, 4.9434993 wide and 1.516986 high.
In this embodiment, the steps S201 and S202 are not limited by the described operation sequence, and the steps S201 and S202 may be performed in other sequences or simultaneously.
S203, obtaining the barrier image characteristics output by the plurality of anchor frames and the tail end characteristic output layer of the preset model, wherein the preset model comprises a plurality of characteristic output layers.
Step S203 is similar to the step S101 in the embodiment of fig. 1, and this embodiment is not described herein again.
And S204, obtaining a category characteristic diagram and a foreground and background characteristic diagram of each anchor frame according to the characteristics of the plurality of anchor frames and the obstacle images.
Step S204 is similar to step S102 in the embodiment of fig. 1, and is not described herein again.
Optionally, the method further comprises: acquiring the number of obstacle types; and determining the number of the class characteristic graphs and the number of the foreground background characteristic graphs according to the number of the anchor frames and the number of the obstacle classes.
Fig. 6 is a schematic diagram of a drive test image acquired by a camera. As shown in fig. 6, the obstacle category is two pedestrians. In the present embodiment, the number of obstacle class categories is 7, and the number of anchor boxes under each category is 16. Therefore, in 7 obstacle category channels, with 16 anchor frames of different sizes and sizes, a total of 112 category feature maps and 16 foreground-background feature maps are obtained. Each channel includes 16 class feature maps. The channel in which the pedestrian is an obstacle is channel _ 6.
S205, according to the class characteristic diagram of each anchor frame, determining the class characteristic diagram with the responsivity larger than a preset threshold value as a target characteristic diagram for missed detection.
More specifically, the preset threshold may be determined according to the actual situation of the class feature map. And selecting the class characteristic diagram with the responsibility larger than a preset threshold value from the class characteristic diagrams of the anchor frames under each class as a target characteristic diagram for missed detection.
And selecting a category characteristic diagram with the responsiveness greater than a preset threshold from the category characteristic diagrams of the anchor frames under the channel _ 6. Fig. 7 is a pedestrian category feature map corresponding to anchor _ 2. Fig. 8 is a pedestrian category feature map corresponding to anchor _ 14. As shown in fig. 7 and 8, in the anchorms _2 and _14, the responsivity of the visible pedestrian region in the corresponding class characteristic diagram of the pedestrian is high, and darker colors indicate weaker responsivity, whereas stronger responsivity indicates stronger responsivity. The responsivity of the highest response region reached 0.9. The anchor frame size of anchor _2 is (19.452609, 17.815241), and the anchor frame size of anchor _14 is (10.0207692, 6.877788). Anchor _2 and Anchor _14 are generated for the obstacle category of pedestrians, and have the highest width-height ratio and size response to the height of the semi-human. Therefore, one category feature map is selected from the anchors _2 and anchors _14 as the missing target feature map.
S206, determining a foreground background characteristic diagram with the same size as the anchor frame of the undetected target characteristic diagram according to the foreground background characteristic diagram of each anchor frame, and taking the foreground background characteristic diagram as a characteristic diagram to be processed.
More specifically, according to an anchor frame corresponding to the missed detection target feature map, determining a feature map to be processed in the foreground and background feature maps of each anchor frame.
For example, the anchor frame corresponding to the undetected target feature map is anchor _2, and the foreground-background feature map corresponding to the anchor _2 is the feature map to be processed. Fig. 9 is a foreground-background feature map corresponding to anchor _ 2. And if the anchor frame corresponding to the undetected target feature map is anchor _14, the foreground background feature map corresponding to the anchor _14 is the feature map to be processed. Fig. 10 is a foreground-background feature map corresponding to anchor _ 14. As shown in fig. 9 and 10, the pedestrian region has a weak response, and in the case where the threshold of the responsiveness is 0.4, the pedestrian or the like is to be screened out, resulting in a false detection. However, for a missing pedestrian, the class profile response of its corresponding anchor frame is still prominent. Therefore, the method and the device solve the problem by reinforcing the feature map to be processed through the missed detection of the target feature map.
And S207, determining a missing detection target characteristic area according to the missing detection target characteristic diagram.
More specifically, the region with the responsivity higher than the preset value in the target feature map of the missed detection is determined as the target feature region of the missed detection.
For example, a region with responsivity greater than 0.6 in the class feature map corresponding to anchor _2 or anchor _14 is marked as a, and a is the region of the missed detection target feature.
And S208, determining a region to be strengthened of the feature map to be processed according to the missing detection target feature region.
More specifically, the region corresponding to the missed detection target feature region in the feature map to be processed is determined as the region to be enhanced.
For example, the region corresponding to a in the foreground-background feature map corresponding to anchor _2 or anchor _14 is labeled as B, and B is the region to be enhanced.
And S209, enhancing the responsivity of the region to be enhanced.
More specifically, the responsivity of the foreground and background characteristic diagram of the missed detection type obstacle is strengthened, and the missed detection is improved.
For example, if a region where B is present exists, the responsivity of the region where B is present is improved by 0.5. Since the responsivity is not more than 1, if the responsivity of the region in which B is present is more than 1, it is set to 1.
And S210, outputting the type of the obstacle according to the strengthened characteristic diagram to be processed.
Fig. 11 is a schematic diagram of a model detecting a pedestrian. As shown in fig. 11, the model outputs that the obstacle category is a pedestrian in the case where the responsiveness threshold is 0.4 according to the feature map to be processed for which the reinforcement is completed.
Fig. 12 is a schematic structural diagram of an obstacle missed-detection repair apparatus according to an exemplary embodiment of the present application. As shown in fig. 12, the present application provides an obstacle missed-detection repair apparatus 40, the apparatus 40 including:
and an obtaining module 41, configured to obtain image characteristics of the obstacle output by the terminal characteristic output layers of the multiple anchor frames and a preset model, where the preset model includes multiple characteristic output layers.
And the processing module 42 is configured to obtain a category feature map and a foreground-background feature map of each anchor frame according to the features of the plurality of anchor frames and the obstacle images.
The processing module 42 is further configured to determine a missed-detection target feature map and a feature map to be processed according to the category feature map and the foreground-background feature map of each anchor frame.
The processing module 42 is further configured to enhance the feature map to be processed according to the missed detection target feature map.
Specifically, the present embodiment may refer to the above method embodiments, and the principle and the technical effect are similar, which are not described again.
Fig. 13 is a schematic diagram of a hardware structure of an electronic device according to an exemplary embodiment of the present application. As shown in fig. 13, the electronic apparatus 50 of the present embodiment includes: a processor 51 and a memory 52; wherein the content of the first and second substances,
a memory 52, a memory for storing processor-executable instructions.
The processor 51 is configured to implement the obstacle missed detection repairing method in the above embodiment according to executable instructions stored in the memory. Reference may be made in particular to the description relating to the method embodiments described above.
Alternatively, the memory 52 may be separate or integrated with the processor 51.
When the memory 52 is provided separately, the electronic device 50 further includes a bus 53 for connecting the memory 52 and the processor 51.
The present application also provides a computer readable storage medium, in which computer instructions are stored, and the computer instructions are executed by a processor to implement the methods provided by the above-mentioned various embodiments.
The computer-readable storage medium may be a computer storage medium or a communication medium. Communication media includes any medium that facilitates transfer of a computer program from one place to another. Computer storage media may be any available media that can be accessed by a general purpose or special purpose computer. For example, a computer readable storage medium is coupled to the processor such that the processor can read information from, and write information to, the computer readable storage medium. Of course, the computer readable storage medium may also be integral to the processor. The processor and the computer-readable storage medium may reside in an Application Specific Integrated Circuit (ASIC). Additionally, the ASIC may reside in user equipment. Of course, the processor and the computer-readable storage medium may also reside as discrete components in a communication device.
The computer-readable storage medium may be implemented by any type of volatile or nonvolatile Memory device or combination thereof, such as Static Random-Access Memory (SRAM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk or optical disk. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.
The present application also provides a computer program product comprising execution instructions stored in a computer readable storage medium. The at least one processor of the device may read the execution instructions from the computer-readable storage medium, and the execution of the execution instructions by the at least one processor causes the device to implement the methods provided by the various embodiments described above.
Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.
Claims (10)
1. A method for repairing missing obstacle detection is characterized by comprising the following steps:
acquiring barrier image characteristics output by a plurality of anchor frames and a terminal characteristic output layer of a preset model, wherein the preset model comprises a plurality of characteristic output layers;
obtaining a category feature map and a foreground background feature map of each anchor frame according to the plurality of anchor frames and the obstacle image features;
determining a missed detection target feature map and a feature map to be processed according to the category feature map and the foreground background feature map of each anchor frame;
and according to the missed detection target feature map, strengthening the feature map to be processed.
2. The method according to claim 1, wherein the enhancing the feature map to be processed according to the missing target feature map comprises:
determining a missed detection target characteristic area according to the missed detection target characteristic diagram;
determining a region to be enhanced of the feature map to be processed according to the missing detection target feature region;
and intensifying the responsivity of the region to be intensified.
3. The method according to claim 1, wherein the determining a missed detection target feature map and a feature map to be processed according to the class feature map and the foreground-background feature map of each anchor frame comprises:
determining a category characteristic diagram with the responsibility being larger than a preset threshold value according to the category characteristic diagram of each anchor frame, and taking the category characteristic diagram as the undetected target characteristic diagram;
and determining a foreground background characteristic diagram with the same size as the anchor frame of the undetected target characteristic diagram as the characteristic diagram to be processed according to the foreground background characteristic diagram of each anchor frame.
4. The method of any of claims 1-3, wherein the obtaining a plurality of anchor frames comprises:
marking the dimension information of all obstacles in the training data;
obtaining a clustering center of the obstacles of a plurality of categories by using a clustering algorithm according to the dimension information, wherein the clustering center is a central value of the size of each obstacle of each category;
and obtaining a plurality of anchor frames with different scales and sizes according to the clustering center.
5. The method according to any one of claims 1-3, further comprising:
acquiring the number of obstacle types;
and determining the number of the class characteristic graphs and the number of the foreground and background characteristic graphs according to the number of the obstacle classes and the number of the anchor frames under each obstacle class.
6. The method according to any one of claims 1-3, further comprising:
and outputting the type of the obstacle according to the strengthened characteristic diagram to be processed.
7. An obstacle missed-detection repair device, the device comprising:
the acquisition module is used for acquiring barrier image characteristics output by a plurality of anchor frames and a terminal characteristic output layer of a preset model, wherein the preset model comprises a plurality of characteristic output layers;
the processing module is used for obtaining a category characteristic diagram and a foreground background characteristic diagram of each anchor frame according to the plurality of anchor frames and the obstacle image characteristics;
the processing module is further used for determining a missed detection target feature map and a feature map to be processed according to the category feature map and the foreground background feature map of each anchor frame;
and the processing module is also used for strengthening the characteristic diagram to be processed according to the missed detection target characteristic diagram.
8. An electronic device, comprising: a memory, a processor;
a memory; a memory for storing the processor-executable instructions;
a processor for implementing the obstacle missed detection repair method of any one of claims 1 to 6 according to executable instructions stored by the memory.
9. A computer-readable storage medium having stored thereon computer-executable instructions for implementing the method of obstacle missed detection repair of any one of claims 1 to 6 when executed by a processor.
10. A computer program product comprising instructions which, when executed by a processor, implement the method of obstacle missed detection repair of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110486374.9A CN113095288A (en) | 2021-04-30 | 2021-04-30 | Obstacle missing detection repairing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110486374.9A CN113095288A (en) | 2021-04-30 | 2021-04-30 | Obstacle missing detection repairing method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113095288A true CN113095288A (en) | 2021-07-09 |
Family
ID=76681233
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110486374.9A Pending CN113095288A (en) | 2021-04-30 | 2021-04-30 | Obstacle missing detection repairing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113095288A (en) |
Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109541632A (en) * | 2018-09-30 | 2019-03-29 | 天津大学 | A kind of target detection missing inspection improved method based on four line laser radars auxiliary |
| CN109815886A (en) * | 2019-01-21 | 2019-05-28 | 南京邮电大学 | A kind of pedestrian and vehicle checking method and system based on improvement YOLOv3 |
| CN109977774A (en) * | 2019-02-25 | 2019-07-05 | 中国科学技术大学 | A kind of fast target detection method based on adaptive convolution |
| CN110309792A (en) * | 2019-07-04 | 2019-10-08 | 电子科技大学 | Indoor human detection method based on component model |
| CN110502983A (en) * | 2019-07-11 | 2019-11-26 | 平安科技（深圳）有限公司 | The method, apparatus and computer equipment of barrier in a kind of detection highway |
| CN110633594A (en) * | 2018-06-21 | 2019-12-31 | 北京京东尚科信息技术有限公司 | Target detection method and device |
| CN110717933A (en) * | 2019-10-10 | 2020-01-21 | 北京百度网讯科技有限公司 | Post-processing method, device, equipment and medium for moving object missed detection |
| CN111046743A (en) * | 2019-11-21 | 2020-04-21 | 新奇点企业管理集团有限公司 | Obstacle information labeling method and device, electronic equipment and storage medium |
| CN111160379A (en) * | 2018-11-07 | 2020-05-15 | 北京嘀嘀无限科技发展有限公司 | Training method and device of image detection model and target detection method and device |
| CN111444364A (en) * | 2020-03-04 | 2020-07-24 | 中国建设银行股份有限公司 | Image detection method and device |
| CN111626987A (en) * | 2020-04-29 | 2020-09-04 | 南京邮电大学 | Vehicle detection method based on unmanned aerial vehicle and anchor-frame-free network |
| CN111951212A (en) * | 2020-04-08 | 2020-11-17 | 北京交通大学 | Method for identifying defects of contact network image of railway |
| CN112070079A (en) * | 2020-07-24 | 2020-12-11 | 华南理工大学 | X-ray contraband package detection method and device based on feature map weighting |
| US20210012116A1 (en) * | 2019-07-08 | 2021-01-14 | Uatc, Llc | Systems and Methods for Identifying Unknown Instances |
| CN112287859A (en) * | 2020-11-03 | 2021-01-29 | 北京京东乾石科技有限公司 | Object recognition method, device and system, computer readable storage medium |
| CN112287860A (en) * | 2020-11-03 | 2021-01-29 | 北京京东乾石科技有限公司 | Training method and device of object recognition model, and object recognition method and system |
| CN112505684A (en) * | 2020-11-17 | 2021-03-16 | 东南大学 | Vehicle multi-target tracking method based on radar vision fusion under road side view angle in severe environment |
-
2021
- 2021-04-30 CN CN202110486374.9A patent/CN113095288A/en active Pending
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110633594A (en) * | 2018-06-21 | 2019-12-31 | 北京京东尚科信息技术有限公司 | Target detection method and device |
| CN109541632A (en) * | 2018-09-30 | 2019-03-29 | 天津大学 | A kind of target detection missing inspection improved method based on four line laser radars auxiliary |
| CN111160379A (en) * | 2018-11-07 | 2020-05-15 | 北京嘀嘀无限科技发展有限公司 | Training method and device of image detection model and target detection method and device |
| CN109815886A (en) * | 2019-01-21 | 2019-05-28 | 南京邮电大学 | A kind of pedestrian and vehicle checking method and system based on improvement YOLOv3 |
| CN109977774A (en) * | 2019-02-25 | 2019-07-05 | 中国科学技术大学 | A kind of fast target detection method based on adaptive convolution |
| CN110309792A (en) * | 2019-07-04 | 2019-10-08 | 电子科技大学 | Indoor human detection method based on component model |
| US20210012116A1 (en) * | 2019-07-08 | 2021-01-14 | Uatc, Llc | Systems and Methods for Identifying Unknown Instances |
| CN110502983A (en) * | 2019-07-11 | 2019-11-26 | 平安科技（深圳）有限公司 | The method, apparatus and computer equipment of barrier in a kind of detection highway |
| CN110717933A (en) * | 2019-10-10 | 2020-01-21 | 北京百度网讯科技有限公司 | Post-processing method, device, equipment and medium for moving object missed detection |
| CN111046743A (en) * | 2019-11-21 | 2020-04-21 | 新奇点企业管理集团有限公司 | Obstacle information labeling method and device, electronic equipment and storage medium |
| CN111444364A (en) * | 2020-03-04 | 2020-07-24 | 中国建设银行股份有限公司 | Image detection method and device |
| CN111951212A (en) * | 2020-04-08 | 2020-11-17 | 北京交通大学 | Method for identifying defects of contact network image of railway |
| CN111626987A (en) * | 2020-04-29 | 2020-09-04 | 南京邮电大学 | Vehicle detection method based on unmanned aerial vehicle and anchor-frame-free network |
| CN112070079A (en) * | 2020-07-24 | 2020-12-11 | 华南理工大学 | X-ray contraband package detection method and device based on feature map weighting |
| CN112287859A (en) * | 2020-11-03 | 2021-01-29 | 北京京东乾石科技有限公司 | Object recognition method, device and system, computer readable storage medium |
| CN112287860A (en) * | 2020-11-03 | 2021-01-29 | 北京京东乾石科技有限公司 | Training method and device of object recognition model, and object recognition method and system |
| CN112505684A (en) * | 2020-11-17 | 2021-03-16 | 东南大学 | Vehicle multi-target tracking method based on radar vision fusion under road side view angle in severe environment |
Non-Patent Citations (2)
| Title |
|---|
| ZHENBO REN等: "Real-Time Target Detection in Visual Sensing Environments Using Deep Transfer Learning and Improved Anchor Box Generation", 《IEEE》 * |
| 聂鑫等: "复杂场景下基于增强YOLOv3的船舶目标检测", 《计算机应用》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| TWI742382B (en) | Neural network system for vehicle parts recognition executed by computer, method for vehicle part recognition through neural network system, device and computing equipment for vehicle part recognition | |
| CN109117876B (en) | Dense small target detection model construction method, dense small target detection model and dense small target detection method | |
| CN109886066B (en) | Rapid target detection method based on multi-scale and multi-layer feature fusion | |
| CN109086785B (en) | Training method and device for image calibration model | |
| Makantasis et al. | Deep convolutional neural networks for efficient vision based tunnel inspection | |
| CN111738110A (en) | Remote sensing image vehicle target detection method based on multi-scale attention mechanism | |
| EP3796228A1 (en) | Device and method for generating a counterfactual data sample for a neural network | |
| US11308714B1 (en) | Artificial intelligence system for identifying and assessing attributes of a property shown in aerial imagery | |
| EP3806064A1 (en) | Method and apparatus for detecting parking space usage condition, electronic device, and storage medium | |
| US20220092856A1 (en) | Crack detection, assessment and visualization using deep learning with 3d mesh model | |
| Wen et al. | Automated pavement distress segmentation on asphalt surfaces using a deep learning network | |
| CN111275026A (en) | Three-dimensional point cloud combined semantic and instance segmentation method | |
| CN114723709A (en) | Tunnel disease detection method and device and electronic equipment | |
| CN111368634B (en) | Human head detection method, system and storage medium based on neural network | |
| CN111523558A (en) | Ship shielding detection method and device based on electronic purse net and electronic equipment | |
| CN115471487A (en) | Insulator defect detection model construction and insulator defect detection method and device | |
| CN113420619A (en) | Remote sensing image building extraction method | |
| CN110909656A (en) | Pedestrian detection method and system with integration of radar and camera | |
| CN112784494B (en) | Training method of false positive recognition model, target recognition method and device | |
| CN112487861A (en) | Lane line recognition method and device, computing equipment and computer storage medium | |
| CN113095288A (en) | Obstacle missing detection repairing method, device, equipment and storage medium | |
| Ranyal et al. | Enhancing pavement health assessment: An attention-based approach for accurate crack detection, measurement, and mapping | |
| CN110084190B (en) | Real-time unstructured road detection method under severe illumination environment based on ANN | |
| Li et al. | Wooden spoon crack detection by prior knowledge-enriched deep convolutional network | |
| EP3702971A1 (en) | Method for estimating the uncertainty of a recognition process in a recognition unit; electronic control unit and motor vehicle |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |