CN117495692A - Image data enhancement method and device based on 3D (three-dimensional) drivable area - Google Patents
Image data enhancement method and device based on 3D (three-dimensional) drivable area Download PDFInfo
- Publication number
- CN117495692A CN117495692A CN202311356907.7A CN202311356907A CN117495692A CN 117495692 A CN117495692 A CN 117495692A CN 202311356907 A CN202311356907 A CN 202311356907A CN 117495692 A CN117495692 A CN 117495692A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- frame
- current scene
- image
- rare
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 230000002708 enhancing effect Effects 0.000 claims description 12
- 238000012545 processing Methods 0.000 claims description 10
- 235000004522 Pentaglottis sempervirens Nutrition 0.000 claims description 8
- 230000011218 segmentation Effects 0.000 claims description 6
- 238000012800 visualization Methods 0.000 claims description 4
- 238000001914 filtration Methods 0.000 claims description 3
- 238000001514 detection method Methods 0.000 abstract description 31
- 230000008569 process Effects 0.000 description 9
- 238000012549 training Methods 0.000 description 9
- 238000004590 computer program Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 240000004050 Pentaglottis sempervirens Species 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000004575 stone Substances 0.000 description 2
- 230000001131 transforming effect Effects 0.000 description 2
- 238000013136 deep learning model Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000009191 jumping Effects 0.000 description 1
- 230000002393 scratching effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/53—Querying
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention discloses an image data enhancement method and device based on a 3D (three-dimensional) drivable area, comprising the following steps: establishing a rare object database; acquiring multi-frame point clouds and multi-frame images of a current scene, and determining a drivable area corresponding to each frame point cloud of the current scene; the multi-frame point cloud and the multi-frame image of the current scene are data corresponding to each other at one time; selecting a position to be enhanced in a drivable area corresponding to each frame of point cloud, and searching rare object data meeting the conditions in a rare object database according to the position to be enhanced; and merging the rare object data into the corresponding image of the current scene to obtain the enhanced image data of the current scene. According to the invention, the rare object is constructed in the 3D drivable area, so that the detection accuracy of the 3D detection model on the rare object is improved.
Description
Technical Field
The invention relates to the technical field of automatic driving, in particular to an image data enhancement method and device based on a 3D (three-dimensional) drivable area.
Background
In an automatic driving scene, the image-based 3D object detection model can provide obstacle position and posture information for an automatic driving vehicle and is used for planning and controlling the driving track of the vehicle, but due to real-world complexity, a labeling data set used for training the object detection model is limited, and is an important challenge for detecting rare objects, and the main reason is that the data quantity of the rare objects has a very low ratio in a label data set, even does not appear, such as triangle warning boards, road construction warning boards, inverted cone barrels, objects such as cartons and stones, which leads to the fact that the object detection model cannot learn the characteristics of the rare objects well, and finally leads to missed detection and seriously influences the safety of automatic driving.
In the prior art, the method for improving the rare object detection performance is more effective in a data enhancement mode, wherein the representative scheme is that a mask which is proposed by google and utilizes marked objects is used for picking the objects from a current image, the objects are randomly scaled in a copy-paste mode and then inserted into random positions of other image frames, and a new image is constructed so as to improve the occurrence frequency of the marked objects in the model training process.
However, the scheme aims at the problem that the scheme aims at an image 2D detection application scene, a stuck 2D object cannot acquire accurate 3D bounding box information, and an accurate 3D bounding box true value of a target object is required to be used as a learning target during training of an image 3D detection model, so that the application scene is limited; on the other hand, the mode that the scheme is randomly pasted at any position is easy to cause false detection, so that the accuracy of the marked object in the model training process is low.
Accordingly, there is a need in the art for improvement.
Disclosure of Invention
The invention aims to solve the technical problem that aiming at the defects of the prior art, the invention provides an image data enhancement method and device based on a 3D (three-dimensional) drivable area so as to solve the problem of low accuracy in the existing rare object detection scheme.
The technical scheme adopted for solving the technical problems is as follows:
in a first aspect, the present invention provides an image data enhancement method based on a 3D drivable region, comprising:
establishing a rare object database;
acquiring multi-frame point clouds and multi-frame images of a current scene, and determining a drivable area corresponding to each frame point cloud of the current scene; wherein, the multi-frame point cloud and multi-frame image of the current scene are data corresponding to each other in time;
selecting a position to be enhanced in a drivable area corresponding to each frame of point cloud, and searching rare object data meeting a condition in the rare object database according to the position to be enhanced;
and merging the rare object data into the corresponding image of the current scene to obtain the enhanced image data of the current scene.
In one implementation, the building a rare object database includes:
acquiring an image frame and a point cloud frame of a rare object; the image frames of the rare objects and the point cloud frames of the rare objects are data at the same moment after time synchronization processing;
acquiring a rare object local image in an image frame of the rare object according to a first instruction;
performing visualization processing on the point cloud frame of the rare object according to a second instruction, and acquiring 3D bounding box information corresponding to the rare object;
and establishing the rare object database according to the rare object local image and the corresponding 3D bounding box information.
In one implementation manner, the determining the drivable area corresponding to each frame of the point cloud of the current scene includes:
performing three-dimensional reconstruction according to the multi-frame point cloud of the current scene, extracting ground points according to the reconstructed three-dimensional model, and estimating and obtaining a ground plane equation according to the extracted ground points;
and according to a ray tracing algorithm, obtaining a travelable area unoccupied by the object in each frame of point cloud of the current scene.
In one implementation manner, the three-dimensional reconstruction is performed according to the multi-frame point cloud of the current scene, ground points are extracted according to the reconstructed three-dimensional model, and a ground plane equation is estimated according to the extracted ground points, including:
extracting ground points in all point cloud frames of the current scene according to a ground segmentation algorithm;
acquiring a current vehicle pose, converting ground points in all point cloud frames into a world coordinate system according to the current vehicle pose, and acquiring a local dense point cloud map of a preset road section;
and fitting by utilizing point cloud coordinates according to the local dense point cloud map to obtain the ground plane equation.
In one implementation, the obtaining the drivable area unoccupied by the object in each frame of the point cloud of the current scene includes:
filtering the extracted ground points and the points with preset height from the ground in each frame of point cloud of the current scene;
projecting the filtered residual three-dimensional point cloud into a bird's eye view with preset resolution, and marking grids occupied by the point cloud in the bird's eye view;
and determining unlabeled lattices, and obtaining the travelable areas not occupied by the objects according to the unlabeled lattices.
In one implementation manner, the selecting a position to be enhanced in the drivable area corresponding to each frame of point cloud, searching the rare object data meeting the conditions in the rare object database according to the position to be enhanced, and includes:
randomly selecting a position to be enhanced on a ground plane in a travelable area corresponding to each frame of point cloud;
querying a 3D bounding box nearest to the selected location from the rare object database according to the selected location;
and inquiring the corresponding rare object local image according to the inquired 3D bounding box.
In one implementation, the merging the rare object data into the corresponding image of the current scene to obtain the enhanced image data of the current scene includes:
projecting the queried 3D bounding box into a corresponding image of the current scene according to the external parameters and the internal parameters of the camera;
and obtaining a projection image area, and fusing the queried rare object local image with the corresponding image of the current scene in a weighted sum mode to obtain the enhanced image data of the current scene.
In a second aspect, the present invention provides an image data enhancement device based on a 3D drivable region, comprising:
a rare object database module for creating a rare object database;
the system comprises a drivable area determining module, a driving area determining module and a driving area determining module, wherein the drivable area determining module is used for acquiring multi-frame point clouds and multi-frame images of a current scene and determining a drivable area corresponding to each frame point cloud of the current scene; wherein, the multi-frame point cloud and multi-frame image of the current scene are data corresponding to each other in time;
the rare object data searching module is used for selecting a position to be enhanced in a drivable area corresponding to each frame of point cloud, and searching rare object data meeting the conditions in the rare object database according to the position to be enhanced;
and the image enhancement processing module is used for merging the rare object data into the corresponding image of the current scene to obtain the enhanced image data of the current scene.
In a third aspect, the present invention provides a terminal comprising: a processor and a memory storing a 3D drivable region-based image data enhancement program which, when executed by the processor, is operable to implement the operation of the 3D drivable region-based image data enhancement method as described in the first aspect.
In a fourth aspect, the present invention also provides a medium, which is a computer-readable storage medium storing a 3D drivable region-based image data enhancement program, which when executed by a processor, is configured to implement the operations of the 3D drivable region-based image data enhancement method as described in the first aspect.
The technical scheme adopted by the invention has the following effects:
the invention can determine the drivable area by acquiring the multi-frame point cloud and the multi-frame image of the current scene; and by selecting the position to be enhanced, the rare object data meeting the conditions can be searched in the rare object database, so that the rare object data is blended into the drivable area of the current scene image, and the enhanced scene image data is obtained. According to the invention, by constructing rare objects in the 3D drivable area, accurate 3D information of the rare objects can be obtained, and rare objects can be added in a reasonable area, so that unnecessary false detection is avoided; and by establishing a database, data of various sources are collected, so that the rare object database can be enriched rapidly, and the detection rate of the 3D detection model on rare objects is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to the structures shown in these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of an image data enhancement method based on a 3D travelable region in one implementation of the present invention.
FIG. 2 is a schematic diagram of a three-dimensional point cloud projected on a bird's eye view in one implementation of the invention.
Fig. 3 is a schematic view of a travelable region in one implementation of the invention.
Fig. 4 is a functional schematic of a terminal in one implementation of the invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more clear and clear, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Exemplary method
In the prior art, the method for improving the rare object detection performance is more effective in a data enhancement mode, wherein the representative scheme is that a mask which is proposed by google and utilizes marked objects is used for picking the objects from a current image, the objects are randomly scaled in a copy-paste mode and then inserted into random positions of other image frames, and a new image is constructed so as to improve the occurrence frequency of the marked objects in the model training process. However, the scheme aims at the problem that the scheme aims at an image 2D detection application scene, a stuck 2D object cannot acquire accurate 3D bounding box information, and an accurate 3D bounding box true value of a target object is required to be used as a learning target during training of an image 3D detection model, so that the application scene is limited; on the other hand, the mode that the scheme is randomly pasted at any position is easy to cause false detection, so that the accuracy of the marked object in the model training process is low.
Aiming at the technical problems, the embodiment of the invention provides an image data enhancement method based on a 3D (three-dimensional) drivable area, which can determine the drivable area by acquiring multi-frame point clouds and multi-frame images of a current scene; and by selecting the position to be enhanced, the rare object data meeting the conditions can be searched in the rare object database, so that the rare object data is blended into the drivable area of the current scene image, and the enhanced scene image data is obtained. Therefore, according to the embodiment of the invention, by constructing the rare object in the 3D drivable area, not only can accurate 3D information of the rare object be obtained, but also the rare object can be added in a reasonable area, so that unnecessary false detection is avoided; and by establishing a database, data of various sources are collected, so that the rare object database can be enriched rapidly, and the detection rate of the 3D detection model on rare objects is improved.
As shown in fig. 1, an embodiment of the present invention provides an image data enhancement method based on a 3D drivable region, including the steps of:
step S100, a rare object database is built.
In this embodiment, a virtual 3D bounding box of a rare object can be constructed on a ground plane in a 3D travelable area by means of a laser radar three-dimensional ranging sensor, then the 3D bounding box is projected into an image by using a camera aperture imaging model to obtain a projection area, and finally the projection area is fused with a local image of the rare object in a weighted sum manner, so that accurate 3D information (the 3D information is used for 3D detection training) is obtained, and rare objects are added in a reasonable area, thereby avoiding unnecessary false detection after training.
Moreover, the manner of constructing the rare object database proposed in the present embodiment may collect the partial images of various rare objects and the corresponding 3D bounding box information in the rare object database; the rare object information in the rare object database can be obtained from various ways such as massive unlabeled data, various public data sets, an internet database and the like, is not limited to the labeled data sets, and can theoretically contain any object.
It is worth mentioning that rare objects in this embodiment include, but are not limited to: triangular warning board, road construction warning board, inverted cone, carton, stone and other objects. Moreover, in other implementations of the present embodiment, the rare object may also be other moving or non-moving objects on the vehicle's travel path.
Specifically, in one implementation of the present embodiment, step S100 includes the steps of:
step S101, acquiring image frames and point cloud frames of rare objects; the image frames of the rare objects and the point cloud frames of the rare objects are data at the same moment after time synchronization processing;
step S102, acquiring a local image of the rare object in the image frame of the rare object according to a first instruction;
step S103, performing visualization processing on the point cloud frame of the rare object according to a second instruction, and acquiring 3D bounding box information corresponding to the rare object;
step S104, the rare object database is built according to the rare object local image and the corresponding 3D bounding box information.
In this embodiment, a rare object database needs to be built according to the acquired image frames and point cloud frames of the rare object, and the specific process is as follows:
first, image frames and point cloud frames containing rare objects are collected, wherein the image frames and the point cloud frames are data at the same moment after time synchronization processing, and it is ensured that two sensors see the scene at the same moment.
Then, in order to obtain the local image area of the rare object, the rare object in the image can be scratched by using the scratching software; wherein, the matting software includes but is not limited to: photoshop and other image processing software; of course, the image frame containing the rare object may also be segmented based on the deep learning model to obtain a local image containing the rare object.
It will be appreciated that when the rare object in the image is to be scratched, the lasso tool of Photoshop software may be used to obtain the mask region of the rare object through the first instruction (i.e. the instruction of the matting operation), that is, only the image region of the rare object is reserved, and other objects are excluded, so that the local image containing the rare object can be obtained.
Then, in order to acquire 3D bounding box information of the rare object, the 3D bounding box of the object can be marked by utilizing 3D marking software; wherein the 3D bounding box information includes: the location (x, y, z), size (length, width, height), orientation angle, etc. of rare objects in the real world.
It can be appreciated that when acquiring 3D bounding box information of rare objects, 3D labeling software is used to load point cloud frame data and perform visualization, and labeling personnel utilize a software editing function to add a 3D bounding box in a point cloud frame through a second instruction (i.e., a bounding box adding instruction).
Finally, by creating a database and recording the rare object partial images and 3D bounding box information in the database, a rare object database can be obtained.
It should be noted that the rare object database in this embodiment may provide a rich sample for subsequent data enhancement, and the data of the rare object database may be updated by massive unlabeled data, various public data sets, internet data, and the like.
As shown in fig. 1, in one implementation manner of the embodiment of the present invention, the image data enhancing method based on the 3D drivable area further includes the following steps:
step S200, acquiring multi-frame point clouds and multi-frame images of a current scene, and determining a drivable area corresponding to each frame point cloud of the current scene; and the multi-frame point cloud and the multi-frame image of the current scene are data corresponding to each other in time one by one.
In the embodiment, multiple-frame point clouds and multiple-frame images of the current scene are required to be acquired, then three-dimensional reconstruction is carried out by utilizing the multiple-frame point clouds, a ground point estimation ground plane equation is extracted, and the ground of the current scene is identified; meanwhile, a travelable region unoccupied by the object is obtained in the single-frame point cloud by utilizing a ray tracing algorithm, so that rare objects and virtual 3D bounding boxes thereof can be constructed in the travelable region later.
Specifically, in one implementation of the present embodiment, step S200 includes the steps of:
step S201, performing three-dimensional reconstruction according to the multi-frame point cloud of the current scene, extracting ground points according to the reconstructed three-dimensional model, and estimating and obtaining a ground plane equation according to the extracted ground points.
In the embodiment, three-dimensional reconstruction is performed by utilizing multi-frame point clouds, and then a ground point estimation ground plane equation is extracted; wherein estimating the ground plane equation includes, but is not limited to: RANSAC, etc. The reason for acquiring the ground plane equation by using the three-dimensional reconstruction is that the ground distance which can be scanned by a single-frame point cloud is limited, the ground at a far distance can be obtained by using the three-dimensional reconstruction, and the range of the distance which can be enhanced is enlarged.
In one implementation of the present embodiment, step S201 includes the steps of:
step S201a, extracting ground points in all point cloud frames of the current scene according to a ground segmentation algorithm;
step S201b, obtaining a current vehicle pose, and converting ground points in all point cloud frames to a world coordinate system according to the current vehicle pose to obtain a local dense point cloud map of a preset road section;
and step S201c, according to the local dense point cloud map, fitting by utilizing point cloud coordinates to obtain the ground plane equation.
In this embodiment, ground points in all the point cloud frames may be extracted by using a ground segmentation algorithm including, but not limited to, a line_group_segment (ground point cloud fast segmentation algorithm), a region_growing_segment (region growing segmentation algorithm), and the like, and after the ground points are obtained, the ground points of all the frames may be converted into a world coordinate system by using pose (position (x, y, z), orientation angle, and the like) obtained by a vehicle GPS, so as to obtain a local dense point cloud map of a certain road section.
After the ground point cloud map is obtained, a plane equation can be fitted with point cloud coordinates by using algorithms including, but not limited to, least squares, RANSAC and the like; wherein the parameters of the plane equation include a, B, C, D (ax+by+cz+d=0).
Considering that there is a fluctuation in the ground, the ground point cloud map may be divided into a plurality of sub-maps with a certain resolution, for example: 10m by 10m, estimating plane equations in each subgraph respectively; the reason why the GPS pose is utilized to construct the ground point cloud map of a certain road section is that the ground distance scanned by the laser radar at a certain moment is limited, and the ground information at a far distance can be obtained by reconstructing the pose of the vehicle, so that the distance range which can be enhanced is enlarged.
Specifically, in one implementation of the present embodiment, step S200 further includes the following steps:
step S202, obtaining a travelable area unoccupied by an object in each frame of point cloud of the current scene according to a ray tracing algorithm.
In this embodiment, a light ray tracing algorithm is used in a single-frame point cloud to obtain a travelable region unoccupied by an object, as shown in fig. 3, and the region indicated by a dashed circle is the travelable region.
In one implementation of the present embodiment, step S202 includes the steps of:
step S202a, filtering the extracted ground points and the points with preset height from the ground in each frame of point cloud of the current scene;
step S202b, projecting the filtered residual three-dimensional point cloud into a bird 'S-eye view with preset resolution, and marking grids occupied by the point cloud in the bird' S-eye view;
step S202c, determining unlabeled lattices, and obtaining the travelable regions not occupied by the objects according to the unlabeled lattices.
In this embodiment, in the single-frame point cloud, the ground points obtained in step S201 are filtered, and in addition, the points at a preset height (for example, 2 meters or more from the ground) are filtered in consideration of the influence of objects at heights such as trees. Then, the three-dimensional point cloud is projected into the bird's eye view with a certain resolution, and the lattice occupied by the point cloud is marked black. As shown in fig. 2, the process of obtaining the drivable area is:
and respectively judging whether the grids at the most edge in the aerial view are occupied by point clouds or not on the connecting line of the grids from the vehicle to the edge according to the arrow direction in the left image, if not, marking as Free (blank), and after the occupied grids appear on the connecting line, interrupting the judging process, and jumping to the next edge grid according to the arrow, so that a travelable area in the ground plane can be obtained.
As shown in fig. 1, in one implementation manner of the embodiment of the present invention, the image data enhancing method based on the 3D drivable area further includes the following steps:
and step S300, selecting a position to be enhanced in a drivable area corresponding to each frame of point cloud, and searching rare object data meeting the conditions in the rare object database according to the position to be enhanced.
In this embodiment, a location to be enhanced is randomly selected on a ground plane within a driving area, a 3D bounding box closest to the location is queried in a rare object database by using the selected location, and then a local image of a corresponding rare object in the rare object database is queried according to the 3D bounding box, so that the rare object data can be queried.
Specifically, in one implementation of the present embodiment, step S300 includes the steps of:
step S301, randomly selecting a position to be enhanced on a ground plane in a travelable area corresponding to each frame of point cloud;
step S302, inquiring a 3D bounding box nearest to the selected position from the rare object database according to the selected position;
step S303, inquiring the corresponding rare object local image according to the inquired 3D bounding box.
In this embodiment, the selected position to be enhanced is selected randomly from unoccupied positions, a distance is calculated by using the selected 3D coordinates and 3D coordinates of each sample in the rare object database, then, in the samples satisfying a distance threshold and having a space within the 3D bounding box completely falling within the drivable region, a 3D bounding box is selected randomly, and according to the randomly selected 3D bounding box, a local image of the corresponding rare object in the rare object database can be found.
As shown in fig. 1, in one implementation manner of the embodiment of the present invention, the image data enhancing method based on the 3D drivable area further includes the following steps:
step S400, the rare object data are merged into the corresponding image of the current scene, and the image data of the current scene after enhancement are obtained.
Specifically, in one implementation of the present embodiment, step S400 includes the following steps:
step S401, according to the parameters outside the camera and the parameters inside the camera, projecting the queried 3D bounding box into the corresponding image of the current scene;
step S402, obtaining a projection image area, and fusing the queried rare object local image with the corresponding image of the current scene in a weighted sum mode to obtain the enhanced image data of the current scene.
In this embodiment, projecting the queried 3D bounding box into the corresponding image of the current scene refers to projecting the point of the 3D bounding box into the corresponding image of the current scene, i.e. transforming the 3D coordinates into the image 2D coordinates. The projection area is determined by:
the method comprises the steps of transforming 8 angular point 3D coordinates of a 3D bounding box into a coordinate system of a corresponding image of a current scene by using an external reference matrix and an internal reference matrix of a camera, calculating minimum row coordinate values and minimum column coordinate values of 8 projection points, and enclosing a rectangular area which is a projection area of the 3D bounding box by using the minimum row coordinate values and the minimum column coordinate values and the maximum row coordinate values and the maximum column coordinate values.
Obtaining a projection image area, and then fusing the queried rare object local image with a corresponding image of the current scene by using a weighted sum mode, so as to generate a new image frame;
the specific process is as follows: after the projection area is obtained, the local image of the rare object corresponding to the 3D bounding box in the rare object database is scaled to be just put into the projection area, and then RBG value fusion is carried out by using the following formula:
R＝m*R0+(1-m)*R1
G＝m*G0+(1-m)*G1
B＝m*B0+(1-m)*B1
wherein R0/G0/B0 is the RGB value of the original image, R1/G1/B1 is the RGB value of the partial image of the rare object inserted, and m represents the specific gravity of the original image.
And after fusion, obtaining the image data of the enhanced current scene.
It is worth mentioning that in the present embodiment, if a plurality of rare objects need to be inserted within the same frame, the processes of determining the drivable region, querying the 3D bounding box, projecting the 3D bounding box, and fusing the partial images of the rare objects need to be repeated, i.e., the drivable region needs to be recalculated every time one rare object is added.
The following technical effects are achieved through the technical scheme:
according to the embodiment, by constructing rare objects in the 3D drivable area, accurate 3D information of the rare objects can be obtained, and the rare objects can be added in a reasonable area, so that unnecessary false detection is avoided; and by establishing a database, data of various sources are collected, so that the rare object database can be enriched rapidly, and the detection rate of the 3D detection model on rare objects is improved.
Exemplary apparatus
Based on the above embodiment, the present invention further provides an image data enhancement device based on a 3D drivable area, including:
a rare object database module for creating a rare object database;
the system comprises a drivable area determining module, a driving area determining module and a driving area determining module, wherein the drivable area determining module is used for acquiring multi-frame point clouds and multi-frame images of a current scene and determining a drivable area corresponding to each frame point cloud of the current scene; wherein, the multi-frame point cloud and multi-frame image of the current scene are data corresponding to each other in time;
the rare object data searching module is used for selecting a position to be enhanced in a drivable area corresponding to each frame of point cloud, and searching rare object data meeting the conditions in the rare object database according to the position to be enhanced;
and the image enhancement processing module is used for merging the rare object data into the corresponding image of the current scene to obtain the enhanced image data of the current scene.
The following technical effects are achieved through the technical scheme:
according to the embodiment, by constructing rare objects in the 3D drivable area, accurate 3D information of the rare objects can be obtained, and the rare objects can be added in a reasonable area, so that unnecessary false detection is avoided; and by establishing a database, data of various sources are collected, so that the rare object database can be enriched rapidly, and the detection rate of the 3D detection model on rare objects is improved.
Based on the above embodiment, the present invention also provides a terminal, and a functional block diagram thereof may be shown in fig. 4.
The terminal comprises: the system comprises a processor, a memory, an interface, a display screen and a communication module which are connected through a system bus; wherein the processor of the terminal is configured to provide computing and control capabilities; the memory of the terminal comprises a storage medium and an internal memory; the storage medium stores an operating system and a computer program; the internal memory provides an environment for the operation of the operating system and computer programs in the storage medium; the interface is used for connecting with external equipment; the display screen is used for displaying corresponding information; the communication module is used for communicating with a cloud server or other devices.
The computer program is executed by a processor to implement the operations of a 3D travelable region-based image data enhancement method.
It will be appreciated by those skilled in the art that the functional block diagram shown in fig. 4 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the terminal to which the present inventive arrangements may be applied, and that a particular terminal may include more or less components than those shown, or may combine some of the components, or have a different arrangement of components.
In one embodiment, a terminal is provided, including: the image data enhancement device comprises a processor and a memory, wherein the memory stores an image data enhancement program based on a 3D drivable region, and the image data enhancement program based on the 3D drivable region is used for realizing the operation of the image data enhancement method based on the 3D drivable region.
In one embodiment, a storage medium is provided, wherein the storage medium stores a 3D drivable region-based image data enhancement program, which when executed by a processor is operative to implement the 3D drivable region-based image data enhancement method as above.
Those skilled in the art will appreciate that implementing all or part of the above-described methods may be accomplished by way of a computer program comprising instructions for the relevant hardware, the computer program being stored on a non-volatile storage medium, the computer program when executed comprising the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory.
In summary, the invention provides an image data enhancement method and device based on a 3D drivable region, wherein the method comprises the following steps: establishing a rare object database; acquiring multi-frame point clouds and multi-frame images of a current scene, and determining a drivable area corresponding to each frame point cloud of the current scene; wherein, the multi-frame point cloud and multi-frame image of the current scene are data corresponding to each other in time; selecting a position to be enhanced in a drivable area corresponding to each frame of point cloud, and searching rare object data meeting a condition in the rare object database according to the position to be enhanced; and merging the rare object data into the corresponding image of the current scene to obtain the enhanced image data of the current scene. According to the invention, the rare object is constructed in the 3D drivable area, so that the detection accuracy of the 3D detection model on the rare object is improved.
It is to be understood that the invention is not limited in its application to the examples described above, but is capable of modification and variation in light of the above teachings by those skilled in the art, and that all such modifications and variations are intended to be included within the scope of the appended claims.
Claims (10)
1. A method for enhancing image data based on a 3D drivable region, comprising:
establishing a rare object database;
acquiring multi-frame point clouds and multi-frame images of a current scene, and determining a drivable area corresponding to each frame point cloud of the current scene; wherein, the multi-frame point cloud and multi-frame image of the current scene are data corresponding to each other in time;
selecting a position to be enhanced in a drivable area corresponding to each frame of point cloud, and searching rare object data meeting a condition in the rare object database according to the position to be enhanced;
and merging the rare object data into the corresponding image of the current scene to obtain the enhanced image data of the current scene.
2. The 3D travelable region-based image data enhancing method as claimed in claim 1, wherein the creating of the rare object database comprises:
acquiring an image frame and a point cloud frame of a rare object; the image frames of the rare objects and the point cloud frames of the rare objects are data at the same moment after time synchronization processing;
acquiring a rare object local image in an image frame of the rare object according to a first instruction;
performing visualization processing on the point cloud frame of the rare object according to a second instruction, and acquiring 3D bounding box information corresponding to the rare object;
and establishing the rare object database according to the rare object local image and the corresponding 3D bounding box information.
3. The 3D drivable region-based image data enhancement method as set forth in claim 1, wherein the determining a drivable region corresponding to each frame point cloud of the current scene comprises:
performing three-dimensional reconstruction according to the multi-frame point cloud of the current scene, extracting ground points according to the reconstructed three-dimensional model, and estimating and obtaining a ground plane equation according to the extracted ground points;
and according to a ray tracing algorithm, obtaining a travelable area unoccupied by the object in each frame of point cloud of the current scene.
4. The method for enhancing image data based on 3D travelable region as claimed in claim 3, wherein the three-dimensional reconstruction is performed according to multi-frame point clouds of the current scene, ground points are extracted according to the reconstructed three-dimensional model, and a ground plane equation is estimated according to the extracted ground points, comprising:
extracting ground points in all point cloud frames of the current scene according to a ground segmentation algorithm;
acquiring a current vehicle pose, converting ground points in all point cloud frames into a world coordinate system according to the current vehicle pose, and acquiring a local dense point cloud map of a preset road section;
and fitting by utilizing point cloud coordinates according to the local dense point cloud map to obtain the ground plane equation.
5. A 3D travelable region-based image data enhancing method as claimed in claim 3, wherein said obtaining a travelable region unoccupied by an object in each frame of point cloud of the current scene comprises:
filtering the extracted ground points and the points with preset height from the ground in each frame of point cloud of the current scene;
projecting the filtered residual three-dimensional point cloud into a bird's eye view with preset resolution, and marking grids occupied by the point cloud in the bird's eye view;
and determining unlabeled lattices, and obtaining the travelable areas not occupied by the objects according to the unlabeled lattices.
6. The method for enhancing image data based on 3D drivable region according to claim 1, wherein selecting a position to be enhanced in the drivable region corresponding to each frame of point cloud, and searching for rare object data satisfying a condition in the rare object database according to the position to be enhanced comprises:
randomly selecting a position to be enhanced on a ground plane in a travelable area corresponding to each frame of point cloud;
querying a 3D bounding box nearest to the selected location from the rare object database according to the selected location;
and inquiring the corresponding rare object local image according to the inquired 3D bounding box.
7. The method for enhancing image data based on 3D drivable region according to claim 1, wherein the merging the rare object data into the corresponding image of the current scene to obtain the enhanced image data of the current scene comprises:
projecting the queried 3D bounding box into a corresponding image of the current scene according to the external parameters and the internal parameters of the camera;
and obtaining a projection image area, and fusing the queried rare object local image with the corresponding image of the current scene in a weighted sum mode to obtain the enhanced image data of the current scene.
8. An image data enhancing apparatus based on a 3D drivable region, comprising:
a rare object database module for creating a rare object database;
the system comprises a drivable area determining module, a driving area determining module and a driving area determining module, wherein the drivable area determining module is used for acquiring multi-frame point clouds and multi-frame images of a current scene and determining a drivable area corresponding to each frame point cloud of the current scene; wherein, the multi-frame point cloud and multi-frame image of the current scene are data corresponding to each other in time;
the rare object data searching module is used for selecting a position to be enhanced in a drivable area corresponding to each frame of point cloud, and searching rare object data meeting the conditions in the rare object database according to the position to be enhanced;
and the image enhancement processing module is used for merging the rare object data into the corresponding image of the current scene to obtain the enhanced image data of the current scene.
9. A terminal, comprising: a processor and a memory storing a 3D drivable region-based image data enhancement program which, when executed by the processor, is operable to implement the 3D drivable region-based image data enhancement method as claimed in any one of claims 1 to 7.
10. A computer-readable storage medium storing an image data enhancing program based on a 3D drivable region, which when executed by a processor is operable to implement the 3D drivable region-based image data enhancing method as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311356907.7A CN117495692A (en) | 2023-10-18 | 2023-10-18 | Image data enhancement method and device based on 3D (three-dimensional) drivable area |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311356907.7A CN117495692A (en) | 2023-10-18 | 2023-10-18 | Image data enhancement method and device based on 3D (three-dimensional) drivable area |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117495692A true CN117495692A (en) | 2024-02-02 |
Family
ID=89679119
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311356907.7A Pending CN117495692A (en) | 2023-10-18 | 2023-10-18 | Image data enhancement method and device based on 3D (three-dimensional) drivable area |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117495692A (en) |
-
2023
- 2023-10-18 CN CN202311356907.7A patent/CN117495692A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10964054B2 (en) | Method and device for positioning | |
| CN110869974B (en) | Point cloud processing method, equipment and storage medium | |
| CN109470254B (en) | Map lane line generation method, device, system and storage medium | |
| CN105512646B (en) | A kind of data processing method, device and terminal | |
| CN113421289B (en) | High-precision vehicle track data extraction method for overcoming unmanned aerial vehicle shooting disturbance | |
| US20220044474A1 (en) | Method for constructing grid map by using binocular stereo camera | |
| CN111338382B (en) | Unmanned aerial vehicle path planning method guided by safety situation | |
| KR20120132704A (en) | Bird's-eye image forming device, bird's-eye image forming method, and recording medium | |
| CN112753038B (en) | Method and device for identifying lane change trend of vehicle | |
| CN111402414A (en) | Point cloud map construction method, device, equipment and storage medium | |
| CN113989450A (en) | Image processing method, image processing apparatus, electronic device, and medium | |
| CN113240734B (en) | Vehicle cross-position judging method, device, equipment and medium based on aerial view | |
| CN113763569B (en) | Image labeling method and device used in three-dimensional simulation and electronic equipment | |
| CN112700486B (en) | Method and device for estimating depth of road surface lane line in image | |
| CN114089330B (en) | Indoor mobile robot glass detection and map updating method based on depth image restoration | |
| US20230222688A1 (en) | Mobile device positioning method and positioning apparatus | |
| CN113096181B (en) | Method and device for determining equipment pose, storage medium and electronic device | |
| CN116978010A (en) | Image labeling method and device, storage medium and electronic equipment | |
| CN114140592A (en) | High-precision map generation method, device, equipment, medium and automatic driving vehicle | |
| CN113223064A (en) | Method and device for estimating scale of visual inertial odometer | |
| CN115077563A (en) | Vehicle positioning accuracy evaluation method and device and electronic equipment | |
| CN114565906A (en) | Obstacle detection method, obstacle detection device, electronic device, and storage medium | |
| CN117495692A (en) | Image data enhancement method and device based on 3D (three-dimensional) drivable area | |
| CN113256756B (en) | Map data display method, device, equipment and storage medium | |
| KR102540634B1 (en) | Method for create a projection-based colormap and computer program recorded on record-medium for executing method therefor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |