CN117854046B - An integrated positioning system and device based on visual fusion - Google Patents

Abstract

The present invention relates to the field of vehicle positioning, and specifically to an integrated positioning system and device based on visual fusion. It includes an image acquisition module, an image processing module and an image grading module; the image acquisition module is used to acquire an image of an object within the image acquisition direction range; the image processing module is communicatively connected to the image acquisition module, and is used to receive the image data acquired by the image acquisition module, and extract the main structure line of the object in the image; the image grading module is communicatively connected to the image processing module, and is used to receive the main structure line of the object in the image, and place the main structure line of the object in the image in a pre-set driving safety grading positioning range map, and the driving safety grading positioning range map includes a safe area, a warning area and a dangerous area, and the safe area, the warning area and the dangerous area are all fan-shaped areas, and two warning areas and two dangerous areas are set. The present invention can also determine whether the platform vehicle is in a safe driving state based only on the acquired image.

Description

An integrated positioning system and device based on visual fusion

Technical Field

The present invention relates to the field of vehicle positioning, and in particular to an integrated positioning system and device based on vision fusion.

Background technique

The intelligent mobile platform is a comprehensive system that integrates multiple functions such as environmental perception, dynamic decision-making and planning, behavior control and execution. The intelligent mobile platform includes platforms such as intelligent vehicles and intelligent robots. Environmental perception is the basic requirement and prerequisite for the intelligent mobile platform to make safe decisions and control, and can provide effective data support for the safe driving of the platform, such as installing radars and cameras on the platform. In the field of logistics and distribution, unmanned intelligent logistics vehicles are intelligent mobile platforms. The vehicles drive autonomously, automatically avoid obstacles, and can safely deliver goods to the destination.

Chinese patent publication number CN112731371A discloses an integrated target tracking system and method of laser radar and vision fusion. The system includes a solid-state laser radar, a monocular vision sensor and a fusion tracker. The fusion tracker includes a monocular vision target detection module, a laser radar target detection module, a laser radar-vision fusion tracking module and a communication module. The monocular vision target detection module obtains target information from the image; the laser radar target detection module obtains target information from the point cloud; the laser radar-vision fusion tracking module performs spatial registration on the measurement, establishes a target state model and a measurement model, establishes a tracking gate through one-step prediction of the target state, and then completes the fusion tracking of the image target and the point cloud target through data association and target state filtering, thereby improving the integration and development efficiency of the intelligent mobile platform and the accuracy of the fusion tracking results.

However, the above technical solution has the following shortcomings:

Environmental perception requires the coordinated operation of multiple devices such as solid-state lidar, monocular vision sensors and fusion trackers. When the lidar or vision sensor fails or cannot collect data normally, the entire system will not be able to operate normally, and the operating stability and reliability will be low.

Summary of the invention

The purpose of the present invention is to address the problems existing in the background technology and to propose an integrated positioning system and device based on visual fusion that can determine whether a platform vehicle is in a safe driving state based only on the collected images, with higher operating stability and reliability.

On the one hand, the present invention proposes an integrated positioning device based on visual fusion, which includes an image acquisition module, an image processing module and an image grading module; the image acquisition module is used to acquire an image of an object within the image acquisition direction range; the image processing module is communicated with the image acquisition module, and is used to receive the image data acquired by the image acquisition module and extract the main structure line of the object in the image; the image grading module is communicated with the image processing module, and is used to receive the main structure line of the object in the image, and place the main structure line of the object in the image in a pre-set driving safety grading positioning range map, and the driving safety grading positioning The range map includes a safe zone, a warning zone and a dangerous zone. The safe zone, the warning zone and the dangerous zone are all fan-shaped areas. Two warning zones and two dangerous zones are set. The two warning zones are respectively connected at the two ends of the safe zone, and the two dangerous zones are respectively connected at one end of each warning zone far from the safe zone. When the main structural lines of the objects in the image are all in the safe zone, the vehicle driving state is determined to be a safe driving state; when the main structural lines of the objects in the image invade the warning zone but do not invade the dangerous zone, the vehicle driving state is determined to be a warning driving state; when the main structural lines of the objects in the image invade the dangerous zone, the vehicle driving state is determined to be a dangerous driving state.

Preferably, the image acquisition module includes front and rear vision cameras, side vision cameras, a bracket and a mounting cover. The collecting directions of the front and rear vision cameras and the side vision cameras are vertical. The front and rear vision cameras face forward or backward, and the side vision cameras face left or right. The front and rear vision cameras and the side vision cameras are all arranged on the bracket, and the bracket is arranged on the mounting cover.

Preferably, the image acquisition ranges of the front and rear vision cameras and the side vision cameras are both conical ranges.

On the other hand, the present invention proposes an integrated positioning system based on visual fusion, which includes a radar ranging module, an image fusion module and the above-mentioned integrated positioning device based on visual fusion; four groups of image acquisition modules are installed in a rectangular shape on the top of the target vehicle, including two front and rear visual cameras facing the front of the vehicle, two front and rear visual cameras facing the rear of the vehicle, two side visual cameras facing the left side of the vehicle and two side visual cameras facing the right side of the vehicle, and a total of eight radar ranging modules are arranged, and two radar ranging modules are installed on the mounting cover in each group of image acquisition modules, and the orientations of the two radar ranging modules are respectively the same as the orientations of the front and rear visual cameras and the side visual cameras in the group of image acquisition modules;

The image processing module is in communication connection with the image fusion module, and is used to transmit the main structure line data of the object in the image to the image fusion module;

The radar ranging module is connected to the image fusion module for collecting ranging dot matrix data of the object in the target direction and transmitting the ranging dot matrix data to the image fusion module;

For two front and rear vision cameras or two side vision cameras facing the same side of the vehicle, after the image processing module extracts the main structure lines of the objects in the images based on the image data collected by the two front and rear vision cameras or the two side vision cameras, the image fusion module is used to fuse the main structure line data of the objects in the two images with the ranging dot matrix data collected by the radar ranging module in the corresponding direction into an integrated image of the structure lines and ranging dot matrix.

Preferably, the image fusion module includes a point-line fusion module and a transformation fusion module;

The point-line fusion module is used to perform point-line fusion on the main structure lines of the objects in the image extracted based on the image data collected by the two front and rear vision cameras or the two side vision cameras facing the same side of the vehicle and the ranging dot matrix data collected by the radar ranging module in the corresponding direction, so as to obtain two sets of point-line fusion images;

The transformation fusion module is communicatively connected with the point-line fusion module, and is used for receiving the point-line fusion images, dividing the two groups of point-line fusion images into an image maintenance area distributed outerwardly and an image to-be-fused area distributed innerwardly, transforming the same object images in the two inner-distributed image to-be-fused areas of the two groups of point-line fusion images into object reconstructed images in the image fusion area, and connecting the two outer-distributed image maintenance areas of the two groups of point-line fusion images to both sides of the image fusion area, respectively, to form an integrated image of structure lines and ranging dot matrix.

Preferably, when the transformation fusion module performs image transformation on the same object in the areas to be fused of two images, the maximum lateral size of the object in each image fusion area is used as the transformation lateral size, and the maximum vertical size of the object in each image fusion area is used as the transformation vertical size, and the forward display size of the reconstructed image of the object is determined based on the transformation lateral size and the transformation vertical size.

Preferably, it also includes an image output module, which is communicatively connected to the image fusion module and is used to receive the integrated image data of the structure line and the ranging dot matrix and output it visually.

Preferably, it also includes a global positioning navigation module and an inertial navigation module installed on the target vehicle, the global positioning navigation module locates the vehicle position in real time, and the inertial navigation module detects the speed and position of the vehicle.

Compared with the prior art, the present invention has the following beneficial technical effects:

The present invention can determine whether the platform vehicle is in a safe driving state only based on the image collected by the integrated positioning device based on visual fusion, and can also work together with the radar ranging module, with high operating stability and reliability. When not used in combination with the radar ranging module, the image grading module places the main structure line of the object in the image in a pre-set driving safety grading positioning range diagram for comparison, and determines whether the distance between the platform vehicle and the object is within a safe range. When used in combination with the radar ranging module, the main structure line data of the object in the two images and the ranging dot matrix data collected by the radar ranging module in the corresponding direction are correspondingly fused into an integrated image of the structure line and the ranging dot matrix through the image fusion module, thereby obtaining an integrated image including both image data and distance data, which can more intuitively and accurately know the distance between the object and the vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a system structure block diagram of an integrated positioning system based on visual fusion according to an embodiment of the present invention;

FIG2 is a structural block diagram of an image fusion module in an embodiment of the present invention;

FIG3 is a schematic diagram of the installation position of the image acquisition module on the vehicle in an embodiment of the present invention;

FIG4 is a schematic diagram of the structure of an image acquisition module in an embodiment of the present invention;

FIG5 is a schematic diagram of the angle range in which the image classification module performs classification and positioning for driving safety;

FIG6 is a schematic diagram of a point-line fusion module fusing an object structure line and an object ranging dot matrix to obtain a point-line fusion image;

FIG. 7 is a schematic diagram showing that the transformation fusion module transforms a point-line fusion image into a forward viewing angle fusion image.

Figure numerals: 100, image acquisition module; 1, front and rear vision cameras; 2, side vision cameras; 3, bracket; 4, mounting cover.

Detailed ways

Embodiment 1

As shown in FIG. 1 to FIG. 7 , this embodiment proposes an integrated positioning system based on vision fusion, which includes a radar ranging module, an image fusion module and an integrated positioning device based on vision fusion.

The integrated positioning device based on visual fusion includes an image acquisition module 100, an image processing module and an image grading module. The image acquisition module 100 is used to acquire the image of the object within the image acquisition direction range. The image acquisition module 100 includes a front and rear vision camera 1, a side vision camera 2, a bracket 3 and a mounting cover 4. The acquisition directions of the front and rear vision cameras 1 and the side vision cameras 2 are vertical. The front and rear vision cameras 1 face forward or backward, and the side vision camera 2 faces left or right. The front and rear vision cameras 1 and the side vision cameras 2 are both arranged on the bracket 3, and the bracket 3 is arranged on the mounting cover 4. The mounting cover 4 can be installed on the roof of the target vehicle. The image acquisition range of the front and rear vision cameras 1 and the side vision cameras 2 are both conical ranges, and rectangular images of a predetermined size can be acquired within the conical range.

The image processing module is connected in communication with the image acquisition module 100, and is used to receive the image data collected by the image acquisition module 100, and extract the main structure line of the object in the image. The image grading module is connected in communication with the image processing module, and is used to receive the main structure line of the object in the image, and place the main structure line of the object in the image in a pre-set driving safety grading positioning range map, and the driving safety grading positioning range map includes a safe area, a warning area and a dangerous area. The safe area, the warning area and the dangerous area are all fan-shaped areas. There are two warning areas and two dangerous areas. The two warning areas are connected at both ends of the safe area, and the two dangerous areas are connected at one end of each warning area away from the safe area. As shown in Figure 5, the safe area is a fan-shaped area with an angle of α, the warning area is two fan-shaped areas with an angle of β, and the dangerous area is two fan-shaped areas with an angle of γ. These fan-shaped areas are all centered on the visual camera. When the main structural lines of the objects in the image are all in the safe zone, the vehicle driving state is judged to be a safe driving state; when the main structural lines of the objects in the image invade the warning zone but do not invade the danger zone, the vehicle driving state is judged to be a warning driving state; when the main structural lines of the objects in the image invade the danger zone, the vehicle driving state is judged to be a dangerous driving state. This judgment method is based on the position of the main structural lines of the object. Without directly measuring the distance to the object, it can still realize the judgment of vehicle driving safety and grade and characterize the vehicle safety in different situations.

Four groups of image acquisition modules 100 are installed in a rectangular shape on the top of the target vehicle, including two front and rear vision cameras 1 facing the front of the vehicle, two front and rear vision cameras 1 facing the rear of the vehicle, two side vision cameras 2 facing the left side of the vehicle and two side vision cameras 2 facing the right side of the vehicle. A total of eight radar ranging modules are arranged. Two radar ranging modules are installed on the mounting cover 4 in each group of image acquisition modules 100. The directions of the two radar ranging modules are respectively the same as the directions of the front and rear vision cameras 1 and the side vision cameras 2 in the group of image acquisition modules 100.

The image processing module is connected to the image fusion module for transmitting the main structure line data of the object in the image to the image fusion module. The radar ranging module is connected to the image fusion module for collecting the ranging dot matrix data of the object in the target direction and transmitting the ranging dot matrix data to the image fusion module.

For two front and rear vision cameras 1 or two side vision cameras 2 facing the same side of the vehicle, after the image processing module extracts the main structure lines of the objects in the images based on the image data collected by the two front and rear vision cameras 1 or the two side vision cameras 2, the image fusion module is used to fuse the main structure line data of the objects in the two images with the ranging dot matrix data collected by the radar ranging module in the corresponding direction into an integrated image of the structure lines and ranging dot matrix.

The integrated positioning system based on visual fusion also includes an image output module, a global positioning navigation module and an inertial navigation module installed on the target vehicle. The image output module is connected to the image fusion module for receiving the integrated image data of the structure line and the distance measurement dot matrix and outputting it visually, so that the displayed data is more intuitive. The global positioning navigation module locates the vehicle position in real time, and the inertial navigation module detects the speed and position of the vehicle, further improving the accuracy of positioning.

This embodiment can determine whether the platform vehicle is in a safe driving state only based on the image collected by the integrated positioning device based on visual fusion, and can also work together with the radar ranging module, with high operating stability and reliability. When used without the radar ranging module, for the image data collected by the image acquisition module 100, the image processing module extracts the main structure line of the object in the image from the image data, and the image grading module then places the main structure line of the object in the image in a pre-set driving safety grading positioning range diagram for comparison. When the main structure line of the object in the image is completely in the safe area, it indicates that the distance between the platform vehicle and the object is within the safe range, and the platform vehicle can drive normally; when the main structure line of the object in the image invades the warning area and does not invade the danger zone, it indicates that the distance between the vehicle and the object is small, and attention should be paid to the driving speed to prevent the speed from being too high and unable to stop in time; when the main structure line of the object in the image invades the danger zone, the vehicle driving state is determined to be a dangerous driving state, indicating that the distance between the vehicle and the object is further reduced to a degree that may affect driving safety, and the vehicle needs to slow down or even stop to expand the distance with the object to ensure the safety of vehicle driving. When used in conjunction with a radar ranging module, the image fusion module fuses the main structural line data of objects in the two images with the ranging dot matrix data collected by the radar ranging module in the corresponding direction into an integrated image of structural lines and ranging dot matrixes, thereby obtaining an integrated image that includes both image data and distance data. This enables a more intuitive and accurate understanding of the distance between objects and vehicles, providing data support for safe vehicle driving.

Embodiment 2

As shown in FIG. 1 to FIG. 7 , this embodiment proposes an integrated positioning system based on visual fusion. Compared with the first embodiment, in this embodiment, the image fusion module includes a point-line fusion module and a transformation fusion module. The operation principles of the point-line fusion module and the transformation fusion module are as follows:

1. The point-line fusion module is used to perform point-line fusion of the main structure line of the object in the image extracted based on the image data collected by the two front and rear vision cameras 1 or the two side vision cameras 2 facing the same side of the vehicle with the ranging dot matrix data collected by the radar ranging module in the corresponding direction, so as to obtain two sets of point-line fusion images, and fuse the ranging dot matrix with the real main structure line of the object, as shown in Figure 6.

2. The transformation fusion module is connected to the point-line fusion module for receiving the point-line fusion images, dividing the two groups of point-line fusion images into an image maintenance area distributed outside and an image to be fused area distributed inside, transforming the same object images in the two image to be fused areas distributed inside in the two groups of point-line fusion images into object reconstruction images in the image fusion area, and connecting the two image maintenance areas distributed outside in the two groups of point-line fusion images to the two sides of the image fusion area, respectively, to form a structure line and ranging dot matrix integrated image, as shown in Figure 7. Because the visual cameras facing the same side are located at different positions on the top of the vehicle, the images collected have perspective, so for the same object in the two image to be fused areas, it is necessary to determine a unified front view image based on two different perspectives, and use the front view image as part of the final structure line and ranging dot matrix integrated image, and then combine the images in the image maintenance areas on both sides for connection, and jointly form a structure line and ranging dot matrix integrated image, so as to obtain an overall image including the image acquisition front and both sides of the acquisition direction.

When the transformation fusion module performs image transformation on the same object in the areas to be fused of two images, the maximum horizontal size of the object in the fusion areas of each image is used as the transformation horizontal size, and the maximum vertical size of the object in the fusion areas of each image is used as the transformation vertical size. The forward display size of the reconstructed image of the object is determined based on the transformation horizontal size and the transformation vertical size. The size of the reconstructed image is maximized, the sensitivity for judging the safe driving of the vehicle is higher, and the driving safety of the vehicle is better.

The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.

Claims (5)

1. The integrated positioning system based on visual fusion is characterized by comprising a radar ranging module, an image fusion module and an integrated positioning device based on visual fusion;

The integrated positioning device based on visual fusion comprises:

An image acquisition module (100) for acquiring an image of an object in a range of image acquisition directions;

the image processing module is in communication connection with the image acquisition module (100) and is used for receiving the image data acquired by the image acquisition module (100) and extracting a main structure line of an object in the image;

The image grading module is in communication connection with the image processing module and is used for receiving a main body structural line of an object in the image, and placing the main body structural line of the object in the image in a preset driving safety grading positioning range diagram, wherein the driving safety grading positioning range diagram comprises a safety zone, a warning zone and a dangerous zone, the safety zone, the warning zone and the dangerous zone are all sector-shaped areas, the warning zone and the dangerous zone are respectively provided with two warning zones, the two warning zones are respectively connected to two ends of the safety zone, and the two dangerous zones are respectively connected to one end of each warning zone far away from the safety zone; when the structural lines of the objects in the image main body are all in the safety zone, judging the running state of the vehicle as a safety running state; when the main body structural line of the object in the image invades the guard zone and does not invade the dangerous zone, judging the running state of the vehicle as the guard running state; when the main body structural line of the object in the image invades into the dangerous area, judging the running state of the vehicle as a dangerous running state;

the image acquisition module (100) comprises a front-back vision camera (1), a lateral vision camera (2), a bracket (3) and a mounting cover (4), wherein the acquisition directions of the front-back vision camera (1) and the lateral vision camera (2) are vertical, the front-back vision camera (1) faces forwards or backwards, the lateral vision camera (2) faces leftwards or rightwards, the front-back vision camera (1) and the lateral vision camera (2) are both arranged on the bracket (3), and the bracket (3) is arranged on the mounting cover (4);

The image acquisition modules (100) are rectangular in top of a target vehicle and comprise two front and rear vision cameras (1) facing the front of the vehicle, two front and rear vision cameras (1) facing the rear of the vehicle, two lateral vision cameras (2) facing the left side of the vehicle and two lateral vision cameras (2) facing the right side of the vehicle, eight radar ranging modules are arranged, two radar ranging modules are arranged on an installation cover (4) in each group of image acquisition modules (100), and the directions of the two radar ranging modules are respectively the same as the directions of the front and rear vision cameras (1) and the lateral vision cameras (2) in the group of image acquisition modules (100);

the image processing module is in communication connection with the image fusion module and is used for transmitting main structure line data of an object in the image to the image fusion module;

the radar ranging module is in communication connection with the image fusion module and is used for acquiring ranging lattice data of the object in the target direction and transmitting the ranging lattice data to the image fusion module;

For two front and back vision cameras (1) or two lateral vision cameras (2) facing the same side of the vehicle, after an image processing module extracts main structural lines of objects in images according to image data collected by the two front and back vision cameras (1) or the two lateral vision cameras (2), an image fusion module is used for correspondingly fusing the main structural lines of the objects in the two images with distance measurement lattice data collected by a radar distance measurement module facing the corresponding direction to form a structural line and distance measurement lattice integrated image;

the image fusion module comprises a dotted line fusion module and a transformation fusion module;

The point-line fusion module is used for carrying out point-line fusion on main structure lines of objects in images extracted according to image data collected by the two front-back vision cameras (1) or the two lateral vision cameras (2) facing the same side of the vehicle and distance measurement lattice data collected by the radar ranging module facing the corresponding direction to obtain two groups of point-line fusion images;

The transformation fusion module is in communication connection with the point-line fusion module and is used for receiving the point-line fusion images, dividing the two groups of point-line fusion images into an image maintenance area which is distributed outwards and an image area to be fused which is distributed inwards, transforming the same object images in the two image areas to be fused in the two groups of point-line fusion images into object reconstruction images in the image fusion area, and respectively connecting the two image maintenance areas which are distributed outwards in the two groups of point-line fusion images on two sides of the image fusion area to form a structural line and distance measurement lattice integrated image.

2. The integrated positioning system based on visual fusion according to claim 1, wherein the image acquisition ranges of the front-rear visual camera (1) and the lateral visual camera (2) are conical ranges.

3. The integrated positioning system based on visual fusion according to claim 1, wherein the transformation fusion module uses the maximum transverse dimension of the object in each image fusion area as a transformation transverse dimension and uses the maximum vertical dimension of the object in each image fusion area as a transformation vertical dimension when performing image transformation on the same object in the two image fusion areas, and determines the forward display dimension of the reconstructed image of the object according to the transformation transverse dimension and the transformation vertical dimension.

4. The integrated positioning system based on visual fusion according to claim 1, further comprising an image output module, wherein the image output module is in communication connection with the image fusion module and is used for receiving and visually outputting the structural line and distance measurement lattice integrated image data.

5. The vision-fusion-based integrated positioning system of claim 1, further comprising a global positioning navigation module and an inertial navigation module mounted on the target vehicle, the global positioning navigation module locating the vehicle position in real time, the inertial navigation module detecting the speed and position of the vehicle.