TWI760128B - Method and system for generating depth image and positioning system using the method - Google Patents

Abstract

The present invention is a generation method and a generation system for a depth image. A pixel image with a relative depth and a radar image with a measured depth is used to generate a magnification, and the pixel image is corrected by the magnification to generate a depth image. That is, the accurate depth is obtained by the fusion of the pixel image and the radar image. In addition, a corresponding positioning system is constructed by using the above method and system in the present invention. The depth image generated by the generation system for a depth image is positioned by the synchronous positioning and map construction module to achieve the purpose of positioning using depth images with absolute depth.

Description

Method and system for generating depth image and positioning system using the method

The present invention relates to an image generation technology and a positioning method, in particular to a depth image generation method and system, and a positioning system applying the method.

In recent years, technologies such as robots and self-driving cars (hereinafter referred to as self-driving cars) have become more and more mature. Taking self-driving cars as an example, it can be seen that self-driving cars are driving on the road in some specific areas. They must rely on the assistance of accurate positioning technology to move smoothly on the predetermined route, so as to avoid self-driving cars from deviating from the predetermined route or self-driving cars. Many accidents are caused by driving away from the intended route, so positioning technology is really important for the driving of self-driving cars.

The positioning technology that has attracted the most attention recently is Simultaneous Localization and Mapping (hereinafter referred to as SLAM). With its precise positioning function, it does not need to rely on the Global Positioning System (GPS). Features, make SLAM technology can be used in outdoor or non-outdoor places, and its application range is wider. General SLAM technology usually needs to set up light detection and ranging (LIDAR) as a sensing device to accurately detect Measuring the distance of objects in the environment and judging the depth of the image, but the LiDAR equipment is expensive, so the current vision SLAM (Vision SLAM) technology is looking forward to replace it.

The visual SLAM technology mainly uses the camera as the sensor, and the current camera has the advantages of low cost, and the erection, setting and calibration are relatively simple, which makes the visual SLAM technology gradually attract more attention. However, there are still some problems that need to be overcome in the visual SLAM technology. One of the problems is that its algorithm is relatively complex and the amount of calculation is relatively large.

In order to solve the above problems, the current visual SLAM technology, in terms of image processing, is roughly divided into the direct method and the feature-based method according to the size of the pixel area used. The method directly uses the entire image to locate and estimate the camera pose, which has the advantage of being less time-consuming, but the direct method is sensitive to changes in brightness and takes up a lot of space to store the map, while the feature point method will affect the entire image. The picture first performs the feature point extraction action. Although it is time-consuming, it has high noise immunity against ambient light, and the map occupies a small storage space.

In addition, in terms of input sensing, visual SLAM technology can be divided into monocular camera (Monocular) SLAM, color depth camera (RGB-D) SLAM and stereo camera (Stereo) SLAM, wherein Monocular SLAM uses a single Image positioning, which uses the position of the feature points between the key frames and the key frames, and the isokinetic model to estimate the distance, that is, when using Monocular SLAM for mapping and positioning, it must rely on the direct disparity between the key frames and the key frames to calculate the feature points. As a result, Monocular SLAM cannot create a map with the same size as the actual trajectory, so the biggest problem of Monocular SLAM is the lack of actual depth information of the environment; secondly, RGB-D SLAM uses a monocular camera with infrared ranging, which only exists In view of the problem that infrared rays cannot operate normally under strong outdoor light, that is, when infrared rays are used in outdoor places, their accuracy will be affected by light, resulting in inaccurate depth measurements. Accordingly, RGB-D SLAM applications are generally limited to indoor environments; in addition, Stereo SLAM uses the parallax of two cameras to estimate the depth for positioning. Specifically, Stereo SLAM uses the parallax between two fisheye cameras to calculate the depth, and then uses the camera's correction to calculate the distance of the actual environment. The problem is poor scalability.

It can be seen from the above that the biggest problem of the current visual SLAM technology is that it cannot effectively provide a depth map with accurate absolute depth, while traditional Monocular SLAM, RGB-D SLAM and Stereo SLAM have their own shortcomings, which makes it difficult for visual SLAM to replace existing applications. The SLAM of the optical radar, therefore, how to overcome the above-mentioned various deficiencies in the prior art will become an urgent problem to be solved at present.

In view of the above problems, the present invention provides a method for generating a depth image, which includes: providing a pixel image including a depth feature point and a relative depth corresponding to the depth feature point; providing a radar point including the depth feature point and a pair of A radar image of the measured depth of the radar point; calculating a magnification between the measured depth and the relative depth; and correcting the pixel image by the magnification to form a depth image.

In one embodiment, the method for generating the depth image further includes analyzing the pixel image through a deep learning network to obtain the depth feature point from the pixel image, so that the relative depth feature point can be obtained by using the depth feature point. depth.

In another embodiment, the radar image and the pixel image are projected so that the radar point corresponds to the depth feature point.

In another embodiment, the pixel image includes a plurality of the depth feature points and a plurality of the relative depths, and the radar image includes a plurality of the radar points and a plurality of the measured depths, through the complex A plurality of magnifications are obtained by calculating between a plurality of the measured depths and a plurality of the relative depths, and the plurality of magnifications are averaged to generate an average magnification for correcting the pixel image using the averaged magnification.

In yet another embodiment, the method for generating the depth image further comprises obtaining a plurality of frame magnifications through a plurality of the pixel images and a plurality of the radar images, and averaging the plurality of frame magnifications to generate a correction magnification, so as to obtain a plurality of frame magnifications. The pixel image is corrected by the correction magnification to form the depth image.

The present invention further provides a system for generating a depth image, comprising: a depth prediction module for receiving and analyzing a pixel image to obtain a depth feature point of the pixel image and a relative depth corresponding to the depth feature point; The radar acquisition module is used to acquire the radar image including the radar point corresponding to the feature point of the depth and the measured depth corresponding to the radar point; and the depth fusion module is used to connect the depth prediction module and the radar The capturing module is used for calculating the magnification between the measured depth and the relative depth, so as to correct the pixel image through the magnification to form a depth image.

In one embodiment, the depth image generation system further includes an image capture module connected to the depth prediction module for capturing the pixel image to provide the pixel image to the depth prediction module .

In another embodiment, the depth prediction module analyzes the pixel image through a deep learning network to obtain the depth feature point from the pixel image, so as to use the depth feature point to obtain the relative depth.

In another embodiment, the generation system of the depth image further includes a correction module connected to the depth prediction module and the radar capture module, which is used to make the radar point and the depth feature point match by projection method. Correspondingly, the projected pixel image and the radar image are transmitted to the deep fusion module.

In another embodiment, the pixel image includes a plurality of the depth feature points and a plurality of the relative depths, the radar image includes a plurality of the radar points and a plurality of the measured depths, and the depth fusion module passes through the plurality of A plurality of magnifications are obtained by calculating between a plurality of the measured depths and a plurality of the relative depths, and the plurality of magnifications are averaged to generate an average magnification for correcting the pixel image using the averaged magnification.

In yet another embodiment, the depth fusion module obtains a plurality of frame magnifications through calculation between a plurality of the pixel images and a plurality of the radar images, and averages the plurality of frame magnifications to generate a correction magnification, so as to use The pixel image is corrected by the correction magnification to form the depth image.

The present invention further provides a positioning system, which includes: a depth prediction module for receiving and analyzing a pixel image to obtain a depth feature point of the pixel image and a relative depth corresponding to the depth feature point; a radar capture module The group is used to capture the radar image including the radar point corresponding to the feature point of the depth and the measured depth corresponding to the radar point; the depth fusion module is used to connect the depth prediction module and the radar capture module, is used for calculating the magnification between the measured depth and the relative depth, and correcting the pixel image through the magnification to form a depth image; and a synchronizing positioning and map building module for comparing the data obtained by the depth fusion module The formed depth image and the pre-stored area map are used for positioning in the area map according to the comparison result.

In one embodiment, the positioning system further includes an image capture module and a calibration module, wherein the image capture module is used to capture the pixel image to provide the pixel image to the depth prediction a module, and the correction module receives the pixel image from the depth prediction module and the radar image from the radar capture module, so as to use a projection method to make the radar point correspond to the depth feature point , so as to transmit the projected pixel image and the radar image to the deep fusion module.

In another embodiment, the synchronous positioning and map construction module further includes: a feature point extraction unit for extracting image feature points from the depth image; a tracking or relocation unit, connected to The feature point extraction unit is connected to compare the image feature point of the depth image with the previous image feature point of the previous depth image, so as to carry out feature point tracking; and an optimization and area positioning unit is connected to the The tracking or repositioning unit is used for analyzing the tracking result of the tracking or repositioning unit to obtain the positioning position corresponding to the depth image.

In another embodiment, the synchronized positioning and map construction module further includes a camera pose output unit for outputting the positioning position corresponding to the depth image.

In another embodiment, the camera gesture output unit is an output port of a Video Graphics Array (VGA), a Digital Video Interface (DVI), a High-Definition Multimedia Interface (HDMI) or a Display Port.

In another embodiment, the simultaneous positioning and map building module further includes an area map unit for storing the area map, so that the tracking or relocation unit cannot match the image feature points of the depth image with the previous area map. When the feature points of the previous image of a depth image are tracked, the depth image is compared with the map of the area for repositioning.

In another embodiment, the optimization and area positioning unit updates the area map synchronously when the positioning position corresponding to the depth image is obtained.

In another embodiment, the synchronous positioning and map construction module further includes a key frame generating unit for inserting a key frame in the area map, so as to compare the key frame and the area map when inserting the key frame The other key frames are used to form a three-dimensional map by using the image feature points when the comparison result matches.

In yet another embodiment, the synchronous positioning and map construction module further includes a loop closure detection unit for calculating the similarity between the key frame and the other key frames of the area map, so that when the similarity exceeds a threshold value , optimize the map of the area.

To sum up, the method and system for generating a depth image of the present invention fuse the pixel image with relative depth and the radar image with measured depth, and calculate the ratio of the difference between the two, so as to correct the ratio through the ratio The pixel image is used to generate a depth image, thereby obtaining an accurate depth. In addition, the present invention proposes a positioning system using the above-mentioned generation method, which is to further locate the generated depth image through a synchronous positioning and map construction module. In order to achieve the purpose of providing a depth image with absolute depth for localization, it not only solves the possible problems of visual SLAM, but also replaces the high cost of SLAM using existing LiDAR.

10,10': Depth image generation system

11: Depth prediction module

12: Radar capture module

13: Deep Fusion Module

14: Image capture module

15: Calibration module

201-211: Process

60: Positioning system

61: Depth prediction module

62: Radar Capture Module

63: Deep Fusion Module

64: Simultaneous positioning and map building module

641: Feature point extraction unit

642: Track or reposition unit

643: Optimization and Regional Positioning Unit

644: Camera pose output unit

645: Area Map Unit

646: Keyframe generation unit

647: Loopback detection unit

65: Image capture module

66: Calibration module

801~807: Process

901~907: Process

S101~S104: Steps

FIG. 1 is a flow chart of the steps of the method for generating a depth image of the present invention.

FIG. 2 is a flowchart of depth fusion performed by the method for generating a depth image of the present invention.

FIG. 3 is a schematic diagram of radar image sampling according to the present invention.

FIG. 4 is a schematic diagram of the structure of the first embodiment of the depth image generation system of the present invention.

FIG. 5 is a schematic diagram of the structure of the second embodiment of the depth image generation system of the present invention.

FIG. 6 is a schematic diagram of the system architecture of the positioning system of the present invention.

FIG. 7 is a schematic structural diagram of a synchronous positioning and map building module in the positioning system of the present invention.

FIG. 8 is a flow chart of forming a projection matrix according to the present invention.

FIG. 9 is a flow chart of camera pose projection of the positioning system of the present invention.

10A-10D are locating effect trajectory diagrams of the present invention.

The following describes the technical content of the present invention through specific embodiments, and those skilled in the art can easily understand the advantages and effects of the present invention from the content disclosed in this specification. However, the present invention can also be implemented or applied by other different specific embodiments.

FIG. 1 is a flow chart of the steps of a method for generating a depth image according to the present invention. As shown in the figure, in step S101, a pixel image including a depth feature point and a relative depth corresponding to the depth feature point is provided. Specifically, the pixel image is a picture or image data captured by a device such as a camera, a video camera, etc. for capturing environmental images. By analyzing the pixel image, depth feature points are obtained from the pixel image. And the relative depth information corresponding to the depth feature point. In other words, the present invention analyzes the pixel image through a deep learning network to obtain depth feature points from the pixel image, and uses the depth feature points to obtain the relative depth of the pixel image, wherein the depth described in the present invention is Feature points refer to specific regions, objects, pixels, or coordinate points in a pixel image, which can be used for analysis by deep learning networks.

In step S102, a radar image including a radar point corresponding to the depth feature point and a measured depth corresponding to the radar point is provided. Specifically, the radar image is a radar point cloud captured by the radar to the environment, which has one or more radar points, and the corresponding distance information can be obtained through the radar points. Further, the radar image and the pixel image use a projection method to make the radar point correspond to the depth feature point, wherein the pixel image and the radar image have respective coordinate axes, and the coordinate axes of the radar image are corresponding to each other. To the coordinate axis of the pixel image, that is, project the radar image to the pixel image, so that the radar point of the radar image can correspond to the pixel point on the pixel image.

In step S103, the magnification between the measurement depth and the relative depth is calculated. This step is based on the relative depth to calculate the ratio between the measured depth and the relative depth. For example, the relative depth obtained by analyzing the depth feature points in the pixel image is 5 meters, and the radar image If the measured depth of the corresponding depth feature points in the image is 10 meters, the magnification between them is 2 times.

In step S104, the pixel image is corrected by the magnification to form a depth image. After obtaining the magnification between the relative depth and the measured depth, the pixel image is corrected by the magnification, so that the pixel image forms a depth image with absolute depth, that is, the pixel image has a lot of relative distance information, It is a dense depth map, while a radar image has accurate distance information but a small amount of data, so it is a sparse point cloud image, and a depth image is a depth map with a lot of distance information such as pixel images, but at the same time absolute depth information. Following the above example, when the magnification between the relative depth and the measured depth is 2 times, the relative depth in the pixel image is corrected to the measured depth in the radar image using this magnification, so that the pixel image is changed by the magnification. After correction, it becomes a depth image with absolute depth.

In one embodiment, pixel images of a plurality of frames and a plurality of radar images corresponding to the pixel images of the plurality of frames can be captured in a period of time, and then, the pixel images and radar images of the corresponding frames are firstly analyzed. The image takes the magnification of its corresponding frame to obtain multiple frame magnifications, and then further averages the multiple frame magnifications to obtain the correction magnification, which can be used to correct each pixel image to form the corresponding depth map For example, according to this, the present invention can achieve the purpose of reducing the large error that may be caused by a single magnification.

In another embodiment, each pixel image includes a plurality of depth feature points, and the plurality of depth feature points correspond to a plurality of relative depths, and each radar image includes a plurality of radar points corresponding to the pixel image. , and a plurality of radar points correspond to a plurality of measurement depths. Accordingly, the present invention can make the corresponding relative depths and The measured depths all generate corresponding magnifications, so as to obtain multiple magnifications in a single pixel image, and then average the multiple magnifications to generate an average magnification, which can be used to correct the pixel image, which can reduce the Using only a single magnification in a single image may produce erroneous results.

FIG. 2 is a flow chart of deep fusion performed by the method for generating a depth image of the present invention. As shown in the figure, the radar image of the present invention is the radar point cloud of the corresponding environment captured by the radar, wherein the radar image There are multiple radar points in the point cloud and the measured depths corresponding to each radar point. In the process 201, radar images are provided; in the processes 202-203, the radar images are screened to reduce the number of radar images. The image generated by the noise in the image, specifically, first confirm whether the radar point cloud is a valid projection, that is, project the radar image onto the pixel image, and confirm whether each radar point is a valid projection, and can find the corresponding If the feature point is not valid projection, the radar image will be discarded. If it is valid projection, then confirm whether the distance range of the radar point in the radar image belongs to the depth range of the radar used. If not, discard the feature point. Remove the radar image to eliminate the unsuitable radar image. If the radar point belongs to the specified depth range, enter the process 204, that is, after the radar image is filtered, use the filtered radar image with depth to compare the relative depth map. Sampling, wherein, FIG. 3 is a schematic diagram of the radar image sampling of the present invention. As shown in the figure, the present invention can achieve the purpose of eliminating noise by sampling around the radar point of the radar image. After that, The relative depth value corresponding to each effective radar point will be obtained; in the process 205, the value of the relative depth corresponding to the effective radar point is used to calculate the magnification of a single frame; in the process 206, the pixel images of multiple frames and corresponding to calculate multiple magnifications of multiple frames (for example, M frames); in process 207, the multiple magnifications formed by the multiple frames are averaged to obtain the final corrected magnification; in process 208, The obtained final correction magnification is used to correct the relative depth; in the process 209, the depth range is limited, that is, the relative depth is required to be within the depth range after the correction; in the process 210, the median alignment is adjusted again ; Finally, in the process 211, a depth image with absolute depth is obtained.

FIG. 4 is a schematic structural diagram of the first embodiment of the depth image generation system of the present invention. As shown in the figure, in this embodiment, the depth image generation system 10 of the present invention includes a depth prediction module 11 , a radar capture module 12 , and a module connected to the depth prediction module 11 and the radar capture module 12 . The deep fusion module 13, wherein the deep fusion module 13 obtains the phase data from the depth prediction module 11 and the radar capture module 12 respectively. The depth and the measured depth corresponding to the relative depth are calculated and corrected to form a depth image. The generation system 10 of the present invention will be described in detail below.

The depth prediction module 11 uses a single RGB image in the pixel image to generate depth information, that is, the depth prediction module 11 analyzes the pixel image after receiving the pixel image to obtain the depth feature points of the pixel image and the corresponding depth feature points. the relative depth. Specifically, the depth prediction module 11 analyzes the pixel image by using the pre-trained deep learning network to obtain depth feature points from the pixel image, and uses the depth feature points to obtain the relative depth, and further , the analysis of pixel images by the deep learning network can be divided into two stages: encoding and decoding. Among them, the encoding part is to find the corresponding feature points for the pixel image, and the decoding part uses the feature points to calculate the corresponding feature points. In-depth information.

The radar capture module 12 is provided with an image capture device that can be connected to the embedded system device, and mainly transmits the radar point cloud information to the system through serial transmission for other operations. The radar capture module 12 can capture radar points including depth feature points corresponding to pixel images and radar images corresponding to the measured depths of the radar points, that is, the present invention obtains radar points and measured depths through the radar capture module Radar image.

The depth fusion module 13 is used to fuse the relative depth obtained by the depth prediction module 11 and the measured depth in the radar image obtained by the radar capture module 12 . The depth fusion module 13 of the present invention connects the depth prediction module 11 and the radar capture module 12 and receives a pixel image with relative depth and a radar image with measured depth, the depth fusion module 13 calculates the measured depth The magnification between the relative depth and the magnification is used to correct the pixel image, that is, the relative depth of the pixel image is corrected to the absolute depth by the magnification, thereby forming a depth image with absolute depth. It should be noted that the depth fusion module 13 is mainly used to process radar point clouds and relative depth maps, wherein the radar images are radar point clouds, which are only sparse point cloud images, but have accurate distance information. The image only has the object-to-object Relative depth, but it is a dense depth map, the present invention fuses the relative depth and the measured depth through the depth fusion module 13 to form a depth image.

In one embodiment, after receiving a plurality of pixel images and a plurality of corresponding radar images, the depth fusion module 13 first obtains the magnification of each frame by calculating the pixel images and radar images of each corresponding frame, so as to obtain the ratio of each frame. Obtain a plurality of frame magnifications, and then average the plurality of frame magnifications to obtain a correction magnification. Accordingly, the depth fusion module 13 can use the correction magnification to correct the pixel image, thereby forming a corresponding depth image. The data averaging of the images can reduce the situation of large error values caused by using only a single image.

In another embodiment, the pixel image includes a plurality of depth feature points and a plurality of relative depths, and the radar image includes a plurality of radar points corresponding to each depth feature point and a plurality of measurement depths. The depth fusion module 13. A plurality of magnifications can be obtained through the calculation between a plurality of measurement depths and a plurality of relative depths, and then the plurality of magnifications can be averaged to generate an average magnification. Accordingly, the present invention can utilize the average magnification to correct pixel images, It can also reduce the error problem that may be caused by only taking a single magnification for a single image.

FIG. 5 is a schematic diagram of the structure of the second embodiment of the depth image generation system of the present invention. As shown in the figure, the depth prediction module 11 , the radar capture module 12 and the depth fusion module 13 of the depth image generation system 10 ′ of this embodiment are the same as those in FIG. In this embodiment, the depth image generation system 10 ′ further includes an image capturing module 14 and a calibration module 15 .

The image capture module 14 is used to capture pixel images for analysis by the depth prediction module 11. In short, it is a module that can be connected to embedded devices, such as network cameras, digital cameras, smart For electronic equipment such as mobile phones, driving recorders, etc., the present invention can obtain the pixel image of the environment in real time through the image capturing module 14 .

The correction module 15 is connected to the depth prediction module 11 , the radar capture module 12 and the depth fusion module 13 to receive the pixel image and the radar image, wherein the correction module 15 projects the radar image onto the radar image by means of projection. The pixel image makes the radar point and the depth feature point correspond, and the projected pixel image and the radar image are then sent to the depth fusion module 13 for depth fusion. In other words, the correction module 15 is used to project the coordinates of the radar point from the radar image to the coordinates of the pixel image, so that the radar point corresponds to the depth feature point of the depth image, that is, the pixel image displays the location of the radar point. The position of the pixel in the corresponding image image. Specifically, the image capture module 14 takes a camera as an example. Before the calibration module 15 performs projection, the internal parameters of the camera and the relative position of the radar to the camera must be calculated. Under the premise that the relative position of the camera is not changed, no further correction is required, that is, only one correction is required between the radar and the camera, and the pre-established projection matrix can be directly used for conversion in the projection stage to achieve the corresponding purpose.

FIG. 6 is a schematic diagram of a system architecture of a positioning system applying a depth image generation method. As shown in the figure, the positioning system 60 of the present invention includes a depth prediction module 61, a radar capture module 62, a depth fusion module 63, and a simultaneous positioning and map construction module 64. Further, it may further include image capture. The module 65 and the calibration module 66, wherein the functions and operations of the depth prediction module 61, the radar capture module 62, the depth fusion module 63, the image capture module 65 and the calibration module 66, such as the front depth The image generation system is described above, so it will not be repeated here. The following only describes the synchronized positioning and map construction module 64 .

The synchronous positioning and map construction module 64 is used for comparing the depth image and the synchronous positioning received from the deep fusion module 63 with the regional map pre-stored by the map construction module 64, so as to create the regional map according to the comparison result. Positioning is carried out in the map to obtain the location on the map of the area. In short, the positioning system 60 performs positioning in the area map based on the depth image generated by the depth fusion module 63 . The specific positioning procedure will be described in detail later.

FIG. 7 is a schematic structural diagram of the synchronous positioning and map construction module 64 in the positioning system 60 of the present invention. As shown in the figure, the synchronized localization and map construction module 64 of the present invention includes a feature point extraction unit 641 , a tracking or relocation unit 642 , and an optimization and area localization unit 643 .

The feature point extraction unit 641 obtains the depth image from the depth fusion module 63 in the simultaneous positioning and map construction module 64, and performs image feature point extraction on each received frame of the depth image.

The tracking or repositioning unit 642 is connected to the feature point extraction unit 641, so that when the feature point extraction unit 641 extracts image feature points from the currently received depth image, the image feature points of the depth image are used for The previous image feature points of the depth image of one frame are tracked. In short, the tracking or relocation unit 642 performs feature point tracking by comparing the image feature points of the depth image of the current frame with the previous image feature points of the previous depth image.

The optimization and area positioning unit 643 is connected to the tracking or repositioning unit 642, and after the tracking or repositioning unit 642 succeeds in tracking, it analyzes according to the tracking result to obtain the positioning position corresponding to the depth image, and then obtains the positioning of the present invention The camera pose of the system. In other words, after the tracking or repositioning unit 642 successfully tracks, the positioning system of the present invention calculates the camera pose through the optimization and regional positioning unit 643, and the camera pose information is simultaneously output and the current camera pose is recorded in the regional map.

Accordingly, the present invention can generate a depth image through the depth prediction module 61, the radar capture module 62, and the depth fusion module 63 in FIG. 6, and provide the synchronous positioning and map construction module 64 to track the depth image, and then Determine the actual location on the regional map to achieve the purpose of positioning.

Further, the synchronized positioning and map construction module 64 shown in FIG. 7 further includes a camera pose output unit 644 , an area map unit 645 , a key frame generation unit 646 and a loop closure detection unit 647 .

The camera pose output unit 644 is used for outputting the camera pose corresponding to the positioning position of the depth image, through a video graphics array (VGA), a digital video interface (DVI), a high-definition multimedia interface (HDMI) or a display port (DisplayPort) Port, DP) and other output ports for the connection of embedded or external display devices, and correspondingly connected to the image display device through the aforementioned output ports to provide simultaneous positioning and the video posture of the map building module 64 . In short, the present invention can provide visual output information through the camera posture output unit 644, and the camera posture output unit 644 can be compatible with the display device set by the positioning system of the present invention through, such as VGA, DVI, HDMI or Display Port, etc. The output interface is connected to transmit the visual output information. In other words, the camera posture unit 644 can provide an interface channel for other modules or devices to obtain the camera posture information or further perform data visualization on the camera posture. In addition, the camera pose output unit 644 also provides a transformation matrix of the camera pose in the three-dimensional space. The transformation matrix includes the current position information of the camera pose, which can be provided to other systems for use or track recording.

The area map unit 645 is used to store the area map, and when the tracking or relocation unit 642 cannot track the image feature points of the depth image and the previous image feature points of the previous depth image, provide the area map and the area map. The depth image is compared to relocate using the result of the comparison between the area map and the depth image. As can be seen from the above, when the positioning system is relocated as described above, the optimization and area positioning unit 643 will obtain the positioning position corresponding to the depth image, and at this time, the area map in the area map unit 645 can be updated synchronously.

In addition, after the tracking or relocation unit 642 successfully tracks the positioning system of the present invention, the camera posture is calculated by the optimization and area positioning unit 643, and the camera posture information will be simultaneously sent to the camera posture output unit 644 and the area map unit 645 for recording. Current camera pose.

The key frame generation unit 646 is used to insert a key frame in the area map, and when inserting the key frame, compare the key frame with other key frames in the area map, so that when the comparison result matches, use the key frame. Image feature points form a three-dimensional map. Specifically, the key frame generation unit 646 accesses the area map in the area map unit 645 to insert key frames at certain intervals (eg, time intervals or frame intervals) during the tracking process of the tracking or repositioning unit 642, And three-dimensional (3D) map points are generated from the feature points matching the current key frame and the key frame in the area map.

The simultaneous positioning and map construction module 64 uses the loop closure detection unit 647 to calculate the similarity between the key frame and the other key frames of the area map, so as to optimize the area map when the similarity exceeds a threshold. In other words, the loop closure detection unit 647 will calculate the similarity between the newly generated map points and key frames, and if the similarity is higher than the threshold, all map information will be optimized to eliminate drift errors. Then save the optimized results back into the area map.

FIG. 8 is a flow chart of forming a projection matrix according to the present invention. As shown in the figure, in order to make the trajectory of the synchronized positioning and map construction module 64 of the present invention approach the accuracy of the Global Positioning System (GPS), the present invention further utilizes a projection matrix to project each camera pose. In process 801, a ROSBAG database is built or pre-recorded; in processes 802-804, a SLAM map is generated using the built ROSBAG database, and the SLAM map is stored as a SLAM map and a preliminary map-building track is obtained; In the process 805, the GPS track is obtained; in the processes 806-807, the correspondence between the mapping track and the GPS track is calculated to generate a projection matrix. Accordingly, after the positioning stage, the present invention generates a projection matrix corresponding to the mapping trajectory and the GPS trajectory for subsequent projection of each camera pose and GPS trajectory to obtain a more accurate trajectory map.

FIG. 9 is a flow chart of camera pose projection of the positioning system of the present invention. As shown in the figure, in the processes 901-902, the SLAM mapping and the projection matrix generated in FIG. 8 are provided to the positioning system, and in the process In 903-904, the positioning system of the present invention receives the SLAM map and the projection matrix for analysis to output the camera pose.

10A-10D are locating effect trajectory diagrams of the present invention. As shown in the figure, on four different routes, the positioning effect and map construction result obtained by the positioning system of the present invention are almost the same as those of the conventional LIDAR SLAM and GPS using optical radar, so the present invention The technology can indeed replace the existing SLAM application of LiDAR, and its accuracy is also similar to the GPS trajectory.

To sum up, the present invention establishes a system for generating a depth image fused by radar and camera sensors, wherein the present invention utilizes the advantages of radar and camera sensors (such as monocular cameras) to achieve such fusion, and through the fusion of monocular cameras Depth estimation can only use a single pixel image to estimate the dense depth of each pixel of the environment, but the pixel image only lacks absolute depth information. For this, the present invention introduces radar to make up for its lack. The radar has the advantage of being cheap, and It can provide sparse depth information about the surrounding environment, therefore, the present invention uses the sparse depth information provided by the radar to fuse with the estimated depth generated by the depth estimation of the monocular camera, thereby creating an absolute depth map that can provide the surrounding environment. picture. Furthermore, the present invention further imports the depth image into the SLAM system, and the SLAM system only needs to use the depth image generated by the present invention to obtain depth information without using triangulation in the key points of the subsequent frame to calculate the surrounding depth.

The present invention can benefit a SLAM system in two ways, namely, in the mapping task of a SLAM system, the present invention can have more accurate depth information about the surrounding environment, and therefore, can create a LIDAR SLAM comparable scale map; in localization tasks, it can use not only feature matching between subsequent frames for localization, but also visual trajectories for localization and reduce the chance of losing tracking due to pure rotation, which increases SLAM stability of the system and increase the chances of repositioning and successful positioning. To sum up, the present invention can generate the A depth map with absolute depth can be obtained, so as to achieve the effect of a map with a scale equivalent to that of GPS or LIDAR SLAM, which will help reduce the cost of system construction.

The above-mentioned embodiments merely illustrate the principles and effects of the present invention, but are not intended to limit the present invention. Any person skilled in the art can modify and change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention should be as listed in the patent application scope described later.

S101~S104: Steps

Claims (17)

A method for generating a depth image, comprising: providing a pixel image including a depth feature point and a relative depth corresponding to the depth feature point; providing a radar point corresponding to the depth feature point and a measured depth corresponding to the radar point. radar image; calculating the magnification between the measured depth and the relative depth; and correcting the pixel image through the magnification to form a depth image, wherein the pixel image includes a plurality of the depth feature points and a plurality of For the relative depth, the radar image includes a plurality of the radar points and a plurality of the measurement depths, and obtains a plurality of magnifications by calculating between the plurality of the measurement depths and the plurality of the relative depths, and averages the plurality of magnifications to generate an average magnification for correcting the pixel image using the average magnification. The method for generating a depth image according to claim 1, further comprising analyzing the pixel image through a deep learning network to obtain the depth feature point from the pixel image, so as to obtain the depth feature point by using the depth feature point. relative depth. The method for generating a depth image according to claim 1, wherein the radar image and the pixel image are projected by means of a projection method, so that the radar point corresponds to the depth feature point. The method for generating a depth image according to claim 1, further comprising obtaining a plurality of frame magnifications through a plurality of the pixel images and a plurality of the radar images, and averaging the plurality of frame magnifications to generate a correction magnification, so as to obtain a plurality of frame magnifications. The pixel image is corrected by the correction magnification to form the depth image. A system for generating a depth image, comprising: a depth prediction module for receiving and analyzing a pixel image to obtain a depth feature point of the pixel image and a relative depth corresponding to the depth feature point; The radar acquisition module is used to acquire the radar image including the radar point corresponding to the feature point of the depth and the measured depth corresponding to the radar point; and the depth fusion module is used to connect the depth prediction module and the radar a capture module for calculating a magnification between the measured depth and the relative depth, so as to correct the pixel image through the magnification to form a depth image, wherein the pixel image includes a plurality of the depth feature points and A plurality of the relative depths, the radar image includes a plurality of the radar points and a plurality of the measured depths, and the depth fusion module obtains a plurality of magnifications by calculating between the plurality of the measured depths and the plurality of the relative depths , and the plurality of magnifications are averaged to generate an average magnification for correcting the pixel image using the averaged magnification. The system for generating a depth image according to claim 5, further comprising an image capturing module connected to the depth prediction module for capturing the pixel image to provide the depth prediction module with the pixel image Group. The system for generating a depth image according to claim 5, wherein the depth prediction module analyzes the pixel image through a deep learning network, so as to obtain the depth feature point from the pixel image, so as to use The depth feature point obtains the relative depth. The system for generating a depth image as claimed in claim 5, further comprising a correction module connected to the depth prediction module and the radar capture module, and used for making the radar point and the depth feature point match by projection. Correspondingly, the projected pixel image and the radar image are sent to the deep fusion module. The system for generating a depth image according to claim 5, wherein the depth fusion module obtains a plurality of frame magnifications through calculation between a plurality of the pixel images and a plurality of the radar images, and averages the plurality of frame magnifications. The frame magnification is used to generate a correction magnification, so that the pixel image is corrected by the correction magnification to form the depth image. A positioning system, comprising: a depth prediction module for receiving and analyzing a pixel image to obtain a depth feature point of the pixel image and a relative depth corresponding to the depth feature point; a radar capture module for using In capturing the radar image including the radar point corresponding to the feature point of the depth and the measured depth corresponding to the radar point; the depth fusion module is connected to the depth prediction module and the radar capture module to calculate the measuring a magnification between the depth and the relative depth, to correct the pixel image through the magnification to form a depth image; and a synchronous positioning and map building module for comparing the depth image formed by the depth fusion module The depth image and the pre-stored area map are used for positioning in the area map according to the comparison result, wherein, the synchronous positioning and map construction module further includes: a feature point extraction unit, which is used for the depth image. Image feature point extraction; a tracking or repositioning unit is connected to the feature point extraction unit for comparing the image feature point of the depth image with the previous image feature point of the previous depth image to perform feature extraction. The point tracking; and the optimization and area positioning unit are connected to the tracking or repositioning unit for analyzing the tracking result of the tracking or repositioning unit to obtain the positioning position corresponding to the depth image. The positioning system of claim 10, further comprising an image capture module and a calibration module, wherein the image capture module is used to capture the pixel image to provide the pixel image to the depth a prediction module, and the correction module receives the pixel image from the depth prediction module and the radar image from the radar capture module, so as to make the radar point correspond to the depth feature point by means of projection , so as to transmit the projected pixel image and the radar image to the deep fusion module. The positioning system of claim 10, wherein the simultaneous positioning and map building module comprises an area map unit for storing the area map, so that the image of the depth image cannot be obtained by the tracking or relocation unit When the feature point is tracked with the feature point of the previous image of the previous depth image, the depth image is compared with the map of the area to perform repositioning. The positioning system of claim 10, wherein the optimization and area positioning unit updates the area map synchronously when the positioning position corresponding to the depth image is obtained. The positioning system of claim 10, wherein the synchronous positioning and map construction module further comprises a key frame generating unit for inserting a key frame in the area map, so as to compare the key frame when inserting the key frame The frame and other key frames in the area map are used to form a three-dimensional map using the image feature points when the comparison result matches. The positioning system of claim 14, wherein the synchronized positioning and map construction module further comprises a loop closure detection unit for calculating the similarity between the key frame and the other key frames of the area map, so as to determine the similarity between the key frame and the other key frames of the area map. When the similarity exceeds the threshold, optimize the area map. The positioning system of claim 10, wherein the simultaneous positioning and map construction module comprises a camera pose output unit for outputting the positioning position corresponding to the depth image. The positioning system of claim 16, wherein the camera attitude output unit is an output of a video graphics array (VGA), a digital video interface (DVI), a high-definition multimedia interface (HDMI) or a display port (Display Port) port.

Images