WO2020237553A1 - 图像处理方法、系统及可移动平台 - Google Patents
图像处理方法、系统及可移动平台 Download PDFInfo
- Publication number
- WO2020237553A1 WO2020237553A1 PCT/CN2019/089177 CN2019089177W WO2020237553A1 WO 2020237553 A1 WO2020237553 A1 WO 2020237553A1 CN 2019089177 W CN2019089177 W CN 2019089177W WO 2020237553 A1 WO2020237553 A1 WO 2020237553A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- view
- disparity
- point cloud
- feature map
- map
- Prior art date
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 24
- 238000000034 method Methods 0.000 claims abstract description 43
- 238000000605 extraction Methods 0.000 claims description 40
- 238000012545 processing Methods 0.000 claims description 33
- 230000015654 memory Effects 0.000 claims description 30
- 230000009466 transformation Effects 0.000 claims description 17
- 238000005457 optimization Methods 0.000 claims description 6
- 238000004891 communication Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 8
- 230000009286 beneficial effect Effects 0.000 description 6
- 230000009471 action Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 238000004590 computer program Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000001364 causal effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/06—Topological mapping of higher dimensional structures onto lower dimensional surfaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
Definitions
- This application relates to the field of image processing technology, and in particular to an image processing method, system and movable platform.
- intelligent control technology With the continuous iterative development of intelligent control technology, some vehicles have begun to be equipped with automatic driving systems or assisted driving systems, which can bring a lot of convenience to operators. Similarly, intelligent control technology is also used on other movable platforms to realize automatic motion or auxiliary motion functions, such as robots, smart cars, drones, etc.
- a very important function is to automatically recognize the distance between the movable platform (such as autonomous vehicles, smart cars, drones, etc.) and objects in the surrounding environment.
- the left eye view and the right eye view of the surrounding environment of the movable platform are usually collected by a binocular camera device, and then the disparity map between the left eye view and the right eye view is determined. Then determine the distance between the movable platform and the objects in the surrounding environment according to the disparity map.
- the error of this method increases with the increase of the distance of the object, resulting in lower accuracy of the disparity map determined for the far object, and thus the distance to the far object cannot be accurately determined. Therefore, how to accurately determine the disparity map is a problem to be solved urgently.
- This application discloses an image processing method, equipment and a movable platform, which are beneficial to improve the accuracy of determining the disparity map.
- this application provides an image processing method, including:
- the present application provides an image processing system, the image processing system includes: a memory, a processor, a binocular camera device and a point cloud sensor, wherein:
- the memory is used to store program instructions
- the binocular camera device is used to collect the first view and the second view of the environment
- the point cloud sensor is used to collect a three-dimensional point cloud of the environment
- the processor calls the program instructions for:
- the present application provides a movable platform, which includes: a memory, a processor, a binocular camera device, and a point cloud sensor, wherein:
- the memory is used to store program instructions
- the binocular camera device is used to collect the first view and the second view of the environment
- the point cloud sensor is used to collect a three-dimensional point cloud of the environment
- the processor calls the program instructions for:
- the image processing method, equipment and movable platform collect a first view and a second view of the environment through a binocular camera device, and collect a three-dimensional point cloud of the environment through a point cloud sensor. Then the three-dimensional point cloud is projected to the first view and matched with part of the pixels of the first view to obtain a priori disparity, that is, the prior disparity is determined according to the three-dimensional point cloud to be accurate for some pixels in the first view The parallax value. Finally, a disparity map between the first view and the second view is obtained according to the prior disparity, which improves the accuracy of determining the disparity map and is beneficial to improving the accuracy of determining the distance.
- FIG. 1 is a schematic diagram of the principle of an existing distance measurement based on a binocular camera device provided by an embodiment of the present application;
- FIG. 2 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
- Fig. 3 is a schematic diagram of a first view provided by an embodiment of the present application.
- FIG. 4 is a schematic flowchart of another image processing method provided by an embodiment of the present application.
- FIG. 5 is a schematic flowchart of another image processing method provided by an embodiment of the present application.
- Fig. 6 is a schematic structural diagram of an image processing system provided by an embodiment of the present application.
- Fig. 7 is a schematic structural diagram of a movable platform provided by an embodiment of the present application.
- the embodiments of the present invention provide an image processing method, equipment and a movable platform.
- the image processing method can be executed by an image processing system, or the image processing method can be executed by a movable platform.
- the movable platform may include, but is not limited to, drones, unmanned ships, ground robots, smart cars, unmanned vehicles, and the like.
- the image processing system may be included in the movable platform.
- the image processing system can be a specific image processing device, and can establish a communication connection with the movable platform through a wireless communication connection, or establish a communication connection with the movable platform through a wired communication connection. .
- the image processing system can also be in a distributed form, and the various components or devices contained in it can be distributed on a movable platform, and the various components or devices can be wired, wireless, or communicated with each other.
- the bus is connected, and the image processing system and the movable platform can also be connected in communication.
- the image processing system includes a binocular camera device.
- the binocular camera device is used to collect the left-eye view and the right-eye view of the surrounding environment, including but not limited to a visible light camera, a grayscale camera, or an infrared camera.
- the binocular camera device may be configured on the fuselage of the image processing system through the carrying device.
- the image processing system may further include a point cloud sensor.
- the point cloud sensor is used to collect the three-dimensional point cloud of the environment.
- the three-dimensional point cloud includes the feature information of each three-dimensional point, that is, the three-dimensional information of the feature point in the environment.
- the point cloud sensor includes, but is not limited to, lidar sensors, millimeter wave radar sensors, and ultrasonic radar sensors.
- the mobile platform may include the above-mentioned binocular camera device and point cloud sensor.
- the binocular camera device and point cloud sensor in the image processing system. No longer.
- the movable platform may further include a communication device for communicating with the control terminal.
- the control terminal is used to control the movable platform.
- the control terminal may be a mobile phone, a tablet computer, a remote control, or other wearable devices (watches or bracelets), etc., which are not limited in the embodiment of the present application.
- FIG. 1 is a schematic diagram of a conventional distance measurement based on a binocular camera device disclosed in an embodiment of the present invention.
- the left-eye camera device in the binocular camera device captures a left-eye view 110 of the environment
- the right-eye camera device in the binocular camera device captures a right-eye view 120 of the environment.
- L1 is the optical axis of the left-eye camera
- L2 is the optical axis of the right-eye camera.
- the baseline distance B is the distance between the projection center C1 of the left-eye camera device and the projection center C2 of the right-eye camera device.
- the point P(x c , y c , z c ) is the same feature point of the left-eye camera device and the right-eye camera device viewing the space-time object at the same time.
- the disparity can be obtained from the depth information of the P(x c , y c , z c ) point.
- the error of this method increases as the distance of the object increases.
- Matching the left-eye view 110 and the right-eye view 120 captured by the binocular camera device cannot accurately determine the point P(x c , y c , z c ) In-depth information.
- embodiments of the present application provide an image processing method, system, and movable platform.
- the image processing method is further described in detail below.
- FIG. 2 is a schematic flowchart of an image processing method disclosed in an embodiment of the present invention.
- the image processing method may include steps 201-203.
- the above steps 201 to 203 may be executed by an image processing system, or may be executed by a movable platform.
- it can be specifically executed by an image processing system of a movable platform. among them:
- 201 Collect a first view and a second view of the environment through a binocular camera device, and collect a three-dimensional point cloud of the environment through a point cloud sensor.
- the binocular camera device is a left and right binocular camera device.
- the second view is the right-eye view; when the first view is the right-eye view, the second view is Left eye view.
- the arrangement of the binoculars can also be arranged in other directions, such as vertical binoculars. In this case, the first view and the second view are similar to the foregoing.
- the a priori disparity is the disparity value of some pixels of the first view
- the a priori disparity can be understood as an accurate disparity value obtained from the three-dimensional point cloud. Since the 3D point cloud and the first view are respectively the collection of feature points collected by the binocular camera and the point cloud sensor in the same environment, the 3D point cloud is projected to the first view, and the first view can be matched with the 3D point cloud. Of pixels. Then, according to the three-dimensional information of the three-dimensional point cloud corresponding to some pixels of the first view, the prior disparity of some pixels of the first view can be calculated.
- FIG. 3 is a schematic diagram of a first view provided by an embodiment of this application.
- the first view includes a plurality of pixels.
- the solid points are represented as part of the pixels in the first view that match the three-dimensional point cloud
- the hollow points are represented as the pixels in the first view that do not match the three-dimensional point cloud.
- point P is a three-dimensional point in a three-dimensional point cloud.
- the three-dimensional point cloud is projected to the first view, and the pixel point in the first view that matches the three-dimensional point cloud is P left , and the left-eye view 110 can be calculated according to the three-dimensional information corresponding to the P point The prior disparity of P left in the middle.
- the disparity map is based on any image in the image, its size is the size of the reference image, and the element value is an image with disparity values.
- the disparity map between the first view and the second view uses the first image as a reference image to describe the disparity value between the second view and the first view.
- the accurate disparity value of some pixels in the first view is determined according to the three-dimensional point cloud, that is, the prior disparity. According to the prior disparity, an accurate disparity map between the first view and the second view can be obtained. Therefore, by implementing the method described in FIG. 2, an accurate disparity map between the first view and the second view can be calculated, which is beneficial to improve the accuracy of determining the distance.
- FIG. 4 is a schematic flowchart of another image processing method disclosed in an embodiment of the present invention.
- steps 402 to 404 are specific implementations of the above step 202.
- the image processing method may include steps 401-405.
- the above steps 401 to 405 may be executed by the image processing system, or may be executed by the mobile platform.
- it can be specifically executed by an image processing system of a movable platform. among them:
- step 401 reference may be made to the description of step 201, which will not be repeated here.
- step 402 includes: according to the positional relationship between the binocular camera device and the point cloud sensor, projecting the three-dimensional point cloud to the first view, and the difference between the three-dimensional point cloud and the first view. Some pixels are matched.
- the above steps do not constitute a limitation to the embodiment of the present application.
- other implementation manners may also be used to project the three-dimensional point cloud to the first view.
- the three-dimensional point cloud is projected to the first view, which is compared with some pixels of the first view. match.
- the external parameters of the binocular camera device include the positional relationship between the left-eye camera device and the right-eye camera device in the binocular camera device, such as a translation vector and a rotation matrix, which are not limited here.
- projecting the three-dimensional point cloud to the first view can further improve the matching degree between the three-dimensional point cloud and the first view.
- the depth reference information of some pixels may be understood as accurate depth information of some pixels, and may be one-dimensional information in the three-dimensional information of the three-dimensional point cloud.
- the depth reference information may be the value of the Z axis in the three-dimensional information of the three-dimensional point cloud.
- point P is a three-dimensional point in the three-dimensional point cloud.
- the first view is the left-eye view 110
- the three-dimensional point cloud is projected to the first view, then the first view and the three-dimensional point cloud
- the matched pixel is P left .
- the prior depth corresponding to P left is the value of the Z axis of the P point detected by the point cloud sensor.
- a specific implementation manner of determining the a priori disparity according to the prior depth corresponding to the part of pixels is: determining the prior disparity according to the internal parameters of the binocular camera device and the depth information of the part of the pixels.
- the internal parameters of the binocular camera device may include focal length, projection center, tilt coefficient and distortion coefficient, etc., which are not limited here. In general, the internal parameters of the camera do not change over time.
- determining the prior disparity based on the internal parameters of the binocular camera device and the depth information of some pixels can improve the accuracy of determining the prior disparity.
- the first view and the second view of the environment are collected by the binocular camera device, and the three-dimensional point cloud of the environment is collected by the point cloud sensor. Then the 3D point cloud is projected to the first view, matched with part of the pixels of the first view, and the prior depth corresponding to the part of pixels is determined according to the 3D information of the 3D points corresponding to the part of pixels, that is, the The prior depth is the accurate depth information of some pixels in the first view determined according to the three-dimensional point cloud.
- the prior disparity is determined according to the prior depth corresponding to the part of the pixels, and the disparity map between the first view and the second view is obtained according to the prior disparity, and the difference between the first view and the second view can be calculated Accurate disparity map helps to improve the accuracy of distance determination.
- FIG. 5 is a schematic flowchart of another image processing method disclosed in an embodiment of the present invention.
- step 503 and step 504 are specific implementations of step 203 above.
- the image processing method may include steps 501-504.
- the above steps 501 to 504 may be executed by an image processing system, or may be executed by a mobile platform.
- it can be specifically executed by an image processing system of a movable platform. among them:
- 501 Collect a first view and a second view of an environment through a binocular camera device, and collect a three-dimensional point cloud of the environment through a point cloud sensor.
- step 501 and step 502 can refer to the description of step 201 and step 202 respectively, which will not be repeated here.
- the target similarity is the similarity between the first view and the second view.
- Step 503 may include the following steps A1 and A2, where:
- A1. Perform feature extraction on the first view to obtain a first feature map, and perform feature extraction on the second view to obtain a second feature map.
- feature extraction is used to identify feature points in the view, and extract the feature value corresponding to the feature point, so that the feature map obtained according to the feature point and its corresponding feature value can be distinguished from other views.
- feature points include parts of the vehicle that can be clearly distinguished from other objects. For example, corners of vehicle boundaries, lights, rearview mirrors, etc.
- a feature map in the vehicle can be obtained to identify the feature map as an image of the vehicle.
- step A1 includes: performing feature extraction on the first view according to the census transformation algorithm to obtain the first feature map, and performing feature extraction on the second view according to the census transformation algorithm to obtain the second feature map.
- the census transformation algorithm is a kind of non-parametric image transformation, which can better detect the local structural features in the image, such as edge and corner features. Its essence is to encode the gray value of the image pixel into a binary code stream to obtain the relationship between the gray value of the neighborhood pixel and the gray value of the center pixel.
- the central pixel is used as the reference pixel, and a rectangular window is defined in the image area. Compare the gray value of each pixel in the rectangular window with the gray value of the reference pixel.
- the pixels whose gray value is less than or equal to the reference value are marked as 0, the pixels greater than the reference value are marked as 1, and finally they are bitwise Connect to get the transformed result, and the transformed result is a binary code stream composed of 0 and 1.
- feature extraction is performed on the first view and the second view according to the census transformation algorithm, which retains the location features of the pixels in the window, can reduce mismatches caused by illumination differences, and improve the extraction efficiency and accuracy of local features, thereby Improve the accuracy of the first feature map and the second feature map.
- A2. Determine the target similarity between the first feature map and the second feature map.
- step A2 includes: calculating the Hamming distance between the first feature map and the second feature map, and determining the Hamming distance as the target similarity between the first feature map and the second feature map.
- the Hamming distance indicates the number of different bits corresponding to two (same length) words. Perform an XOR operation on two strings, and count the number of 1, then this number is the Hamming distance. It should be noted that the smaller the Hamming distance, the higher the similarity.
- obtaining the first view and the second view based on the Hamming distance between the first view and the second view can improve the accuracy of determining the target similarity.
- steps A1 and A2 By implementing steps A1 and A2, a specific implementation algorithm is provided, which can improve the stability and accuracy of determining the target similarity.
- step 503 includes: performing feature extraction on the first view according to the census transformation algorithm to obtain the first feature map, and performing feature extraction on the second view according to the census transformation algorithm to obtain the second feature map;
- the Hamming distance between the first feature map and the second feature map is determined to be the target similarity between the first feature map and the second feature map.
- the census transformed image uses the Hamming distance to calculate the similarity, which is to find the point with the highest similarity to the reference pixel in the disparity map, and the Hamming distance is the measure of the similarity between the disparity map pixel and the reference pixel. . In this way, the accuracy of determining the target similarity can be further improved.
- step 504 includes step B1 and step B2, where:
- the optimization solution model is a model in which the prior disparity and the target similarity are known parameters to solve the disparity map between the first view and the second view.
- the optimization solution model may be a conditional probability distribution model, and the mathematical expression formula of the conditional probability distribution model is: P(Y
- X is a known variable, that is, the prior disparity and target similarity in the embodiment of the present application
- Y is a random variable.
- the conditional probability distribution model can be understood as a causal derivation model in an uncertain environment, that is, solving the maximum probability of Y, which is the optimal disparity map.
- conditional probability distribution model may be a conditional random field (CRF).
- the conditional random field is a discriminative probability model, a kind of conditional probability distribution model, which represents a Markov random field of a given set of input random variables X and another set of output random variables Y.
- the conditional random field is used to calculate the disparity map between the first view and the second view, which can improve the accuracy of obtaining the view difference.
- the view difference between the first view and the second view can be calculated based on the optimized solution model obtained by the prior disparity and the target similarity, which improves the accuracy of obtaining the view difference.
- the first view and the second view of the environment are collected by the binocular camera device, and the three-dimensional point cloud of the environment is collected by the point cloud sensor. Then the three-dimensional point cloud is projected to the first view and matched with part of the pixels of the first view to obtain a priori disparity, that is, the prior disparity is determined according to the three-dimensional point cloud to be accurate for some pixels in the first view The parallax value. Finally, the target similarity is obtained according to the first view and the second view, and the disparity map between the first view and the second view is obtained according to the prior disparity and the target similarity, thereby further improving the determination of the disparity map. Accuracy helps improve the accuracy of determining the distance.
- FIG. 6 is a schematic structural diagram of an image processing system provided in an embodiment of the application.
- the image processing system includes a memory 601, a processor 602, a binocular camera 603, and a point cloud sensor 604.
- the memory 601, the processor 602, the binocular camera 603, and the point cloud sensor 604 may be connected through a communication system 605.
- the memory 601 is used to store program instructions.
- the memory 601 may include a volatile memory (volatile memory), such as a random-access memory (random-access memory, RAM); the memory 601 may also include a non-volatile memory (non-volatile memory), such as a flash memory (flash memory). memory), solid-state drive (SSD), etc.; the memory 601 may also include a combination of the foregoing types of memories.
- the processor 602 may include a central processing unit (CPU).
- the processor 602 may further include a hardware chip.
- the aforementioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), etc.
- ASIC application-specific integrated circuit
- PLD programmable logic device
- the above-mentioned PLD may be a field-programmable gate array (FPGA), a generic array logic (GAL), etc.
- the binocular camera device 603 is used to collect the first view and the second view of the environment
- the point cloud sensor 604 is used to collect a three-dimensional point cloud of the environment
- the processor 602 calls the program instructions in the memory 601 to execute the following steps:
- the processor 602 projects the three-dimensional point cloud to the first view to match some pixels of the first view to obtain the prior disparity specifically as follows:
- the prior disparity is determined according to the prior depth corresponding to the partial pixels.
- the processor 602 projects the three-dimensional point cloud to the first view, and the method for matching with some pixels of the first view is specifically:
- the three-dimensional point cloud is projected to the first view to match some pixels of the first view.
- the method for the processor 602 to determine the a priori disparity according to the depth information of the partial pixels is specifically as follows:
- the prior disparity is determined according to the internal parameters of the binocular camera device and the depth information of the partial pixels.
- the processor 602 obtains the disparity map between the first view and the second view according to the prior disparity specifically as follows:
- a disparity map between the first view and the second view is obtained.
- the way that the processor 602 obtains the target similarity according to the first view and the second view is specifically as follows:
- the processor 602 performs feature extraction on the first view to obtain a first feature map, and performs feature extraction on the second view to obtain the second feature map specifically as follows:
- the processor 502 determines the target similarity between the first feature map and the second feature map specifically as follows:
- the Hamming distance is determined as the target similarity between the first feature map and the second feature map.
- the way for the processor 602 to obtain the disparity map between the first view and the second view according to the prior disparity and the target similarity is specifically as follows:
- a disparity map between the first view and the second view is obtained.
- the optimization solution model is a conditional probability distribution model.
- conditional probability distribution model is a conditional random field.
- the principle of the image processing system provided in the embodiment of this application to solve the problem is similar to the method embodiment of this application, so the implementation of the image processing system can refer to the implementation of the method, and the beneficial effects of the image processing system can refer to the benefits of the method. The effect is described briefly and will not be repeated here.
- FIG. 7 is a schematic structural diagram of a movable platform provided in an embodiment of the application.
- the movable platform can be a vehicle, drone, ground robot, smart car, etc.
- the mobile platform includes a memory 701, a processor 702, a binocular camera 703 and a point cloud sensor 704.
- the memory 701, the processor 702, the binocular camera 703, and the point cloud sensor 704 may be connected through a communication system 605.
- the memory 701 and the processor 702 can refer to the description in FIG. 6, which is not repeated here.
- the binocular camera device 703 is used to collect the first view and the second view of the environment
- the point cloud sensor 704 is used to collect a three-dimensional point cloud of the environment
- the processor 702 calls the program instructions in the memory 701 to execute the following steps:
- the processor 702 projects the three-dimensional point cloud to the first view to match some pixels of the first view to obtain the prior disparity specifically as follows:
- the prior disparity is determined according to the prior depth corresponding to the partial pixels.
- the processor 702 projects the three-dimensional point cloud to the first view, and the method for matching with some pixels of the first view is specifically as follows:
- the three-dimensional point cloud is projected to the first view to match some pixels of the first view.
- the method for the processor 702 to determine the a priori disparity according to the depth information of the partial pixels is specifically as follows:
- the prior disparity is determined according to the internal parameters of the binocular camera device and the depth information of the partial pixels.
- the processor 702 obtains the disparity map between the first view and the second view according to the prior disparity specifically as follows:
- a disparity map between the first view and the second view is obtained.
- the method for the processor 702 to acquire the target similarity according to the first view and the second view is specifically:
- the processor 702 performs feature extraction on the first view to obtain a first feature map, and performs feature extraction on the second view to obtain the second feature map specifically as follows:
- the processor 702 specifically determines the target similarity between the first feature map and the second feature map as follows:
- the Hamming distance is determined as the target similarity between the first feature map and the second feature map.
- the way for the processor 702 to obtain the disparity map between the first view and the second view according to the prior disparity and the target similarity is specifically:
- a disparity map between the first view and the second view is obtained.
- the optimization solution model is a conditional probability distribution model.
- conditional probability distribution model is a conditional random field.
- the principles of the mobile platform provided in the embodiments of this application to solve the problem are similar to the method embodiments of this application, so the implementation of the mobile platform can refer to the implementation of the method, and the beneficial effects of the mobile platform can be referred to the benefits of the method. The effect is described briefly and will not be repeated here.
- a computer-readable storage medium is also provided, and the computer-readable storage medium stores a computer program.
- the embodiment of the present application corresponds to FIG. 2, FIG. 4, and FIG. 5. The image processing method described in the embodiment will not be repeated here.
- the computer-readable storage medium may be an internal storage unit of the image processing system or a removable platform described in any of the foregoing embodiments, such as a hard disk or a memory.
- the computer-readable storage medium may also be an image processing system or an external storage device of a removable platform, such as a plug-in hard disk equipped on a removable platform, a smart memory card (Smart Media Card, SMC), and a secure digital (Secure Digital) , SD) card, flash card (Flash Card), etc.
- the computer-readable storage medium may also include both an internal storage unit of the control image processing system or a movable platform and an external storage device.
- the computer-readable storage medium is used to store the computer program and other programs and data required by the image processing system or the movable platform.
- the computer-readable storage medium can also be used to temporarily store data that has been output or will be output.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
本申请公开了一种图像处理方法、设备及可移动平台，其中方法包括：通过双目摄像装置采集环境的第一视图和第二视图，并通过点云传感器采集所述环境的三维点云；将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配，得到先验视差；根据所述先验视差，得到所述第一视图和所述第二视图之间的视差图。可见，通过实施本申请，有利于提升确定视差图的准确性。
Description
本申请涉及图像处理技术领域，尤其涉及一种图像处理方法、系统及可移动平台。
随着智能控制技术的不断迭代发展，一些车辆开始配置自动驾驶系统或辅助驾驶系统，自动驾驶系统或辅助驾驶系统可以给操作者带来许多便利。类似的，智能控制的技术也被运用在其他的可移动平台上来实现自动运动或辅助运动的功能，例如机器人、智能小车、无人机等。
在这些实现智能控制的系统中，一个很重要的功能是自动识别可移动平台(如自动驾驶车辆、智能小车、无人机等)与周围环境中物体之间的距离。在可移动平台的运动过程中，通常通过双目摄像装置采集可移动平台周围环境的左目视图和右目视图，再确定左目视图和右目视图之间的视差图。再根据该视差图确定可移动平台与周围环境中物体之间的距离。然而，该方法的误差随着物体的距离增加而增加，导致对于较远的物体确定出的视差图的精度较低，进而导致无法准确地确定出与较远的物体之间的距离。因此，如何准确地确定视差图是目前亟待解决的问题。
发明内容
本申请公开了一种图像处理方法、设备及可移动平台，有利于提升确定视差图的准确性。
第一方面，本申请提供了一种图像处理方法，包括：
通过双目摄像装置采集环境的第一视图和第二视图，并通过点云传感器采集所述环境的三维点云；
将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配，得到先验视差；
根据所述先验视差，得到所述第一视图和所述第二视图之间的视差图。
第二方面，本申请提供了一种图像处理系统，所述图像处理系统包括：存储器、处理器、双目摄像装置和点云传感器，其中：
所述存储器，用于存储程序指令；
所述双目摄像装置，用于采集环境的第一视图和第二视图；
所述点云传感器，用于采集所述环境的三维点云；
所述处理器，调用所述程序指令以用于：
将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配，得到先验视差；
根据所述先验视差，得到所述第一视图和所述第二视图之间的视差图。
第三方面，本申请提供了一种可移动平台，所述可移动平台包括：存储器、处理器、双目摄像装置和点云传感器，其中：
所述存储器，用于存储程序指令；
所述双目摄像装置，用于采集环境的第一视图和第二视图；
所述点云传感器，用于采集所述环境的三维点云；
所述处理器，调用所述程序指令以用于：
将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配，得到先验视差；
根据所述先验视差，得到所述第一视图和所述第二视图之间的视差图。
本申请实施例中提供的图像处理方法、设备及可移动平台，通过双目摄像装置采集环境的第一视图和第二视图，并通过点云传感器采集该环境的三维点云。然后将该三维点云投影至该第一视图，与该第一视图的部分像素点进行匹配得到先验视差，即该先验视差为根据三维点云确定第一视图中的部分像素点准确的视差值。最后根据该先验视差得到该第一视图和该第二视图之间的视差图，提高了确定视差图的准确性，有利于提高确定距离的准确性。
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本申请实施例提供的一种现有的基于双目摄像装置进行距离测量的原理示意图；
图2是本申请实施例提供的一种图像处理方法的流程示意图；
图3是本申请实施例提供的一种第一视图的示意图；
图4是本申请实施例提供的另一种图像处理方法的流程示意图；
图5是本申请实施例提供的又一种图像处理方法的流程示意图；
图6是本申请实施例提供的一种图像处理系统的结构示意图。
图7是本申请实施例提供的一种可移动平台的结构示意图。
为了使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请实施例的技术方案进行描述。
本发明实施例提出了一种图像处理方法、设备及可移动平台。其中，该图像处理方法可由图像处理系统执行，或者，该图像处理方法可由可移动平台执行。其中，该可移动平台可以包括但不限于无人机、无人船、地面机器人、智能小车、无人车等。当该图像处理方法由图像处理系统执行时，该图像处理系统可包括于该可移动平台中。在一些实施方式中，图像处理系统可以为一个具体的图像处理设备，并且与可移动平台之间可以通过无线通信连接方式建立通信连接，或者与可移动平台之间通过有线通信连接方式建立通信连接。在另一些实施方式中，图像处理系统还可以为分布式的形式，其所包含的各个部件或装置可以分布式地设置在可移动平台，并且各个部件或装置之间可以通过有线、无线、通信总线等方式连接，并且图像处理系统与可移动平台也可以通信连接。
在本申请实施例中，图像处理系统包括双目摄像装置。该双目摄像装置用于采集周围环境的左目视图和右目视图，包括但不限于可见光相机、灰度相机或红外相机等。可选的，双目摄像装置可以通过承载装置配置在图像处理系统的机身上。
在本申请实施例中，图像处理系统还可包括点云传感器。该点云传感器用于采集环境的三维点云。其中，三维点云包括各个三维点的特征信息，即环境中特征点的三维信息。该点云传感器包括但不限于激光雷达传感器、毫米波雷达传感器和超声波雷达传感器等。
当图像处理方法由可移动平台执行时，该可移动平台可包括上述的双目摄像装置和点云传感器，其描述可参照图像处理系统中的双目摄像装置和点云传感器的描述，在此不再赘述。
在某些实施例中，该可移动平台还可包括通信装置，用于与控制终端进行通信。该控制终端用于对该可移动平台进行控制。该控制终端可以为手机、平板电脑、遥控器或其他穿戴式设备(手表或手环)等，本申请实施例不做限定。
为了便于对本申请的方案进行理解，下面先对现有的基于双目摄像装置进行距离测量的原理进行介绍。
请参阅图1，图1为本发明实施例公开的一种现有的基于双目摄像装置进行距离测量的原理示意图。如图1所示，双目摄像装置中的左目摄像装置采集环境 的左目视图110，双目摄像装置中的右目摄像装置采集环境的右目视图120。L1为左目摄像装置的光轴，L2为右目摄像装置的光轴。基线距离B为左目摄像装置的投影中心C1和右目摄像装置的投影中心C2之间的距离。P(x
c，y
c，z
c)点为左目摄像装置和右目摄像装置在同一时刻观看时空物体的同一特征点。P(x
c，y
c，z
c)点在左目视图110中的坐标为P
left＝(x
left，y
left)，P(x
c，y
c，z
c)点在右目视图120中的坐标为P
right＝(x
right，y
right)。而左目视图110和右目视图120处于同一个Y平面，则特征点P的图像坐标中的Y坐标相等，即y
left＝y
right＝y。由三角几何关系可以得到如下公式(1)：
其中，f为双目摄像装置的焦距。由于左目视图110和右目视图120之间对于P(x
c，y
c，z
c)点的视差Disparity＝x
left-x
right，则特征点P在该双目摄像装置的坐标系下的三维坐标，可通过如下所示的公式(2)进行计算得到。
可见，视差可根据P(x
c，y
c，z
c)点的深度信息得到。然而，该方法的误差随着物体的距离增加而增加，对双目摄像装置拍摄的左目视图110和右目视图120进行匹配，无法准确地确定出P(x
c，y
c，z
c)点的深度信息。进而无法根据P(x
c，y
c，z
c)点的深度信息准确地确定出P(x
c，y
c，z
c)点对应的视差，进而导致无法准确地确定出与P(x
c，y
c，z
c)点之间的距离。
因此，为了能够准确地确定出双目摄像装置拍摄的左目视图110和右目视图120之间的视差图，本申请实施例提供了一种图像处理方法、系统及可移动平台。以下进一步对该图像处理方法进行详细地介绍。
请参阅图2，图2为本发明实施例公开的一种图像处理方法的流程示意图。如图2所示，该图像处理方法可包括步骤201～203。上述步骤201～203可以由图像处理系统执行，也可以由可移动平台执行。可选的，具体可以由可移动平台的图像处理系统执行。其中：
201、通过双目摄像装置采集环境的第一视图和第二视图，并通过点云传感 器采集该环境的三维点云。
在本申请实施例中，双目摄像装置为左右双目摄像装置，相应的，当第一视图为左目视图时，第二视图为右目视图；当第一视图为右目视图时，第二视图为左目视图。可以理解的是，双目的排列方式也可以为其他方向的排列，例如竖排的双目，此时第一视图与第二视图与前述类似。
202、将该三维点云投影至该第一视图，与该第一视图的部分像素点进行匹配得到先验视差。
在本申请实施例中，先验视差为第一视图的部分像素点的视差值，该先验视差可以理解为根据三维点云得到的准确的视差值。由于三维点云和第一视图分别为双目摄像装置和点云传感器在同一环境下采集的特征点的集合，将三维点云投影至第一视图，可获取第一视图与三维点云中匹配的像素点。然后，根据第一视图的部分像素点对应的三维点云的三维信息，可计算第一视图的部分像素点的先验视差。
例如，请参照图3，图3为本申请实施例提供的一种第一视图的示意图。如图3所示，该第一视图中包括多个像素点。其中，实心点表示为第一视图中与三维点云匹配的部分像素点，空心点表示为第一视图中与三维点云不匹配的像素点。结合图1进行举例说明，P点为三维点云中的一个三维点。在第一视图为左目视图110时，将三维点云投影至第一视图，则第一视图中与三维点云中匹配的像素点为P
left，可根据P点对应的三维信息计算左目视图110中P
left的先验视差。
203、根据该先验视差得到该第一视图和该第二视图之间的视差图。
视差图是以图像中的任一图像为基准，其大小为该基准图像的大小，元素值为视差值的图像。在本申请实施例中，第一视图和第二视图之间的视差图以第一图像为基准图像，用于描述第二视图与第一视图之间的视差值。
可以理解，根据三维点云确定第一视图中的部分像素点准确的视差值，即先验视差。再根据先验视差就可得到第一视图和该第二视图之间准确的视差图。因此，通过实施图2所描述的方法，能够计算出第一视图和该第二视图之间准确的视差图，有利于提高确定距离的准确性。
请参阅图4，图4为本发明实施例公开的另一种图像处理方法的流程示意图。其中，步骤402～404为上述步骤202的具体实施方式。如图4所示，该图像处理方法可包括步骤401～405。上述步骤401～405可以由图像处理系统执行，也可以由可移动平台执行。可选的，具体可以由可移动平台的图像处理系统执行。其中：
401、通过双目摄像装置采集环境的第一视图和第二视图，并通过点云传感器采集该环境的三维点云。
其中，步骤401可参照步骤201的描述，在此不在赘述。
402、将该三维点云投影至该第一视图，与该第一视图的部分像素点进行匹配。
在本申请实施例中，步骤402的具体实施方式包括：根据该双目摄像装置与该点云传感器之间的位置关系，将该三维点云投影至该第一视图，与该第一视图的部分像素点进行匹配。
可以理解，根据双目摄像装置与点云传感器之间的位置关系，将三维点云投影至第一视图，可提高三维点云与第一视图的匹配程度，从而有利于提高确定先验视差的准确性。
需要说明的是，上述步骤并不构成对本申请实施例的限定，实际应用中，还可以采用其他实施方式将三维点云投影至第一视图。例如，根据该双目摄像装置与该点云传感器之间的位置关系和该双目摄像装置的外参，将该三维点云投影至该第一视图，与该第一视图的部分像素点进行匹配。其中，双目摄像装置的外参包括双目摄像装置中左目摄像装置和右目摄像装置之间的位置关系，例如平移向量和旋转矩阵等，在此不做限定。可见，根据双目摄像装置与点云传感器之间的位置关系和该双目摄像装置的外参，将三维点云投影至第一视图，可进一步提高三维点云与第一视图的匹配程度。
403、根据该部分像素点对应的三维点云的三维信息确定该部分像素点对应的先验深度，该先验深度为部分像素点的深度参考信息。
在本申请实施例中，部分像素点的深度参考信息可以理解为部分像素点的准确的深度信息，可以是三维点云的三维信息中的一维信息。例如，深度参考信息可以是三维点云的三维信息中Z轴的值。
例如，如图1所示，P点为三维点云中的一个三维点，在第一视图为左目视图110时，将三维点云投影至第一视图，则第一视图中与三维点云中匹配的像素点为P
left。P
left对应的先验深度为点云传感器检测到的P点的Z轴的值。
404、根据该部分像素点对应的先验深度确定先验视差。
可选的，根据该部分像素点对应的先验深度确定先验视差的具体实施方式为：根据双目摄像装置的内参和部分像素点的深度信息确定先验视差。
其中，双目摄像装置的内参可包括焦距、投影中心、倾斜系数和畸变系数等，在此不做限定。一般情况下，相机的内参是不随时间变化的。
可以理解，根据双目摄像装置的内参和部分像素点的深度信息确定先验视差，可提高确定先验视差的准确性。
405、根据该先验视差得到该第一视图和该第二视图之间的视差图。
通过实施图4所描述的方法，通过双目摄像装置采集环境的第一视图和第二视图，并通过点云传感器采集该环境的三维点云。然后将该三维点云投影至该第一视图，与该第一视图的部分像素点进行匹配，根据该部分像素点对应的三维点的三维信息确定该部分像素点对应的先验深度，即该先验深度为根据三维点云确定第一视图中的部分像素点准确的深度信息。最后根据该部分像素点对应的先验深度确定先验视差，根据该先验视差得到该第一视图和该第二视图之 间的视差图，能够计算出第一视图和该第二视图之间准确的视差图，有利于提高确定距离的准确性。
请参阅图5，图5为本发明实施例公开的又一种图像处理方法的流程示意图。其中，步骤503和步骤504为上述步骤203的具体实施方式。如图5所示，该图像处理方法可包括步骤501～504。上述步骤501～504可以由图像处理系统执行，也可以由可移动平台执行。可选的，具体可以由可移动平台的图像处理系统执行。其中：
501、通过双目摄像装置采集环境的第一视图和第二视图，并通过点云传感器采集该环境的三维点云。
502、将该三维点云投影至该第一视图，与该第一视图的部分像素点进行匹配，得到先验视差。
其中，步骤501和步骤502可分别参照步骤201和步骤202的描述，在此不在赘述。
503、根据该第一视图和该第二视图获取目标相似度。
在本申请实施例中，目标相似度为第一视图和第二视图之间的相似度。步骤503可包括以下步骤A1和步骤A2，其中：
A1、对该第一视图进行特征提取，得到第一特征图，并且对该第二视图进行特征提取，得到第二特征图。
其中，特征提取用于识别视图中的特征点，并提取该特征点对应的特征值，以使根据该特征点和其对应的特征值得到的特征图可与其它的视图进行区别。以车辆进行举例说明，特征点包括车辆中可明显与其它物体进行区分的部位。例如，车辆边界的角、车灯、后视镜等。通过特征提取可得到该车辆中的特征图，以识别该特征图为车辆的图像。
可选的，步骤A1的具体实施方式包括：按照census变换算法对第一视图进行特征提取，得到第一特征图，并且按照census变换算法对第二视图进行特征提取，得到第二特征图。
其中，census变换算法属于非参数图像变换的一种，它能够较好地检测出图像中的局部结构特征，如边缘、角点特征等。其实质是将图像像素的灰度值编码成二进制码流，以此来获取邻域像素灰度值相对于中心像素灰度值的大小关系。具体的，将中心像素作为参考像素，在图像区域定义一个矩形窗口。将矩形窗口中每个像素的灰度值与参考像素的灰度值进行比较，灰度值小于或等于参考值的像素标记为0，大于参考值的像素标记为1，最后再将它们按位连接得到变换后的结果，变换后的结果是由0和1组成的二进制码流。
可以理解，分别按照census变换算法对第一视图和第二视图进行特征提取，保留了窗口中像素的位置特征，能够减少光照差异引起的误匹配，提高了局部特征的提取效率和准确率，从而提高第一特征图和第二特征图的准确性。
A2、确定该第一特征图和该第二特征图之间的目标相似度。
可选的，步骤A2的具体实施方式包括：计算第一特征图和第二特征图之间的汉明距离，确定汉明距离为第一特征图和第二特征图之间的目标相似度。
其中，汉明距离表示两个(相同长度)字对应位不同的数量。对两个字符串进行异或运算，并统计结果为1的个数，那么这个数就是汉明距离。需要说明的是，汉明距离越小即相似度越高。
可以理解，基于第一视图和第二视图之间的汉明距离得到第一视图和第二视图，可提高确定目标相似度的准确性。
通过实施步骤A1和步骤A2，提供了具体的实施算法，可提高确定目标相似度的稳定性和准确性。
可选的，步骤503的具体实施方式包括：按照census变换算法对第一视图进行特征提取，得到第一特征图，并且按照census变换算法对第二视图进行特征提取，得到第二特征图；计算第一特征图和第二特征图之间的汉明距离，确定汉明距离为第一特征图和第二特征图之间的目标相似度。
可以理解，经过census变换后的图像使用汉明距离计算相似度，就是在视差图中找出与参考像素点相似度最高的点，而汉明距离正是视差图像素与参考像素相似度的度量。如此，可进一步提高确定目标相似度的准确性。
504、根据该先验视差和该目标相似度得到该第一视图和该第二视图之间的视差图。
可选的，步骤504的具体实施方式包括步骤B1和步骤B2，其中：
B1、根据该先验视差和该目标相似度构造优化求解模型。
其中，优化求解模型是以先验视差和目标相似度为已知参数，求解第一视图和第二视图之间的视差图的模型。可选的，该优化求解模型可以为条件概率分布模型，该条件概率分布模型的数学表达公式为：P(Y|X)。其中，X是已知变量，即本申请实施例中的先验视差和目标相似度，Y都是随机变量。该条件概率分布模型可以理解为在不确定环境下的因果推导模型，即求解Y的概率最大值，也就是最优的视差图。
可选的，该条件概率分布模型可以为条件随机场(conditional random field，CRF)。该条件随机场是一种鉴别式机率模型，是条件概率分布模型的一种，表示的是给定一组输入随机变量X的条件下另一组输出随机变量Y的马尔可夫随机场。在本申请实施例中，以条件随机场计算第一视图和第二视图之间的视差图，可提高获取视图差的准确性。
B2、根据该优化求解模型，得到该第一视图和该第二视图之间的视差图。
通过实施步骤B1和步骤B2，可根据先验视差和目标相似度得到的优化求解模型，计算第一视图和第二视图之间的视图差，提高了获取视图差的准确性。
通过实施图5所描述的方法，通过双目摄像装置采集环境的第一视图和第二 视图，并通过点云传感器采集该环境的三维点云。然后将该三维点云投影至该第一视图，与该第一视图的部分像素点进行匹配得到先验视差，即该先验视差为根据三维点云确定第一视图中的部分像素点准确的视差值。最后根据该第一视图和该第二视图获取目标相似度，根据该先验视差和该目标相似度得到该第一视图和该第二视图之间的视差图，从而进一步提高了确定视差图的准确性，有利于提高确定距离的准确性。
请参见图6，图6为本申请实施例中提供的一种图像处理系统的结构示意图，该图像处理系统包括存储器601、处理器602、双目摄像装置603和点云传感器604。可选的，存储器601、处理器602、双目摄像装置603和点云传感器604可通过通信系统605相连。
存储器601，用于存储程序指令。存储器601可以包括易失性存储器(volatile memory)，例如随机存取存储器(random-access memory，RAM)；存储器601也可以包括非易失性存储器(non-volatile memory)，例如快闪存储器(flash memory)，固态硬盘(solid-state drive，SSD)等；存储器601还可以包括上述种类的存储器的组合。
处理器602可以包括中央处理器(central processing unit，CPU)。处理器602还可以进一步包括硬件芯片。上述硬件芯片可以是专用集成电路(application-specific integrated circuit，ASIC)，可编程逻辑器件(programmable logic device，PLD)等。上述PLD可以是现场可编程逻辑门阵列(field-programmable gate array，FPGA)，通用阵列逻辑(generic array logic，GAL)等。
在本申请实施例中，所述双目摄像装置603，用于采集环境的第一视图和第二视图；
所述点云传感器604，用于采集所述环境的三维点云；
所述处理器602调用所述存储器601中的程序指令用于执行以下步骤：
将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配，得到先验视差；
根据所述先验视差，得到所述第一视图和所述第二视图之间的视差图。
可选的，处理器602将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配，得到先验视差的方式具体为：
将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配；
根据所述部分像素点对应的三维点云的三维信息确定所述部分像素点对应 的先验深度，所述先验深度为所述部分像素点的深度参考信息；
根据所述部分像素点对应的先验深度确定先验视差。
可选的，处理器602将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配的方式具体为：
根据所述双目摄像装置与所述点云传感器之间的位置关系，将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配。
可选的，处理器602根据所述部分像素点的深度信息确定先验视差的方式具体为：
根据所述双目摄像装置的内参和所述部分像素点的深度信息确定先验视差。
可选的，处理器602根据所述先验视差，得到所述第一视图和所述第二视图之间的视差图的方式具体为：
根据所述第一视图和所述第二视图获取目标相似度；
根据所述先验视差和所述目标相似度，得到所述第一视图和所述第二视图之间的视差图。
可选的，处理器602根据所述第一视图和所述第二视图获取目标相似度的方式具体为：
对所述第一视图进行特征提取，得到第一特征图，并且对所述第二视图进行特征提取，得到第二特征图；
确定所述第一特征图和所述第二特征图之间的目标相似度。
可选的，处理器602对所述第一视图进行特征提取，得到第一特征图，并且对所述第二视图进行特征提取，得到第二特征图的方式具体为：
按照census变换算法对所述第一视图进行特征提取，得到第一特征图，并且按照census变换算法对所述第二视图进行特征提取，得到第二特征图。
可选的，处理器502确定所述第一特征图和所述第二特征图之间的目标相似度的方式具体为：
计算所述第一特征图和所述第二特征图之间的汉明距离；
确定所述汉明距离为所述第一特征图和所述第二特征图之间的目标相似度。
可选的，处理器602根据所述先验视差和所述目标相似度，得到所述第一视图和所述第二视图之间的视差图的方式具体为：
根据所述先验视差和所述目标相似度构造优化求解模型；
根据所述优化求解模型，得到所述第一视图和所述第二视图之间的视差图。
可选的，所述优化求解模型为条件概率分布模型。
可选的，所述条件概率分布模型为条件随机场。
基于同一发明构思，本申请实施例中提供的图像处理系统解决问题的原理与本申请方法实施例相似，因此图像处理系统的实施可以参见方法的实施，图 像处理系统的有益效果可以参见方法的有益效果，为简洁描述，在这里不再赘述。
请参见图7，图7为本申请实施例中提供的一种可移动平台的结构示意图。可移动平台可以为车辆、无人机、地面机器人、智能小车等。该可移动平台包括存储器701、处理器702、双目摄像装置703和点云传感器704。可选的，存储器701、处理器702、双目摄像装置703和点云传感器704可通过通信系统605相连。
其中，存储器701和处理器702可参照图6中的描述，在此不在赘述。
在本申请实施例中，所述双目摄像装置703，用于采集环境的第一视图和第二视图；
所述点云传感器704，用于采集所述环境的三维点云；
所述处理器702调用所述存储器701中的程序指令用于执行以下步骤：
将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配，得到先验视差；
根据所述先验视差，得到所述第一视图和所述第二视图之间的视差图。
可选的，处理器702将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配，得到先验视差的方式具体为：
将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配；
根据所述部分像素点对应的三维点云的三维信息确定所述部分像素点对应的先验深度，所述先验深度为所述部分像素点的深度参考信息；
根据所述部分像素点对应的先验深度确定先验视差。
可选的，处理器702将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配的方式具体为：
根据所述双目摄像装置与所述点云传感器之间的位置关系，将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配。
可选的，处理器702根据所述部分像素点的深度信息确定先验视差的方式具体为：
根据所述双目摄像装置的内参和所述部分像素点的深度信息确定先验视差。
可选的，处理器702根据所述先验视差，得到所述第一视图和所述第二视图之间的视差图的方式具体为：
根据所述第一视图和所述第二视图获取目标相似度；
根据所述先验视差和所述目标相似度，得到所述第一视图和所述第二视图之间的视差图。
可选的，处理器702根据所述第一视图和所述第二视图获取目标相似度的方 式具体为：
对所述第一视图进行特征提取，得到第一特征图，并且对所述第二视图进行特征提取，得到第二特征图；
确定所述第一特征图和所述第二特征图之间的目标相似度。
可选的，处理器702对所述第一视图进行特征提取，得到第一特征图，并且对所述第二视图进行特征提取，得到第二特征图的方式具体为：
按照census变换算法对所述第一视图进行特征提取，得到第一特征图，并且按照census变换算法对所述第二视图进行特征提取，得到第二特征图。
可选的，处理器702确定所述第一特征图和所述第二特征图之间的目标相似度的方式具体为：
计算所述第一特征图和所述第二特征图之间的汉明距离；
确定所述汉明距离为所述第一特征图和所述第二特征图之间的目标相似度。
可选的，处理器702根据所述先验视差和所述目标相似度，得到所述第一视图和所述第二视图之间的视差图的方式具体为：
根据所述先验视差和所述目标相似度构造优化求解模型；
根据所述优化求解模型，得到所述第一视图和所述第二视图之间的视差图。
可选的，所述优化求解模型为条件概率分布模型。
可选的，所述条件概率分布模型为条件随机场。
基于同一发明构思，本申请实施例中提供的可移动平台解决问题的原理与本申请方法实施例相似，因此可移动平台的实施可以参见方法的实施，可移动平台的有益效果可以参见方法的有益效果，为简洁描述，在这里不再赘述。
需要说明的是，对于前述的各个方法实施例，为了简单描述，故将其都表述为一系列的动作组合，但是本领域技术人员应该知悉，本申请并不受所描述的动作顺序的限制，因为依据本申请，某一些步骤可以采用其他顺序或者同时进行。其次，本领域技术人员也应该知悉，说明书中所描述的实施例均属于优选实施例，所涉及的动作和模块并不一定是本申请所必须的。
本领域技术人员应该可以意识到，在上述一个或多个示例中，本申请所描述的功能可以用硬件、软件、固件或它们的任意组合来实现。
在本申请实施例中还提供了一种计算机可读存储介质，计算机可读存储介质存储有计算机程序，计算机程序被处理器执行时，实现本申请实施例图2、图4和图5所对应实施例中描述的图像处理方法，在此不再赘述。
所述计算机可读存储介质可以是前述任一实施例所述的图像处理系统或可移动平台的内部存储单元，例如硬盘或内存。所述计算机可读存储介质也可以是图像处理系统或可移动平台的外部存储设备，例如可移动平台上配备的插接 式硬盘，智能存储卡(Smart Media Card，SMC)，安全数字(Secure Digital，SD)卡，闪存卡(Flash Card)等。进一步地，所述计算机可读存储介质还可以既包括所述控图像处理系统或可移动平台的内部存储单元也包括外部存储设备。所述计算机可读存储介质用于存储所述计算机程序以及所述图像处理系统或所述可移动平台所需的其他程序和数据。所述计算机可读存储介质还可以用于暂时地存储已经输出或者将要输出的数据。
以上所述的具体实施方式，对本申请的目的、技术方案和有益效果进行了进一步详细说明，所应理解的是，以上所述仅为本申请的具体实施方式而已，并不用于限定本申请的保护范围，凡在本申请的技术方案的基础之上，所做的任何修改、等同替换、改进等，均应包括在本申请的保护范围之内。
Claims (34)
- 一种图像处理方法，其特征在于，包括：通过双目摄像装置采集环境的第一视图和第二视图，并通过点云传感器采集所述环境的三维点云；将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配，得到先验视差；根据所述先验视差，得到所述第一视图和所述第二视图之间的视差图。
- 根据权利要求1所述的方法，其特征在于，所述将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配，得到先验视差，包括：将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配；根据所述部分像素点对应的三维点云的三维信息确定所述部分像素点对应的先验深度，所述先验深度为所述部分像素点的深度参考信息；根据所述部分像素点对应的先验深度确定先验视差。
- 根据权利要求2所述的方法，其特征在于，所述将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配，包括：根据所述双目摄像装置与所述点云传感器之间的位置关系，将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配。
- 根据权利要求2或3所述的方法，其特征在于，所述根据所述部分像素点的深度信息确定先验视差，包括：根据所述双目摄像装置的内参和所述部分像素点的深度信息确定先验视差。
- 根据权利要求1-4任一项所述的方法，其特征在于，所述根据所述先验视差，得到所述第一视图和所述第二视图之间的视差图，包括：根据所述第一视图和所述第二视图获取目标相似度；根据所述先验视差和所述目标相似度，得到所述第一视图和所述第二视图之间的视差图。
- 根据权利要求5所述的方法，其特征在于，所述根据所述第一视图和所述第二视图获取目标相似度，包括：对所述第一视图进行特征提取，得到第一特征图，并且对所述第二视图进行特征提取，得到第二特征图；确定所述第一特征图和所述第二特征图之间的目标相似度。
- 根据权利要求6所述的方法，其特征在于，所述对所述第一视图进行特征提取，得到第一特征图，并且对所述第二视图进行特征提取，得到第二特征图，包括：按照census变换算法对所述第一视图进行特征提取，得到第一特征图，并且按照census变换算法对所述第二视图进行特征提取，得到第二特征图。
- 根据权利要求6或7所述的方法，其特征在于，所述确定所述第一特征图和所述第二特征图之间的目标相似度，包括：计算所述第一特征图和所述第二特征图之间的汉明距离；确定所述汉明距离为所述第一特征图和所述第二特征图之间的目标相似度。
- 根据权利要求5-8任一项所述的方法，其特征在于，所述根据所述先验视差和所述目标相似度，得到所述第一视图和所述第二视图之间的视差图，包括：根据所述先验视差和所述目标相似度构造优化求解模型；根据所述优化求解模型，得到所述第一视图和所述第二视图之间的视差图。
- 根据权利要求9所述的方法，其特征在于，所述优化求解模型为条件概率分布模型。
- 根据权利要求10所述的方法，其特征在于，所述条件概率分布模型为条件随机场。
- 一种图像处理系统，其特征在于，所述图像处理系统包括：存储器、处理器、双目摄像装置和点云传感器，其中：所述存储器，用于存储程序指令；所述双目摄像装置，用于采集环境的第一视图和第二视图；所述点云传感器，用于采集所述环境的三维点云；所述处理器，调用所述程序指令以用于：将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配，得到先验视差；根据所述先验视差，得到所述第一视图和所述第二视图之间的视差图。
- 根据权利要求12所述的系统，其特征在于，所述处理器将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配，得到先验视差的方式具体为：将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配；根据所述部分像素点对应的三维点云的三维信息确定所述部分像素点对应的先验深度，所述先验深度为所述部分像素点的深度参考信息；根据所述部分像素点对应的先验深度确定先验视差。
- 根据权利要求13所述的系统，其特征在于，所述处理器将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配的方式具体为：根据所述双目摄像装置与所述点云传感器之间的位置关系，将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配。
- 根据权利要求13或14所述的系统，其特征在于，所述处理器根据所述部分像素点的深度信息确定先验视差的方式具体为：根据所述双目摄像装置的内参和所述部分像素点的深度信息确定先验视差。
- 根据权利要求12-15任一项所述的系统，其特征在于，所述处理器根据所述先验视差，得到所述第一视图和所述第二视图之间的视差图的方式具体为：根据所述第一视图和所述第二视图获取目标相似度；根据所述先验视差和所述目标相似度，得到所述第一视图和所述第二视图之间的视差图。
- 根据权利要求16所述的系统，其特征在于，所述处理器根据所述第一视图和所述第二视图获取目标相似度的方式具体为：对所述第一视图进行特征提取，得到第一特征图，并且对所述第二视图进行特征提取，得到第二特征图；确定所述第一特征图和所述第二特征图之间的目标相似度。
- 根据权利要求17所述的系统，其特征在于，所述处理器对所述第一视图进行特征提取，得到第一特征图，并且对所述第二视图进行特征提取，得到第二特征图的方式具体为：按照census变换算法对所述第一视图进行特征提取，得到第一特征图，并且按照census变换算法对所述第二视图进行特征提取，得到第二特征图。
- 根据权利要求17或18所述的系统，其特征在于，所述处理器确定所述第一特征图和所述第二特征图之间的目标相似度的方式具体为：计算所述第一特征图和所述第二特征图之间的汉明距离；确定所述汉明距离为所述第一特征图和所述第二特征图之间的目标相似度。
- 根据权利要求16-19任一项所述的系统，其特征在于，所述处理器根据所述先验视差和所述目标相似度，得到所述第一视图和所述第二视图之间的视差图的方式具体为：根据所述先验视差和所述目标相似度构造优化求解模型；根据所述优化求解模型，得到所述第一视图和所述第二视图之间的视差图。
- 根据权利要求20所述的系统，其特征在于，所述优化求解模型为条件概率分布模型。
- 根据权利要求21所述的系统，其特征在于，所述条件概率分布模型为条件随机场。
- 一种可移动平台，其特征在于，所述可移动平台包括：存储器、处理器、双目摄像装置和点云传感器，其中：所述存储器，用于存储程序指令；所述双目摄像装置，用于采集环境的第一视图和第二视图；所述点云传感器，用于采集所述环境的三维点云；所述处理器，调用所述程序指令以用于：将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配，得到先验视差；根据所述先验视差，得到所述第一视图和所述第二视图之间的视差图。
- 根据权利要求23所述的可移动平台，其特征在于，所述处理器将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配，得到先验视差的方式具体为：将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配；根据所述部分像素点对应的三维点云的三维信息确定所述部分像素点对应的先验深度，所述先验深度为所述部分像素点的深度参考信息；根据所述部分像素点对应的先验深度确定先验视差。
- 根据权利要求24所述的可移动平台，其特征在于，所述处理器将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配的方式具体为：根据所述双目摄像装置与所述点云传感器之间的位置关系，将所述三维点云投影至所述第一视图，与所述第一视图的部分像素点进行匹配。
- 根据权利要求24或25所述的可移动平台，其特征在于，所述处理器根据所述部分像素点的深度信息确定先验视差的方式具体为：根据所述双目摄像装置的内参和所述部分像素点的深度信息确定先验视差。
- 根据权利要求23-26任一项所述的可移动平台，其特征在于，所述处理器根据所述先验视差，得到所述第一视图和所述第二视图之间的视差图的方式具体为：根据所述第一视图和所述第二视图获取目标相似度；根据所述先验视差和所述目标相似度，得到所述第一视图和所述第二视图之间的视差图。
- 根据权利要求27所述的可移动平台，其特征在于，所述处理器根据所述第一视图和所述第二视图获取目标相似度的方式具体为：对所述第一视图进行特征提取，得到第一特征图，并且对所述第二视图进行特征提取，得到第二特征图；确定所述第一特征图和所述第二特征图之间的目标相似度。
- 根据权利要求28所述的可移动平台，其特征在于，所述处理器对所述第一视图进行特征提取，得到第一特征图，并且对所述第二视图进行特征提取，得到第二特征图的方式具体为：按照census变换算法对所述第一视图进行特征提取，得到第一特征图，并且按照census变换算法对所述第二视图进行特征提取，得到第二特征图。
- 根据权利要求28或29所述的可移动平台，其特征在于，所述处理器确定所述第一特征图和所述第二特征图之间的目标相似度的方式具体为：计算所述第一特征图和所述第二特征图之间的汉明距离；确定所述汉明距离为所述第一特征图和所述第二特征图之间的目标相似度。
- 根据权利要求27-30任一项所述的可移动平台，其特征在于，所述处理器根据所述先验视差和所述目标相似度，得到所述第一视图和所述第二视图之间的视差图的方式具体为：根据所述先验视差和所述目标相似度构造优化求解模型；根据所述优化求解模型，得到所述第一视图和所述第二视图之间的视差图。
- 根据权利要求31所述的可移动平台，其特征在于，所述优化求解为条件概率分布模型。
- 根据权利要求32所述的可移动平台，其特征在于，所述条件概率分布 模型为条件随机场。
- 根据权利要求23-33所述的可移动平台，其特征在于，所述可移动平台为车辆。
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2019/089177 WO2020237553A1 (zh) | 2019-05-30 | 2019-05-30 | 图像处理方法、系统及可移动平台 |
| CN201980007886.3A CN111656404B (zh) | 2019-05-30 | 2019-05-30 | 图像处理方法、系统及可移动平台 |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2019/089177 WO2020237553A1 (zh) | 2019-05-30 | 2019-05-30 | 图像处理方法、系统及可移动平台 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2020237553A1 true WO2020237553A1 (zh) | 2020-12-03 |
Family
ID=72350339
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2019/089177 WO2020237553A1 (zh) | 2019-05-30 | 2019-05-30 | 图像处理方法、系统及可移动平台 |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111656404B (zh) |
| WO (1) | WO2020237553A1 (zh) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112581542A (zh) * | 2020-12-24 | 2021-03-30 | 北京百度网讯科技有限公司 | 自动驾驶单目标定算法的评估方法、装置及设备 |
| CN114879377A (zh) * | 2022-04-11 | 2022-08-09 | 北京邮电大学 | 水平视差三维光场显示系统的参数确定方法、装置及设备 |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105160702A (zh) * | 2015-08-20 | 2015-12-16 | 武汉大学 | 基于LiDAR点云辅助的立体影像密集匹配方法及系统 |
| CN106796728A (zh) * | 2016-11-16 | 2017-05-31 | 深圳市大疆创新科技有限公司 | 生成三维点云的方法、装置、计算机系统和移动设备 |
| KR20170100229A (ko) * | 2016-02-25 | 2017-09-04 | (주)앤미디어 | 다시점 영상 디스플레이 시스템 |
| CN107886477A (zh) * | 2017-09-20 | 2018-04-06 | 武汉环宇智行科技有限公司 | 无人驾驶中立体视觉与低线束激光雷达的融合矫正方法 |
| CN109615652A (zh) * | 2018-10-23 | 2019-04-12 | 西安交通大学 | 一种深度信息获取方法及装置 |
| CN109640066A (zh) * | 2018-12-12 | 2019-04-16 | 深圳先进技术研究院 | 高精度稠密深度图像的生成方法和装置 |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| BRPI0822142A2 (pt) * | 2008-01-29 | 2015-06-30 | Thomson Licensing | Método e sistema para converter dados de imagem em 2d para dados de imagem estereoscópica |
| CN108961383B (zh) * | 2017-05-19 | 2021-12-14 | 杭州海康威视数字技术股份有限公司 | 三维重建方法及装置 |
-
2019
- 2019-05-30 CN CN201980007886.3A patent/CN111656404B/zh active Active
- 2019-05-30 WO PCT/CN2019/089177 patent/WO2020237553A1/zh active Application Filing
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105160702A (zh) * | 2015-08-20 | 2015-12-16 | 武汉大学 | 基于LiDAR点云辅助的立体影像密集匹配方法及系统 |
| KR20170100229A (ko) * | 2016-02-25 | 2017-09-04 | (주)앤미디어 | 다시점 영상 디스플레이 시스템 |
| CN106796728A (zh) * | 2016-11-16 | 2017-05-31 | 深圳市大疆创新科技有限公司 | 生成三维点云的方法、装置、计算机系统和移动设备 |
| CN107886477A (zh) * | 2017-09-20 | 2018-04-06 | 武汉环宇智行科技有限公司 | 无人驾驶中立体视觉与低线束激光雷达的融合矫正方法 |
| CN109615652A (zh) * | 2018-10-23 | 2019-04-12 | 西安交通大学 | 一种深度信息获取方法及装置 |
| CN109640066A (zh) * | 2018-12-12 | 2019-04-16 | 深圳先进技术研究院 | 高精度稠密深度图像的生成方法和装置 |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112581542A (zh) * | 2020-12-24 | 2021-03-30 | 北京百度网讯科技有限公司 | 自动驾驶单目标定算法的评估方法、装置及设备 |
| CN114879377A (zh) * | 2022-04-11 | 2022-08-09 | 北京邮电大学 | 水平视差三维光场显示系统的参数确定方法、装置及设备 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111656404B (zh) | 2024-03-01 |
| CN111656404A (zh) | 2020-09-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112861653B (zh) | 融合图像和点云信息的检测方法、系统、设备及存储介质 | |
| EP3876141A1 (en) | Object detection method, related device and computer storage medium | |
| CN110325818B (zh) | 经由多模融合的联合3d对象检测和取向估计 | |
| US10891500B2 (en) | Method and apparatus for acquiring traffic sign information | |
| US10427304B2 (en) | Robotic charger alignment | |
| CN110148185B (zh) | 确定成像设备坐标系转换参数的方法、装置和电子设备 | |
| Alismail et al. | Automatic calibration of a range sensor and camera system | |
| WO2020102944A1 (zh) | 点云处理方法、设备及存储介质 | |
| CN110176032B (zh) | 一种三维重建方法及装置 | |
| CN111563923A (zh) | 获得稠密深度图的方法及相关装置 | |
| US9959625B2 (en) | Method for fast camera pose refinement for wide area motion imagery | |
| EP3716210B1 (en) | Three-dimensional point group data generation method, position estimation method, three-dimensional point group data generation device, and position estimation device | |
| WO2020083307A1 (en) | Method, apparatus, and storage medium for obtaining depth image | |
| WO2022262160A1 (zh) | 传感器标定方法及装置、电子设备和存储介质 | |
| US20220156954A1 (en) | Stereo matching method, image processing chip and mobile vehicle | |
| CN111123242B (zh) | 一种基于激光雷达和相机的联合标定方法及计算机可读存储介质 | |
| WO2021195939A1 (zh) | 一种双目拍摄装置的外参的标定方法、可移动平台及系统 | |
| CN113256718B (zh) | 定位方法和装置、设备及存储介质 | |
| US20200082641A1 (en) | Three dimensional representation generating system | |
| WO2023155353A1 (zh) | 深度图像的获取方法、装置、深度系统、终端和存储介质 | |
| WO2020237553A1 (zh) | 图像处理方法、系统及可移动平台 | |
| CN114812558A (zh) | 一种结合激光测距的单目视觉无人机自主定位方法 | |
| CN110673607A (zh) | 动态场景下的特征点提取方法、装置、及终端设备 | |
| CN112560769B (zh) | 用于检测障碍物的方法、电子设备、路侧设备和云控平台 | |
| Ou et al. | Targetless extrinsic calibration of camera and low-resolution 3-d lidar |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19930306 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 19930306 Country of ref document: EP Kind code of ref document: A1 |