TWI793584B - Mapping and localization system for automated valet parking and method thereof - Google Patents

Abstract

A mapping and localization system for automated valet parking and method thereof is disclosed. It includes an image information receiving module, for receiving continuous image information from at least two monocular cameras to capture the surrounding environment of a vehicle; a vehicle information assembly interface that receives data from the vehicle; a processing unit, generates feature points with depth information in each frame of the continuous image information of the surrounding environment of the vehicle, and after matching the feature points with the driving data information of the vehicle, a three-dimensional point cloud map is constructed; and a storage module, for storing the three-dimensional point cloud map, characterized in that when the vehicle is repositioned, the processing unit can receive real-time image information of the surrounding environment of the vehicle captured by the at least two monocular cameras according to the image information receiving module. The real-time image information is used to be compared with the stored three-dimensional point cloud map to determine the positioning poses of the at least two monocular cameras.

Description

System and method for automatic parking mapping and positioning

The invention relates to the technical field of automatic driving, in particular to a method for automatic parking mapping and positioning.

With the development of in-vehicle artificial intelligence, the functional requirements for self-driving vehicles and automatic parking are increasing, especially the automatic parking technology in specific parking areas. The most important thing for automatic parking is to locate and build a map of the parking area. Outdoor areas are usually positioned by the Global Positioning System (GPS), but the relative accuracy is not good. As for indoor parking areas, GPS signals cannot be positioned due to buildings blocking them. Therefore, Simultaneous Localization and Mapping (Simultaneous Localization and Mapping) technology is used to realize vehicle positioning and build a parking lot map by means of visual recognition. The general practice is that the self-driving vehicle uses a single monocular camera to create a point cloud map that matches the real environment through computer vision on the path of the vehicle driving in the parking area. When performing subsequent automatic parking functions , the position of the vehicle itself in the map can be determined by the establishment of the map. However, such a map often produces poor recognition results for the actual positioning of the vehicle, especially for parking areas without specific structures or scenes. A condition with low variability.

In order to solve the above technical problems, a system and method for automatic parking mapping and positioning are provided.

The purpose of the present invention can be achieved through the following technical solutions: a system for automatic parking mapping and positioning, including: an image signal receiving module, which receives continuous image signals from at least two monocular cameras to capture the surrounding environment of the vehicle, and the vehicle The signal assembly interface receives the driving data signal from the vehicle, and the processing unit generates feature points with depth information in each frame of the continuous image signal of the surrounding environment of the vehicle, and matches the feature points with the driving of the vehicle After the data signal, build a three-dimensional point cloud map, store the module, and store the three-dimensional point cloud map. It is characterized in that when the vehicle is repositioned, the processing unit can receive the at least two single points according to the image signal receiving module. The real-time image signal of the surrounding environment of the vehicle captured by the eye camera in real time is used to compare the stored three-dimensional point cloud map to determine the positioning pose of the at least two monocular cameras.

According to the idea of the present invention, the continuous image signals of the surrounding environment of the vehicle at least include image signals in front of the vehicle and behind the vehicle.

According to the idea of the present invention, the continuous image signals of the surrounding environment of the vehicle at least include image signals in front of the vehicle, image signals behind the vehicle, and image signals on the left and right sides of the vehicle.

According to the concept of the present invention, the processing unit has semantic operations, which can remove dynamic feature points in the three-dimensional point cloud map.

According to the concept of the present invention, the image signals on the left and right sides of the vehicle, the image signals in front of the vehicle and the image signals behind the vehicle have image overlapping areas.

In order to achieve the purpose of the present invention, the present invention provides a method for automatic parking mapping and positioning, including: receiving continuous image signals from at least two monocular cameras to capture the surrounding environment of the vehicle, receiving driving data signals from the vehicle, and A feature point with depth information is generated in each frame of the continuous image signal of the surrounding environment of the vehicle, and after matching the feature point with the driving data signal of the vehicle, a 3D point cloud map is constructed, and the constructed 3D point cloud map is stored. The point cloud map is characterized in that when the vehicle is repositioned, the real-time image signals of the surrounding environment of the vehicle captured by the at least two monocular cameras are received according to the image signal receiving module, and are used to compare the stored three-dimensional The point cloud map is used to determine the positioning poses of the at least two monocular cameras.

According to the idea of the present invention, the continuous image signals of the surrounding environment of the vehicle at least include image signals in front of the vehicle and behind the vehicle.

According to the idea of the present invention, the continuous image signals of the surrounding environment of the vehicle at least include image signals in front of the vehicle, image signals behind the vehicle, and image signals on the left and right sides of the vehicle.

According to the conception of the present invention, the above method further includes semantic operation, which can remove dynamic feature points in the three-dimensional point cloud map.

According to the concept of the present invention, the image signals on the left and right sides of the vehicle, the image signals in front of the vehicle and the image signals behind the vehicle have image overlapping areas.

The present invention has the following advantages: 1. When automatic parking is used to establish a parking area map, since the system of the present invention receives continuous image signals from at least two monocular cameras to capture the surrounding environment of the vehicle, it can cover all the visual fields of the surrounding environment of the vehicle, that is, constitute a panoramic visual field To improve the completeness of the three-dimensional point cloud map in the construction of the parking area, and at the same time, in order to achieve a more accurate three-dimensional point cloud map of the parking area, the present invention can remove the dynamic feature points in the three-dimensional point cloud map through semantic operations. 2. Since the monocular cameras installed in the front of the vehicle, the rear of the vehicle, and the left and right sides of the vehicle can reduce the distortion of the edge of the image caused by the lens through the replacement of the front and rear image frames, and improve the construction of the three-dimensional point cloud map of the parking area. completeness and accuracy. 3. In the positioning of automatic parking, such as facing a highly repetitive parking area, or when the vehicle's driving angle is offset, due to the difference in the image signals captured by these monocular cameras in real time, these monocular cameras can be quickly found. The target camera corresponds to the 3D point cloud map positioning pose of the parking area, and then the vehicle positioning pose is obtained.

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments It is some embodiments of the present invention, but not all of them. Based on the implementation manners in the present invention, all other implementation manners obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention. Accordingly, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. Based on the implementation manners in the present invention, all other implementation manners obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "upper", "lower", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description. It is not intended to indicate or imply that the referred device or element must have a particular orientation, be constructed in a particular orientation, and operate in a particular orientation, and thus should not be construed as limiting the invention.

In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

In the present invention, unless otherwise clearly specified and limited, terms such as "installation", "connection", "connection" and "fixation" should be understood in a broad sense, for example, it can be a fixed connection or a detachable connection , or integrated; it can be mechanically connected or electrically connected; it can be directly connected or indirectly connected through an intermediary, and it can be the internal communication of two components or the interaction relationship between two components. Those of ordinary skill in the art can understand the specific meanings of the above terms in the present invention according to specific situations.

Below in conjunction with accompanying drawing and specific embodiment the present invention is described in further detail:

Please refer to FIG. 1 , the present invention discloses an automatic parking mapping and positioning system 100 including an image signal receiving module 101 , a vehicle signal assembly interface 102 , a processing unit 103 , and a storage module 104 .

The image signal receiving module 101 is used to receive continuous image signals, and has an image processing unit (ISP, Image Signal Processor), which can process such as lens correction, pixel correction, color interpolation, Bayer noise removal, white balance correction, color correction , gamma correction, color space conversion and other functions. The video signal receiving module 101 can generally have LVDS (Low Voltage Differential Signaling, Low Voltage Differential Signaling) or MIPI CSI transmission interface (not shown). In the present invention, the continuous image signal comes from at least two monocular cameras installed on the vehicle, and the monocular camera is usually installed on the outside of the vehicle to avoid being blocked by the vehicle's own field of vision. It is mainly used to photograph the surrounding environment of the vehicle. Continuous image signal, in order to obtain better image quality, the image signal can be LVDS format signal.

The vehicle signal assembly interface 102 is connected to the vehicle CAN bus to receive driving data signals from the vehicle. The driving data signal of the vehicle includes, for example, vehicle speed, engine speed, steering angle, acceleration, gear position, etc., and may even include information from the Inertial Measurement Unit (Inertial Measurement Unit) loaded on the vehicle, wheel speedometer, GPS, etc. that can measure the movement of the vehicle. distance sensing signal.

The processing unit 103 is the main computing unit of the present invention, and usually the processor of the processing unit 103 is a DSP (digital signal processor, digital signal processor). DSP is suitable for various multiplication and addition operations (SOP: Sum of Products), such as: Finite Impulse Response Filtering Operation (FIR: Finite Impulse Response), Infinite Impulse Response Filtering Operation (IIR: Infinite Impulse Response), Discrete Fourier Transform (DFT : Discrete Fourier Transform), discrete cosine transform (DCT: Discrete Cosine Transform), dot product (Dot product), convolution (Convolution), and matrix polynomial evaluation operations, etc. The processing unit 103 is connected to the image signal receiving module 101 to calculate the image signal converted from the image signal receiving module 101. The processing unit 103 is also connected to the vehicle signal assembly interface 102 to receive the driving data signal from the vehicle. The continuous image signal of the environment and the driving data signal can start to build a 3D point cloud map when the automatic parking map building function is turned on, and compare the 3D point cloud map when the vehicle is repositioned to obtain at least two monocular cameras The positioning pose of .

The main function of the storage module 104 is to store image signals or/and various driving data signals, as well as store 3D point cloud maps. The storage module 105 can be a built-in integrated circuit memory, or an external storage device, such as SSD or SD card. Meanwhile, the storage module 104 is connected with the processing unit 103 and can store data according to the signal generated by the processing unit 103 .

Please refer to FIG. 1 , FIG. 2 and FIG. 3 , which illustrate the operation of an automatic parking mapping and positioning system 100 according to the present invention. The automatic parking mapping and positioning system 100 of the present invention is installed on the vehicle, and performs the following steps:

Step S01 , receiving continuous image signals from at least two monocular cameras shooting the surrounding environment of the vehicle and receiving driving data signals from the vehicle. When the user activates the automatic parking function on the vehicle, he must first perform the first mapping mode. Therefore, after the automatic parking function is activated, the user must drive to the parking grid in the parking lot (usually there are grid lines drawn) Or in a specific parking space, during the process of driving by the user, the system 100 in the present invention uses the image signal receiving module 101 to receive at least two monocular cameras installed in front of the vehicle and behind the vehicle (as shown in Figure 3). The camera captures continuous image signals of the vehicle's surrounding environment. These continuous image signals involve the internal and external parameters of the monocular camera, so they must be corrected. In this embodiment, the monocular camera has a wide viewing angle or a fisheye lens, so that the continuous image signals captured by these monocular cameras installed in the front and rear of the vehicle can cover the field of view of all the surrounding environment of the vehicle, that is, constitute A panoramic view. And these monocular cameras must have a better focal length, and they are clear for shooting within 20 meters, so that users can completely collect scene images of the parking area during self-driving, so that the vehicle can drive to a specific parking space or On the path of the parking space, each frame of the image signal of the surrounding environment of the vehicle has a corresponding complete 3D point cloud map. At the same time, the vehicle signal assembly interface 102 is connected to the CAN bus of the vehicle to receive the driving data signal from the vehicle. The driving data signals of these vehicles are mainly the sensing signals for measuring the moving distance and speed of the vehicles, as the actual distance signals matching the continuous image signals of the surrounding environment of the vehicles.

Step S02 , generating feature points with depth information in each frame of the continuous image signal of the surrounding environment of the vehicle. Mainly, the processing unit 103 performs feature point extraction on continuous image signals, and the feature point calculation adopted by the present invention is carried out in a manner of replacing computer vision (Computer Vision) with a model base (Model-based), mainly for the image signal In each frame, it has features such as points, corners, and textures of common pillars, walls, road signs, symbols, etc. (but not limited to the above) in the parking area, and the descriptor of each feature point is estimated. According to the one-to-one pairing of the feature point descriptors of the front and back frames, and calculate the projection changes of the feature points on the three-dimensional coordinates after pairing, and finally obtain the depth information in each frame of the continuous image signal Feature points.

Step S03, matching the feature points with the driving data signal of the vehicle. Since the image signal is a record composed of consecutive frames, the front and rear frames of the image reflect the time difference, and in step S02, even if the processing unit 103 generates feature points with depth information for each frame of the continuous image signal, it will be different from the actual There is a certain difference in the distance traveled by the vehicle, so in this embodiment, the processing unit 103 receives the driving data signal from the vehicle through the vehicle signal assembly interface 102 according to step S01, especially the inertial measurement unit (Inertial Measurement Unit) and Sensing signals such as wheel speedometers that can measure the moving distance of the vehicle. When the moving distance of the vehicle corresponding to the front and rear frames of the image is known, the actual Depth information of feature points. In addition, for the marking of all feature points in the image, the processing unit 103 has semantic operations, which can remove the dynamic feature points in the three-dimensional point cloud map. Most dynamic objects, such as parked vehicles, pedestrians, and movable fire-fighting equipment, etc., if these dynamic objects are regarded as feature points during mapping and become part of the 3D point cloud map, automatic parking will be enabled in subsequent users When the position of the dynamic object changes or moves, the positioning of the automatic parking function will fail. Therefore, through the semantic operation of the processing unit 103, the method can delete the dynamic feature points in the area selected by the semantic operation in each frame of the video signal, that is, the signals with these dynamic feature points are not stored in the In the 3D point cloud map, when the user executes the automatic parking function next time, these deleted dynamic feature points can be placed in the area of the 3D point cloud map again through this system, and the new data generated according to the image signal captured at that time The feature points of the 3D point cloud map are updated appropriately, and the areas that do not have feature points in the 3D point cloud map, thus improving the accuracy of the parking lot map.

Step S04, storing the 3D point cloud map. In this step, the established three-dimensional point cloud map will be stored in the storage module 104, and the stored signals may include continuous image signals taken by each monocular camera, and also include these monocular cameras corresponding to the three-dimensional point cloud map. relative coordinates. Even the storage module 104 can store the 3D point cloud maps created by users in different parking areas, or the 3D point cloud maps of the same parking area at different times. Furthermore, it can store the dynamic feature points of each parking area after semantic calculation. Mapping completeness index, as well as GPS messages from vehicles, inertial measurement unit signals, etc. These signals stored in the storage module 104 can be read, rewritten, or transmitted to the cloud server through the network for optimizing maps and signal exchange.

Step S05 , comparing the real-time image signals of the surrounding environment of the vehicle captured by the monocular camera and the stored 3D point cloud map to determine the positioning poses of the at least two monocular cameras. When the user finishes creating and storing the 3D point cloud map in step S04, when the automatic parking function is activated again and the vehicle is repositioned, the processing unit 103 reads the 3D point cloud map stored in the storage unit 104, and simultaneously receives these orders The real-time video signals of the surrounding environment of the vehicle captured by the real-time camera (in this embodiment, at least two monocular cameras installed at the front of the vehicle and the rear of the vehicle), the feature points of these real-time video signals are calculated, and compared with the stored After the 3D point cloud map, the positioning poses of these at least two monocular cameras can be quickly obtained, and then the coordinate position of the vehicle on the 3D point cloud map can be obtained to achieve rapid positioning.

Since the at least two monocular cameras mounted in the front and rear of the vehicle in this embodiment can cover the field of view of all the surrounding environment of the vehicle, that is, form a panoramic field of view, improve the integrity of the three-dimensional point cloud map in the construction of the parking area, and in addition , when the vehicle is repositioned, the difference in the image signals captured by these monocular cameras in real time can quickly find the positioning pose of these monocular cameras corresponding to the 3D point cloud map of the parking lot, and then obtain the vehicle positioning pose.

Please refer to FIG. 4 , which is a second embodiment of the operation of an automatic parking mapping and positioning system 100 according to the present invention. The automatic parking mapping and positioning system 100 and the operation method of this embodiment are not different from those of the first embodiment, but in this embodiment, the image signal receiving module 101 receives the image signal in front of the vehicle, the image signal behind the vehicle, The image signals on the left and right sides of the vehicle, that is, the monocular cameras are installed on the front, rear, and left and right sides of the vehicle to capture continuous image signals of the surrounding environment of the vehicle to form a panoramic view. Due to the wide viewing angle of the monocular camera, the reliability of the depth information of the relative image will be reduced, and the lens mounted on the monocular camera will easily cause distortion when the image is close to the edge of the image, and the distorted image signal is in the When building a three-dimensional point cloud map of the panoramic view of the parking area, especially when the feature points of the front and rear frames of the image are matched (step S03), it will cause poor matching, and even abnormal matching of feature points on the same object. Therefore, this embodiment In addition to receiving video signals from the front of the vehicle and video signals from the rear of the vehicle, it also receives video signals from the left and right sides of the vehicle at the same time, and these video signals from the left and right sides of the vehicle have overlapping areas with the video signals from the front of the vehicle and the video signals from the rear of the vehicle. Since the image signals from these monocular cameras are synchronous and real-time images, in the part of the image overlapping area, except that the 3D point cloud map that determines the panoramic view can be completely covered by the field of view of these monocular cameras, for these For image overlapping areas, the processing unit 103 marks them without calculating the feature points of the image signals. As the vehicle’s moving pose changes, these overlapping and possibly highly distorted image signals will be replaced by subsequent images on the left and right sides of the vehicle. The signal is replaced, and the replaced image signal can reduce the distortion of the image edge caused by the lens in the monocular camera in front of the vehicle and behind the vehicle, and greatly improve the accuracy of the 3D point cloud map.

Similarly, since the monocular cameras of the present embodiment mounted on the front of the vehicle, the rear of the vehicle, and the left and right sides of the vehicle can cover the field of view of all surrounding environments of the vehicle, that is, form a panoramic field of view, and reduce the distortion of the image, and improve the field of view of the vehicle. The completeness and accuracy of the 3D point cloud map of the construction parking area. In addition, when the vehicle is repositioned, the image signals captured by these monocular cameras in real time are different, and the 3D point cloud corresponding to the parking area of these monocular cameras can be quickly found. The map positioning pose, and then the vehicle positioning pose is obtained.

For users who use the automatic parking function, since the present invention includes at least two monocular cameras, when positioning the automatic parking, such as facing a parking area with high repeatability, or the angle of the vehicle is offset At the same time, due to the difference in real-time image signals captured by these monocular cameras, the positioning poses of these monocular cameras corresponding to the 3D point cloud map of the parking lot can be quickly found, and then the vehicle positioning pose can be obtained. In order to achieve a more accurate 3D point cloud map of the parking area, the present invention can remove the dynamic feature points in the 3D point cloud map through semantic operations. Therefore, the accuracy of the parking area can be achieved by repeating the aforementioned steps S01 to S04. until optimized.

The above description is only the preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as above with the preferred embodiment, it is not intended to limit the present invention. Anyone familiar with this professional technology Personnel, without departing from the scope of the technical solution of the present invention, when the method and technical content disclosed above can be used to make some changes or modifications to equivalent embodiments with equivalent changes, but all the content that does not depart from the technical solution of the present invention, according to this Technical Essence of the Invention Any simple modifications, equivalent changes and modifications made to the above embodiments still fall within the scope of the technical solutions of the present invention.

100: Automatic parking mapping and positioning system 101: Video signal receiving module 102: Vehicle signal assembly interface 103: Processing unit 104: Storage module

Fig. 1 is a schematic structural diagram of a system for automatic parking mapping and positioning of the present invention; Fig. 2 is a flow chart of a method for automatic parking mapping and positioning of the present invention; 3 is a schematic diagram of a monocular camera installed in a vehicle according to the first embodiment of the present invention; FIG. 4 is a schematic diagram of a monocular camera installed in a vehicle according to a second embodiment of the present invention.

100: Automatic parking mapping and positioning system

101: Video signal receiving module

102: Vehicle signal assembly interface

103: Processing unit

104: Storage module

Claims (10)

A system for automatic parking mapping and positioning, installed on a vehicle, including: an image information receiving module, which receives images of the surrounding environment of the vehicle from at least two monocular cameras installed at the front and rear of the vehicle Continuous image information, wherein the continuous image signal of the surrounding environment of the vehicle covers all the field of view of the surrounding environment of the vehicle; the vehicle information assembly interface receives the driving data information from the vehicle; the processing unit, in the surrounding environment of the vehicle A feature point with depth information is generated in each frame of the continuous image information, and after matching the feature point with the driving data information of the vehicle, a three-dimensional point cloud map is constructed; and a storage module stores the three-dimensional point cloud map , it is characterized in that when the vehicle is repositioned, the processing unit can receive the real-time image information of the surrounding environment of the vehicle captured by the at least two monocular cameras according to the image information receiving module, and use it for comparison The stored three-dimensional point cloud map is used to determine the positioning poses of the at least two monocular cameras. The automatic parking mapping and positioning system described in item 1 of the patent application, wherein the continuous image information of the surrounding environment of the vehicle includes at least the image information of the front of the vehicle and the rear of the vehicle. The automatic parking mapping and positioning system described in item 1 of the scope of the patent application, wherein the continuous image information of the surrounding environment of the vehicle at least includes the image information in front of the vehicle, the image information behind the vehicle, the left and right sides of the vehicle Image information on both sides. The system for automatic parking mapping and positioning described in item 1 of the scope of the patent application is characterized in that the processing unit has semantic operations and can remove dynamic feature points in the three-dimensional point cloud map. The system for automatic parking mapping and positioning described in item 3 of the patent application is characterized in that the image information on the left and right sides of the vehicle overlaps with the image information in front of the vehicle and the image information behind the vehicle district. A method for automatic parking mapping and positioning, used on a vehicle, comprising: using an image signal receiving module to receive images of the vehicle from at least two monocular cameras installed at the front and rear of the vehicle The continuous image information of the surrounding environment, wherein the continuous image signal of the surrounding environment of the vehicle covers all the visual field of the surrounding environment of the vehicle; receiving the driving data information from the vehicle through a vehicle signal assembly interface; through a processing A unit is used to generate feature points with depth information in each frame of continuous image information of the surrounding environment of the vehicle, and after matching the feature points with the driving data information of the vehicle, a three-dimensional point cloud map is constructed; and by A storage module to store the constructed three-dimensional point cloud map, characterized in that when the vehicle is repositioned, the processing unit receives the real-time images captured by the at least two monocular cameras according to the image information receiving module The real-time image information of the surrounding environment of the vehicle is compared with the stored three-dimensional point cloud map to determine the positioning poses of the at least two monocular cameras. The method for automatic parking mapping and positioning described in item 6 of the patent application, wherein the continuous image information of the surrounding environment of the vehicle at least includes image information of the front of the vehicle and the rear of the vehicle. The method for automatic parking mapping and positioning as described in item 6 of the patent application, wherein the continuous image information of the surrounding environment of the vehicle at least includes image information in front of the vehicle, image information behind the vehicle, left and right sides of the vehicle Image information on both sides. For the method for automatic parking mapping and positioning described in item 6 of the patent application, the processing unit further includes semantic operations, which can remove dynamic feature points in the three-dimensional point cloud map. The method for automatic parking mapping and positioning described in item 8 of the scope of the patent application is characterized in that the image information on the left and right sides of the vehicle overlaps with the image information in front of the vehicle and the image information behind the vehicle district.

Images