CN112802346A

CN112802346A - Autonomous parking system and method based on cloud sharing and map fusion

Info

Publication number: CN112802346A
Application number: CN202011606568.XA
Authority: CN
Inventors: 殷佳豪; 范圣印; 李雪; 陈禹行
Original assignee: Suzhou Yihang Yuanzhi Intelligent Technology Co Ltd
Current assignee: Suzhou Yihang Yuanzhi Intelligent Technology Co Ltd
Priority date: 2020-12-28
Filing date: 2020-12-28
Publication date: 2021-05-14
Anticipated expiration: 2040-12-28
Also published as: CN115035728A; CN112802346B

Abstract

An autonomous parking system and method based on cloud sharing and map fusion integrally adopts autonomous parking based on tracking, only local path planning is carried out, and the problems of high computing power requirement and poor real-time performance of the existing global path planning method are solved; track map sharing, track map selection and track map fusion are carried out at the cloud, so that a map does not need to be established by self, the track map is multiplexed only after being selected according to GPS coordinates, scene semantic information, track label classification and the like, and a tracking route is optimized; a cloud-vehicle-end-user-terminal cooperative communication mechanism is established, a cloud storage, processing and map reuse are utilized, the requirement of vehicle-end hardware on storage performance is lowered, and meanwhile real-time communication between the vehicle end and the user terminal guarantees real-time performance and reliability of autonomous parking. The parking method has high parking efficiency and high success rate.

Description

Autonomous parking system and method based on cloud sharing and map fusion

Technical Field

The invention relates to the technical field of autonomous parking in the unmanned driving industry and the technical field of computer vision, in particular to an autonomous parking system and method based on cloud sharing and map fusion, and particularly relates to the technical field of long-distance tracing parking.

Background

In recent years, the automatic driving technology is rapidly developed, and the demand for autonomous parking is more and more extensive. Autonomous parking refers to the ability of a vehicle to park automatically without manual intervention. The autonomous parking can provide safe, convenient and time-saving unmanned parking service for users. Because the parking space in a large city is limited, on one hand, driving a vehicle into a narrow space becomes a necessary skill, however, even if a driver parks the vehicle in place with great expense and carelessness, the situation that the vehicle cannot be opened or moved in and out due to narrow position or a road obstacle of a stereo garage causes inconvenience in getting on and off the vehicle for passengers often exists. On the other hand, with the development of three-dimensional transportation, parking lots in large urban comprehensive communities or large leisure and entertainment venues, parks, airports, stations, hospitals and the like are usually large in scale and complex in distribution, so that the distance from a parking lot to a destination is long, and people need to walk for a long distance after parking cars manually, which is very inconvenient. In addition, the autonomous parking can actually save the parking time for the user, and for the user who is not skillful in parking skill, the autonomous parking can also reduce the driving burden for the user. Therefore, the method has very important practical significance in researching autonomous parking, particularly long-distance autonomous tracing parking.

Autonomous parking usually includes functions such as SLAM mapping, multi-map merging, cloud sharing, automatic parking space selection, automatic vehicle recall and the like, and relates to a plurality of technologies such as top view splicing, obstacle detection (vehicles, pedestrians, objects and the like), local path planning, 4G/5G transmission communication, travelable area detection, vehicle control, parking space and parking space number detection, GPS positioning, visual repositioning and the like. The autonomous parking system is a complex system integrating all the complex key technologies.

In order to understand the state of the art, the present invention searches, compares and analyzes the existing patents and papers:

1. the technical scheme 1: chinese patent document CN110304050A, "a memory parking system, method, terminal and cloud server based on feature combination", CN110348341A, "a memory parking system, method, terminal and storage medium", and CN111439257A, "a memory parking system, method, terminal and cloud server" performs automatic tracking parking by uploading a pre-established trajectory to a cloud system, and then downloading the trajectory from the cloud during memory parking. The patent CN110304050A relies on 1 forward looking millimeter wave radar and 4 look around millimeter wave radar to establish a track map, and the system sensor cost is high. In addition, the cloud systems of the three patents only have the function of temporarily storing the own track map, can only build a map by themselves and can not solve the problem of reuse of others by themselves; further, the track maps created in these three patents only include the characteristic data of the track and the surrounding environment for positioning, and the positioning accuracy is not sufficient.

2. The technical scheme 2 is as follows: chinese patent document CN110517526A "a method and system for parking a passenger instead of a car" establishes a passenger-replacing parking system with a car end-cloud end-parking space management server end, obtains configuration information of a parking space at the cloud end according to current vehicle acquisition information and parking space management server end information, then the cloud end sends a target parking space to the car end or instructs the car end to start a roaming mode, and finally, automatic parking is performed by using global path planning. However, since the system uses the global path planning method for passenger-replacing parking, the system has high requirements and dependence on cloud computing power, poor real-time performance and large uncertainty of parking tracks.

3. Technical scheme 3: the foreign patent US20200207338a1 "System, Method, Infrastructure, and Vehicle for Automated Valet Parking" performs tracking memory Parking through a pre-established map, and establishes a communication mechanism of Vehicle end-cloud end-user terminal. Therefore, when the user uses the automatic driving function, the user only needs to get off in the guest falling area, and the rest parking process is completed by the vehicle independently. However, it needs to establish a whole global map of the parking lot, and use the cloud to dynamically save the status of the parking space (free or occupied), so that the cloud dynamically plans a currently free parking space and sends the information to the vehicle end when the vehicle autonomously parks. However, in the case that the entire global map of the parking lot cannot be built, the method cannot complete the parking task. However, some scenes in reality, such as courtyards, garages, underground parking spaces and other places of the user's own home, are often private areas, cannot be drawn by commercial map companies, lack exactly the whole global map of the parking lot, and naturally cannot be tracked, memorized and parked through a pre-established map.

Therefore, the existing autonomous parking research has many technical difficulties:

firstly, the method comprises the following steps: at present, automatic driving based on a high-precision map is applied to automatic parking, so that the acquisition and maintenance cost of the map is too high, partial scenes such as courtyards, garages, underground parking spaces and other places of own homes of users are often private areas and cannot be drawn by commercial map companies, the same user often needs to repeatedly park in the same place, and the path is relatively fixed. For the parking occasions with high demand, the traditional method is often stranded and cannot meet the use requirement.

Secondly, the method comprises the following steps: the existing autonomous parking system mostly only uses the track of the previous parking as a reference, then adopts a global path planning method to carry out automatic parking, and has high system calculation cost, so that the requirement on the calculation capacity of a platform is very high, and the existing vehicle-mounted controller is often difficult to meet the requirement. And because of the uncertainty of the path, the user is often required to monitor in real time in a remote manner to cope with the emergency, which increases the extra computational burden and is difficult to reduce the system cost.

Thirdly, the method comprises the following steps: the existing memory parking system can only use the track established previously, does not have the functions of cloud map sharing, map selection and map fusion, and has a limited application range.

Fourthly: due to the influence of complex method and insufficient system computing power, the existing parking system is limited in real-time performance and cannot meet the requirement of long-distance tracing parking.

Therefore, the existing autonomous parking method is difficult to obtain satisfactory comprehensive effects in application range, positioning accuracy, stability, instantaneity and success rate. The method and the system for autonomous parking are required to be researched, can adapt to various parking lot environments, can ensure positioning accuracy, stability and instantaneity, improve the parking success rate, do not increase extra calculation overhead, can be used for a low-power-consumption vehicle-mounted processor, do not need high-cost sensor system support, and reduce the use cost of the system.

Disclosure of Invention

In order to achieve the above-mentioned object, the present disclosure is improved in several aspects as follows:

1. the technical problems that the existing memory parking system can only use a map which is built previously, a cloud system only has the function of temporarily storing a track map of the parking system, can only build the map and can be used by the parking system, the built track map only comprises the track and characteristic data of the surrounding environment for positioning, and the positioning accuracy is not enough are solved. However, in the scheme of the cloud sharing track map, how to realize effective utilization of the shared track is a key point for solving the problem and is also a technical difficulty. There is no description in the prior art of how to utilize shared traces. The method for parking along tracks is innovatively provided by adopting a cloud map sharing, track selection and track fusion mode, and a vehicle owner can perform tracking parking without map acquisition; the method adopts a mode of establishing a global track map of the same place by the cooperation of multiple vehicles, so as to enrich and optimize the track map of a certain place and meet the personalized parking requirement through track fusion; a track fusion method is innovatively and specifically designed; the fused track determined by the fusion method greatly improves the parking success rate of the subsequent tracking autonomous parking.

2. Aiming at the technical problem of inaccurate positioning in the prior art, the method embeds information such as a parking space layer, a semantic layer and the like into a track map so as to improve the positioning accuracy; and a coarse-to-fine repositioning technology is introduced, so that quick repositioning and cloud map selection and reuse are facilitated, and a quick and accurate positioning effect can be obtained.

3. Aiming at the technical problems that the existing autonomous parking system needs to rely on 1 foresight millimeter wave radar and 4 around-looking millimeter wave radars to establish a track map and the cost of a system sensor is too high, the method disclosed by the invention avoids the dependence on a high-precision laser radar sensor in principle, and can achieve the effects of accurately positioning a target parking space and autonomously parking by only adopting a common sensor to establish the track map without the millimeter wave radars.

4. The system for passenger-assistant parking by using the global path planning method has the technical problems of high system calculation cost, high requirement on platform calculation capacity, high dependence on cloud calculation capacity, poor system real-time performance and high parking track uncertainty. The present disclosure does not use a global path planning method, but employs a local path planning method to reduce system computation overhead. However, in an autonomous parking mode of a vehicle end-cloud end-parking space management server end, local path planning is difficult to realize without using global planning, and in order to solve the technical problem, in the local path planning method, tracking memory parking is performed by using a track map established in advance, so that the real-time performance and the reliability are higher; in addition, in order to cooperate with the local path planning method, the cloud module is improved, the cloud module not only carries out communication with a vehicle end, but also carries out map storage, sharing, selection and fusion, and the bottleneck of local path planning is effectively solved.

5. Aiming at the technical problems that in the prior art, a whole global map of a parking lot needs to be established, a parking task cannot be completed under the condition that the whole global map of the parking lot cannot be established, and a part of scenes in reality, such as places such as courtyards, garages and underground parking spaces of a user's own house are often private areas, cannot be drawn by a commercial map company, and naturally cannot be tracked, memorized and parked through a pre-established map, the SLAM map is stored and processed by a cloud end, the track maps are shared and merged at the cloud end, only the track information needs to be stored or merged to obtain target track information, the global map of the whole parking lot does not need to be established, and when parking is carried out, parking space detection and selection are carried out according to the target track in a way of searching parking spaces while tracking; the driving path is relatively fixed, safe and reliable, public resources of the cloud shared map are fully utilized, the private track map established in the private area is effectively utilized, the public resources and the private track map are organically combined to fuse the track maps, a real-time and accurate target track is obtained, tracking parking is carried out according to the target track, the range of autonomous parking is greatly widened, and the success rate of autonomous parking is improved.

6. In order to improve the safety and the reliability of the use of the system, the system adopts the mode that real-time information transmission is carried out at a vehicle end-user terminal, a top view is checked at a user end in real time, the real-time control of the vehicle is realized when the user leaves the vehicle by utilizing the control of 360-degree panoramic all-round dragging, designated recall destination, midway parking and the like carried out on a screen interface of the user end, and the real-time intervention is carried out on the autonomous parking system under the necessary condition so as to deal with the uncontrollable emergency situation met by the autonomous parking system.

Based on the improved technical scheme, the method has the outstanding advantages in the aspects of solving the following problems:

1. the whole system adopts autonomous parking based on tracking, only local path planning is carried out, and the problems of high calculation force requirement and poor real-time performance of the existing global path planning method are solved;

2. the functions of cloud track map sharing, track map selection and track map fusion are realized, so that a map does not need to be built by self, the track map is multiplexed after selection is carried out according to GPS coordinates, scene semantic information, track label classification and the like, and the multiplexing rate of the map is improved by storing parking space layer information, map fusion, a user-defined recall end point and the like in the map building;

3. the method has the advantages that the relocation technology from coarse to fine is realized by utilizing GPS coordinates, scene semantic information, image global description and image local description, the map fusion and scene relocation efficiency is greatly improved, the real-time performance is ensured, and the probability of relocation failure caused by the change of factors such as illumination, shadow and the like of scene appearance is reduced to a certain extent;

4. the cloud-vehicle-end-user-terminal cooperative communication mechanism is established, the cloud is used for storing, processing and multiplexing the map, the requirement of vehicle-end hardware on storage performance is lowered, meanwhile, the vehicle-end and user-terminal real-time communication enables a vehicle owner to feel relieved, and the vehicle position and the running state can be monitored at any time.

Specifically, to solve the above technical problem, according to an aspect of the present invention, there is provided an autonomous parking method based on cloud sharing and map fusion, including the following steps:

step 1), camera images are collected, a semantic map of a parking route and a parking space is established by using synchronous positioning and mapping (SLAM), and track information is classified to obtain a track map;

step 2), storing the track map in a local and/or cloud terminal;

step 3), generating a target track map according to the selected track map;

the target track map is a new track map generated by fusing the selected track maps;

the fusion is as follows:

when only one local existing track map is selected from the local, the track map is the target track map;

when only one track map shared by others is selected from the cloud, the track map is the target track map;

when the number of the track maps selected from the cloud is more than one, carrying out track combination on the selected more than one track maps to obtain a target track map;

step 4), setting an autonomous parking parameter according to the target track map;

and step 5), carrying out relocation from coarse to fine according to the autonomous parking parameters.

Preferably, the camera is a fisheye camera.

Preferably, the classification of the trajectory information is classified according to parking scene, parking/recall information.

Preferably, the classification of the trajectory information requires a user to classify the trajectory according to parking scenes, parking/recalling, and the like.

Preferably, when the user classifies the tracks according to parking scenes, parking/recall information, the SLAM map is selected by the user to be stored locally and/or uploaded to the cloud sharing.

Preferably, when the track map is selected, one or more track maps shared by others are selected from the cloud according to the GPS coordinates and scene semantic information, so that the track maps are fused.

Preferably, the method further comprises the following vehicle automatic parking or vehicle recalling steps:

the automatic parking of the vehicle comprises the following steps:

(1) automatic tracking driving and obstacle avoidance local planning;

(2) detecting and selecting a parking space;

(3) automatically parking in a parking space;

the vehicle recall includes the steps of:

(1) automatically parking out of the parking place;

(2) automatic tracking driving and obstacle avoidance local planning

(3) Mid-course or recall end stops.

Preferably, the cloud is adopted for map sharing, track selection and track fusion.

Preferably, a global track map of the same place is established by multiple vehicles in cooperation for track map fusion.

Preferably, information such as a parking space layer and a semantic layer is embedded into the global track map so as to improve positioning accuracy.

Preferably, in the local path planning method, a pre-established trajectory map is used for tracking, memorizing and parking; and the cloud module is adopted for communication with the vehicle end, track map storage, track map sharing, track map selection and track map fusion.

Preferably, the SLAM map is saved and processed in the cloud.

Preferably, real-time information transmission is carried out at a vehicle end, namely a user terminal, a top view is checked at a user end in real time, and 360-degree panoramic all-around dragging, designated recall destination and midway parking control are carried out on a screen interface of the user end.

Preferably, the number of the fisheye cameras is 4.

Preferably, the method comprises the following steps: 12 ultrasonic sensor arrays, a wheel speed meter, a Global Positioning System (GPS), a mobile communication terminal and an embedded computing platform.

Preferably, a cloud-vehicle-end-mobile communication terminal collaboration system is adopted, wherein the mobile communication terminal is a 4G/5G communication terminal; the cloud is used for track map storage, classification selection, sharing and fusion; the vehicle end is used for sensing, drawing, planning and controlling; and the 4G/5G communication terminal is responsible for monitoring.

Preferably, the 4G/5G communication terminal is a mobile phone, a tablet or other remote control terminals; and the user checks the current position, the state and the top view of the vehicle in real time on a mobile phone, a tablet or other remote control terminals, and controls the vehicle in a remote control mode on the 4G/5G communication terminal.

Preferably, the trajectory map includes:

a track layer (track layer) which contains a path traveled by the vehicle when the map is built, a track classification label, track starting point and end point information; according to different track labels, the starting point and the end point of the track are one point or one section of track on the track;

a parking space layer (parking space layer) which contains coordinates of a target parking space, a target parking space number and parking space information along a track in a fixed parking space mode; the parking space information comprises a parking space coordinate, a parking space number and parking space information along the track, wherein the parking space coordinate, the parking space number and the parking space information are detected in a parking available area and a parking available area in a moving parking space mode;

a location and tracking layer (layer) containing information required for SLAM relocation and tracking;

a semantic information layer (semantic information layer) containing semantic information capable of assisting positioning and identification, comprising: cell number, floor number, road sign, landmark, lane line, building, and/or textual information.

Preferably, the track information includes:

the method comprises the steps of fixing a parking space track, when a fixed parking space track mode is selected, storing all parking space information detected along the way, generating and displaying a track map after map building is completed, wherein a default track starting point is a map building starting point, and a default track end point is a map building end point; if the parking space number is detected at the starting point or the ending point, the default target parking space number is the parking space number of the current parking space which is automatically detected; the user confirms the parking lot number before autonomously parking, and changes the starting point or the end point of the track by appointing a point on the track; at least one of the starting point or the end point is positioned in the parking space, and after the operation is finished, the operation information is stored in a track map and the track map is stored;

and/or the presence of a gas in the gas,

the method comprises the steps that a mobile parking space track is selected, when a mobile parking space track mode is selected, all parking space information detected along the way is stored, a track map is generated and displayed after map building is completed, the default track starting point is a map building starting point, and the default terminal point of a parking available area is a map building track terminal point; the method comprises the steps that a user changes a starting point of a track and an end point of a parking available area by specifying a point on the track before autonomous parking; and at least one of the starting point or the terminal point is positioned in the parking lot, and a point designated by a user between the starting point and the terminal point is the starting point of the parking available area, so that when the user carries out tracking autonomous parking, the parking can be carried out in the free parking spaces detected during tracking between the starting point of the parking available area and the terminal point of the parking available area.

Preferably, the track information further includes:

bidirectional trajectory: when the mapping track can be used for vehicle parking and vehicle recalling, the track is a bidirectional track;

one-way vehicle parking track: when the mapping track can only be used for parking vehicles, the track is a one-way vehicle parking track, and the terminal point of the track needs to be a parking space or a parking area;

one-way vehicle recall trajectory: when the mapping track can only be used for vehicle recall, the track is a one-way vehicle recall track, and the starting point of the track needs to be a parking space or a parking area;

in the bidirectional track mode, the information stored in the corresponding distributed cameras in the map is automatically exchanged, and the information of the starting point and the end point is automatically exchanged, so that a track is generated, and the two tracks are called as a shadow track; for the bidirectional tracks, when the vehicle is parked in a memory mode, one track with the end point located in the parking space/parking area in the shadow track is automatically selected as a vehicle parking track, and the other track is used as a vehicle recall track.

Preferably, when the detected parking space deviation mapping track is not greater than the specified maximum deviation distance, the parking is judged to be possible.

Preferably, the selection of the trajectory map comprises:

a user specifies a tracking track, wherein the tracking track is selected from locally stored tracks, or one track is selected from a cloud end, or a plurality of tracks are selected from the cloud end to perform map combination to form a new tracking track;

when a user selects a track shared by others from the cloud, firstly, track searching is carried out, and during searching, selection is carried out according to the appointed current GPS coordinate, track category, building location and/or parking space number information; if a plurality of tracks at the same place are appointed, track map fusion is carried out by the cloud, and when the track map fusion is completed, the cloud direct loading, cloud collection adding and local downloading operations can be carried out.

Preferably, after the track selection is finished, loading a track map, and loading track starting points, track end points, parking space information and scale alignment (scale) parameter size information of vision and other sensors in the map; then loading other information of the vehicle sensor, including camera model and parameter information, external parameter matrix between sensors, IMU random walk (random walk) variance and Gaussian noise (noise) variance; after loading is finished, when all necessary sensors are confirmed to be normally started and normally run, parking setting is carried out; selecting a parking mode by a user, wherein the parking mode is a vehicle parking mode or a vehicle recall mode, and the parking mode is a fixed parking space parking mode or a mobile parking space parking mode; the vehicle parking mode is the same as the loaded track type if the loaded track type is the fused track or the bidirectional track, and if the loaded track type is the fused track or the bidirectional track, the vehicle parking mode is selected to be the vehicle parking mode or the vehicle recalling mode; when the loaded trajectory type is a flow parking trajectory or a trajectory formed by combining at least one flow parking trajectory, whether fixed parking or flow parking is selected, and when the loaded trajectory type is a fixed parking trajectory or a trajectory formed by combining only fixed parking trajectories, the parking mode is set as a fixed parking mode.

Preferably, when the vehicle recall mode is selected, the recall end point defaults to a track end point, and when the recall end point is set by a user, any point on the track may be designated as a track end point.

Preferably, when the user selects the parking mode to park the vehicle, the method includes:

if the parking mode is a fixed parking space parking mode, a user is required to confirm parking space information, the parking space information defaults to a target parking space in the loading track, and when the information of all parking spaces along the track is stored during map building, any parking space stored along the track can be used as a parking space; when the loaded track type is the fused track, the user is required to select to park in the parking space;

if the parking mode is the mobile parking space parking mode, the user is required to confirm the parking available area, the parking available area is the parking available area in the loading track by default, and when the type of the loading track is the fused track, the user is required to select the parking available area.

Preferably, parking is enabled when the detected deviation of the parking space from the loaded trajectory is not greater than the specified maximum deviation distance.

Preferably, the relocation from coarse to fine comprises:

rough positioning:

firstly, positioning to a rough position in a map by using a GPS coordinate of a current vehicle, and acquiring all key frame information within a certain distance nearby;

then, further selecting the key frames according to the global description and semantic information of the current frame acquired image, and carrying out next-step accurate positioning on the key frames which meet the condition that the global description similarity of the image is greater than a fifth threshold or contain the same semantic information;

and (3) accurate positioning:

comparing the key frames returned in the rough positioning one by one, calculating an accurate pose, and if an accurate pose is successfully obtained, considering that the repositioning is successful;

otherwise, reselecting the current state of the vehicle to perform relocation from coarse to fine;

the accuracy degree of the accurate pose is greater than a sixth threshold.

Preferably, the step of calculating the accurate pose comprises:

step 1) obtaining a matching relation between feature points based on a local feature point and descriptor matching method, obtaining a maximum matching set by using methods such as Ranac or DLT and the like, then calculating the current vehicle pose according to a PnP method, and measuring the accuracy degree of the pose according to the number of points in the PnP;

step 2), directly obtaining the pose of the current image according to the deep learning model; and measuring the accuracy degree of the pose according to the confidence coefficient output by the model.

Preferably, the deep learning model is a KFNet deep learning model.

Preferably, the automatic tracking driving and obstacle avoidance local planning includes travelable area detection, object detection and identification, ultrasonic obstacle detection, and top view stitching.

Preferably, the object detection and recognition comprises obstacle detection, vehicle recognition, pedestrian recognition.

Preferably, a GPS coordinate and a 4G/5G communication module are adopted in the whole process of automatic tracking driving and obstacle avoidance local planning to establish a 4G/5G communication terminal connection at a vehicle end, the vehicle end sends real-time data to 4G/5G communication terminal equipment, and a user monitors the position and parking state of the vehicle in real time through the terminal equipment and makes a temporary decision to control the vehicle.

Preferably, when an object exists on a predetermined track in the automatic tracking process, firstly, whether the object is a dynamic obstacle or a static obstacle is judged;

if the object is a dynamic obstacle, the following processing is carried out:

if the dynamic barrier appears suddenly and the distance between the dynamic barrier and the vehicle is less than a seventh threshold value, starting an emergency braking function and stopping for waiting; otherwise, reducing the vehicle speed for running;

if the object is a static obstacle or the parking waiting time exceeds an eighth threshold; the following processing is performed:

and performing detour by adopting local path planning, and then returning to the preset track to continue the tracking driving.

Preferably, when the detour is carried out by adopting local path planning, the detour comprises travelable area detection, object detection and identification, ultrasonic obstacle detection and top view splicing; wherein the object detection and identification comprises obstacle detection, vehicle identification, and pedestrian identification.

Preferably, the parking space detection and selection includes:

if the current state is in the fixed parking space mode: if the target parking space designated by the current position of the vehicle is smaller than a ninth threshold value, firstly, a top view splicing method is adopted, based on the top view, parking space detection and parking space number identification are carried out, and when the error between the detected parking space position and the designated target parking space position is smaller than a tenth threshold value or the detected parking space number of the target parking space is consistent with the designated parking space number, parking availability judgment is carried out; when the target parking space meets the parking accessibility condition, controlling the vehicle to automatically park in the target parking space; otherwise, when the target parking space is not detected or the target parking space does not meet the parking accessibility condition, the mobile communication terminal informs the user, reports the information that the designated parking space cannot be parked to the user, and displays the top view of the current vehicle at the mobile communication terminal;

if the user reassigns the target parking space or sends an instruction of 'parking beside' to wait on the top view of the vehicle of the mobile communication terminal: if the user appoints the target parking space again, the vehicle carries out parking judgment again, and carries out autonomous parking processing according to the judgment result, wherein the processing mode is the same as the above; when a user sends an instruction of 'parking while approaching to wait' or the user does not respond when a certain time threshold is exceeded, the vehicle starts a local path planning step to select a place which does not obstruct the passing to park while approaching nearby, and sends the GPS coordinates of the vehicle, the four-way camera image, the current top view, the semantic information collected around and the relative position of the target parking space to the mobile communication terminal for display; when the fixed parking space is occupied when the vehicle is parked in the target parking space, informing a user, and remotely controlling the vehicle by the user on the mobile communication terminal;

if the current state is in the floating parking space mode: if the current position of the vehicle reaches the starting point of the designated parking available area, firstly, a top view splicing method is adopted, parking space detection is carried out based on the top view, and when a target parking space is detected, firstly, whether the distance between the target parking space and the preset track exceeds the designated maximum deviation distance is judged; if the distance between the target parking space and the preset track does not exceed the maximum deviation distance or the user does not specify the maximum deviation distance, carrying out parking availability judgment; when the target parking space meets the parking accessibility condition, controlling the vehicle to automatically park in the target parking space, identifying the parking space number of the target parking space, storing the identified parking space number, and if the identification is unsuccessful, the parking space number is empty; and if the maximum deviation distance is exceeded or the current parking space does not pass the parking availability judgment although the maximum deviation distance is not exceeded, abandoning the target parking space to continue the target parking space detection.

Preferably, the distance between the target parking space and the predetermined track is detected by an ultrasonic ranging and visual depth estimation method.

Preferably, the maximum deviation distance may be previously designated by a user or set in real time through a mobile communication terminal.

Preferably, the determination of the parking availability comprises:

detecting the target parking space by adopting a target detection and identification module, an ultrasonic ranging module and a visual depth estimation module;

the ultrasonic ranging module and the visual depth estimation module are adopted to detect the length and the width of the target parking space;

if the target detection and identification module detects that a dynamic object exists in the target parking space, sending a parking waiting instruction to stop the vehicle immediately, and timing and recording the parking waiting time;

if the target detection and identification module detects that a static obstacle exists in the target parking space, judging that the target parking space does not pass the parking accessibility judgment;

if the parking waiting time exceeds a certain threshold value or the length and the width of the target parking space detected by the ultrasonic ranging module and the visual depth estimation module are not enough to park the vehicle, judging that the target parking space does not pass the parking availability judgment;

otherwise, the determination is that the determination is passed.

Preferably, if the parking space still cannot be reserved from the current position to the end point of the designated parking available area, the mobile communication terminal informs the user that the user does not find a free parking available space, and the user sends a command of 'searching for a parking space again' or 'waiting for parking beside' through the mobile communication terminal.

Preferably, when the vehicle receives a "find the parking space again" command, first the loop judgment is performed:

if a loop mark exists in a track section from the terminal point of the parking-available area to the terminal point of the track and the track before the starting point of the parking-available area, the vehicle automatically tracks and drives from the terminal point of the parking-available area to the loop mark to enter the original track;

if the track section between the terminal point of the parking-capable area and the track terminal point and the track before the starting point of the parking-capable area do not have a loop mark, the track section between the terminal point of the parking-capable area and the track before the starting point of the parking-capable area is subjected to approximate loop judgment, the minimum value of the GPS distance between the track section and the track before the starting point of the parking-capable area is found out, if the minimum value is smaller than a set approximate loop threshold value, the two points are considered to have approximate loops, the point on the track section between the terminal point of the parking-capable area and the track terminal point is taken as an approximate loop starting point, the point on the original track is taken as an approximate loop terminal point, the vehicle is automatically tracked and driven to the approximate loop starting point from the terminal point of the parking-capable area, and the vehicle is automatically driven to the approximate loop terminal point to enter;

and after the vehicle enters the original track, the vehicle is driven according to the steps of the automatic tracking driving and the obstacle avoidance local planning until the vehicle is driven to the starting point of a parking available area, and the vehicle re-determines the target parking space according to the steps of parking space detection and selection on the basis of the automatic tracking driving and the obstacle avoidance local planning.

Preferably, when the user sends an instruction of 'parking while approaching to wait', or no loop mark and no approximate loop exist, or the user does not respond for a long time, the vehicle adopts a local path planning method to select nearby places which do not obstruct the passage to park while approaching, and sends the GPS coordinates of the vehicle, the four-way camera image, the current top view, the semantic information collected around, and the relative position of the starting point/the ending point of the parking available area to the mobile communication terminal for display.

Preferably, the automatic parking space mode firstly needs to accurately target the type of a parking space and the geometric parameters of the parking space, adopts an ultrasonic sensor and a wheel odometer to detect the parking space, adopts a deep learning-based method to detect the angular point and the line of the parking space, identifies the sign line based on Hough transform and clustering and identifies the line of the parking space through a one-dimensional filter;

and after the parking space information is accurate, parking the vehicle into a target parking space by adopting local path planning, sending the GPS coordinate, the number of the parked vehicle and the current top view of the current vehicle to the mobile communication terminal after parking is finished, and then automatically extinguishing the vehicle.

Preferably, the types of the target parking spaces include horizontal spaces, vertical spaces and inclined spaces.

Preferably, the automatic parking-out parking space mode firstly needs to calculate the minimum distance between the current vehicle GPS coordinate and the recall trajectory, and if the distance is greater than an eleventh threshold, the recall trajectory is considered to be unable to be recalled and needs to be selected again; otherwise, parking the parking space by adopting a local path planning method and returning to the designated recall track, and then tracking driving according to the steps of automatic tracking driving and obstacle avoidance local planning.

Preferably, the local path planning method includes: the method comprises the steps of establishing a road force planning method of an automobile kinematic model based on Ackermann steering geometry, parking path planning based on B spline theory, local path planning based on A method, road force planning based on RBF neural network model, two-step trajectory planning method or bidirectional trajectory planning method or polynomial curve planning method.

Preferably, in the midway or recall terminal parking step, in the vehicle recall mode, after the vehicle is parked out, tracking driving is carried out according to automatic tracking driving and obstacle avoidance local planning until the vehicle reaches the designated recall terminal, and in the midway or recall terminal parking step, a user can send a midway parking instruction through a 4G/5G communication terminal to control the vehicle so as to realize midway passenger pickup and boarding.

In order to solve the above technical problems, according to another aspect of the present invention, there is provided an autonomous parking system based on cloud sharing and map fusion,

the method comprises the following steps:

the image acquisition device is used for acquiring camera images, establishing a semantic map of a parking route and a parking space by using synchronous positioning and mapping (SLAM), and classifying track information to obtain a track map;

the map storage device stores the track map in the local and/or cloud end;

a target track map generation means for generating a target track map based on the selected track map;

wherein the target track map is a new track map generated by fusion of the selected track maps;

wherein the fusion is:

the parameter setting device is used for setting the autonomous parking parameters according to the target track map;

and the repositioning device performs repositioning from coarse to fine according to the autonomous parking parameters.

In order to solve the above technical problem, according to still another aspect of the present invention, there is provided a method for trajectory map fusion in autonomous parking, including the steps of:

step 1), fusion judgment: inputting a plurality of track maps, judging the selected plurality of track maps pairwise, and judging that the two tracks are possibly fused if the minimum distance of GPS coordinates of two track key frames is smaller than a first threshold value or the same semantic information exists; if at least one map with fusion possibility exists, the selected track maps are judged to be fused, otherwise, the selected track maps are considered to be incapable of being fused and need to be selected again;

step 2), public area detection: firstly, extracting a key frame pair with GPS coordinates smaller than a first threshold value or the same semantic information from two maps needing to be combined; judging whether the global image description similarity is greater than a second threshold value or not for each key frame pair, if not, judging the next key frame pair, otherwise, matching local feature points with descriptors, and if the number of matched point pairs is greater than a third threshold value, entering step 3);

step 3), calculating an alignment transformation matrix between the two track maps: SE3 transformation (i.e., euro transformation) if the camera uses a binocular camera, Sim3 transformation (i.e., similarity transformation) if the camera uses a monocular camera; performing initial estimation by using map points corresponding to the matched local feature points in the step 2), improving the precision of an estimated value by using a random sampling consistency (Ransanc) method or a Direct Linear Transformation (DLT) method, optimizing a reprojection error by using nonlinear optimization to obtain a final alignment transformation matrix, returning to the step 2) to judge the next key frame pair if the number of optimized inner points is less than a fourth threshold, and otherwise, entering the step 4);

step 4), map merging: aligning the two maps by using the alignment transformation matrix obtained by calculation, namely performing alignment transformation on the poses of all key frames, the poses of parking spaces, map points and the poses of semantic information; then, fusing repeated information in the two maps and updating the common-view relationship between the key frames; the repeated information comprises parking space layer information, semantic layer information, map points in a positioning and tracking layer and key frame related information;

step 5), local nonlinear optimization: according to the common-view relation of key frames of two track maps positioning tracking layers, extracting all key frames having the common-view relation with the key frames to establish a local window, and performing local Beam Adjustment (BA) optimization on the window;

step 6), optimizing the track pose: and (4) carrying out Pose Graph Optimization (position Graph Optimization) on the fused track to obtain a track path and a Pose after map fusion.

Preferably, if the number of the specified track maps needing to be merged is greater than two, the merged map is continuously merged with the rest track maps in sequence, and the map merging process is repeated until the specified new track maps of all the tracks needing to be merged are finally returned.

Preferably, the same semantic information includes: cell number, floor number, road sign, landmark, lane line, building, and/or textual information.

Preferably, the first and second substrates are, among others,

the alignment transformation matrix in the SE (3) transformation is as follows:

the alignment transformation matrix in the Sim (3) transformation is:

where s is a scale factor for scale alignment, T is a pose matrix, R is a rotation matrix, T is a translation vector, R is a rotation matrix, and^3×3represents a real matrix set of 3 × 3 size, | r | represents a determinant of r.

Preferably, the first and second substrates are, among others,

taking the alignment transformation matrix as T_dThen the alignment change is:

x'＝T_d·x

wherein x is the coordinate under the world coordinate system before transformation, and x' is the coordinate under the world coordinate system after alignment transformation.

Preferably, the first and second substrates are, among others,

the SLAM model used consists of an equation of motion and an observation equation, as follows:

wherein u is_kFor motion sensor input, w_kFor motion sensor noise, v_k,jTo observe noise, y_jIs a road marking point, x_k-1And x_kIs the coordinate position of the two frames before and after, z_k,jTo count the observationsAccording to the formula, if f () is a motion equation and h () is an observation equation, then local BA optimization, i.e. simultaneous optimization of camera pose and observation data, using least square description, then:

wherein R is^-1 _kAnd Q^-1 _kIs an information matrix which is the inverse of a covariance matrix of the noise distribution of the motion sensor and the observation noise distribution, J (x, y) is a target function, and superscript T represents transposition operation; e.g. of the type_u,kAnd e_z,j,kRepresenting displacement error and observation error, respectively, which are calculated as follows:

wherein e_u,kRepresenting displacement error, e_z,j,kRepresenting the observation error.

Preferably, the first and second substrates are, among others,

the pose graph optimization is to fix observation data, only to perform nonlinear optimization on the pose of the camera, the objective function is:

wherein

For the information matrix, ε is the set of edges formed by the connection of all keyframes with common view relationships, e_i，jAs a residual of the camera pose, it is defined as follows:

wherein T is_iRepresents the i-th frame pose, T_jRepresents the j frame pose, T_i,,jRepresenting pose transformation from jth frame to ith frameFunction y ═ ln (t)^νRepresenting the transformation relationship of lie groups to lie algebra.

The invention has the beneficial effects that:

1. the realization cost is low. The autonomous parking system based on the track learning and tracking is adopted, so that the global path planning is not needed, the requirement on computing power is low, the dependence on a cloud platform is low, the expensive hardware platform is not needed to support, and the expensive RTK and Lidar modules are not needed to support;

2. the track map may be shared. The cloud map sharing is introduced, so that autonomous parking can be carried out without establishing a track map by self, and tracking parking can be carried out by using a map shared by other people even if the parking system is positioned in a scene strange by an owner (such as a hotel parking lot which is not yet visited);

3. the track map can be selected at the cloud, GPS information is fused, a user can conveniently and quickly select tracks shared by other people at the specified position at the cloud, tracks of different parking modes are selected, in addition, if the established map contains a semantic information layer, the selection of a certain floor number and a certain building position and the track of a certain landmark position are also supported, the functions are rich, and the humanization degree is high.

4. The distance for supporting the autonomous parking is long, and the coverage area of the site for starting the autonomous parking is large. Due to the introduction of the cloud map merging technology, a parking path is not limited to a place where a certain track passes, and the merging of multiple tracks can be supported, so that the autonomous parking memory system can be started in more places. Meanwhile, due to the fact that the path is relatively fixed, compared with a parking scheme based on global path planning, the autonomous parking scheme based on the track learning and tracking is higher in safety; meanwhile, a cloud track merging technology is introduced, so that the tracking distance of autonomous parking is further increased, and the requirement of long-distance tracking parking is met.

5. The cloud track map is strong in reusability, and firstly, due to the fact that the established track map contains information of parking places, parking place numbers, floor numbers, GPS coordinates and the like along the way, other users can independently park in the parking places along the way appointed by the cloud, and do not need to park according to the original track completely, and reusability of the track map is enhanced; secondly, support the function of fusing of high in the clouds orbit map, be convenient for establish the more abundant global semantics map in same place to can include more along the way parking stall information, further strengthen the reusability of orbit map, also be convenient for can more effectively find idle parking stall when parking at the parking stall of flowing parking stall in the more public region in parking area

6. The method can be used for repositioning in various scenes, has high repositioning success rate and accuracy, and is slightly influenced by factors such as illumination, weather, seasons and the like. Firstly, a candidate key frame selection strategy which integrates GPS coordinates and image global description is used, so that the scenes needing to be retrieved for relocation are few, the speed is high, and the success rate is high; and secondly, a coarse-to-fine repositioning technology is introduced, so that repositioning precision is high, and influence of factors such as illumination is small.

7. The parking type is various, multiple modes such as fixed parking stall parking, mobile (public) parking stall parking, automatic parking of vehicle and recall are supported, and the user only needs to drive the car to a certain region and can get off, need not the driver and just can accomplish independently parking in the car, wait until need use with the vehicle recall can to be applicable to multiple scene parking, like private parking stall, public parking area, the parking area of access & exit difference etc..

8. The vehicle end-cloud-4G/5G communication terminal cooperative system is established, a user can check the current position, the state and the top view of a vehicle in real time at a mobile phone, a tablet computer or other remote control terminals, so that a vehicle owner is relieved, the vehicle owner can be timely informed and remotely controlled to control the vehicle when meeting emergency conditions, if the parking in a fixed parking space meets the situation that the parking space is occupied, the vehicle owner can temporarily select other parking spaces to park at the terminal through the top view, the vehicle can park midway when the vehicle is recalled, and the like.

9. The method is suitable for the detection occasion of the fisheye camera, and can overcome the influence of image distortion caused by the fisheye camera. Even in the case of using a fisheye camera, a very satisfactory autonomous parking effect can be obtained. The autonomous parking control method and the autonomous parking control device particularly solve the autonomous parking control problem in the environments of narrow road sections, such as getting out of the hall and being close to the wall surface.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the invention and together with the description serve to explain the principles of the invention. The above and other objects, features and advantages of the present invention will become more apparent from the detailed description of the embodiments of the present invention when taken in conjunction with the accompanying drawings.

FIG. 1 is an overall system flow of the present invention;

FIG. 2 is a schematic diagram of a cloud-vehicle-end-4G/5G communication terminal system;

FIG. 3 is a schematic view of a track map-1 (a moving car track);

FIG. 4 is a schematic view of the track map-2 (fixed parking track);

FIG. 5 is a diagram of a semantic segmentation result of a driving area of a four-way camera;

FIG. 6 is a top view of a tiled display effect diagram;

FIG. 7 is a diagram of object detection and recognition results;

FIG. 8 is a schematic view of a 4G/5G remote control terminal display for a user;

FIG. 9 is a top view of a user 4G/5G communication terminal displaying a 360 ring vehicle supporting rotational dragging;

FIG. 10 is a diagram of parking space detection and angular point identification of a parking space;

FIG. 11 carport cataract detection (determination of parking availability);

FIG. 12 is an illustration of an automatic parking present for a plurality of stall types;

FIG. 13 extracts road sign semantic information (SLAM map semantic information layer);

FIG. 14 track map fusion example 1;

FIG. 15 track map fusion example 2;

fig. 16 track map fusion example 3 (building a global track map of the same location using "multi-vehicle fusion map").

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are for purposes of illustration only and are not to be construed as limitations of the invention. It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings.

In addition, the embodiments of the present invention and the features of the embodiments may be combined with each other without conflict. Technical solutions of the present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present application, "plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.

The invention aims to provide an autonomous parking system and method based on cloud sharing and map fusion. FIG. 1 depicts the overall system flow diagram of the present invention.

1 overview of the overall scheme

As shown in fig. 1, the system flow of the present invention mainly includes the following steps:

step 1), building a semantic map of a parking route and a parking space by using SLAM (Simultaneous Localization and mapping), and classifying track information for subsequent track selection;

classifying the track information according to parking scenes, parking/recalling and other information; further, the classification of the track information may be that the user is required to classify the track according to information such as parking scene, parking/recall, etc.;

step 2), storing the SLAM map in a local and/or cloud end;

when the user classifies the tracks according to information such as parking scenes, parking/recalling and the like, the SLAM map can be stored locally and/or in a cloud; further, the user selects to store the SLAM map locally and/or upload the SLAM map to the cloud sharing;

step 3), selecting and fusing a track map;

before the autonomous parking step is started, the existing local track map can be selected, the track map shared by others can be selected from the cloud according to GPS coordinates, scene semantic information and the like, and map fusion of multiple track maps is supported;

step 4), starting the autonomous parking step and setting the autonomous parking parameters;

step 5), carrying out relocation from coarse to fine;

after the track loading is finished, the relocation from coarse to fine is automatically carried out, after the relocation is finished, manual operation of a system is not needed, namely a user can get off the vehicle, the vehicle automatically finishes the parking or recalling process of the vehicle, in the whole parking or recalling process of the vehicle, the vehicle only carries out local path planning and does not need global path planning, meanwhile, the user can observe the running state of the vehicle in real time on a 4G/5G communication terminal, preferably, the top view of the vehicle can be observed in real time on the terminal and control instructions can be sent at any time, and the control instructions comprise deceleration, parking, recalling and the like; the vehicle can monitor the control instruction in real time in the process of autonomously completing parking or recalling the vehicle, preferentially execute the control instruction, and then continuously and automatically execute the step of automatically parking or recalling the vehicle. Therefore, the system has strong real-time performance and high safety.

The automatic parking of the vehicle comprises the following steps:

step 1.1), automatic tracking driving and obstacle avoidance local planning;

step 1.2), detecting and selecting a parking space;

step 1.3), automatically parking in a parking space;

the vehicle recall includes the steps of:

step 2.1), automatically parking out of the parking place;

step 2.2), automatic tracking driving and obstacle avoidance local planning

Step 2.3), mid-way or recall end-stop.

When the autonomous parking system works, the situation that the scene environment is complex and the working condition changes in real time is often faced, especially under the conditions of narrow road sections such as driving in a hallway and driving close to a wall surface, vehicle-mounted ordinary cameras such as ordinary perspective cameras, binocular cameras and RGBD cameras are almost completely shielded by buildings, and effective information cannot be acquired; at the moment, a fisheye camera is needed to be used for information acquisition; the fish-eye camera can acquire more observation information and effective information due to the larger visual angle range; for example, a plurality of fisheye cameras with each FOV of about 190 ° may be used to form a 360 ° blind spot-free all-round viewing area around the vehicle, so as to achieve the purpose of no blind spot. In addition, in the automatic parking process, the system needs to accurately sense a close range, that is, the image acquisition device can accurately detect objects and landmarks within 10 meters, and roughly sense objects within 25 meters. Therefore, on the one hand, an on-vehicle camera based on a fisheye camera is necessary in an automatic parking system; on the other hand, however, the scene captured by the fisheye camera is very different from the real world scene inevitably due to the inherent defects of the fisheye camera, namely the larger distortion necessarily accompanied by the larger photographic visual angle; this causes additional difficulty in autonomous parking using a fisheye camera. The traditional method cannot well solve the influence caused by image distortion detected by a fisheye camera, and is difficult to ensure the control precision of autonomous parking, so that the parking success rate is limited. Therefore, aiming at the technical difficulty, the novel autonomous parking system and method based on cloud sharing and map fusion can well solve the technical problem, all the methods of the method are realized on a distorted image acquired by a fisheye camera, the method has a wider view angle (FOV), the surrounding situation of a vehicle body can be observed, visual dead angles do not exist, and the method has strong robustness.

The method of the present disclosure is performed on a fisheye camera, which is a great feature and advantage of the cloud sharing and map fusion based autonomous parking system and method of the present disclosure, and means that the cloud sharing and map fusion based autonomous parking system and method of the present disclosure is superior in processing images acquired by the fisheye camera having distortion, but it does not mean that the cloud sharing and map fusion based autonomous parking system and method of the present disclosure can only be applied to an automatic parking system using the fisheye camera, and the cloud sharing and map fusion based autonomous parking system and method can also process images acquired by image acquisition systems of other general cameras including a general perspective camera, a binocular camera, an RGBD camera, etc., and on the basis that images acquired by image acquisition systems of other general cameras including the general perspective camera, the binocular camera, the RGBD camera, etc. satisfy the requirement of being able to acquire effective information, the autonomous parking system and method based on cloud sharing and map fusion are higher in parking precision and success rate. Accordingly, the present disclosure is a substantial improvement to an autonomous parking system and method that has strong robustness and high success rate of parking, and is able to overcome the effects of image distortion caused by the use of fisheye cameras.

The autonomous parking system and method based on cloud sharing and map fusion can use a common perspective camera, a binocular camera, an RGBD camera and the like, and can also use a fisheye camera; the autonomous parking method based on cloud sharing and map fusion is suitable for occasions using a common perspective camera, a binocular camera, an RGBD (red, green and blue) camera and the like to acquire information, and is also suitable for occasions using a fisheye camera to acquire information.

The autonomous parking system based on cloud sharing and map fusion comprises the following hardware devices:

(1)4 fisheye cameras for visual perception, such as parking space detection, visual SLAM, target detection, and the like;

(2)12 ultrasonic sensor arrays for parking space distance measurement, collision prevention and the like;

(3) a wheel speed meter for scale alignment of the track and the world coordinate system, vehicle control, and the like;

(4) global Positioning System (GPS) for relocation, trajectory selection, etc.;

(5) the 4G/5G communication module is used for communication of a vehicle end, a cloud end and a user terminal, uploading and downloading of a track map and the like;

(6) embedded hardware computing platforms, such as TDAx, Nvidia Xavier, renewas V3H, and the like.

(7) Other optional devices: such as an inertial navigation unit (IMU), a laser/millimeter wave radar (LiDar), a real-time dynamic carrier-phase differential (RTK) unit, etc., for assisting in establishing a semantic map, improving positioning accuracy, etc.

The solution described in this disclosure does not require a millimeter wave radar and thus the sensor is low cost, but those skilled in the art will appreciate that the lack of a millimeter wave radar in this disclosure does not mean that the present disclosure cannot use a millimeter wave radar.

Fig. 2 is a schematic system diagram of a cloud-vehicle-end-mobile communication terminal. The present disclosure constructs a cloud-vehicle-end-mobile communication terminal collaboration system, preferably, wherein the mobile communication terminal is a 4G/5G communication terminal; the cloud is used for track map storage, classification selection, sharing and fusion; the method comprises the steps of obtaining a cloud point, calculating a calculation load of a vehicle-mounted processor at the vehicle end, and carrying out track fusion on the cloud point. The cloud track fusion can quickly obtain a proper track by means of strong computing power of the cloud, and meanwhile, extra computing overhead of a vehicle end is not increased. The vehicle end is responsible for sensing, map building, planning and controlling; the 4G/5G communication terminal is responsible for monitoring; the 4G/5G communication terminal is a mobile phone, a tablet or other remote control terminals; the user can check the current position, the state and the top view of the vehicle in real time at a mobile phone, a tablet or other remote control terminals, and can timely inform the vehicle user when encountering an emergency situation and remotely control the vehicle on the mobile communication terminal.

Map building and trajectory classification based on SLAM

1) The SLAM (Simultaneous Localization and Mapping) method includes but is not limited to:

1.1 SLAM based on the feature point method: such as orbSLAM, MonoSLAM, PTAM, etc.;

1.2 SLAM based on direct method/optical flow method: DSO, DTAM, LSD-SLAM, etc.;

1.3 SLAM based on the semi-direct method: SVO, MSCKF, OKVIS, etc.;

1.4 SLAM based on sensor fusion: for example, IMU blends with vision (either loosely or tightly), represented as VINS, VIOs, orbSLAM3, etc., wheel speed blends with vision, etc.;

1.5 SLAM based on deep learning: for example, the traditional feature points are changed into feature points subjected to deep learning such as SuperPoint, R2D2, Hfnet and the like, and loop detection is performed by using the deep learning.

2) The map includes the following parts:

2.1 track layer (track layer): the method comprises the steps of constructing a map, wherein the map comprises a driving path of a vehicle, a track classification label, track starting point and ending point information, and the track starting point and the ending point are one point or one section of track on the track according to different track labels;

2.2 parking space layer (parking slot layer):

for the parking space layer, the fixed parking space mode comprises coordinates of a target parking space, a target parking space number and parking space information along the track. The target parking space number is automatically identified and can be confirmed by a user, and the target parking space number can be in a default state; the parking space information comprises a parking space number, parking space coordinates, parking space images and the like.

For the parking space layer, the mobile parking space mode comprises a parking available area, all parking space coordinates and parking space numbers detected in the parking available area and parking space information along the track. Wherein, the parking available area is generally a section of track or a section of track plus a specified deviation distance; the parking space information comprises a parking space number, parking space coordinates, parking space images and the like.

2.3 location and tracking layer:

the location tracking layer contains information required by SLAM repositioning and tracking, which varies according to the SLAM system used, taking a characteristic point method orbSLAM as an example, and contains map point coordinates, key frame poses (which can be represented by rotation matrix R and translation t, quaternion and translation t, xy coordinates and yaw angle yaw, etc.), key frame GPS coordinates, local feature points and descriptors of key frames and corresponding relations with map points, global descriptions of key frames (the global descriptions are based on vectors of DBOW bag model and/or global descriptors based on NetVLAD, etc.), common view relations between key frames, and corresponding relations of vision and other sensors (the sensors are wheel speed meters and/or IMUs, etc.) (the corresponding relations include scale parameter size of track alignment and/or external reference matrix, etc.).

2.4 semantic information layer (semantic information layer):

the semantic information layer comprises semantic information which can assist positioning and identification, such as unit numbers, floor numbers, road signs, landmarks, lane lines, buildings, text information and the like. The semantic information layer is optional and need not be included.

3) The trajectory classification includes the following parts:

3.1 fixing the parking space track: when a fixed parking space track mode is selected, the vehicle can start the parking space detection and automatic parking space number identification functions, and automatically store all parking space information (including the parking space number, the parking space coordinates, the parking space images and the like) detected along the way. Generating and displaying a track map after map building is finished, wherein a default track starting point is a map building starting point, a default track end point is a map building end point, and if a parking space number is detected at the starting point or the end point, a default target parking space number is the parking space number of the current parking space which is automatically detected; then, the parking space number is required to be confirmed; the user can change the starting point or the end point of the track through a certain point on the designated track, and at least one of the starting point or the end point is positioned in the parking space; the user can also directly change the parking space number information. And after the operation is finished, storing the operation information into a map and storing the map.

3.2 mobile parking space track: when the mobile parking space track mode is selected, the vehicle can start a parking space detection function and a parking space number automatic identification function, and automatically store all parking space information detected along the way (the parking space information comprises a parking space number, parking space coordinates, parking space images and the like). Generating a track map after the map building is finished, and displaying the track map, wherein the default track starting point is a map building starting point, and the default terminal point of the parking available area is a map building track terminal point; then, the user is required to operate, the user can change the starting point of the track and the end point of the parking available area by specifying a certain point on the track, at least one of the starting point and the end point is required to be positioned in the parking lot (namely, a plurality of parking spaces exist nearby), then the user is required to specify a point between the starting point and the end point as the starting point of the parking available area, and then the parking can be carried out on the free parking spaces detected during the tracking between the starting point and the end point of the parking available area during the tracking automatic parking. In addition, optionally, a maximum offset distance can be set, that is, when the detected carport deviates from the mapping track and is not greater than the specified maximum offset distance, the carport can be parked. After the operation is completed and confirmed, the information is stored in a map and the map is saved.

3.3 bidirectional trajectory: when the bidirectional track mode is selected, four cameras are required to be complete and distributed correspondingly (the corresponding distribution refers to that one camera is distributed at the front, back, left and right, wherein the cameras can be respectively corresponding to the left and right, and can also be respectively corresponding to the front and back), the types of the cameras distributed correspondingly (if the cameras are corresponding to the left and right, or corresponding to the front and back) are required to be the same, if the bidirectional track mode is selected, the specified mapping track can be used for parking the vehicle and recalling the vehicle, and the track type is usually used for roads without distinguishing the driving direction. In the bidirectional track mode, the information stored in the corresponding distributed cameras in the map is automatically exchanged, and the information of the starting point and the end point is automatically exchanged, so that a track is generated, and the two tracks are called shadow tracks. For the bidirectional tracks, when the vehicle is parked in a memory mode, one track with the end point located in the parking space/parking area in the shadow track is automatically selected as a vehicle parking track, and the other track is used as a vehicle recall track.

3.4 one-way vehicle parking track: the one-way vehicle parking track is applicable to the condition that the entrances and exits of some parking lots are different, and the one-way vehicle parking track is specified to be only used for parking vehicles, and the terminal point of the required track is a parking space or a parking area.

3.5 one-way vehicle recall trajectory: the one-way vehicle recall trajectory is applicable to certain parking lot exits and entrances that are different, and is designated as being available for vehicle recall only, when the starting point of the required trajectory must be a parking space or a parking area.

The track classification, namely the classification of the track information, can be that a user is required to classify the track according to information such as parking scenes, parking/recalling and the like; a user specifies a track type to be established before drawing is started, wherein one of a fixed parking space track and a moving parking space track needs to be selected, and the other one of a bidirectional track, a one-way vehicle parking track and a one-way vehicle recalling track needs to be selected. Fig. 3 is a schematic diagram of a track map-1 (a flowing car track). Fig. 4 is a schematic diagram of a track map-2 (fixed parking track).

3 local storage or cloud sharing map

After the map is built, the user can select to store the map locally or upload the map to the cloud (private or public sharing can be selected), and when the system is in a netless environment, the map is stored locally; when the system is in an internet environment, the map is stored locally or uploaded to the cloud according to user selection, and when the map is uploaded to the cloud, one of a private state and a public sharing state can be selected for storage and use.

After the cloud public sharing is selected, the cloud automatically reads track classification tags, starting point and end point information, on-way parking space information, GPS information and/or semantic layer information from an uploaded map, and integrates the information to form a tag list for describing the track, so that the track can be quickly classified and selected.

According to the method and the device, due to the introduction of the map sharing of the cloud, the step of establishing the track map by the user is unnecessary, the operation steps of the system are further simplified, the operation process is simpler, and the use experience of the user is improved. The user can independently park by using the shared map at the cloud, and can also use the map shared by other people to perform tracking parking even in a scene which is unknown to the user (such as hotels, shopping malls and hospital parking lots which have never been visited), so that the parking success rate is improved, and the parking efficiency is improved; in addition, the user can establish a track map of the user at any time and share the track map to the cloud so as to enrich the map library of the whole system and further improve the parking reliability and parking efficiency of the whole system.

Selection and fusion of 4-track maps

Although the shared track map provides a basis for improving the system parking effect on a theoretical framework, in specific implementation, if the system parking effect is improved only by simply multiplexing the shared map at the cloud end, and the parking scene is changed at any time, in order to further solve the problems that resources of the shared track map can be effectively utilized and the change of the existing parking scene can be adapted in real time, so that the autonomous parking efficiency and the success rate are improved, the method adopting track fusion is innovatively provided, the problems are effectively solved, and the quite successful parking effect is obtained.

In the modern urban parking layout, due to the fact that parking lots of large communities, large commercial areas, large traffic hubs, parks and the like are distributed dispersedly, large in scale and complex in structure, the distance of autonomous parking is too far, the coverage area of the autonomous parking place is too large, and new pressure is brought to autonomous parking. On the one hand, parking environments of parking lots vary more, and on the other hand, path planning is also more complicated. The cloud map merging technology can well solve the technical problems, and due to the introduction of the cloud map merging technology, a parking path is not limited to a place where a certain track passes, the merging of multiple tracks can be supported, and therefore the autonomous memory parking system can be started in more places. Meanwhile, the path is relatively fixed, so the autonomous parking scheme based on the track learning and tracking is innovatively provided, and the scheme has higher safety compared with a parking scheme based on global path planning; furthermore, a cloud track merging technology is introduced more innovatively in the autonomous parking scheme based on track learning and tracking, so that the trackable distance of autonomous parking is further increased. Therefore, the autonomous parking system and method based on cloud sharing and map fusion can support long distance of autonomous parking, and the coverage area of a place where autonomous parking can be started is large.

The specific implementation process comprises the following steps:

before the memory parking function is started, a user specifies a tracking track, and the tracking track can be selected from locally stored tracks, or a track is selected from a cloud end, or a plurality of tracks are selected from the cloud end to perform map combination to form a new tracking track.

When a user selects a track shared by others from the cloud, firstly, track searching is carried out, and during searching, selection is carried out according to information such as the appointed current GPS coordinate, track category, building location and/or parking space number; if a plurality of tracks at the same place are appointed, the cloud end can automatically perform track map fusion, when the track map fusion is completed, the operations of direct cloud end loading, cloud end addition collection, local downloading and the like can be performed, the reuse of the track map can be ensured, and the phenomenon that the track map merging operation needs to be performed again in autonomous parking under the same condition every time is avoided.

After the track is selected from the cloud, the track can be directly loaded, and can also be loaded after being downloaded to the local, so that the track can be used subsequently.

The track map fusion changes according to the change of the used SLAM system, and mainly comprises the following steps:

1) fusibility determination: judging the selected multiple track maps pairwise, and if the minimum distance between the GPS coordinates of the two track key frames is smaller than a certain threshold value or the same semantic information exists (the same semantic information can be the parking lot number, the same landmark and the like), judging that the two maps are possibly fused; if at least one map with fusion possibility exists, the selected track maps are considered to pass the fusion judgment, otherwise, the selected track maps are considered to be incapable of being fused and need to be selected again.

2) Common area detection: firstly, extracting a key frame pair with GPS coordinates smaller than a certain threshold or the same semantic information from two maps needing to be combined. Firstly, judging whether the description similarity of the global image (wherein the description similarity can be based on a DBOW word bag model method, a deep learning NetVLAD/Calc/Hfnet method and the like) is greater than a certain threshold value or not for each key frame pair, if not, judging the next key frame pair, otherwise, matching local feature points with descriptors, and if the number of matched point pairs is greater than a certain threshold value, entering the step 3);

3) computing an alignment transformation matrix between two trajectory maps: SE3 transform (euro transform) if the camera uses a binocular camera, Sim3 transform (similarity transform) if the camera uses a monocular camera; performing initial estimation by using map points corresponding to the matched local feature points in the step 2), improving the precision of an estimated value by using a Ransanc (random sampling consistency) method or a DLT (direct linear transformation) method, optimizing a reprojection error by using nonlinear optimization to obtain a final alignment transformation matrix, returning to the step 2) to judge the next key frame pair if the number of optimized inner points is less than a certain threshold value, and otherwise, entering a step 4);

the alignment transformation matrix in the SE (3) transformation is as follows:

the alignment transformation matrix in the Sim (3) transformation is:

4) Merging maps: aligning the two maps by using the alignment transformation matrix obtained by calculation, namely performing alignment transformation (SE3 transformation or Sim3 transformation) on all key frame poses, parking space poses, map points and semantic information poses; then, fusing repeated information in the two maps and updating the common-view relationship between the key frames; the repeated information comprises parking space layer information, semantic layer information, map points in a positioning and tracking layer and key frame related information.

Taking the alignment transformation matrix as T_dThen the alignment change is:

x'＝T_d·x

5) Local nonlinear optimization: according to the common-view relation of key frames of the two track maps positioning and tracking layers, extracting all key frames with the common-view relation with the key frames to establish a local window, and carrying out local BA (Beam Adjustment) optimization on the window.

The SLAM model used consists of a motion equation (e.g., wheel speed, IMU, etc. inputs) and an observation equation (e.g., camera, lidar, etc. inputs) as follows:

wherein u is_kFor motion sensor input, w_kFor motion sensor noise, v_k,jTo observe noise, y_jIs a road marking point, x_k-1And x_kIs the coordinate position of the two frames before and after, z_k,jFor observed data, f () is a motion equation, h () is an observation equation, then local BA optimization, i.e. optimizing camera pose and observed data simultaneously, using least squares description then:

6) Optimizing the track pose: and performing Pose Graph Optimization (position Graph Optimization) on the fused track (for example, the fused track is represented by using a key frame Pose), so as to obtain a track path, a Pose and the like after map fusion.

wherein

wherein T is_iRepresents the i-th frame pose, T_jRepresents the j frame pose, T_i,,jRepresenting the pose transformation from the j frame to the i frame, function y ═ ln (T)^νRepresenting the transformation relationship of lie groups to lie algebra.

And if the number of the appointed track maps needing to be combined is more than two, using the combined map to continue to be combined with the rest track maps in sequence, and repeating the map combining process until finally returning to the appointed new track maps of all the tracks needing to be combined.

According to the method, the track map is selected at the cloud end, the GPS information is fused, so that a user can conveniently and quickly select tracks shared by others at the specified position at the cloud end, and the tracks of different parking modes are selected; in addition, if the established map also comprises a semantic information layer, a certain floor number, a certain building or a track of a certain road sign can be further selected, so that the parking efficiency and the success rate are further improved, and the user experience is improved. Fig. 13 shows that road sign semantic information (SLAM map semantic information layer) is extracted, and a track at a certain road sign can be obtained after extraction, and the map information is taken into consideration when an actual map is merged, so that the safety of autonomous parking can be further guaranteed, and the parking efficiency and the success rate can be improved.

Preferably, the track map comprises information such as parking spaces, parking space numbers, floor numbers, GPS coordinates and the like along the way, other users can independently park the vehicle at the parking spaces along the way appointed by the cloud end without completely parking according to the original track, and therefore reusability of the track map can be enhanced; fuse high in the clouds orbit map for the global semantics map of same place is abundanter, including more along the way parking stall information, can further strengthen the reusability of orbit map, also is convenient for can find idle parking stall more effectively when public areas such as parking area flow the parking stall and park.

The track map fusion example diagrams are shown in fig. 14, fig. 15 and fig. 16, respectively. Fig. 14 illustrates fusion of two trajectory maps, which respectively show the two trajectory maps and the trajectory map after fusion, and shows the corresponding relationship between the trajectory maps after fusion and before fusion. Fig. 15 exemplarily illustrates the fusion of three trajectory maps, showing the fused trajectory, and the fused trajectory map. Fig. 16 is an exemplary illustration of sequentially merging and repeatedly fusing three or more trajectory maps until a final fused trajectory map is obtained.

5 initiating autonomous parking and performing a parking setup

After the track selection is completed, firstly, a track map needs to be loaded (from local or cloud), and necessary information such as a track starting point, a track end point, parking space information, scale parameter size of vision and other sensors (such as a wheel speed meter, an IMU and the like) in the map is loaded. Then, the information of the vehicle sensor, such as camera model and parameter information, an external parameter matrix between sensors, IMU random walk (random walk) variance, Gaussian noise (noise) variance and the like, is loaded. After the loading is finished, when all necessary sensors are confirmed to be normally started and operate normally, the autonomous parking function is started, and at the moment, parking setting needs to be carried out. Firstly, a user is required to select a parking mode, namely vehicle parking or vehicle recall, fixed parking or mobile parking; it should be noted that the vehicle parking or the vehicle recall can be selected only if the loaded trajectory category is the fused trajectory or the bidirectional trajectory, otherwise, the parking mode can only be the same as the loaded trajectory category; in addition, it should be noted that when the loaded trajectory type is a mobile parking trajectory or a trajectory that includes at least one merged mobile parking trajectory, it may be selected whether to park in a fixed parking space or a mobile parking space, and when the loaded trajectory type is a fixed parking space or a trajectory that is only merged by fixed parking spaces, it may be selected only to park in a fixed parking space.

Secondly, when the user selects the vehicle recall mode, a recall end point needs to be confirmed, the recall end point can be set by the user, the recall end point is defaulted to be a track end point, and the user can also appoint any point on the track as the track end point again.

When the user selects the parking mode to park the vehicle, the following conditions are included:

if the parking mode is a fixed parking space parking mode, a user is required to confirm parking space information, the parking space information defaults to a target parking space in the loading track, and because information of all parking spaces along the track, such as images, parking space numbers, coordinates and the like, is stored during map building, the user can select any parking space stored along the track as the parking space, and the reusability of the track is enhanced; and when the loaded track type is the fused track, the user is required to select.

If the parking mode is a mobile parking space parking mode, a user is required to confirm a parking available area, the parking available area is the parking available area in the loading track by default, and when the type of the loading track is a fused track, the user is required to select, because the fused track is synthesized by at least one mobile parking track, the track comprises a plurality of starting points of the parking available area and a plurality of end points of the parking available area, the user can select two points from the track, and the vehicle can park when detecting an idle parking space between the two points.

In addition, the user may preferably specify a maximum offset distance, i.e., may park the vehicle when the detected deviation of the vehicle from the loaded trajectory is not greater than the specified maximum offset distance.

Generally, the parking setting has a default value, and the operation of a user is not needed. Every time the default value is changed or the track without the default value (such as the track after fusion) is set for the first time, the user needs to set whether to save the current parking setting as the default value or not so as to reuse the current parking setting for the next time.

6 relocation from coarse to fine

After the parking setting is finished, vehicle relocation is needed, in order to improve the positioning accuracy and positioning efficiency, the method adopts a coarse-to-fine visual relocation technology, greatly reduces the calculated amount, ensures the real-time performance, not only ensures the relocation success rate, but also ensures the high positioning precision, and simultaneously defines the relocation range by fusing GPS coordinates, greatly reduces candidate key frames needing comparison, reduces the scenes needing retrieval for relocation, effectively reduces the calculation consumption of the system, and ensures that the relocation consumes less time and is fast; on the other hand, the relocation strategy from coarse to fine leads the relocation precision to be high, the influence of factors such as illumination is small, and the success rate is high.

Rough positioning: the method comprises the following steps: firstly, positioning to a rough position in a map by using a GPS coordinate of a current vehicle, and acquiring all key frame information within a certain distance nearby; and then, further selecting the key frames according to the global description and semantic information of the current acquired image, and carrying out next-step accurate positioning on the key frames which meet the condition that the global description similarity of the image is greater than a certain threshold value or contain the same semantic information. The global description similarity is obtained by a method based on a DBOW bag-of-words model, a method based on deep learning NetVLAD/Calc/Hfnet and the like. The same semantic information includes floor numbers, buildings, etc.

And (3) accurate positioning: comparing the key frames returned in the rough positioning one by one, calculating an accurate pose, and if an accurate pose is successfully obtained, considering that the repositioning is successful; otherwise, the current state of the vehicle is reselected to perform relocation from coarse to fine. The accuracy degree of the accurate pose is greater than a certain threshold.

The calculation method of the accurate pose is as follows:

1) a method based on local feature point and descriptor matching:

the local feature points and descriptors include, but are not limited to, ORB, SuperPoint, R2D2, Hfnet, aslpeak, blbid, etc.;

the matching methods include, but are not limited to: descriptor matching based on a DBOW word bag model, descriptor matching based on machine learning methods such as knn/kmeans and the like, and descriptor matching methods based on deep learning such as SuperGlue, D2Net and the like are used.

And after the matching relation among the characteristic points is obtained, obtaining a maximum matching set by using methods such as Ranac or DLT and the like, calculating the current vehicle pose according to a PnP method, and measuring the accuracy degree of the pose according to the number of points in the PnP.

Preferably, on the basis, a BA (bulb adjustment) method is used for global optimization, and the accuracy degree of the pose is measured according to the number of the optimized interior points.

2) Directly obtaining the pose of the current image according to a KFNet equal deep learning model; and measuring the accuracy degree of the pose according to the confidence coefficient output by the model.

When the parking device is in a vehicle parking mode, the user can leave the vehicle after the relocation is successful, and the vehicle can finish the parking process independently. When the vehicle is in the vehicle recall mode, the vehicle can autonomously complete the recall as long as the vehicle is automatically parked or the distance between the current GPS coordinate of the vehicle and the preset track is less than the threshold value.

7 automatic tracking driving and obstacle avoidance local planning

In the automatic tracking driving and obstacle avoidance local planning process, a vehicle carries out tracking driving along a loaded track without global path planning, and the automatic tracking driving and obstacle avoidance local planning comprises travelable area detection, object detection and identification, ultrasonic obstacle detection and top view splicing; wherein the object detection and recognition includes obstacle detection, vehicle recognition, pedestrian recognition, and the like; in the whole process of automatic tracking driving and obstacle avoidance local planning, a GPS coordinate and a 4G/5G communication module are adopted to establish a vehicle end-4G/5G communication terminal connection, the vehicle end sends real-time data to 4G/5G communication terminal equipment, a user monitors the position and parking state of the vehicle in real time through the terminal equipment and can make a temporary decision to control the vehicle, namely, the user can observe the top view of the vehicle in real time on the terminal equipment and can send control instructions at any time, and the control instructions comprise deceleration, parking, recall and the like; the vehicle can monitor the control instruction in real time in the process of autonomously completing parking or recalling the vehicle, preferentially execute the control instruction, and then continuously and automatically execute the step of automatically parking or recalling the vehicle. The real-time data comprises data such as GPS coordinates, a current vehicle top view and the like; the terminal equipment comprises a mobile phone, a tablet personal computer and the like; the temporary decision includes parking, changing parking space, etc.

In order to prevent the discontinuity of the pose caused by the short-term loss of a certain sensor and ensure the continuity and the success rate of tracking to the maximum extent, the automatic tracking driving disclosed by the invention adopts a mode of fusing visual tracking, GPS coordinates, a wheel speed meter and an IMU (inertial measurement Unit) to calculate the current vehicle pose, so that even if the visual sensor is lost due to the visual positioning caused by factors such as illumination change, the GPS coordinates, the wheel speed meter and other sensors can be used for transition, and the robustness of the method can be enhanced.

When an object exists on a preset track in the automatic tracking process, whether the object is a dynamic obstacle or a static obstacle is judged first. If the object is a dynamic obstacle, the following processing steps are carried out: if the dynamic barrier appears suddenly and the distance between the dynamic barrier and the vehicle is less than a certain threshold value, starting an emergency braking function and stopping for waiting; otherwise, the vehicle speed is reduced for running.

If the object is a static obstacle or the parking waiting time exceeds a certain threshold value, starting local path planning to bypass, and then returning to the preset track to continue the tracking driving. When starting the local path planning to detour, executing the above-mentioned obstacle avoidance local planning steps, namely, detecting a travelable area, detecting and identifying an object, detecting an ultrasonic obstacle and splicing a top view; wherein the object detection and recognition includes obstacle detection, vehicle recognition, pedestrian recognition, and the like; in the whole process of automatic tracking driving and obstacle avoidance local planning, a GPS coordinate and a 4G/5G communication module are adopted to establish a vehicle end-4G/5G communication terminal connection, the vehicle end sends real-time data to 4G/5G communication terminal equipment, a user monitors the position and parking state of the vehicle in real time through the terminal equipment and can make a temporary decision to control the vehicle, namely, the user can observe the top view of the vehicle in real time on the terminal equipment and can send control instructions at any time, and the control instructions comprise deceleration, parking, recall and the like; the vehicle can monitor the control instruction in real time in the process of autonomously completing parking or recalling the vehicle, preferentially execute the control instruction, and then continuously and automatically execute the step of automatically parking or recalling the vehicle. The real-time data comprises data such as GPS coordinates, a current vehicle top view and the like; the terminal equipment comprises a mobile phone, a tablet personal computer and the like; the temporary decision includes parking, changing parking space, etc. The obstacle avoidance local planning ensures that a locally planned path is located in a travelable area and does not deviate from a preset track too far while avoiding objects such as obstacles, pedestrians, vehicles and the like. FIG. 5 is a diagram of semantic segmentation results of a driving area of a four-way camera. Fig. 6 is a top view of a tiled display effect. Fig. 7 is a diagram showing the result of object detection and recognition. FIG. 8 is a schematic diagram of a 4G/5G remote control terminal display for a user. FIG. 9 is a top view of a user 4G/5G communication terminal showing a 360 annular vehicle supporting rotational dragging.

The local path planning method includes, but is not limited to, a teb method (time Elastic Band), a DWA method (Dynamic Window Approach), a sampling-based op _ planer method, and the like.

8 parking stall detection and selection

1) If the current state is in the fixed parking spot mode: when the target parking space appointed by the current position of the vehicle is smaller than a certain threshold value, parking space detection and selection are needed, firstly, a top view splicing method is adopted, parking space detection and parking space number identification are carried out on the basis of the top view, and parking accessibility judgment is carried out when the error between the detected parking space position and the appointed target parking space position is smaller than a certain threshold value or the detected parking space number of the target parking space is consistent with the appointed parking space number. When the target parking space meets the parking accessibility condition, controlling the vehicle to automatically park in the target parking space; otherwise, when the target parking space is not detected or the target parking space does not meet the parking accessibility condition, the mobile communication terminal (the mobile communication terminal is a 4G/5G mobile phone and/or a tablet and the like) informs the user, reports the information that the designated parking space cannot be parked to the user, displays the current top view of the vehicle at the mobile communication terminal, and the user can re-designate the target parking space or send an instruction of 'parking beside for waiting' on the top view of the vehicle at the mobile communication terminal. If the user appoints the target parking space again, the vehicle carries out parking judgment again, and carries out autonomous parking processing according to the judgment result, wherein the processing mode is the same as the above; when a user sends an instruction of 'parking while approaching' or the user does not respond when a certain time threshold is exceeded, the vehicle starts a local path planning step to select a place which does not obstruct the passing to park while approaching nearby, and sends the GPS coordinates of the vehicle, the four-way camera image, the current top view, the semantic information collected around and the relative position of the target parking space to the mobile communication terminal for display. Preferably, when the fixed parking space is occupied when the parking in the target parking space is met, the user is informed, the user remotely controls the vehicle on the mobile communication terminal, further, the user temporarily selects other parking spaces to park in the mobile communication terminal through the top view, and further, a midway parking instruction is sent out when the vehicle is recalled.

2) If the current state is in flowing parking space mode: when the current position of the vehicle reaches the starting point of the designated parking available area, the vehicle carries out parking space detection and selection, firstly, a top view splicing method is adopted, based on the top view, parking space detection is carried out, and when a target parking space is detected, firstly, whether the distance between the target parking space and the preset track exceeds the designated maximum deviation distance or not is judged. Preferably, the distance between the target parking space and the preset track is detected by an ultrasonic ranging and visual depth estimation method; preferably, the maximum deviation distance may be previously designated by a user or set in real time through a mobile communication terminal. And if the distance between the target parking space and the preset track does not exceed the maximum deviation distance or the user does not specify the maximum deviation distance, carrying out parking availability judgment. And when the target parking space meets the parking accessibility condition, controlling the vehicle to automatically park in the target parking space, identifying the parking space number of the target parking space, and storing the identified parking space number (if the identification is unsuccessful, the parking space number is empty). And if the maximum deviation distance is exceeded or the current parking space does not pass the parking availability judgment although the maximum deviation distance is not exceeded, abandoning the target parking space to continue the target parking space detection.

Fig. 10 is a view of parking space detection and angular point identification.

The determination of the parking availability includes:

and detecting the target parking space by adopting a target detection and identification module, an ultrasonic ranging module, a visual depth estimation module and the like. And the ultrasonic ranging module and the visual depth estimation module are adopted to detect the length and the width of the target parking space. And if the target detection and identification module detects that dynamic objects (such as pedestrians and animals) exist in the target parking space, sending a parking waiting instruction to enable the vehicle to stop immediately, and timing and recording the parking waiting time. And if the target detection and identification module detects that static obstacles (such as other vehicles, ground locks, guideboards, garbage cans and the like) exist in the target parking space, judging that the target parking space does not pass the parking availability judgment. And if the parking waiting time exceeds a certain threshold value or the length and the width of the target parking space detected by the ultrasonic ranging module and the visual depth estimation module are not enough to park the vehicle, judging that the target parking space fails to pass the parking availability judgment. Otherwise, the determination is that the determination is passed. Fig. 11 is a diagram showing detection of an obstacle in the vehicle space during the parking availability determination process.

If the parking space still cannot be reserved from the current position to the terminal of the designated parking available area, the mobile communication terminal informs the user of not finding a free parking available parking space, and the user can send a command of 'searching for a parking space again' or 'waiting for parking beside' through the mobile communication terminal.

When the vehicle receives the 'seek the parking stall again' instruction, firstly carry out the loop judgment: if a loop mark exists in a track section from the terminal point of the parking-available area to the terminal point of the track and the track before the starting point of the parking-available area, the vehicle automatically tracks and drives from the terminal point of the parking-available area to the loop mark to enter the original track; if there is no loopback mark in the track map between the end point of the parking-enabled area and the track before the start point of the parking-enabled area, then, performing approximate loop judgment on a track segment from the terminal point of the parking available area to the terminal point of the track (in some embodiments, the terminal point of the parking available area and the terminal point of the track may be the same point), finding out the minimum value of the GPS distance between the track segment and the track before the starting point of the parking available area, if the minimum value is smaller than a set approximate loop threshold value, then the two points are considered to have approximate loop, the point on the track section between the terminal point of the parking available area and the terminal point of the track is taken as the approximate loop starting point, the point on the original track is taken as the approximate loop terminal point, the vehicle is automatically tracked and driven from the terminal point of the parking available area to the approximate loop starting point, and the vehicle is automatically driven to the approximate loop ending point by adopting local path planning to enter the original track.

When a user sends an instruction of 'parking while approaching to wait', or no loop mark and no approximate loop exist, or the user does not respond for a long time, the vehicle adopts a local path planning method to select nearby places which do not obstruct the passage to park while approaching, and sends the GPS coordinates of the vehicle, the four-way camera image, the current top view, the semantic information collected around and the relative position of the starting point/the ending point of the parking available area to the mobile communication terminal for display.

When the traditional autonomous parking method is used for parking space detection and selection, because the method modules needed in the link are more and the road condition of a parking lot is more complex, the method for reducing the vehicle speed is selected more for searching, detecting and selecting the target parking space, at the moment, the tracking driving method for global path planning is poor in timeliness due to large calculation consumption, and snow frosting is particularly performed on the target parking space detection of the vehicle with reduced speed in the parking lot, so that the parking space detection and selection efficiency is low. In order to solve the technical problem, the method for detecting and selecting the new parking space is adjusted, and the parking space is detected and selected by the vehicle on the basis of the automatic tracking driving and obstacle avoidance local planning; the autonomous parking system and the autonomous parking method based on cloud sharing and map fusion are relatively accurate in path planning, small in system calculation cost, short in adjustment time and fast in response time, and real-time performance, safety and reliability of the autonomous parking system and the autonomous parking method based on cloud sharing and map fusion are guaranteed in principle. The autonomous parking system based on the trajectory learning and tracking does not need global path planning, has low computational power requirement, low dependence on a cloud platform, no need of expensive hardware platform support and no need of expensive RTK and Lidar module support, so that the autonomous parking method and the system disclosed by the invention have the advantages of low implementation cost, quick response, high parking success rate and good user experience.

9 automatic parking/parking position

1) Automatically parking into a parking space:

fig. 12 shows an automatic parking pull-in view of various parking space types. The automatic parking mode firstly needs to accurately target the type (such as horizontal parking space, vertical parking space, inclined parking space and the like) of the parking space and the geometric parameters (such as length, width and the like) of the parking space, and the adopted methods include but are not limited to a method for detecting the parking space by adopting an ultrasonic sensor and a wheel odometer, a method for detecting the angular point and the line of the parking space by adopting a visual method based on deep learning, a parking space detection method for identifying a mark line based on Hough transformation and clustering and identifying a line of the parking space by a one-dimensional filter, a parking space detection method for fusing the ultrasonic sensor and the visual information and the like.

After the parking space information is accurate, the vehicle starts a local path planning method to park in a target parking space, and after parking is completed, the vehicle sends the GPS coordinate, the number of the parked vehicle and the current top view of the current vehicle to the mobile communication terminal, and then the vehicle is automatically flamed out.

2) Automatic parking out of the parking space:

in the automatic parking-out parking space mode, firstly, the minimum distance between the current vehicle GPS coordinate and a recall track needs to be calculated, if the distance is greater than a threshold value, recall is not possible, and a user needs to reselect the recall track; otherwise, starting a local path planning method to park out the parking space and return to the designated recall track, and then tracking driving according to the steps of automatic tracking driving and obstacle avoidance local planning.

The local path planning method comprises the following steps: the method comprises the steps of establishing a road stiffness planning method of an automobile kinematic model based on Ackermann steering geometry, parking path planning based on B spline theory, local path planning based on A method, road stiffness planning based on RBF neural network model, two-step trajectory planning method, bidirectional trajectory planning method or polynomial curve planning method and the like.

10 mid-way or recall end stops

In the vehicle recall mode, after the vehicle is parked out, tracking driving is carried out according to automatic tracking driving and obstacle avoidance local planning until the vehicle reaches a designated recall terminal, and a user can also send a midway parking instruction through a 4G/5G communication terminal (such as a mobile phone, a tablet and the like) to control the vehicle, so that the purpose of receiving midway passengers and getting on the vehicle is achieved.

In addition, in the whole parking process, the user can stop the ongoing autonomous parking at any time; if the user is on the vehicle, the automatic parking can be stopped at any time and the automatic parking can be converted into manual control of the vehicle.

Therefore, the method constructs a cloud-vehicle-end-4G/5G communication terminal collaboration system, a user can check the current position, the state and the top view of the vehicle in real time through a mobile phone, a tablet or other remote control terminals, the user can be timely notified when encountering an emergency, and the vehicle can be controlled through remote control on the mobile communication terminal.

In conclusion, in the autonomous parking, the conventional method has difficulty in obtaining satisfactory comprehensive effects in real-time performance, stability and success rate. The autonomous parking system and the method based on cloud sharing and map fusion, which are provided by the disclosure, are improved in the following aspects, so that the autonomous parking efficiency is high, and the success rate is higher:

the realization cost is low. The autonomous parking system based on the track learning and tracking is adopted, so that the global path planning is not needed, the requirement on computing power is low, the dependence on a cloud platform is low, the expensive hardware platform is not needed to support, and the expensive RTK and Lidar modules are not needed to support;

the track map may be shared. The cloud map sharing is introduced, so that autonomous parking can be carried out without establishing a track map by self, and tracking parking can be carried out by using a map shared by other people even if the parking system is positioned in a scene strange by an owner (such as a hotel parking lot which is not yet visited);

the track map can be selected at the cloud, GPS information is fused, a user can conveniently and quickly select tracks shared by other people at the specified position at the cloud, tracks of different parking modes are selected, in addition, if the established map contains a semantic information layer, the selection of a certain floor number and a certain building position and the track of a certain landmark position are also supported, the functions are rich, and the humanization degree is high.

The distance for supporting the autonomous parking is long, and the coverage area of the site for starting the autonomous parking is large. Due to the introduction of the cloud map merging technology, a parking path is not limited to a place where a certain track passes, and the merging of multiple tracks can be supported, so that the autonomous parking memory system can be started in more places. Meanwhile, due to the fact that the path is relatively fixed, compared with a parking scheme based on global path planning, the autonomous parking scheme based on the track learning and tracking is higher in safety; meanwhile, a cloud track merging technology is introduced, so that the tracking distance of the autonomous parking is further increased.

The cloud track map is strong in reusability, and firstly, due to the fact that the established track map contains information of parking places, parking place numbers, floor numbers, GPS coordinates and the like along the way, other users can independently park in the parking places along the way appointed by the cloud, and do not need to park according to the original track completely, and reusability of the track map is enhanced; secondly, support the function of fusing of high in the clouds orbit map, be convenient for establish the more abundant global semantics map in same place to can include more along the way parking stall information, further strengthen the reusability of orbit map, also be convenient for can more effectively find idle parking stall when parking at the parking stall of flowing parking stall in the more public region in parking area

The method can be used for repositioning in various scenes, has high repositioning success rate and accuracy, and is slightly influenced by factors such as illumination, weather, seasons and the like. Firstly, a candidate key frame selection strategy which integrates GPS coordinates and image global description is used, so that the scenes needing to be retrieved for relocation are few, the speed is high, and the success rate is high; and secondly, a coarse-to-fine repositioning technology is introduced, so that repositioning precision is high, and influence of factors such as illumination is small.

The parking type is various, multiple modes such as fixed parking stall parking, mobile (public) parking stall parking, automatic parking of vehicle and recall are supported, and the user only needs to drive the car to a certain region and can get off, need not the driver and just can accomplish independently parking in the car, wait until need use with the vehicle recall can to be applicable to multiple scene parking, like private parking stall, public parking area, the parking area of access & exit difference etc..

The vehicle end-cloud-4G/5G communication terminal cooperative system is established, a user can check the current position, the state and the top view of a vehicle in real time at a mobile phone, a tablet computer or other remote control terminals, so that a vehicle owner is relieved, the vehicle owner can be timely informed and remotely controlled to control the vehicle when meeting emergency conditions, if the parking in a fixed parking space meets the situation that the parking space is occupied, the vehicle owner can temporarily select other parking spaces to park at the terminal through the top view, the vehicle can park midway when the vehicle is recalled, and the like.

The method is suitable for the detection occasion of the fisheye camera, and can overcome the influence of image distortion caused by the fisheye camera. Even in the case of using a fisheye camera, a very satisfactory autonomous parking effect can be obtained. The autonomous parking control method and the autonomous parking control device particularly solve the autonomous parking control problem in the environments of narrow road sections, such as getting out of the hall and being close to the wall surface.

Therefore, the autonomous parking system and method based on cloud sharing and map fusion, which are provided by the disclosure, can overcome the distortion influence of a fisheye camera detection image, reduce the calculation consumption of the system, adapt to indoor and outdoor parking lot environments, do not increase extra calculation overhead, can be used for a low-power-consumption vehicle-mounted processor, do not need high-cost sensor system support, are rapid in autonomous parking, have high success rate, and have wide application prospects.

So far, the technical solutions of the present invention have been described with reference to the preferred embodiments shown in the drawings, but it should be understood by those skilled in the art that the above embodiments are only for clearly illustrating the present invention, and not for limiting the scope of the present invention, and it is apparent that the scope of the present invention is not limited to these specific embodiments. Equivalent alterations and substitutions of related technical features may be made by those skilled in the art without departing from the principles of the present invention, and such alterations and substitutions are intended to be within the scope of the present disclosure.

Claims

1. An autonomous parking method based on cloud sharing and map fusion is characterized by comprising the following steps:

step 2), storing the track map in a local and/or cloud terminal;

step 3), generating a target track map according to the selected track map;

the fusion is as follows:

2. The cloud sharing and map fusion based autonomous parking method according to claim 1, wherein the camera is a fisheye camera.

3. Cloud sharing and map fusion based autonomous parking method according to claim 1 or 2,

the classification of the trajectory information is classified according to parking scenes and parking/recall information.

4. The cloud sharing and map fusion based autonomous parking method according to any one of claims 1-3,

the classification of the trajectory information requires the user to classify the trajectory according to parking scenes, parking/recall information.

5. The cloud sharing and map fusion based autonomous parking method of claim 4,

when the user classifies the tracks according to parking scenes and parking/recalling information, the user selects to store the SLAM map locally and/or upload the SLAM map to the cloud sharing.

6. The cloud sharing and map fusion based autonomous parking method of claim 1,

when the track map is selected, one or more track maps shared by others are selected from the cloud according to the GPS coordinates and scene semantic information so as to be used for track map fusion.

7. The cloud sharing and map fusion based autonomous parking method of claim 1,

the method also comprises the following steps of vehicle automatic parking or vehicle recall:

the automatic parking of the vehicle comprises the following steps:

(1) automatic tracking driving and obstacle avoidance local planning;

(2) detecting and selecting a parking space;

(3) automatically parking in a parking space;

the vehicle recall includes the steps of:

(1) automatically parking out of the parking place;

(2) automatic tracking driving and obstacle avoidance local planning;

(3) mid-course or recall end stops.

8. The cloud sharing and map fusion based autonomous parking method of claim 7,

and carrying out map sharing, track selection and track fusion by adopting a cloud terminal.

9. An autonomous parking system based on cloud sharing and map fusion is characterized in that,

the method comprises the following steps:

the map storage device stores the track map in the local and/or cloud end;

wherein the fusion is:

10. A method for track map fusion in autonomous parking is characterized by comprising the following steps: