SK1272022U1 - Method of locating an autonomous vehicle and the connection of a visual system for locating an autonomous vehicle - Google Patents

Abstract

The method of locating an autonomous vehicle is characterized by the steps of: determining the initial position (6) of the vehicle using a GNSS sensor system (4); calculating the error ellipse (7) of the vehicle position by fusing data from the sensor system (4) GNSS to determine the approximate position (6) of the vehicle and the sensor system (5) odometry to determine the error ellipse estimate (7) of the vehicle position; visual scanning of the road identifier (2) R-ID on the lane in front of the vehicle in the field of view of the visual device; image processing (16) by a sampling algorithm; recognition (17) to recognize the road identifier (2) R-ID; determination, by the selection algorithm (11), of a specific recognized road identifier (2) R-ID with its exact position and at the same time the relative position (12) of the vehicle with respect to the recognized road identifier (2) R-ID based on the intersection of the ellipse estimation (7) error the position of the vehicle with the surroundings (9) of this R-ID in the virtual map (10) and the R-ID database; calculating the relative coordinates of a specific recognized road indicator (2) R-ID in the direction of the vehicle axis x and perpendicular to the vehicle axis y in millimetre scale and the relative position (12) of the vehicle with respect to this R-ID; calculation of the precise position (13) of the vehicle, by the conversion algorithm and the sum (15), from the known position (14) of the specific recognized road identifier (2) R-ID and from the relative position (12) of the vehicle with respect to this R-ID.

Description

The field of technology

The technical solution relates to the method of increasing the accuracy of the localization of the autonomous vehicle and the involvement of the visual system for the localization of the autonomous vehicle. The technical solution falls into the field of localization systems of autonomous vehicles.

Current state of the art

In the current state of the art, it is known that GPS or multi-system GNSS (GPS, GLONASS, Galileo, Beidou) in combination with IMU (Inertial Measuring Unit) and odometry is often used as the key technology for the localization of autonomous vehicles. GNSS can provide a global, high-precision positioning service, but due to satellite orbital error, timing error with the influence of signal propagation errors, and so on, the positioning accuracy of GNSS can only reach the level of 1 meter. Although it is possible to use the method of phase shift and post-processing (without moving the vehicle) and thus increase the positioning accuracy to a few centimeters, in intensive building construction or in interiors - halls, interference or signal interference occurs. In such a case, the use of GNSS for the purposes of localization and navigation of an autonomous vehicle is impossible.

To improve positioning accuracy, position information from other sensors is often integrated into the calculations for navigation purposes. Vehicle position can be determined using vehicle wheel speed encoders (wheel odometry). However, such a method is susceptible, in addition to the encoder error itself, to the accumulation of error in certain environmental conditions (deposition of soil on the wheels, wheel slippage), when it is not possible to ensure obtaining the exact position of the vehicle and estimating the position. Another method for determining position is an inertial navigation system (INS), which determines position relative to an initial inertial space. The exact determination of the position using the INS system is burdened by an integration error that grows over time, making this system unsuitable for independent use in autonomous vehicles.

From the state of the art, several methods and ways of visual localization of the vehicle and increasing the accuracy of the localization are known. For example, US20200318976 A1 provides systems and methods for controlling a vehicle. In one embodiment, the method includes: receiving landmark data obtained from a vehicle image sensor by a processor; fusing the landmark data using the processor with the vehicle position data to form the fused lane data, the fusing being based on a Kalman filter; obtaining, using the processor, map data from the lane map based on the vehicle position data; selectively updating the lane map by the processor based on a change in the associated lane data from the map data; and controlling the vehicle by the processor based on the updated lane map.

Furthermore, document WO2014130854 A1 describes various techniques for use in an inertial navigation system with the support of a visual system. It is an inertial navigation system with the support of a visual system, which includes: an image source that generates image data containing a number of images; and an inertial measurement unit (IMU) that generates inertial measurement unit data indicative of motion of the visually assisted inertial navigation system when the visually assisted inertial navigation system generates an image, the image data capturing features of the external calibration target. The vision-aided inertial navigation system further includes a processing unit including an estimator, wherein the processing unit includes processing of inertial measurement unit data and image data to calculate calibration parameters for the vision-aided inertial navigation system in calculating the tilt and inclination of the calibration target, wherein the calibration parameters define relative positions and orientations of the inertial measurement unit and image source of the vision-aided inertial navigation system.

A method for a vehicle control system with a perspective view camera generating perspective images of vehicles in the direction of traffic, including performing an anti-perspective transformation on the perspective images is presented in document US9928426 B1. The method divides each anti-perspective image into sub-images along one of the radial direction and the tangential direction with respect to the traffic direction, and determines the scale factor for each sub-image based on measuring the scale of each vehicle at multiple positions in one image using the frame difference method. Subsequently, scale transformation is performed for each sub-image using the corresponding scale factors for the sub-image. By combining each of the scaled transformed sub-images for each vehicle into the corresponding enhanced anti-perspective images, it is then possible to perform vehicle detection for each vehicle based on the combination of the enhanced anti-perspective images, in which the tracking is enhanced by an optimized detection box size range determined by the enhanced anti-perspective images.

SK 127-2022 U1

The vehicle can then be tracked based on a combination of enhanced anti-perspective images in which the detection is enhanced with an optimized detection box size range determined by the enhanced anti-perspective images.

Document US10691963 B2 describes a method of locating a vehicle, where the vehicle sensorically detects the first surrounding objects. The localization of the vehicle is then carried out by matching the data of the sensorially detected first surrounding objects with the map data of the first digital map. If this matching is not possible, a second digital map is created for the local area and the vehicle's position is determined using the second digital map.

The solution described in document CN107229063 A relates to a type of unmanned automotive navigation and positioning accuracy improvement based on GNSS and visual odometry (VO). The steps of visual odometry are as follows: It is lane detection based on monocular vision. Next comes the optimization of the odometer positioning accuracy for monocular vision. Finally, it is a correction of the positioning accuracy of the integrated GNSS navigation system and visual odometry. The presented solution improves visual odometry using the lane line, then merges it with GNSS positioning to improve the reliability of the vehicle positioning system. GNSS/VO has a very strong complementary characteristic, GNSS can obtain long-term stable positioning result, and VO can obtain short-term high-precision position data. The calculation is filtered by the difference of position measurements using two types of sensors.

The shortcomings of the solutions known in the state of the art evoked the design of another system that would provide a global location for the navigation of autonomous vehicles in both outdoor and indoor environments (outdoor + indoor) without the need for lanes. The result of this effort is the further described method of locating an autonomous vehicle and the inclusion of a visual system for locating an autonomous vehicle in the presented technical solution.

The essence of the technical solution

Deficiencies from the state of the art are eliminated by the method of locating an autonomous vehicle and the involvement of a visual system for locating an autonomous vehicle according to the presented technical solution. It is possible to estimate the position of the vehicle using various methods or sensors. The simplest implementation of rough positioning is odometry. When using typical incremental sensors to determine the position of an autonomous vehicle, the estimation of the error in the angle of rotation of the vehicle is around 1°/m, while the estimation of the error in the traveled distance is roughly within a few decimeters per ten meters. The estimation of the error in the turn and the traveled path of the vehicle can be visualized/realized in the global map. by the error estimation ellipse. This method assumes a known approximate global position of the vehicle, which can be determined by various other procedures that are not the essence of this method. The error estimation ellipse determines the distribution of the measured points (positions) from the actual point (position) of the vehicle.

The minimum error estimation ellipse and the approximate position of the vehicle can be determined in several ways known in the art. If a GNSS signal is available, in a sufficiently long time (e.g. 1 to 2 minutes) it is possible to determine the position by statistical evaluation of GNSS signals and using the RTK method (Real Time Kinematic, which is provided as a service - SK POS) and thus obtain higher accuracy of vehicle position or minimum error estimation ellipse. Another option is to obtain an exact position (or an ellipse with a minimum dimension) by manually entering a known position by the operator or automatically, e.g. when recharging the vehicle in a docking station with a known global position. Another possibility is the use of other sensor systems that enable global localization of the vehicle, e.g. visual triangulation, unique identifier recognition, WiFi trilateration, UWB localization, etc.

The proposed method to increase the localization accuracy uses the road identifier (R-ID), which is recognized by the visual sensor system. The road identifier has a graphic form known in advance, preferably e.g. a triangle, which can be depicted either directly on the road or applied graphically to another material (e.g. paper, plastic or metal plate, etc.) and place this material on the road. The graphic form is preferably chosen so that it is possible to determine the position of the road identifier relative to the vehicle. The position of each road identifier is known on the global map. For each road identifier, its associated neighborhood is defined. The size of the ellipse of the estimation of the vehicle localization error determined according to the proposed method is minimized by a sufficient density or by the distance of individual R-IDs on the road. If the R-ID is detected using a visual sensory system, it is evaluated in which neighborhood of the R-ID this error estimation ellipse falls. Based on the identification of a given R-ID, it is possible to determine the precise position of the vehicle and the azimuth (thus, the error estimation ellipse is minimized at this point), since each R-ID has a precisely determined position and rotation in the virtual global map stored in the control computer. The essence of the solution when measuring the position consists in periodic imaging of the camera, which, if it captures the R-ID, uses several algorithms to recognize the image of the R-ID road identifier and determine its position in the field of view of the camera and

SK 127-2022 U1 then calculates its relative position to the vehicle. Since each R-ID has a known global position, the global position of the vehicle is obtained using the relative position. When locating the R-ID, two coordinates are primarily evaluated: in the direction of the vehicle axis (length x) and perpendicular to the direction of the vehicle axis (width y). Image processing involves two basic phases: the first phase is mark recognition/identification preferably using a convolutional neural network, a cascade classifier, and the like. The output of recognized objects from the camera is the position of the recognized R-ID. In the second phase, using the R-ID coordinates (in pixels), the calculation of the relative R-ID coordinates takes place in the plane of the track and in the millimeter scale.

The essence of the involvement of a visual system for the localization of an autonomous vehicle using the method of localization of an autonomous vehicle consists in the fact that it consists of a visual sensing device - a camera connected to a computing device - a control computer, an algorithm - image processing software and an R-ID road identifier placed on the track in the form of a specific 2D image fixed or drawn on the road. The visual sensing device is fixed at the front of the vehicle at a defined height and with an inclination so as to sense the road in front of the vehicle. In this position, the visual sensing device must be calibrated. Calibration consists in placing a calibration pattern (usually a black-and-white checkerboard) in the field of view of the visual sensing device in the plane of the vehicle, photographing this pattern and comparing the image with the real dimensions of the checkerboard. After placing the road identifier R-ID on the track, calibration of each R-ID is necessary, which consists in measuring the absolute, global position of the R-ID, e.g. using a geodetic instrument or a precise GNSS receiver and statistical evaluation. The global position of each R-ID is recorded in the database and in the environment map in the control computer. Furthermore, the GNSS sensor system for determining the approximate position of the vehicle and the odometry sensor system for determining the estimation of the ellipse of the vehicle position error are connected to the control computer via a data connection. The function of the visual sensing device in cooperation with other localization systems (e.g.: GNSS, odometry, IMU,...) when driving autonomous vehicles are complementary to each other and at the same time redundant in order to increase reliability and ultimately ensure functional safety. GNSS provides absolute global position, odometry and IMU relative position. The proposed method of visual localization provides a global position in a physically independent (from GNSS) way. It achieves a high level of reliability by the fact that R-IDs are advantageously placed according to the user's needs directly in the driving path of vehicles. After capturing the R-ID identifier, it is certain that the vehicle is located in a suitable place in the space.

In order for the visual system to locate an autonomous vehicle to work properly, the conditions for selecting the individual components of this visual system must be met. When choosing a visual sensing device - here preferably a camera, it is necessary to take into account the number of pixels, but mainly the number of frames per second so that at the highest speed of the vehicle in the field of vision, at least one R-ID road identifier is captured and scanned. Image processing speed increases with fewer pixels, but resolution decreases. Image processing speed increases if the minimum permissible distance between two R-ID road identifiers is ensured so that only one R-ID road identifier is in the field of view of the camera at the same time. For example, the minimum distance will preferably be 2 m between two R-ID road identifiers. With the distance chosen in this way, the size of the ellipse of the localization error estimate determined according to the proposed method will be optimal.

The camera is connected to a computing device - a control computer for the purpose of object recognition. A resolution of 640 x 480 pixels at a scanning speed of 30 images per second is advantageously used for recognition. This resolution represents an advantageous compromise between the camera's ability to capture detail and the size of the captured image or as fast as it is possible to process this image for the needs of the technical solution described here. The camera recognizes a simple R-ID road identifier. The advantages of the method of increasing the accuracy of the localization of the autonomous vehicle and the involvement of the visual system for the localization of the autonomous vehicle according to the technical solution are apparent from its external effects. The effects consist in the use of a monocular camera on the vehicle to capture the position of the road identifier R-ID on the track in front of the vehicle in order to evaluate the relative coordinates of the R-ID to the vehicle. The R-ID has a pre-known, calibrated position in global coordinates on a global map stored in the control computer. Using the camera and image processing, the exact global coordinates of the vehicle are obtained. This original sensor system provides a global position for the navigation of autonomous vehicles in both outdoor and indoor environments (outdoor + indoor). Unlike GNSS (GPS, Galileo,...) it measures the global position even in the "satellite shadow of buildings, shelters or directly in buildings. Lanes are not required for navigation and autonomous driving of vehicles using the method according to the presented technical solution. The proposed localization method provides a global position in a physically independent way from GNSS. A high level of reliability is achieved by the fact that R-IDs are placed according to the user's needs directly in the driving path of the vehicles. After capturing the R-ID identifier, it is certain and reliable that the vehicle is located in the appropriate place in the space.

SK 127-2022 U1

Overview of images on drawings

The method of locating the autonomous vehicle and the involvement of the visual system for locating the autonomous vehicle according to the technical solution will be further explained in the drawings. In fig. 1 shows a flowchart of the sequence of the method. In fig. 2 shows the recognition of the R-ID road identifier and the overlay of the R-ID surroundings with an ellipse of the vehicle position error estimation. In fig. 3 shows one possible form of road identifier R-ID. In fig. 4 shows the recognition of the road identifier R-ID and the determination of the relative position of the vehicle with respect to the R-ID. In fig. 5 shows an illustration of the connection of a visual system for the localization of an autonomous vehicle.

Implementation examples

It is understood that individual embodiments of the technical solution are given for illustration and not as limitations of the solutions. Those skilled in the art will find or be able to find, using no more than routine experimentation, many equivalents to specific embodiments of the technical solution. Even such equivalents will fall within the scope of protection claims.

Example 1

In this example of a preferred embodiment, the method of locating an autonomous vehicle is described according to the technical solution as illustrated in fig. 1 to 4. It is based on the situation when the approximate position (6) of the vehicle is determined by the GNSS sensor system 4 with a known error of the vehicle position. Further, by fusing data from the GNSS sensor system 4 for determining the approximate position of the vehicle and the odometry sensor system 5 for determining the estimate of the vehicle position error ellipse, an estimate of the vehicle position error ellipse 7 is calculated. The approximate position of the vehicle and the minimum error estimation ellipse can be advantageously determined in several ways. If a GNSS signal is available, in a sufficiently long time (e.g. 1 to 2 minutes) it is possible to determine the position by statistical evaluation of GNSS signals and using the RTK method (Real Time Kinematic, which is provided as a SK POS service) and thus obtain a higher the accuracy of the vehicle's position or minimum error estimation ellipse. Another advantageous option is to obtain an exact position (or an ellipse with a minimum dimension) by manually entering a known position by the operator or automatically, e.g. when recharging the vehicle in a docking station with a known global position. Another advantageous option is the use of other sensor systems that enable global localization of the vehicle, e.g. visual triangulation, unique identifier recognition, WiFi trilateration, UWB localization, etc.

At the same time, the presence of the R-ID road identifier 2 in the visual field 8 of the visual sensing device 1 is detected by the visual sensing device 1 with the determined image sampling 16. The recognition 17 of the R-ID follows. If the road identifier 2 R-ID is recognized, based on the intersection of the ellipse estimate of the vehicle position error with the surroundings 9 of the R-ID in the virtual map 10 and the R-ID database, the selection algorithm 11 determines which specific road identifier has been recognized. At the same time, calculation 18 determines the relative position 12 of the vehicle with respect to the recognized R-ID.

By using the known position 14 of the specific recognized road identifier 2 R-ID and the relative position 12 of the vehicle with respect to this R-ID, the conversion and sum algorithm 15 calculates the precise position 13 of the vehicle with increased accuracy or estimation of the vehicle position error ellipse. The procedure of the method is subsequently advantageously repeated for further recognitions of R-ID road identifiers, for example during the movement of the vehicle, as described. In the event that the estimate of the error ellipse is larger than the neighborhood of the R-ID in the virtual map, the approximate position of the vehicle is determined by the above-mentioned procedure again, or determined by another method known in the state of the art. After re-determining the approximate position of the vehicle, the procedure of the method is carried out as already described.

Example 2

In this example of a specific embodiment, the connection of a visual system for the localization of an autonomous vehicle according to the technical solution shown in fig. 5, in the manner described in example 1. It consists of a visual sensing device 1 with an algorithm for recognizing the road identifier 2 R-ID, which has a visual range of the road identifier 2 R-ID located on the road. One possible realization of the road identifier 2 R-ID in the shape of a triangle is shown in fig. 3. The visual sensing device 1 is connected to the control computer 3 by a data connection. The GNSS sensor system 4 for determining the approximate position of the vehicle and the odometry sensor system 5 for determining the estimation of the ellipse of the vehicle position error are connected to the control computer 3 by a data connection.

SK 127-2022 U1

The FSCAM_CU135 camera was used for the visual capture device 1 with a buffer memory, which allows fast and cheap processing of images, which ensures the storage and processing of each captured image. This camera has a unique built-in leveling mechanism. Effective separation of camera image capture from USB communication enables high speed, flexibility and stability during transfer without loss of 5 frames. The camera has a 13 MP sensor and connects to the device using a USB interface. The sensor has a fixed focus lens. The printed circuit board has an advanced multiple buffer memory for higher reliability. It has 2Gb DDR3 SDRAM to save and load the entire frame without data loss even at very high speeds.

Industrial applicability

The method of locating an autonomous vehicle and connecting a visual system for increased accuracy of locating an autonomous vehicle using a camera system according to the technical solution is usable for all 15 autonomous vehicles and transport systems that need automatic navigation and precise control along a designated path on roads or service areas such as logistics centers , production areas, recreation areas, airports, etc. or in buildings such as production halls, warehouses, hangars, etc.

SK 127-2022 U1

List of visual sensing device related tags

R-ID control computer sensor system GNSS sensor system for determining vehicle position error ellipse estimation approximate vehicle position vehicle position error estimation ellipse field of vision of the visual system surroundings R-ID virtual map and database R-ID selection R-ID relative position of the vehicle with respect to R -ID precise position of vehicle known position R-ID conversion and sum image recognition R-ID position calculation

Claims (3)

1. A method of locating an autonomous vehicle, characterized in that it includes the steps: determining the approximate position (6) of the vehicle using the GNSS sensor system (4); calculating the ellipse estimate (7) of the vehicle position error by fusing data from the sensor system (4) GNSS to determine the approximate position (6) of the vehicle and the sensor system (5) odometry to determine the ellipse estimate (7) of the vehicle position error; visual scanning of the road identifier (2) R-ID on the lane in front of the vehicle in the field of view of the visual device; image processing (16) by a sampling algorithm; recognition (17) to recognize the road identifier (2) R-ID; determination, by the selection algorithm (11), of a specific recognized road identifier (2) R-ID with its exact position and at the same time the relative position (12) of the vehicle with respect to the recognized road identifier (2) RID based on the intersection of the ellipse estimate (7) of the vehicle position error with the surroundings (9) of this R-ID in the virtual map (10) and the R-ID database; calculating the relative coordinates of a specific recognized road indicator (2) R-ID in the direction of the vehicle axis x and perpendicular to the vehicle axis y in millimeter scale, and the relative position (12) of the vehicle with respect to this R-ID; calculation of the precise position (13) of the vehicle, by the conversion algorithm and the sum (15), from the known position (14) of the specific recognized road identifier (2) R-ID and from the relative position (12) of the vehicle with respect to this R-ID. 2. The method of locating an autonomous vehicle according to claim 1, characterized in that the calculation of the relative coordinates of a specific recognized road indicator (2) R-ID is carried out using the coordinates of this R-ID in pixels in the field of view of the visual device. 3. A system for locating an autonomous vehicle, for carrying out a method of locating an autonomous vehicle according to any of the preceding claims, characterized in that it consists of a visual sensing device (1) with an image sampling algorithm for recognizing a road identifier (2) R -ID, , while the visual sensing device (1) is connected to the control computer (3) by a data link, while the sensor system (4) for determining the approximate position of the vehicle and the sensor system (5) for determination of the ellipse estimate of the vehicle position error.