KR20210037893A - Vehicle navigation method based on vision sensor - Google Patents

Abstract

The present invention relates to a vehicle navigation method based on a vision sensor. According to an embodiment of the present invention, the vehicle navigation method based on the vision sensor comprises: a step of acquiring a pair of continuous images with differences in time outside the vehicle; a step of using a modified normalized phase correlation and matching the pair of continuous images; a step of calculating a pixel displacement between the pair of continuous images based on the results of matching; and a step of calculating the latitude and the longitude based on the pixel displacement and renewing the current position of the vehicle. The step of acquiring the continuous images includes: a step of using the vision sensor mounted on the vehicle and acquiring a pair of continuous images; a step of disassembling each of the pair of continuous images into a plurality of quadtree image patches by using the quadtree disassembly method for selecting a fixed boundary area in an image with less distorted substances; and a step of decimating the quadtree image patches for each of the pair of continuous images. The step of calculating the pixel displacement between the pair of continuous images includes a step of calculating a pixel displacement between the decimated image patches of each of the pair of continuous images. The present invention aims to provide a vehicle navigation method based on the vision sensor, which is able to improve the efficiency and accuracy of the image matching process.

Description

Vehicle navigation method based on vision sensor{VEHICLE NAVIGATION METHOD BASED ON VISION SENSOR}

The present invention relates to a vehicle navigation method based on a vision sensor.

The concept of a smart city is based on the sustainable development and management of urban assets and possessions using information and communication technology (ICT) along with the Internet of Things (IoT). Smart City aims to provide technology-based integrated solutions to improve the quality of life of individuals. In the development of smart cities, numerous challenges such as technological ignorance, lack of infrastructure, ecological sustainability with socio-economic modernization, participation in public governance for improving public services, prediction of solutions to social and environmental problems, collective policy and decision-making have to be solved. do. The use of technology to make life easier and safer in urban smart environments can be fundamental in all other aspects of life related to the management of city assets such as educational institutions, public libraries, and health centers. Socio-economic development in smart cities is achieved through effective control and efficient use of resources by integrating technology into all areas of their lives.

In the digital smart city, the change of the existing urban environment has become a reality, which aims to transform daily life activities into automated processes in order to facilitate human effort and shorten the effort time. Vision-based sensors are commonly used to monitor cities, and cities collect vast amounts of different data and store them for further computer vision processing.

On the other hand, vehicle navigation is an important issue in cities, since there is no adequate navigation infrastructure in a city with a large population, and it is more appropriate to use autonomous navigation in urban environments and smart cities in particular. The main task of autonomous navigation systems for automobiles or mobile robots is the error-free self-positioning calculations. Self-positioning measurement refers to the ability of a mobile robot to determine its active position and orientation relative to the inertial frame of reference. The most common technology used for positioning is based on the Global Positioning System (GPS). GPS-based positioning calculates the absolute position and position of an object on the Earth's surface in terms of longitude, latitude and altitude based on satellite signals. One of the drawbacks of GPS-based positioning is the high cost of installing landmarks in the robot's operating area. Likewise, GPS access may not be possible or denied in enclosed environments such as large forests, mountains, tunnels and tall buildings.

An alternative to GPS-based technology is to use an Inertial Navigation System (INS). Inertial navigation is an independent navigation technology that uses data from an accelerometer and gyroscope to track the body's position and orientation relative to a known starting position, velocity, and direction. In this system, on-board sensor data helps to estimate the position. The baseline of the calculation is the initial position and the estimated position is provided by the velocity, acceleration and drift parameters. The disadvantage of this type of technology is that it cannot accurately calculate the exact position due to cumulative errors due to wheel slip or uneven areas.

Many researchers have successfully used vision-based navigation techniques, especially in off-road mobile robots, such as space missions, underwater navigation and exploration. Visual odometry (VO) is an increasing online approximation of robot motion in an image sequence using an on-board camera. This idea has improved in terms of location accuracy and robustness with respect to the complexity of the environment.

In Wong, Deguchi, Ide, and Murase (2014), the authors used SURF-based features with dynamic time distortion to achieve vehicle position measurements in a traffic environment. SURF-based features have different characteristics, but if there are too many unique features for each input image, the system is inefficient in terms of calculation and accuracy. In contrast, the approach proposed in the present invention uses efficient image matching using a Gram polynomial basis function, which is efficient by reducing computational complexity in the extraction of features from the input image.

In Xie, Gu, and Kamijo (2017), the authors proposed a lane-level positioning method specifically designed in a bridge or enclosed environment using a stereo vision system with an inertial sensor. The downside of this method is that real-time stereo vision systems have poor accuracy when determining depth information, resulting in poor positioning. In addition, since stereo vision-based systems rely heavily on low-level pixel details of an image such as lighting and contrast, they are susceptible to inaccuracies, making depth mapping decisions poor.

In Vivacqua, Bertozzi, Cerri, Martins and Vassallo (2018), the authors proposed a method of constructing a registry by detecting near-vision lane markers along with existing odometer information. One of the limitations of these systems is the use of existing information in a map of the environment to provide efficient location measurements. Compared to the above-described approach, the technique proposed in the present invention is more suitable under any conditions since it does not require prior information about the environment.

Image registration is the process of correlating and geometrically aligning two images of the same area with each other. Due to its wide range of applications, image registration has been an active research field for many years. In classical image-based matching, the spectral domain function is used to reconstruct the signal using frequency transformation. When the intensity of the data (textured area) or discontinuities (edges or edges) are included in the data, the performance of these systems is degraded and reconstruction is not performed properly.

Therefore, in the navigation based on the vision sensor, there is a need for a method capable of efficiently and accurately providing the position of the vehicle by improving the efficiency and accuracy of the image matching process even in an area with high intensity change or discontinuity.

KR 10-1096113 B1

The problem to be solved by the present invention is to improve the efficiency and accuracy of the image matching process even in areas with high intensity changes or discontinuities, and thus, a vision sensor capable of accurately providing the location of a vehicle based on a vision sensor even in an environment where GPS service is denied. It is to provide a vehicle navigation method based on.

A vehicle navigation method based on a vision sensor according to an embodiment of the present invention for solving the above problem,

Acquiring a temporally different continuous image of a pair of outside the vehicle;

Matching the pair of consecutive images using the corrected normalized phase correlation;

Calculating a pixel displacement between the pair of consecutive images based on the matching result; And

Computing latitude and longitude based on the pixel displacement to update the current location of the vehicle,

The step of obtaining the continuous image,

Acquiring a pair of continuous images using a vision sensor mounted on the vehicle;

Decomposing each of the pair of consecutive images into a plurality of quad-tree image patches using a quad-tree decomposition method in order to select a fixed boundary area within an image having less distortion; And

Decimating quad-tree image patches for each of the pair of consecutive images,

Calculating the pixel displacement between the pair of successive images includes calculating the pixel displacement between decimated image patches of each of the pair of successive images.

In the vehicle navigation method based on a vision sensor according to an embodiment of the present invention, the decimating each image patch (D) includes obtaining a _{spectrum S pg based on Equation 7,}

[Equation 7]

F _x and F _y are matrices containing Fourier basis functions for the x and y directions, respectively, T denotes the transpose operator,

D _pg is obtained based on Equation 5,

[Equation 5]

The columns of G _x and G _y may include Gram basis functions for the x and y directions, respectively.

)는, 수학식 6에 의해 주어지고,In the vehicle navigation method based on a vision sensor according to an embodiment of the present invention, the Gram basis function (

) Is given by Equation 6,

[Equation 6]

,

,

이며,

,

,

이고,

이며, n<m이고, 상기 수학식 6에서 x는 [-1,1] 범위 내의 값이며, n은 1 이상인 소정의 정수일 수 있다.From above

Is,

,

,

ego,

Is, n<m, in Equation 6, x is a value within the range of [-1,1], and n may be a predetermined integer equal to or greater than 1.

In addition, in the vehicle navigation method based on the vision sensor according to an embodiment of the present invention, before the step of matching the pair of consecutive images using the modified normalized phase correlation,

In order to provide an approximation value to a portion of the image patches having no features, each of the image patches (D) is reconstructed based on [Equation 10] to obtain a spectrum (S) of the reconstructed image patch. can do.

[Equation 10]

.

.

In addition, in the vehicle navigation method based on a vision sensor according to an embodiment of the present invention, the step of matching the pair of consecutive images using the modified normalized phase correlation includes the surface P( and obtaining n),

[Equation 12]

IFFT represents the inverse fast Fourier transform, M(w) and R(w) are 2D Fourier spectra of modified image patches for the pair of consecutive images, respectively,

The 2D Fourier spectrum of the modified image patches is obtained by projecting the 2D Fourier spectrum of the image patches onto an orthogonal complement of a Gram polynomial of order 1,

The pixel displacement can be obtained by the position of the peak at the surface P(n).

In addition, in the vehicle navigation method based on the vision sensor according to an embodiment of the present invention, after the step of acquiring a pair of continuous images outside the vehicle,

It may further include the step of obtaining a normalized image (N(x, y)) by applying Equation 1 to the pair of consecutive images.

[Equation 1]

I(x, y) is the input image, μI is the local mean of I(x, y), and σI is the local standard deviation.

According to the vehicle navigation method based on the vision sensor according to an embodiment of the present invention, the efficiency and accuracy of the image matching process are improved even in areas with high intensity change or discontinuity, and the vehicle is based on the vision sensor even in an environment where GPS service is denied. You can provide the location exactly.

1 is a schematic flowchart of a vehicle navigation method based on a vision sensor according to an embodiment of the present invention.
2 is a detailed flowchart of a vehicle navigation method based on a vision sensor according to an embodiment of the present invention.
3 is a block diagram of an apparatus to which a vehicle navigation method based on a vision sensor according to an embodiment of the present invention can be applied.
4 is a view showing the camera position and height from the ground level d.
5 shows latitude and longitude calculations in four consecutive images.
6(a) to 6(d) show the layer 0 correlation peak, FIGS. 6(e) to 6(h) show the layer 1 correlation peak, and FIGS. 6(i) to 6(l) show the layer 1 sub Figure 1 shows the correlation peak.
7(a) to 6(d) show layer 2 correlation peaks, and FIGS. 7(e) to 6(h) show layer 3 correlation peaks.
8 is a diagram showing a distortion map.
Fig. 9(a) is a 3D plot showing a pixel shift displayed in an image, and Fig. 9(b) is a 3D plot showing a peak value due to a corrected normalized phase correlation.
FIG. 10(a) is a diagram showing a trajectory for a test drive for 1200 images and an evaluation of GPS and VO, and FIG. 10(b) is a diagram showing an error graph.

Objects, specific advantages and novel features of the present invention will become more apparent from the following detailed description and preferred embodiments associated with the accompanying drawings.

Prior to this, terms or words used in the present specification and claims should not be interpreted in a conventional and dictionary meaning, and the inventor can appropriately define the concept of the term in order to describe his own invention in the best way. It should be interpreted as a meaning and concept consistent with the technical idea of the present invention based on the principles that exist.

In adding reference numerals to elements of each drawing in the present specification, it should be noted that, even though they are indicated on different drawings, only the same elements are to have the same number as possible.

In addition, terms such as "first", "second", "one side", and "the other side" are used to distinguish one component from other components, and the component is limited by the terms. It is not.

Hereinafter, in describing the present invention, detailed descriptions of related known technologies that may unnecessarily obscure the subject matter of the present invention will be omitted.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

In the present invention, it is intended to explore whether and how to search for a vehicle using a cost-effective means (vision-based sensor) in a smart city without using a calibrated sensor and a global positioning system (GPS). Vehicle positioning and navigation require on-board calibration sensors and a stable GPS link. In urban environments, these sensors do not work well in indoor environments (tunnels) or in crowded and congested areas and severe weather conditions. The most effective technology used for vision-based navigation depends on image registration. The challenges of successful and effective alignment are as follows. Sufficient lighting in the environment, the dominance of the static scene over moving objects, the apparent movement between successive frames, and a high texture ratio that allows for the necessary overlap of the scene. The present invention proposes a new approach to vehicle navigation based on a vision sensor using a modified normalized phase correlation. In the proposed approach, the distinction between textured and less textured surfaces is based on identification of the corresponding features. In this regard, the Gram polynomial basis function is used to eliminate the Gibbs error problem created due to peaks in the registration process. Similarly, an entropy-based tensor approximation is used to remove outliers for strong image matching. Experiments conducted in real time during the test drive show excellent results with respect to the estimated location accuracy compared to GPS calculation data.

The approach proposed in the present invention uses a Gram polynomial basis function to provide the efficiency and accuracy of the image matching process even in areas with high intensity variations or discontinuities.

In the present invention, a vision-based sensor utilizing image matching technology is used to solve the vehicle navigation problem in an environment (both indoors and outdoors) where there is no GPS function or GPS service is denied. In this approach, features are extracted to estimate and identify transformations between data representing different locations. The data is further processed to estimate the vehicle's position in relation to the vehicle's latitude and longitude. The main contributions of the present invention can be summarized as follows.

-Providing an autonomous driving system capable of navigating vehicles using vision-based sensors in urban environments using cost-effective means.

-Provides a vehicle navigation solution through image matching using normalized phase correlation modified based on Fast Fourier Transform (FFT). Likewise, an innovative step in the approach proposed in the present invention is to use a Gram polynomial basis function to improve image matching through phase correlation.

-Considering sub-pixel matching, it creates a system that can be used for low-resolution images, reduces the processing time of vehicle navigation, and provides accurate results of vision-based navigation.

Proposed Modified Normalized Phase Correlation Based Approach

The most important step for automatic navigation by machine vision is to find pixel movement between two consecutive scenes. Most of the methods these days match images with a correlation method. The correlation-based method is widely used in both domains, spatial and spectral. Two successive images in the spatial domain are correlated by points, lines, corners and edges between the corners, which results in false peaks or many peaks or corresponding points where corresponding points occur. Also, finding important edges, corners, and points in a blur image is difficult and problematic.

The spectral domain correlation incorporates the complete processing of the image in the frequency domain, and in the case of noisy data, phase correlation is performed to provide robustness. The phase correlation based matching allows a shift between images with sub-pixel accuracy from the location of the correlation peak. Also, a blurred image without an appropriate ideal boundary can be correlated. Fourier-based functions can be used to efficiently process and reconstruct the analysis of data or signals with smooth properties (no discontinuities or gradients). In general, the image matching process uses a Fourier basis function that gives fairly satisfactory results in a consistent and uniform image, but in situations where the data contains high entropy, variance, irregularities, and discontinuities, the Fourier basis function is the reconstruction of the original signal or data. Performance is degraded. Thus, the Fourier basis is not efficient for representing discontinuities or sharp edges that require a large number of Fourier components to represent discontinuities. This discontinuity point at which the Fourier sum overshoot is called Gibbs Phenomenon.

1 is a schematic flowchart of a vehicle navigation method based on a vision sensor according to an embodiment of the present invention, and FIG. 3 is a block diagram of an apparatus to which a vehicle navigation method based on a vision sensor according to an embodiment of the present invention can be applied. to be.

3, the vision sensor 300 is a sensor for acquiring an image of the outside of the vehicle, such as a camera, and the memory 304 is a program code for a vehicle navigation method based on a vision sensor according to an embodiment of the present invention. It is stored in the form, and the control unit 302 executes a vehicle navigation method based on a vision sensor according to an embodiment of the present invention stored in the memory 304, and the vision sensor 300, the display 306, and the memory 304 ), and the display 306 displays the position of the vehicle output from the control unit 302.

Referring to FIGS. 1 and 3, in step S100, the control unit 302 acquires a continuous image of a pair of outside the vehicle with temporal difference using the vision sensor 300.

In step S102, the control unit 302 matches the pair of consecutive images using the corrected normalized phase correlation, and in step S104, the control unit 302 matches the pair of consecutive images based on the matching result. Compute the pixel displacement between images.

In step S106, the controller 302 updates the current position of the vehicle by calculating latitude and longitude based on the pixel displacement.

A detailed block diagram of a vehicle navigation method based on a vision sensor according to an embodiment of the present invention is shown in FIG. 2, and image matching in the present invention is used to extract a global position for determining an object position. The entire procedure of a vehicle navigation method based on a vision sensor according to an embodiment of the present invention can be summarized as the following steps.

-A pair of consecutive images are acquired using a vision sensor (camera) mounted on a ground vehicle, and an image taken at time (t-1) is selected.

-Match the image at time (t-1) and the image at time t, which is the current image, using the corrected normalized phase correlation.

-Estimate the pixel displacement between the template and the maximum correlation point.

-Calculate the distance in meters through the pixel-level image resolution.

-Convert the distance to ΔLat and ΔLong and use these values to update the vehicle position.

-Repeat the above process for each successive image.

The basic idea of a vehicle navigation method based on a vision sensor according to an embodiment of the present invention is to find a position change of two consecutive images using a modified normalized phase correlation. Changes in location are converted to the corresponding latitude and longitude.

In step S200, the controller 302 captures a sequence of images I(x, y) using the vision sensor 300, that is, a pair of temporally different continuous images outside the vehicle (images (t- 1), an image (t)) is acquired. Here, the first image is considered the starting point of the method. The next image in the sequence is considered a shifted image.

Meanwhile, in order to correct uneven lighting or shading artifacts, the control unit 302 locally normalizes the two images to equalize the local average and variance of the images. The resulting normalized image N(x, y) is extracted using the following equation.

Where I(x, y) is the input image, μI is the local mean of I(x, y) and σI is the local standard deviation.

Due to the use of conventional image capturing devices, the acquired images have different levels of distortion at the boundaries of each image. Normalization prevents irregular lighting, but still the distortion component negatively affects the underlying process. To solve this problem, a fixed boundary area inside the image with less distortion is selected. These boundaries are created by decomposing the image using a quad-tree decomposition method.

Referring to FIG. 2, the controller 302 uses a quad-tree decomposition method for a plurality of quad-tree images in order to select a fixed boundary area inside an image with a small distortion component. It is decomposed into patches (step S202, step S206). The image is decomposed into equally sized patches by dividing the image into 4, 16, and 64 blocks, respectively, using a quad-tree decomposition method.

The control unit 302 decimates the quad-tree image patches for each of the pair of consecutive images (steps S204 and S208).

Decimation of each patch based on the Gibbs polynomial eliminates the Gibbs error problem. The methods widely used for decimation are interpolation, smoothing, compression, enhancement, and feature detection. While the image is represented as a set of discrete values with evenly spaced nodes, the coefficients of this polynomial represent a linear transformation of the image pixels. For transforms, the computational requirements are much broader, but you can use two-dimensional coefficients to accurately represent the desired matrix in polynomials. Decimating the polynomial coefficients using interpolation and least squares approximation reduces the computational cost.

To compute the expansion of a vector, it is more appropriate to have a normal orthogonal basis for a given vector space. Thus, in the case of a non-normal orthogonal basis, it is necessary to obtain a normal orthogonal basis, and this transformation is provided by Gram-Schmidt orthogonalization. Gram-Schmidts' orthogonalization process is a recursive technique in which vectors are transformed into regular orthogonal vectors. Once transformed, the difference between the original vector and its projection to the space spanned by the first normal orthogonal vector is normalized to obtain a vector in the normal orthogonal collection.

A vehicle navigation method based on a vision sensor according to an embodiment of the present invention presents a Gram polynomial transformation and shows that the Gram polynomial transformation can be used for image decimation. Decimation helps reduce execution time for each remaining step.

Gram polynomial basis function

Decimation is applied to the image at various scales for a non-rigid matching technique, even coarser and finer. Since the images change non-linearly and the image changes are not similar to each other, the stiffness matching method cannot solve the problem accurately. Thus, using hierarchical segmentation of the image, the cut and divide rules are applied and the registration is done in a global to local way. On the other hand, Gram polynomial decimation is applied to the image at variable speed at all levels.

In summary, it has a maximum decimation ratio at the top or zero level, where the size of the image is the maximum (original image) at the higher level. As the quad-tree decomposition process proceeds further into the sub-image hierarchy, the size of the image decreases as it reaches the last level, reducing the decimation rate. There are two main sources of error through decimation of images that reduce resolution. First, a Gibbs error that cannot describe the characteristics of the data occurs due to the basis function, and the second cause of the error is aliasing. In general, aliasing is only considered for periodic basis functions, but the same problem exists for polynomial basis functions. This problem is further complicated by the presence of Gaussian noise in the image data.

의 열(column)을 형성하는 k개의 이산 정규 직교 기저 함수

가 고려된다.Considering the signal Y with n data points, to find its spectrum S, the matrix

K discrete normal orthogonal basis functions forming a column of

Is considered.

Here, T stands for the transpose operator. The inverse transformation is as follows.

는 오류 없는 불연속 신호를 나타낸다. 그러나 수학식 4에 주어진 바와 같이, 깁스 오차는 감소된 세트의 기저 함수에 의해 또는 가중에 의해 신호를 닮아서 정확한 기저와 특히 관련되지는 않는다.When the basis is complete, i.e. when some set of basis functions B are complete,

Represents an error-free discontinuous signal. However, as given in Equation 4, the Gibbs error resembles a signal by weight or by a reduced set of basis functions and thus is not particularly relevant to the exact basis.

Here, H denotes a weight vector added to each spectral component.

The best set of basis functions for image decimation in 2D image registration depends on the characteristics of the image being considered. The Fourier basis is suitable for the case of the periodic structure of the image, while the Gram basis is suitable for an essentially geometric image. Thus, when calculating the 2D Fourier spectrum, the image patch D is projected onto the orthogonal complement of the _{truncated Gram polynomial D pg set as given in the following equation:}

Where _{the columns of G x} and G _y contain Gram basis functions for the x and y directions, respectively. This significantly removes the sub-harmonic component from the patch and consequently reduces the associated cast error.

)는 수학식 6에 의해 주어진다.The Gram basis function (

) Is given by Equation 6.

이며,

,

,

이고,

이며, n<m이고, 상기 그람 기저 함수에 대한 수학식에서 x는 [-1,1] 범위 내의 값이며, n은 1 이상인 소정의 정수이다.From above

Is,

,

,

ego,

And n<m, in the equation for the Gram basis function, x is a value within the range [-1,1], and n is a predetermined integer equal to or greater than 1.

In general, the spectrum S _pg is calculated as follows.

Where F _x and F _y are matrices containing Fourier basis functions for the x and y directions, respectively. However, in the Gram polynomial basis function, the spectrum can be calculated as follows using the modified basis function given by _{B x} and B _y:

The signal-to-noise ratio is better than the fit calculation.

Entropy weighted tensor polynomial regularization

Successive patches of the reference image and the shifted image are matched using the corrected normalized phase correlation, and the control unit 302 is characterized in the image patches before matching the successive patches of the reference image and the shifted image. Each of the image patches D is reconstructed using entropy weighted tensor polynomial normalization to provide an approximation to the missing part.

That is, in order to improve the matching process (that is, regions with fewer features are matched), the control unit 302 applies an entropy-based tensor approximation to a neighboring feature with a weight and then applies it to strong matching.

In a vehicle navigation method based on a vision sensor according to an embodiment of the present invention, a feature/texture/information available in an image is used as a weighting function. There may be fewer features available in some areas of the image, and phase correlation will be required to accurately match the image. As a result, entropy plays a role in this regard by providing an approximation assigned to a non-characteristic part.

의 차이를 나타내는 오차 E에 의해 주어진다.It cannot be guaranteed that the coordinates of the patches are in the grid corresponding to the elastic deformation. Normalizing the coordinates to a grid that matches the solution of the partial differential equation for elastic deformation can be achieved by approximating by the tensor product of the global polynomial basis functions. The function for this tensor product approximation is the original image patch D and the reconstructed image patch

Is given by the error E representing the difference in

Where S is the spectrum of the reconstructed patch and F is the Frobenius norm. The above equation depends on the value of S. That is, the smaller the value of S, the smaller the error E will be. The goal is to determine the entry value of matrix S that minimizes the function. S can be calculated as follows:

An additional issue to consider is that not all patches have the same information content. It is preferable to weight the coordinates of the patch by the information contained in the patch. In other words, patches with higher information content are more important during the least squares approximation process. Given a weighting matrix W, the error E _w for the weighted tensor product approximation can be formulated as follows.

Here, is the Hadamard product operator.

Modified Normalized Phase Correlation

In step S212, the controller 302 matches the pair of consecutive images using the corrected normalized phase correlation, and calculates a pixel displacement between the pair of consecutive images based on the matching result.

Phase correlation is used to find the shift in the image. To match two images by phase correlation, first the discrete 2D Fourier transform of the two images is calculated, then the cross-power spectrum is calculated by taking the complex conjugate of the shifted image, which undergoes further normalization. Finally, the inverse Fourier transform is used to obtain the required shift in both images.

The corrected normalized phase correlation is applied to two consecutive images to find the relative motion. The core idea of the present invention is based on the Fourier shift theory, which states that the shift of the two functions in the spatial domain is transformed into a linear phase difference in the Fourier spectral domain. The phase correlation method calculates the phase difference including an ideal single peak. The position of the peak represents the relative transformation between two successive images. The phase correlation method is robust to noise and image defects and is easily automated.

A modified normalized phase correlation method in a vehicle navigation method based on a vision sensor according to an embodiment of the present invention is presented to perform matching. The Fourier basis function is modified by projecting it onto the orthogonal complement of a Gram polynomial of degree 1. This eliminates the influence of intensity and intensity peaks within the patch. The mathematically normalized phase correlation is:

Where IFFT denotes the inverse fast Fourier transform, and M(w) and R(w) are the 2D Fourier spectra of the modified template image and the modified reference image, respectively. Pixel shift, i.e. pixel displacement, is given by the position of the peak at the surface P(n).

Finally, in step S214, the control unit 302 updates the current position of the vehicle by calculating latitude and longitude based on the pixel displacement.

Experiment setup and results

The vehicle is equipped with a simple digital camera in order to compare the reported position and the recorded GPS value through the vehicle navigation method based on the vision sensor according to an embodiment of the present invention. The horizontal viewing angle of the camera is 15.5°, and the acquired image size is 480×640. The image pairs provided in FIG. 5 (i.e., reference images and shifted images) resulting in an overlap of 50% or more are used for experimental verification purposes. During the experimental setup, the ground vehicle moved at a maximum speed of 30 kilometers/hour (km/h), or 8.333 meters/second (m/s). Therefore, the total distance is 200×8.33=1666.6m or 1.6km on the general road. In a vehicle navigation method based on a vision sensor according to an embodiment of the present invention, information available in an image is maintained as a weighting function.

Estimation of vehicle displacement

The displacement [dx, dy] is estimated using an image captured by the camera at an angle toward the ground as shown in FIG. 4. At the sampling moment t-1, the camera photographs the ground. Similarly, another image is taken at the next sampling instant t, the corrected normalized phase correlation is calculated using Equation 8, and the transform is calculated. The peak value provided by the corrected normalized phase correlation is used to formulate the shift between the reference image and the current image.

Pixel displacement (Δx, Δy) is calculated from a sequence of images taken with a ground-pointing camera and is used to find the Across and Along distances, respectively. This distance is converted into units of measure such as latitude and longitude to calculate the exact location of the vehicle.

Find latitude and longitude in pixel values

To find the latitude and longitude, you need to find the corrected resolution. In FIG. 4, “height” is defined as Altitude above Ground Level (AGL), and the symbol α represents the angle of view in degrees. The following variables are calculated mathematically.

Where FOV is the field of view and tan is the tangent of the angle.

Once the movements of x and y are calculated, they can be transformed to find the Across and Along distances, respectively. It is necessary to compensate for the influencing components, i.e. the x motion given by the roll and the y motion given by the pitch. Similarly, the side wise distance and the along wise distance are calculated using the following equation.

To compare the horizontal and vertical distances with GPS, you need to resize them in latitude (Lat) and longitude (long) respectively.

To do this, first, a movement matrix is calculated using a clockwise rotation matrix. Second, this shift matrix is converted to degrees using Factor, where Factor = (180/π) × Earth radius (6378137 meters). Finally, ΔLat and ΔLong are accumulated in the seed value (formerly latitude/longitude). For the first sample, the seed value is the latitude/longitude of the trajectory starting point. The starting point is determined according to the current location via GPS. The shift matrix is calculated as follows.

Four consecutive images are shown in Fig. 5, and each image, that is, Fig. 5 (b, e, h, k) is matched with Fig. 5 (a, d, g, j) which is the previous image, Pixel movement is obtained through quad-tree decomposition indicated at 5(c, f, i, l). Considering the initial point of a known position in the image of Fig. 5 (a, d, g, j), for the next images, Fig. 5 (b, e, h, k), Fig. 5 (a, d, g, The images given in j) are used to calculate the latitude and longitude distances. Each successive image, Fig. 5(b, e, hk), is used to calculate the latitude and longitude of the new image, respectively. The direction is calculated using the angle headed with the help compass and the distance (latitude and longitude) is calculated using Equations 18 and 19.

Real-time experiment results

The main goal of a vehicle navigation method based on a vision sensor according to an embodiment of the present invention is to construct a navigation system that provides an accurate location in a GPS rejection environment. To this end, the vehicle was equipped with a camera with a resolution of 480x640 to perform a daytime experiment and a real-time experiment under clear weather conditions.

A common practice used in autonomous navigation systems is to use initial location data extracted from GPS. Before starting navigation, GPS data is provided for initial positioning of the vehicle to determine the vehicle's location.

In the vehicle navigation method based on a vision sensor according to an embodiment of the present invention, only the first location is required from the point where the trip starts, and even without a GPS or INS system, the next navigation point is determined while autonomously moving to the destination along the route. In a real environment, the image captured by the vision sensor may have little information due to noise. This low information content can lead to erroneous phase correlation. In this regard, entropy plays an important role, and approximations are assigned to these parts of which there is no or limited information. During least-squares tensor approximation, images with lower information features do not contribute significantly compared to images with more information features.

In order to evaluate the robustness of a vehicle navigation method based on a vision sensor according to an embodiment of the present invention in real time, an experiment is performed in hardware, and two images are provided to the system to find movement between the two images. In the vehicle navigation method based on the vision sensor according to an embodiment of the present invention, during the experiment, the capture frame rate of the camera is reduced from 28 frames per second to 15 frames per second, thereby improving the computational complexity for frame processing. Similarly, two frames input for the correlation step are considered to determine latitude and longitude as given in the vehicle navigation method based on a vision sensor according to an embodiment of the present invention. After processing of the previous frame, a new input frame is selected for further processing. This process continues until the coverage of the entire navigation path. The calculation time for each result takes about 900 ms regardless of the number of frames. Data is collected from the camera, and pixel movement is converted into position data through pre-processing, image correlation, and post-processing. By incorporating an advanced digital signal processor (DSP), it is possible to improve the performance of a vehicle navigation method based on a vision sensor according to an embodiment of the present invention.

A vehicle navigation method based on a vision sensor according to an embodiment of the present invention takes an initial or a start position from a GPS or a last reported position. In the case of a vehicle navigation method based on a vision sensor according to an embodiment of the present invention, only the initial latitude and longitude values of the starting point are provided using a GPS device, and the vehicle navigation method based on the vision sensor according to an embodiment of the present invention is a drive While calculating the latitude and longitude values of each image. The covered distance is calculated using the pixel shift of the image. Similarly, the direction of the vehicle is found by the direction the vehicle is facing, which is also further used to compensate along and across the distance.

6 is applied to two displaced images, i.e., a reference image (Fig. 6(a, e, i)) and a shifted image (Fig. 6(b, f, j)) at different levels of quad-tree decomposition. An example of application of a vehicle navigation method based on a vision sensor according to an embodiment of the present invention is shown.

For display purposes, the peaks are the 2D plot of Fig. 6(c, g, k) and the 3D plot of Fig. 6(d, h, l) (in the 3D plot, the X-axis and Y-axis represent the pixel position and the Z-axis represents the shift transformation). Was shown in. The phase correlation method provides remarkable sharp peaks at the time of matching, as shown in Fig. 6(d, h, l), respectively.

During the experiment, the vehicle navigation method based on the vision sensor according to an embodiment of the present invention matches the image with the minimum feature (i.e., low texture patch) as in Figs. 7 (a, b) and 7 (e, f). It has been observed that it can be done. That is, layer 2 and layer 3 provide correlation peaks with minimal features. This appears remarkably due to the nature of the phase correlation technique of the vehicle navigation method based on the vision sensor according to an embodiment of the present invention when used with the tensor approximation added with entropy.

The improvement of the signal by normalization in the form of whitening is a second important characteristic, and the phase correlation is remarkably strong for images with large noise, e.g., uniform illumination deviation, general intensity offset, and calibration error. This property also makes the phase correlation suitable for image matching in different spectral bands. At each level of the quad tree, the resolution of the patch is reduced through a Savitzky-Golay approximation that is consistently combined with decimation. On the other hand, since the computational speed can be achieved using a higher degree of decimation, the decimation ratio must be chosen carefully. However, this requires sufficient local information to be maintained for the probabilistic pattern to enable global matching. Matching is performed in each layer through normalized phase correlation based on Fast Fourier Transform (FFT) modified for individual patches. Before calculating the normalized phase correlation, the patch is projected onto the orthogonal component of a low degree of Gram polynomial basis.

Autonomous navigation systems have proven to be accurate and powerful when calculating the exact pixel movement between two consecutive frames. The pixel shift between two successive images is scaled by meters and then by latitude and longitude. As a result, the main periodic part of the data is removed and the cast error is greatly reduced.

Figure 8 shows the total movement (shift) formulated as a distance map, which provides movement insight using Euclidean distances at each node of the quad tree layer in 3D. The distortion map given in Fig. 8 is the actual movement between two images, which determines the distance along and across the road by the vehicle between the two samples. The X axis represents a pixel index representing the movement of the vehicle in the transverse direction, and the Y axis represents the pixel index representing the movement of the vehicle in the longitudinal direction. The Z-axis represents the intensity of the peak value, but the higher the Z-axis value, the more accurate correlation results can be obtained. In the vehicle navigation method based on a vision sensor according to an exemplary embodiment of the present invention, a single peak is found, whereas in spatial domain correlation, more peaks are found, but finding an actual peak is a difficult process.

FIG. 9A shows the pixel movement, and FIG. 9B shows the overall peak formulated as a distance map, which provides insight into the movement at the level of each layer and its sub-images in the 3D plot.

The result calculated by the vehicle navigation method based on the vision sensor according to an embodiment of the present invention is shown by comparing with GPS using the same scale as shown in FIG. 10A. The red line represents a GPS route, and the blue line represents a visual rangefinder route by a vehicle navigation method based on a vision sensor according to an embodiment of the present invention. It can be seen that the obtained results are quite similar to the GPS results. Through extensive tests, the vehicle navigation method based on the vision sensor according to an embodiment of the present invention has proven its effectiveness in a GPS rejection environment as an alternative method. This application only requires sensor details (roll, pitch, and heading) along with the image and camera-specific parameters to provide promising results. The error diagram between the GPS and the vehicle navigation method based on the vision sensor according to an embodiment of the present invention is illustrated in FIG. 10B, and as illustrated in FIG. 10A, the trajectory that the vehicle follows is considered. When the system begins its journey, there are two types of data logs. One is location measurement using data from a GPS device, and the second is location estimation using a vehicle navigation method based on a vision sensor according to an embodiment of the present invention. In the vehicle navigation method based on a vision sensor according to an embodiment of the present invention, only the first position is taken from the GPS and then the remaining position is measured by itself.

While the GPS-based system can obtain noise during the experiment, it has been observed that the vehicle navigation method based on the vision sensor according to an embodiment of the present invention provides an accurate position value without jitter. During the experiment, an error of up to 9m is observed between the GPS and VO given in the error diagram. This error is basically caused by the GPS position jump during the test drive. Otherwise a smooth and accurate position is created with the method of the present invention during navigation. This shows that GPS and VO can be combined to obtain an integrated solution. In some cases, if GPS is not available or gives erroneous results, the VO supports the system while some location is unavailable.

In a vehicle navigation method based on a vision sensor according to an embodiment of the present invention, a method of utilizing a vision-based sensor for navigation in a complex urban environment where classical technology cannot provide an appropriate solution in a smart city is provided. A vehicle navigation method based on a vision sensor according to an embodiment of the present invention utilizes minimal resources and provides accurate results compared to a general expensive method. The method of the present invention can be said to be a featureless correlation, that is, unlike many other methods dealing with other features, only visual data (images) are used and its spectral properties are used. The vehicle navigation method based on a vision sensor according to an embodiment of the present invention provides a single peak representing the actual motion of two images to be matched instead of representing many peaks. An interesting aspect of a vehicle navigation method based on a vision sensor according to an embodiment of the present invention is the processing time using MATLAB for development, and the execution time is less than 1 second per image for image matching. The introduction of the Gram polynomial basis function eliminates the Gibbs error problem and further improves the result. Similarly, entropy-based tensor approximation can match strong image matching, i.e., images with less texture, for example desert regions. The proposed technique can also be used for low-resolution images to improve efficiency by taking into account sub-pixel matching. Thus, it can be concluded that matching in the sub-pixel domain yields better results, especially for low-resolution images.

Although the present invention has been described in detail through specific examples, this is for describing the present invention in detail, and the present invention is not limited thereto, and those of ordinary skill in the art within the technical scope of the present invention It would be clear that the transformation or improvement is possible.

All simple modifications to changes of the present invention belong to the scope of the present invention, and the specific scope of protection of the present invention will be made clear by the appended claims.

300: vision sensor
302: control unit
304: display
306: memory

Claims (6)

Acquiring a temporally different continuous image of a pair of outside the vehicle;
Matching the pair of consecutive images using the corrected normalized phase correlation;
Calculating a pixel displacement between the pair of consecutive images based on the matching result; And
Computing latitude and longitude based on the pixel displacement to update the current location of the vehicle,
The step of obtaining the continuous image,
Acquiring a pair of continuous images using a vision sensor mounted on the vehicle;
Decomposing each of the pair of consecutive images into a plurality of quad-tree image patches using a quad-tree decomposition method in order to select a fixed boundary area within an image having less distortion; And
Decimating quad-tree image patches for each of the pair of consecutive images,
The calculating of the pixel displacement between the pair of consecutive images comprises calculating a pixel displacement between decimated image patches of each of the pair of consecutive images. The method according to claim 1,
Decimating each image patch (D) includes obtaining a _{spectrum S pg based on Equation 7,}
[Equation 7]

F _x and F _y are matrices containing Fourier basis functions for the x and y directions, respectively, T denotes the transpose operator,
D _pg is obtained based on Equation 5,
[Equation 5]

A vehicle navigation method based on a vision sensor, wherein the columns of G _x and G _{y contain Gram basis functions for the x and y directions, respectively.} The method according to claim 2,
The Gram basis function (

) Is given by Equation 6,
[Equation 6]

,
From above

Is,

,

,

ego,

And n<m, in Equation 6, x is a value within the range of [-1,1], and n is a predetermined integer equal to or greater than 1, a vehicle navigation method based on a vision sensor. The method of claim 3,
Before the step of matching the pair of consecutive images using the modified normalized phase correlation,
In order to provide an approximation value to a portion of the image patches having no features, each of the image patches (D) is reconstructed based on [Equation 10] to obtain a spectrum (S) of the reconstructed image patch. The vehicle navigation method based on the vision sensor.
[Equation 10]

. The method of claim 4,
The step of matching the pair of consecutive images using the modified normalized phase correlation includes obtaining a surface P(n) based on Equation 12,
[Equation 12]

IFFT represents the inverse fast Fourier transform, M(w) and R(w) are 2D Fourier spectra of modified image patches for the pair of consecutive images, respectively,
The 2D Fourier spectrum of the modified image patches is obtained by projecting the 2D Fourier spectrum of the image patches onto an orthogonal complement of a Gram polynomial of order 1,
The pixel displacement is obtained by the position of the peak at the surface P(n). The method according to claim 1,
After the step of acquiring a pair of consecutive images outside the vehicle,
A vehicle navigation method based on a vision sensor, further comprising the step of obtaining a normalized image (N(x, y)) by applying Equation 1 to the pair of consecutive images.
[Equation 1]

I(x, y) is the input image, μI is the local mean of I(x, y), and σI is the local standard deviation.

Images