CN116152342A - Guideboard registration positioning method based on gradient - Google Patents

Abstract

The invention relates to a gradient-based road sign registration and positioning method, comprising: A, building a database; B, rough positioning and extraction of street signs; a, obtaining road images; b, using the trained YOLOv8 network model to perform target detection on the road images , realize the coarse positioning of the target street sign, and identify the category of the current street sign, obtain the database information of the current street sign, and obtain the rough positioning area image; c, convert the rough positioning area image from RGB space to HSV space; d, convert the processed The image is thresholded in the HSV space to obtain the ROI image of the road sign area; C, gradient-based registration optimization; D, vehicle pose calculation; the present invention uses the highest version of the YOLO network model to perform rough positioning of road signs, and realizes in complex scenes The problem of detecting the target area of road signs improves the speed and reliability of the detection results, ensures the integrity of the road sign area, and effectively eliminates the interference of other similar areas.

Description

A road sign registration and positioning method based on gradient

Technical Field

The invention relates to a road sign registration and positioning method based on gradient, and belongs to the field of digital image processing and computer vision.

Background Art

With the rapid improvement of the national economic level, the number of cars has increased rapidly, and traffic congestion has become increasingly serious. Intelligent transportation systems and autonomous driving technologies have become hot topics for scholars at home and abroad. Among them, vehicle self-positioning, as a basic and key technology, is extremely important. In underground parking lots, tunnels, or densely built urban areas, the positioning accuracy of satellite-based vehicle positioning systems is greatly affected or cannot work properly due to the obstruction of satellite signals. Based on this background, in-depth research on vehicle positioning systems based on road signs aims to improve the positioning accuracy and reliability of vehicle positioning systems in specific areas.

The earliest developed and most widely used positioning technology is the Global Positioning System (GPS), but its positioning effect in dense urban areas has declined significantly. If a single sensor is used, it is impossible to achieve interference-free high-precision positioning in densely populated areas; the multi-sensor fusion positioning system is costly and inflexible, which hinders the large-scale productization of the system; at the same time, most positioning systems that rely on sensor cascades will have cumulative errors and lead to large positioning errors in complex urban environments and congested road conditions.

Based on the limitations of the above methods, with the development of computer vision, visual sensors are increasingly used for vehicle positioning, and systems combining binocular cameras, monocular cameras and various sensors are emerging in an endless stream. Using a monocular camera to complete vehicle positioning generally requires the construction of a more complex and accurate comparison database, and the accuracy is lower.

Summary of the invention

In view of the deficiencies of the prior art, the present invention provides a gradient-based road sign registration and positioning method.

The present invention adopts binocular vision vehicle self-positioning algorithm, combined with GPS or offline downloaded route planning and road sign database to realize autonomous navigation, especially when there are dense vehicles at intersections and lane lines are covered, the road signs are used to realize accurate self-positioning of the vehicle.

Terminology Note:

1. RGB color space: It consists of three basic color channels: red (R), green (G), and blue (B). The value range of pixels in each channel is [0,255]. By changing the values of the three channels and then superimposing them, different colors can be obtained.

2. HSV color space: describes colors using three characteristics: hue (H), saturation (S), and value (V), which is more similar to the way the human eye perceives colors.

3. getPerspectiveTransform function: Calculates the perspective transformation matrix from the original image to the target avatar based on the four pairs of point coordinates on the source image and the target image.

其中f是焦距，

代表x轴方向焦距的像素长度，

代表y轴方向焦距的像素长度，u₀,v₀为图像主点的实际位置，这些参数只由相机自身属性决定，不因外界环境而改变。4. Camera intrinsic parameter matrix: The 3D camera coordinates can be transformed into 2D homogeneous image coordinates, expressed as

Where f is the focal length,

Represents the pixel length of the focal length in the x-axis direction,

represents the pixel length of the focal length in the y-axis direction, u ₀ ,v ₀ are the actual positions of the principal points of the image, and these parameters are determined only by the camera's own properties and will not change due to the external environment.

其中R是旋转矩阵，它的每一个列向量表示世界坐标系的每一个坐标轴在相机坐标系下的指向；T是平移矩阵，它是世界坐标系原点在相机坐标系下的表示。5. External parameter matrix: realizes the transformation from the world coordinate system to the camera coordinate system, which can be expressed as

Where R is the rotation matrix, each of its column vectors represents the direction of each coordinate axis of the world coordinate system in the camera coordinate system; T is the translation matrix, which is the representation of the origin of the world coordinate system in the camera coordinate system.

6. Perspective transformation matrix: Two cameras image the same scene, and there is a unique geometric correspondence between the two images, which can be expressed in the form of a matrix. One situation that satisfies the perspective transformation is that the light centers of the two images coincide when they are taken, and another situation is that the scene being photographed is on the same plane, such as the road sign used in the present invention.

7. Homogeneous coordinates: A vector that is originally n-dimensional is represented by an n+1-dimensional vector. It is used in the coordinate system in projective geometry. The homogeneous coordinate form of the two-dimensional coordinate (x, y) is usually expressed by three-dimensional coordinates (hx, hy, h). In homogeneous coordinates, h is the scale factor, which can take any value and is generally set to 1 to keep the two coordinates consistent.

8. Image registration: It refers to the process of matching and superimposing two or more images acquired at different times, using different sensors (imaging devices) or under different conditions (weather, illumination, camera position and angle, etc.).

9. Optical flow: It is the instantaneous speed of the pixel movement of a moving object in space on the observation imaging plane. When the time interval is very small (such as between two consecutive frames of a video), it is also equivalent to the displacement of the target point.

The technical solution of the present invention is:

A gradient-based road sign registration and positioning method comprises the following steps:

A. Building a database

The database includes the following information of each road sign: geographic coordinates, distance information between the center of the road sign and the lane, and background color. The geographic coordinates refer to the longitude and latitude of the road sign; the distance information between the center of the road sign and the lane refers to the lateral distance between the center of the road sign and each lane line; the background color refers to the color of the road sign;

B. Rough positioning and extraction of road signs

a. Install a binocular camera in front of the vehicle to obtain road images in real time;

b. Use the trained YOLOv8 network model to perform target detection on the road image obtained in step a, achieve coarse positioning of the target road sign, identify the category of the current road sign, obtain the database information of the current road sign, and obtain the coarse positioning area image;

c. Convert the coarse positioning area image obtained in step b from RGB space to HSV space;

d. Perform threshold processing on the image processed in step c in HSV space, and obtain a mask image through morphological operations and connected domain length-width ratio constraints, thereby obtaining a road sign area ROI image;

C. Gradient-based registration optimization

Based on the obtained mask image, four pairs of initial corresponding points are initially obtained, thereby obtaining the initial perspective transformation matrix. Then, the gradient-based optimization algorithm is used to perform sub-pixel registration on the Gaussian-smoothed left and right road sign area ROI images, and the accurate perspective transformation matrix is obtained by iterative update.

D. Vehicle posture calculation

The overall displacement of the road sign is obtained, that is, the parallax is obtained, and then the vehicle's position information is obtained, including the lateral and normal distances of the camera relative to the road sign, that is, the distance of the vehicle relative to the road sign and the lane it is in.

Preferably, according to the present invention, the specific implementation process of step b includes:

(1) Training YOLOv8 network model:

Get the training set: select pictures containing road signs and label them. The labels include the coordinate information of the road signs in the pictures and the categories of the road signs.

The YOLOv8 network model includes Backbone unit, Neck unit, and Head unit;

The Backbone unit includes a convolution module, a C2f module, and an SPPF module. The convolution module (Conv module) includes a convolution layer, a batch normalization layer, and a SiLU activation function layer; the C2f module includes a convolution module, a Bottleneck module, and a residual structure module; the SPPF module includes a convolution layer and a pooling layer;

The Neck unit includes a convolution module, a C2f module, and an upsampling layer;

The Head unit includes a detection module (Detect module), which includes a convolution module and a convolution layer;

Based on the Backbone unit, each image in the training set is scaled and convolved to obtain the initial feature map; based on the Neck unit, the initial feature map is extracted again to obtain intermediate feature maps of different scales; the intermediate feature maps of different scales are input into the Head unit to obtain the coordinates of the road sign predicted by the YOLOv8 network model;

The loss is calculated using the road sign coordinates predicted by the YOLOv8 network model and the actual road sign coordinates. The gradient of the YOLOv8 network model optimization is obtained through the loss. The weights of the YOLOv8 network model are updated. The loss continues to decrease and the accuracy of network prediction continues to increase, thus obtaining a trained YOLOv8 network model.

The road image obtained in step a is used to perform target detection using the trained YOLOv8 network model to achieve rough positioning of the target road sign, identify the category of the current road sign, and obtain database information;

(2) In the actual reasoning test phase, the road image obtained in step a is input into the trained YOLOv8 network model to obtain the predicted rough positioning coordinates of the road sign and the type of the road sign;

(3) The coarse positioning area obtained by the coarse positioning coordinates of the road sign is set to 1, and the other areas are set to 0 to obtain a coarse positioning area image.

According to the preferred embodiment of the present invention, in step c,

In HSV space, separate pixels that meet the following threshold ranges:

The threshold value range of saturation S is 0.35<S<1, and the threshold value range of brightness V is 0.35<V<1. The threshold value range of hue H is determined by the road sign. For rectangular blue road signs, set 200<H<280. For the four corner red areas of self-made standard road signs, set H>330 or H<30.

According to the preferred embodiment of the present invention, in step d,

Threshold processing means: for pixels that meet the threshold range, set them to 255, and the remaining pixels are set to 0 to obtain a preliminary mask image;

Morphological operation means: calling the morphology library function to remove external noise and internal holes, and solving possible edge discontinuities through closing operations to eliminate most interference areas;

The connected domain length-width ratio constraint means: constraining the target area according to the length-width ratio and area size of the target area to obtain the final target area without interference;

The minimum enclosing rectangle of the target area is obtained, and is set to 255 to obtain the mask image of the road sign area. The mask image and the original road sign image are ANDed together to finally obtain the ROI image of the road sign area.

Preferably, according to the present invention, the specific implementation process of step C includes:

The vertex coordinates of the minimum circumscribed rectangle in the left and right target mask images are extracted as four pairs of initial corresponding points to obtain the initial perspective transformation matrix M. The plane perspective projection transformation relationship is shown in formula (I):

Where x = (x, y, 1) and x' = (x', y', 1) are homogeneous coordinates, ~ indicates proportionality, rewritten as:

Take the right image as the target image, perform perspective projection transformation on the left image to approximate the right image, and use M←(E+D)M to iteratively update the transformation matrix, where:

In formula (V), d ₀ to d ₇ correspond to the update parameters of each iteration of m ₀ to m ₇ in the M matrix;

即：At this time, the left image I ₁ is resampled using the new transformation x'～(E+D)Mx, which is equivalent to the left image resampled using the transformation x"～(E+D)x

Right now:

Where x'=(x',y',1) is a homogeneous coordinate, rewritten as:

The pixel motion is estimated by minimizing the intensity error between the two images. The intensity error equation is as follows:

是重采样后的左图图像

在x_i的图像梯度，x_i的取值范围是路牌ROI区域；In formula (X) and (XI),

The left image is resampled

In the image gradient of _xi , the value range of _xi is the road sign ROI area;

是重采样后的左图图像

和目标图像I₀对应点的强度误差；

The left image is resampled

The intensity error of the corresponding point with the target image I ₀ ;

d = (d ₀ , d ₁ , ..., d ₇ ) is the motion update parameter, _Ji = _Jd ( _xi ) is the Jacobian determinant of the resampled point coordinate _x'i ' with respect to d, corresponding to the optical flow caused by the instantaneous motion of the three-dimensional plane, expressed as:

At this time, the least squares method is used to obtain the analytical solution:

Ad＝-b (XIII)

Among them, the Hessian matrix:

Accumulated gradient:

According to the preferred embodiment of the present invention, in step D,

According to the imaging principle of the binocular camera:

Among them, X refers to the lateral distance from the camera to the center of the road sign. The lane where the vehicle is located is calculated based on the distance information between the center of the road sign and the lane in the database; Z refers to the normal distance from the camera to the road sign plane, that is, the distance from the vehicle to the road sign;

_OL and _OR are the left and right aperture centers of the binocular camera, the distance between _OL and _OR is the baseline b of the binocular camera, and f is the focal length; P(X,Y,Z) is a point in three-dimensional space, and P(X,Y,Z) forms an image in each binocular camera, recorded as _PL and _PR . After correction, the coordinates of _PL and _PR on the x-axis of the imaging plane are _uL and _uR , and the obtained disparity disp = _uL - _uR ;

At this point, combined with the internal and external parameters obtained by camera calibration, the lateral and normal distances of the camera relative to the road sign are obtained, that is, the distance of the vehicle relative to the road sign and the lane it is in are obtained, thus achieving vehicle self-positioning.

A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a gradient-based road sign registration and positioning method when executing the computer program.

A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a gradient-based road sign registration and positioning method.

The beneficial effects of the present invention are:

1. The present invention uses the highest version of the YOLO (You Only Look Once version 8) network model to perform rough positioning of road signs, and combines it with color extraction to solve the problem of detecting the target area of road signs in complex scenes, improve the speed and reliability of the detection results, ensure the integrity of the road sign area, and effectively eliminate the interference of other similar areas.

2. The present invention makes full use of the characteristic that the road sign is a plane, and uses it as a feature point for matching, obtains the perspective transformation matrix of the left and right road sign areas, calculates the overall optical flow between the planes for stereo matching, avoids the redundancy of point-by-point calculation, makes the registration result more robust, and the detection result more accurate and faster.

3. The present invention proposes to use a simple database system with a simple structure, small data volume, and easy later maintenance. The database content mainly includes the location information, background color and lane information of each road sign at the road sign, which can realize the lane-level positioning of the vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a flow chart of a gradient-based road sign registration and positioning method of the present invention;

Figure 2 is a schematic diagram of a blue rectangular road sign;

Figure 3(a) is a schematic diagram of a self-made standard road sign labeled A1;

Figure 3(b) is a schematic diagram of a self-made standard road sign labeled A2;

Figure 3(c) is a schematic diagram of a self-made standard road sign labeled B3;

Figure 3(d) is a schematic diagram of a self-made standard road sign labeled B4;

Figure 3(e) is a schematic diagram of a self-made standard road sign labeled C5;

Figure 3(f) is a schematic diagram of a self-made standard road sign labeled C6;

Figure 4(a) is a simplified schematic diagram of the YOLOv8 network model;

Figure 4(b) is a schematic diagram of the structure of the YOLOv8 network model;

Figure 4(c) is a detailed structural diagram of the Conv module of YOLOv8;

Figure 4(d) is a detailed structural diagram of the C2f module of YOLOv8;

Figure 4(e) is a detailed structural diagram of the Bottleneck module of YOLOv8;

Figure 4(f) is a detailed structural diagram of the SPPF module of YOLOv8;

Figure 4(g) is a detailed structural diagram of the Detect module of YOLOv8;

Figure 5 is a schematic diagram of the effect of YOLOv8 detecting a blue rectangular road sign;

Figure 6 is a schematic diagram of the effect of YOLOv8 detecting a self-made standard road sign;

Figure 7 is a schematic diagram of the ROI image of a blue rectangular road sign;

FIG8 is a schematic diagram of a ROI image of a self-made standard road sign;

FIG9 is a schematic diagram of error variation during iterative optimization;

FIG10 is a schematic diagram of an imaging model of a binocular camera.

DETAILED DESCRIPTION

The present invention will be further described below in conjunction with the embodiments and the accompanying drawings, but is not limited thereto.

Example 1

A road sign registration and positioning method based on gradient, the road sign is any road sign in the database, set on the right side of the road and perpendicular to the road surface, taking a blue rectangular road sign (as shown in FIG2 ) and a self-made standard road sign as examples, the self-made standard road sign is a square plane sign, the background color is white, and square red areas are set at the four top corners, and the indicating characters are letters and numbers, marked in black, as shown in FIG1 , including the following steps:

A. Building a database

The database includes the following information of each road sign: geographic coordinates, distance information between the center of the road sign and the lane, and background color. The geographic coordinates refer to the longitude and latitude of the road sign; the distance information between the center of the road sign and the lane refers to the lateral distance between the center of the road sign and each lane line; the background color refers to the color of the road sign; the present invention uses a binocular camera, which is applicable to different road signs and does not require the size of the road sign to be collected in advance.

B. Rough positioning and extraction of road signs

a. Install a binocular camera in front of the vehicle to obtain road images in real time; the optical axis of the binocular camera is parallel and in the same direction as the vehicle's travel, the focal lengths of the left and right cameras of the binocular camera are equal, and the positive directions of the x-axis coincide;

b. Use the trained YOLOv8 network model to perform target detection on the road image obtained in step a to achieve rough positioning of the target road sign and identify the category of the current road sign. There are 7 types of road signs used in this embodiment, including one type of blue rectangular road signs and six types of self-made standard road signs. The numbers on the road signs are A1, A2, B3, B4, C5, and C6, as shown in Figures 3(a), 3(b), 3(c), 3(d), 3(e), and 3(f), respectively;

c. Convert the roughly positioned area image obtained in step b from the RGB space to the HSV space. In the RGB space, the three color components are highly correlated, difficult to analyze, and easily affected by lighting, while the HSV space can eliminate the influence of lighting by adjusting saturation and brightness, and can more accurately separate a certain color.

d. Perform threshold processing on the image processed in step c in HSV space, and obtain a mask image through morphological operations and connected domain length-width ratio constraints, thereby obtaining a road sign area ROI image;

C. Gradient-based registration optimization

Based on the obtained mask image, four pairs of initial corresponding points are initially obtained, thereby obtaining the initial perspective transformation matrix. Then, the gradient-based optimization algorithm is used to perform sub-pixel registration on the Gaussian-smoothed left and right road sign area ROI images, and the accurate perspective transformation matrix is obtained by iterative update.

D. Vehicle posture calculation

The above steps are iteratively updated to obtain an accurate perspective transformation matrix. The overall displacement of the road sign is calculated according to the coordinates of the center point of the road sign and formula (I), that is, the parallax is calculated, and then the vehicle position information is obtained, including the lateral and normal distances of the camera relative to the road sign, that is, the distance of the vehicle relative to the road sign and the lane it is in.

Example 2

The gradient-based road sign registration and positioning method according to Example 1 is different in that:

The specific implementation process of step b includes:

(1) Training YOLOv8 network model:

Get the training set: select pictures containing road signs and label them. The labels include the coordinate information of the road signs in the pictures and the categories of the road signs.

Because the original model cannot recognize target information such as road signs well, in order to address the practical problems faced by the present invention, 100 pictures containing road signs were selected for each type of road sign for training. The pictures cover a large number of road sign scenes, which can ensure the accuracy of road sign detection in new scenes. The labels in the training process are the coordinate information of the road signs in the pictures and the categories of the road signs. In this way, the required road sign coordinates and category information can be obtained in the inference stage to facilitate the next step of operation.

As shown in Figure 4(a) and Figure 4(b), the YOLOv8 network model includes Backbone unit, Neck unit, and Head unit;

The Backbone unit includes a convolution module, a C2f module, and an SPPF module. As shown in Figure 4(c), the convolution module (Conv module) includes a convolution layer, a batch normalization layer, and a SiLU activation function layer. The specific structure of the C2f module is shown in Figure 4(d). The C2f module includes a convolution module, a Bottleneck module, and a residual structure module. The specific structure of the Bottleneck module is shown in Figure 4(e). The specific structure of the SPPF module is shown in Figure 4(f). The SPPF module includes a convolution layer and a pooling layer.

After the input is scaled to a picture with a height and width of 640, the initial feature maps with height and width of 80, height and width of 40, and height and width of 20 are obtained after passing through the Backbone unit;

The Neck unit includes a convolution module, a C2f module, and an upsampling layer; after passing through the Neck unit, intermediate feature maps with height and width of 80, height and width of 40, and height and width of 20 are obtained.

The Head unit includes a detection module (Detect module), as shown in Figure 4(g), which includes a convolution module and a convolution layer. After passing through the Head unit, the predicted target category information and target coordinate information are obtained;

Based on the Backbone unit, each image in the training set is scaled (scaled to an image with a height and width of 640) and convolved to obtain the initial feature map; based on the Neck unit, the initial feature map is secondary extracted to obtain intermediate feature maps of different scales; the intermediate feature maps of different scales are input into the Head unit to obtain the coordinates of the road sign predicted by the YOLOv8 network model;

The loss is calculated by using the road sign coordinates predicted by the YOLOv8 network model and the actual road sign coordinates, and the gradient of the YOLOv8 network model optimization is obtained through the loss. Figure 9 is a schematic diagram of the error change during the iterative optimization process; the weights of the YOLOv8 network model are updated, the loss continues to decrease, and the accuracy of network prediction continues to increase, thereby obtaining a well-trained YOLOv8 network model;

The road image obtained in step a is used to perform target detection using the trained YOLOv8 network model to achieve rough positioning of the target road sign, identify the category of the current road sign, and obtain database information;

(2) In the actual reasoning test phase, the road image obtained in step a is input into the trained YOLOv8 network model to obtain the predicted rough positioning coordinates of the road sign and the type of the road sign; FIG5 is a schematic diagram of the effect of YOLOv8 detecting a blue rectangular road sign; FIG6 is a schematic diagram of the effect of YOLOv8 detecting a self-made standard road sign.

(3) The coarse positioning area obtained by the coarse positioning coordinates of the road sign is set to 1, and the other areas are set to 0 to obtain a coarse positioning area image.

In step c, the three color components in the RGB space are highly correlated, difficult to analyze, and easily affected by lighting, while the HSV space can eliminate the influence of lighting by adjusting saturation and brightness, and can more accurately separate a certain color. In the HSV space, separate the pixels that meet the following threshold range:

According to prior knowledge and experimental measurements, the threshold range of saturation S is 0.35<S<1, and the threshold range of brightness V is 0.35<V<1. The threshold range of hue H is determined by the road sign. For rectangular blue road signs, 200<H<280 is set. For self-made standard road signs to extract the four red corners, H>330 or H<30 is set.

In step d, threshold processing means: for pixels that meet the threshold range, set them to 255, and set the remaining pixels to 0, to obtain a preliminary mask image;

Morphological operation means: calling the morphology library function to remove external noise and internal holes, and solving possible edge discontinuities through closing operations, eliminating most interference areas, and ensuring that the road sign area is a complete connected area;

The connected domain length-width ratio constraint means: constraining the target region according to the length-width ratio and area size of the target region. Experiments show that the final target region without interference can be obtained.

Since the four corner areas of the self-made red standard road sign used in the present invention are red squares, that is, the aspect ratio is 1, the aspect ratio can be limited to be greater than 0.8 and less than 1.2; at the same time, since the red color of the road sign in the rough positioning area is the largest red area, area constraints can be performed, and the four areas with the largest areas are retained as the final target areas;

At this point, all interference has been eliminated. Now, four target areas are extracted, and their minimum circumscribed rectangle is calculated and set to 255 to obtain the mask image of the entire road sign area. The mask image and the original road sign image are ANDed together to finally obtain the ROI image of the road sign area. Figure 7 is a schematic diagram of the ROI image of the blue rectangular road sign; Figure 8 is a schematic diagram of the ROI image of the homemade standard road sign.

The specific implementation process of step C includes:

Extract the vertex coordinates of the minimum circumscribed rectangle in the left and right target mask images as four pairs of initial corresponding points, and call the getPerspectiveTransform function in OpenCV to obtain the initial perspective transformation matrix M. The plane perspective projection transformation relationship is shown in formula (I):

Where x = (x, y, 1) and x' = (x', y', 1) are homogeneous coordinates, ~ indicates proportionality, rewritten as:

Take the right image as the target image, perform perspective projection transformation on the left image to approximate the right image. In order to optimize the eight parameters of the projection matrix, use M←(E+D)M to iteratively update the transformation matrix, where:

In formula (XXII), _d0 to _d7 correspond to the update parameters of each iteration of _m0 to _m7 in the M matrix;

即：At this time, the left image I ₁ is resampled using the new transformation x'～(E+D)Mx, which is equivalent to the left image resampled using the transformation x"～(E+D)x

Right now:

Where x'=(x',y',1) is a homogeneous coordinate, rewritten as:

In order to restore the accurate perspective transformation relationship, the pixel motion is estimated by minimizing the intensity error between the two images. The intensity error equation is as follows:

是重采样后的左图图像

在x_i的图像梯度，x_i的取值范围是路牌ROI区域；In formula (XXVII) and (XXVIII),

The left image is resampled

In the image gradient of _xi , the value range of _xi is the road sign ROI area;

是重采样后的左图图像

和目标图像I₀对应点的强度误差；

The left image is resampled

The intensity error of the corresponding point with the target image I ₀ ;

d = (d ₀ , d ₁ , ..., d ₇ ) is the motion update parameter, _Ji = J _d ( _xi ) is the Jacobian determinant of the resampled point coordinate x″ _i with respect to d, corresponding to the optical flow caused by the instantaneous motion of the three-dimensional plane, expressed as:

At this time, the least squares method is used to obtain the analytical solution:

Ad＝-b (XXX)

Among them, the Hessian matrix:

Accumulated gradient:

In step D, according to the imaging principle of the binocular camera, we can get:

Among them, X refers to the lateral distance from the camera to the center of the road sign. The lane where the vehicle is located is calculated based on the distance information between the center of the road sign and the lane in the database; Z refers to the normal distance from the camera to the road sign plane, that is, the distance from the vehicle to the road sign;

The imaging model of the binocular camera is shown in Figure 10. _OL and _OR are the left and right aperture centers of the binocular camera, the distance between _OL and _OR is the baseline b of the binocular camera, the box is the imaging plane, and f is the focal length; P(X, Y, Z) is a point in three-dimensional space (the optical center of the left camera is taken as the origin coordinate), and P(X, Y, Z) forms an image in each binocular camera, recorded as _PL and _PR . After correction, the coordinates of _PL and _PR on the x-axis of the imaging plane are _uL and _uR (the image principal point is taken as the origin, so _uR is a negative number), then the parallax disp obtained in the above steps = _uL - _uR ;

At this point, combined with the internal and external parameters obtained by camera calibration, the lateral and normal distances of the camera relative to the road sign are obtained, that is, the distance of the vehicle relative to the road sign and the lane it is in are obtained, thus achieving vehicle self-positioning.

Example 3

A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the gradient-based road sign registration and positioning method of embodiment 1 or 2 are implemented.

Example 4

A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the gradient-based road sign registration and positioning method of embodiment 1 or 2.

Claims (8)

1. The guideboard registration positioning method based on the gradient is characterized by comprising the following steps:

A. building a database

The database includes the following information for each guideboard: geographic coordinates, distance information between the center of the guideboard and the lane and ground color, wherein the geographic coordinates refer to longitude and latitude of the guideboard; the distance information between the center of the guideboard and the lane refers to the transverse distance between the center of the guideboard and each lane line; the ground color refers to the color of the guideboard;

B. rough positioning and extracting method for guideboard

a. Installing a binocular camera in front of a vehicle to acquire road images in real time;

b. c, carrying out target detection on the road image obtained in the step a by using a trained YOLOv8 network model, realizing rough positioning of a target guideboard, identifying the category of the current guideboard, obtaining database information of the current guideboard, and solving a rough positioning area image;

c. b, converting the rough positioning area image obtained in the step b from RGB space into HSV space;

d. c, performing threshold processing on the image processed in the step c in HSV space, and obtaining a mask image through morphological operation and constraint of aspect ratio of the connected domain, thereby obtaining a guideboard region ROI image;

C. gradient-based registration optimization

According to the obtained mask image, four pairs of initial corresponding points are preliminarily obtained, so that an initial perspective transformation matrix is obtained, sub-pixel level registration is carried out on the left and right guideboard region ROI images after Gaussian smoothing by utilizing a gradient-based optimization algorithm, and an accurate perspective transformation matrix is obtained through iterative updating;

D. vehicle pose calculation

Obtaining the overall displacement of the guideboard, namely obtaining parallax, and further obtaining the position information of the vehicle, wherein the position information comprises the transverse and normal distances of the camera relative to the guideboard, namely the distance of the vehicle relative to the guideboard and the lane where the vehicle is located.

2. The gradient-based guideboard registration positioning method of claim 1, wherein the specific implementation process of the step b comprises:

(1) Training a YOLOv8 network model:

acquiring a training set: selecting a picture containing the guideboard, labeling a label, wherein the label comprises coordinate information of the guideboard in the picture and the category of the guideboard;

the YOLOv8 network model comprises a Backbone unit, a Neck unit and a Head unit;

the backbox unit comprises a convolution module, a C2f module and an SPPF module, wherein the convolution module comprises a convolution layer, a batch normalization layer and a SiLU activation function layer; the C2f module comprises a convolution module, a Bottleneck module and a residual error structure module; the SPPF module comprises a convolution layer and a pooling layer;

the Neck unit comprises a convolution module, a C2f module and an up-sampling layer;

the Head unit comprises a detection module, wherein the detection module comprises a convolution module and a convolution layer;

scaling and convolving each picture in the training set based on a Backbone unit, so as to obtain an initial feature map; performing secondary extraction on the obtained initial feature map based on a Neck unit to obtain intermediate feature maps with different scales; inputting the obtained intermediate feature graphs with different scales into a Head unit to obtain guideboard coordinates predicted by a YOLOv8 network model;

calculating loss through the guideboard coordinates predicted by the YOLOv8 network model and the real guideboard coordinates, obtaining a gradient optimized by the YOLOv8 network model through the loss, updating the weight of the YOLOv8 network model, and continuously reducing the accuracy of network prediction and continuously increasing the loss, so as to obtain a trained YOLOv8 network model;

b, performing target detection on the road image obtained in the step a by using a trained YOLOv8 network model, realizing rough positioning of a target guideboard, identifying the category of the current guideboard, and obtaining database information;

(2) In the actual reasoning test stage, inputting the road image obtained in the step a into a trained YOLOv8 network model to obtain predicted rough positioning coordinates of the guideboard and the type of the guideboard;

(3) And setting 1 in the rough positioning area and 0 in the other areas obtained by rough positioning coordinates of the guideboard to obtain a rough positioning area image.

3. The method of gradient-based guideboard registration positioning of claim 1, wherein, in step c,

in HSV space, pixels are isolated that meet the following threshold ranges:

the threshold value range of the saturation S is 0.35< S <1, and the threshold value range of the brightness V is 0.35< V <1; the threshold value range of the hue H is determined by the guideboard, 200< H <280 is set for a rectangular blue guideboard, and H >330 or H <30 is set for a self-made standard guideboard to extract a quadrangle red area.

4. The method of gradient-based guideboard registration positioning of claim 1, wherein in step d,

thresholding refers to: setting 255 for pixels meeting a threshold range, and setting 0 for the rest pixels to obtain a preliminary mask image;

morphological operations refer to: calling a morphism library function, removing external noise points and internal holes, solving the possible edge discontinuity condition in a closed operation mode, and eliminating most of interference areas;

the aspect ratio constraint of the connected domain refers to: constraining the area according to the aspect ratio and the area size of the target area to obtain a final target area without interference;

obtaining the minimum circumscribed rectangle of the target area, setting the minimum circumscribed rectangle as 255, obtaining a mask image of the guideboard area, performing AND operation on the mask image and the original guideboard image, and finally obtaining the ROI image of the guideboard area.

5. The gradient-based guideboard registration positioning method according to claim 1, wherein the specific implementation process of the step C includes:

extracting vertex coordinates of a minimum circumscribed rectangle in the left and right target mask images as four pairs of initial corresponding points to obtain an initial perspective transformation matrix M, wherein the plane perspective projection transformation relation is shown in the formula (I):

where x= (x, y, 1) and x ' = (x ', y ', 1) are homogeneous coordinates, and-represent proportionality, rewritten as:

taking the right image as a target image, performing perspective projection transformation on the left image to approximate the right image, and iteratively updating a transformation matrix by using M≡ (E+D) M, wherein,

in the formula (V), d ₀ To d ₇ Corresponding to M in M matrix ₀ To m ₇ Updating parameters of each iteration;

at this time, the left-hand image I is transformed with new transforms x' - (E+D) Mx ₁ Resampling corresponds to resampling the left image using an x' to (E+D) x transform

Namely: />

Where x "= (x", y ", 1) is a homogeneous coordinate, rewritten as:

the motion of the pixel is estimated by minimizing the intensity error between the two images, then the intensity error equation is as follows:

in the formulas (X) and (XI),

is the resampled left image +.>

At x _i Image gradient, x _i The value range of (2) is the guideboard ROI area;

is the resampled left image +.>

And target image I ₀ Intensity errors of corresponding points;

d＝(d ₀ ,d ₁ ,...,d ₇ ) Is a motion update parameter, J _i ＝J _d (x _i ) Is the resample point coordinates x' _i ' jacobian with respect to d, corresponds to the optical flow caused by the instantaneous motion of a three-dimensional plane, expressed as:

at this time, an analytical solution is obtained by the least square method:

Ad＝-b (XIII)

wherein, hessian matrix:

cumulative gradient:

6. the method of gradient-based guideboard registration and positioning of claim 1, wherein, in step D,

the imaging principle of the binocular camera is as follows:

wherein X is the transverse distance from the camera to the center of the guideboard, and the lane where the vehicle is located is calculated according to the distance information between the center of the guideboard and the lane in the database; z refers to the normal distance from the camera to the guideboard plane, i.e., the distance from the vehicle to the guideboard;

O _L 、O _R is the center of the left aperture and the right aperture of the binocular camera, O _L 、O _R The distance between the two is the base line b and f of the binocular camera, and the focal length is the distance; p (X, Y, Z) is a point in three-dimensional space, and P (X, Y, Z) is imaged in a binocular camera, denoted as P _L And P _R After correction, P _L And P _R The coordinate of the x-axis of the imaging plane is u _L And u _R The obtained parallax disp=u _L -u _R ；

The transverse and normal distances of the camera relative to the guideboard are obtained by combining the internal and external parameters obtained by the calibration of the camera, namely the distance of the vehicle relative to the guideboard and the lane where the vehicle is located are obtained, and the self-positioning of the vehicle is realized.

7. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the gradient-based guideboard registration positioning method of any of claims 1-6 when the computer program is executed.

8. A computer readable storage medium having stored thereon a computer program, characterized in that the computer program when executed by a processor implements the steps of the gradient-based guideboard registration positioning method of any of claims 1-6.

Images