CN113034583A - Vehicle parking distance measuring method and device based on deep learning and electronic equipment - Google Patents
Vehicle parking distance measuring method and device based on deep learning and electronic equipment Download PDFInfo
- Publication number
- CN113034583A CN113034583A CN202110360717.7A CN202110360717A CN113034583A CN 113034583 A CN113034583 A CN 113034583A CN 202110360717 A CN202110360717 A CN 202110360717A CN 113034583 A CN113034583 A CN 113034583A
- Authority
- CN
- China
- Prior art keywords
- image
- distortion
- vehicle
- camera
- white line
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000013135 deep learning Methods 0.000 title claims abstract description 37
- 230000009466 transformation Effects 0.000 claims abstract description 65
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 49
- 230000011218 segmentation Effects 0.000 claims abstract description 43
- 239000011159 matrix material Substances 0.000 claims abstract description 37
- 238000001514 detection method Methods 0.000 claims abstract description 7
- 230000015654 memory Effects 0.000 claims description 17
- 238000013528 artificial neural network Methods 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 5
- 238000003062 neural network model Methods 0.000 claims description 4
- 238000001914 filtration Methods 0.000 claims description 3
- 230000036316 preload Effects 0.000 claims description 2
- 238000000691 measurement method Methods 0.000 claims 1
- 230000001131 transforming effect Effects 0.000 claims 1
- 238000003384 imaging method Methods 0.000 abstract description 10
- 238000012360 testing method Methods 0.000 description 14
- 230000008569 process Effects 0.000 description 12
- 238000010586 diagram Methods 0.000 description 8
- 239000004744 fabric Substances 0.000 description 8
- 238000012545 processing Methods 0.000 description 7
- 230000010339 dilation Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000005286 illumination Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 238000009434 installation Methods 0.000 description 2
- 239000003550 marker Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
- G06T2207/30256—Lane; Road marking
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
- Traffic Control Systems (AREA)
Abstract
The invention discloses a vehicle parking distance measuring method, device and electronic equipment based on deep learning, wherein the method comprises the following steps: determining a distortion coefficient of a camera lens, and determining a homography transformation matrix according to image coordinates and actual physical coordinates of each identification point in a preset identifier based on an installed camera; acquiring an image acquired by a camera when a vehicle is in a side-by-side parking state, detecting a roadside white line in the acquired image by adopting a semantic segmentation algorithm based on deep learning, and outputting a binary image; removing lens distortion in the binary image based on the distortion coefficient and a distortion removal algorithm to obtain a distortion removal image; and carrying out back projection transformation through the homography transformation matrix to obtain an image with uniform resolution, and measuring the distance between the vehicle and the roadside white line. The method can accurately detect the position of the roadside white line in different scenes, improve the accuracy of the detection of the position of the white line and practical application scenes, remove lens distortion in camera imaging, and accurately measure the distance between the vehicle and the white line.
Description
Technical Field
The invention relates to the field of computer vision and advanced driving assistance, in particular to a vehicle parking distance measuring method and device based on deep learning and electronic equipment.
Background
In the driving test subject three-test, the side parking module of the road test instrument measures the distance between the vehicle and the roadside white line in a photographing mode through a camera mounted on the vehicle, so as to judge whether the student reaches the range of 30cm to 50cm from the vehicle to the roadside white line required by the test.
However, most of the existing road test instrument products use traditional image processing methods, such as simple methods like threshold segmentation and hough transformation, to detect white lines from images, and these simple methods cannot cope with illumination, weather and road surface conditions which are complicated and changeable in real environment, and can only correctly detect roadside white lines in very limited environment.
In addition, in the training process of the driving test subject III, the accurate distance between the vehicle and the roadside white line is important feedback to the student when the vehicle parks alongside, the student can adjust the operation of the vehicle according to the feedback, and the progress is continuously made, however, most of the existing road test instrument products can only be judged if not, and tell the student whether the current distance is within the range of 30cm to 50cm, but cannot tell the accurate distance of the student specifically how many centimeters, so that the training of the student cannot be effectively fed back.
Disclosure of Invention
The invention mainly aims to provide a vehicle parking distance measuring method and device based on deep learning, and aims to solve the problems that the conventional road test instrument cannot correctly detect a roadside white line and cannot measure the accurate distance between a vehicle and the white line.
In order to achieve the above object, a first aspect of the present invention provides a vehicle parking ranging method based on deep learning, including:
determining all distortion coefficients of a camera lens in advance through an opencv camera calibration algorithm, and determining a homography transformation matrix according to image coordinates and actual physical coordinates of each identification point in a preset identifier on the basis of an installed camera;
when the vehicle is in a side-by-side parking state, acquiring an image acquired by a camera, detecting a roadside white line in the acquired image by adopting a semantic segmentation algorithm based on deep learning, and outputting a binary image;
removing lens distortion in the binary image based on all distortion coefficients and a distortion removal algorithm to obtain a distortion removal image;
carrying out back projection transformation on the distortion-removed image through the homography transformation matrix to obtain an image with uniform resolution;
and measuring the distance between the vehicle and the roadside white line when the vehicle is in a side-by-side parking state according to the resolution uniform image.
Optionally, the determining, in advance, all distortion coefficients of the camera lens through a camera calibration algorithm of opencv includes:
the lens distortion caused by the camera comprises radial distortion and tangential distortion;
before the camera is installed, determining all distortion coefficients in the radial distortion and the tangential distortion in advance through a camera calibration algorithm of opencv;
all the determined distortion coefficients are saved as preload data.
Further, the radial distortion is described by the following system of equations:
xdistort＝x(1+k1r2+k2r4+k3r6)
ydistort＝y(1+k1r2+k2r4+k3r6)
the tangential distortion is described by the following system of equations:
xdistort2＝x+[2p1xy+p2(r2+2x2)]
ydistort2＝y+[p1(r2+2y2)+2p2xy]
wherein x isdistortIs a transverse distortion of radial distortion, ydistortIs the longitudinal distortion, x, of the radial distortionsdistort2Is a transverse distortion in tangential distortion, ydistort2Is the longitudinal distortion in tangential distortion, x is the abscissa without distortion, y is the ordinate without distortion, r is the distance between each point on the image and the center of the image without distortion, k1、k2、k3、p1And p2Is five distortion coefficients in the lens distortion;
the five distortion coefficients are determined by the camera calibration algorithm of opencv.
Optionally, the determining a homography transformation matrix according to the image coordinates and the actual physical coordinates of each identification point in the preset identifier includes:
acquiring an image by using a camera, wherein the image comprises each identification point in a preset identifier;
establishing an image coordinate system according to the positions of the identification points in the image, and determining the image coordinates of the identification points in the preset identifier;
establishing an actual physical coordinate system according to the position of each identification point in the actual physical world, and determining the actual physical coordinate of each identification point in the preset identifier;
and (3) enabling the image coordinates of each identification point to correspond to the actual physical coordinates one by one, and determining a homography transformation matrix through opencv.
Optionally, the detecting the roadside white line in the acquired image by using a semantic segmentation algorithm based on deep learning, and outputting a binary image includes:
the semantic segmentation neural network model adopted by the semantic segmentation algorithm is a deplab semantic segmentation model based on expansion convolution;
adjusting the number of layers and the number of characteristic channels of the neural network in the deplab semantic segmentation model to meet the requirement of the running speed of the vehicle when the vehicle stops at the side;
detecting a roadside white line in an image acquired by the camera by using the adjusted deplab semantic segmentation model;
and displaying the position of the white line by using a white pixel, filtering other contents except the white line in the acquired image, and outputting a binary image.
Optionally, the performing back projection transformation on the undistorted image through the homography transformation matrix to obtain an image with uniform resolution includes:
carrying out back projection transformation on each identification point in the distortion-removed image to obtain the coordinate of each identification point in a three-dimensional space;
wherein, each identification point in the three-dimensional space is positioned on the plane of the road ground, and the Z-axis coordinate of each identification point in the three-dimensional space is constantly 0;
and determining the X-axis coordinate and the Y-axis coordinate of each identification point in the three-dimensional space according to each identification point in the distortion-removed image and the homography transformation matrix to obtain an image with uniform resolution.
Optionally, the measuring a distance between the vehicle and a roadside white line when the vehicle is in a side parking state according to the resolution uniform image includes:
extracting the left edge of the white line in the resolution uniform image by using a gradient operator in the horizontal direction, and performing straight line fitting;
and measuring the actual physical distance between the vehicle and the roadside white line according to the image resolution of the image with uniform resolution and the pixel number between the middle point of the fitted line segment and the left edge of the image after straight line fitting.
The second aspect of the present invention provides a vehicle parking distance measuring device based on deep learning, comprising:
the determining unit is used for determining all distortion coefficients of a camera lens in advance through an opencv camera calibration algorithm, and determining a homography transformation matrix according to image coordinates and actual physical coordinates of each identification point in a preset identifier based on the installed camera;
the detection unit is used for acquiring an image acquired by the camera when the vehicle is in a side-by-side parking state, detecting a white line on the roadside in the acquired image by adopting a semantic segmentation algorithm based on deep learning, and outputting a binary image;
the distortion removing unit is used for removing lens distortion in the binary image based on all the distortion coefficients and a distortion removing algorithm to obtain a distortion removed image;
the back projection transformation unit is used for carrying out back projection transformation on the distortion-removed image through the homography transformation matrix to obtain an image with uniform resolution;
and the measuring unit is used for measuring the distance between the vehicle and the roadside white line when the vehicle is in a side parking state according to the resolution uniform image.
A third aspect of the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the deep learning-based vehicle parking ranging method provided in any one of the first aspects.
A fourth aspect of the present invention provides an electronic apparatus, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to cause the at least one processor to perform the deep learning-based vehicle parking ranging method provided in any one of the first aspect.
In the vehicle parking distance measuring method based on deep learning provided by the embodiment of the invention, a lens distortion coefficient and a homography transformation matrix are predetermined and are subsequently applied to an acquired image, so that the image processing efficiency is improved; the method has the advantages that the method detects the roadside white lines in the collected images by using the semantic segmentation algorithm based on deep learning, can accurately detect the white line positions on the road surface which is worn at night, rainy days and the white line part, improves the accuracy rate of white line position detection, and simultaneously improves the practical application scene;
by adopting a camera calibration algorithm and a distortion removal algorithm, the lens distortion existing in the camera imaging can be calculated and removed; the homography transformation matrix is used for carrying out back projection transformation to obtain an image with uniform resolution, the actual physical distance of any two points in the image can be calculated, the calculated actual distance error is low, and the error is in the centimeter level.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a schematic flow chart of a vehicle parking distance measuring method based on deep learning according to an embodiment of the present invention;
fig. 2 is an image of a black and white scrim with lens distortion according to an embodiment of the present invention;
fig. 3 is an image of a black and white scrim with lens distortion removed according to an embodiment of the present invention;
fig. 4 is an image in which all inner points in the black-and-white scrim are sequentially connected after distortion is removed according to the embodiment of the present invention;
FIG. 5 is a black and white checkerboard coordinate system provided by an embodiment of the present invention;
fig. 6 is a schematic diagram illustrating that all inner points in the black-and-white scrim provided in the embodiment of the present invention are sequentially connected according to actual physical coordinates;
fig. 7 is an image of a black-and-white scrim after back projection transformation according to an embodiment of the present invention;
FIG. 8 is a binary image output by a semantic segmentation algorithm based on deep learning according to an embodiment of the present invention;
FIG. 9 is a semantic segmentation model based on an encoder-decoder according to an embodiment of the present invention;
FIG. 10 is a deplab semantic segmentation model based on dilation convolution according to an embodiment of the present invention;
FIG. 11 is a schematic diagram of a projective transformation provided by an embodiment of the present invention;
FIG. 12 is a diagram illustrating the extraction of the left edge of a white line in a uniform-resolution image using a horizontal gradient operator according to an embodiment of the present invention;
FIG. 13 is a block diagram of a vehicle parking distance measuring device based on deep learning according to an embodiment of the present invention;
fig. 14 is a block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged under appropriate circumstances in order to facilitate the description of the embodiments of the invention herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In the present invention, the terms "upper", "lower", "left", "right", "front", "rear", "top", "bottom", "inner", "outer", "center", "vertical", "horizontal", "lateral", "longitudinal", and the like indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings. These terms are used primarily to better describe the invention and its embodiments and are not intended to limit the indicated devices, elements or components to a particular orientation or to be constructed and operated in a particular orientation.
Moreover, some of the above terms may be used to indicate other meanings besides the orientation or positional relationship, for example, the term "on" may also be used to indicate some kind of attachment or connection relationship in some cases. The specific meanings of these terms in the present invention can be understood by those skilled in the art as appropriate.
Furthermore, the terms "mounted," "disposed," "provided," "connected," and "sleeved" are to be construed broadly. For example, it may be a fixed connection, a removable connection, or a unitary construction; can be a mechanical connection, or an electrical connection; may be directly connected, or indirectly connected through intervening media, or may be in internal communication between two devices, elements or components. The specific meanings of the above terms in the present invention can be understood by those of ordinary skill in the art according to specific situations.
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict. The present invention will be described in detail below with reference to the embodiments with reference to the attached drawings.
For the driving test subject three vehicles to park by side and detect the distance between the vehicles and the roadside white line, most of the existing road test instrument products use the traditional image processing method to detect the white line from the image, and the simple methods cannot cope with the illumination, weather and road surface conditions which are complicated and changeable in real environment, and can only correctly detect the roadside white line in a very limited environment. In addition, the existing road test instrument product cannot tell the precise distance between the vehicle and the roadside white line to the trainee, so that effective feedback cannot be formed on the training of the trainee.
In order to solve the above problem, an embodiment of the present invention provides a vehicle parking distance measuring method based on deep learning, as shown in fig. 1, the method includes the following steps S101 to S105:
step S101: determining all distortion coefficients of a camera lens in advance through an opencv camera calibration algorithm, and determining a homography transformation matrix according to image coordinates and actual physical coordinates of each identification point in a preset identifier on the basis of an installed camera; due to the requirement of controlling cost, lens distortion exists in the imaging effect of the camera adopted by the road test instrument, and the distortion can be determined by the camera calibration algorithm of opencv.
Specifically, the determining, in step S101, all distortion coefficients of the camera lens by using the camera calibration algorithm of opencv in advance includes:
the lens distortion caused by the camera comprises radial distortion and tangential distortion;
before the camera is installed, determining all distortion coefficients in the radial distortion and the tangential distortion in advance through a camera calibration algorithm of opencv; for cameras of the same model, all distortion coefficients of the cameras before and after installation are the same, so that all the distortion coefficients of the cameras are determined only once before the installation of the cameras;
and storing all the determined distortion coefficients as preloading data, and putting the distortion coefficients as the preloading data into a software system of the road test instrument.
Specifically, the radial distortion is described by the following system of equations:
xdistort＝x(1+k1r2+k2r4+k3r6)
ydistort＝y(1+k1r2+k2r4+k3r6)
the tangential distortion is described by the following system of equations:
xdistort2＝x+[2p1xy+p2(r2+2x2)]
ydistort2＝y+[p1(r2+2y2)+2p2xy]
wherein x isdistortIs a transverse distortion of radial distortion, ydistortIs the longitudinal distortion, x, of the radial distortionsdistort2Is a transverse distortion in tangential distortion, ydistort2Is the longitudinal distortion in tangential distortion, x is the abscissa without distortion, y is the distortion-freeThe ordinate of time, r is the distance between each point on the image and the center of the image in the absence of distortion, k1、k2、k3、p1And p2Is five distortion coefficients in the lens distortion;
the five distortion coefficients are determined by the camera calibration algorithm of opencv.
In the embodiment of the invention, before a camera is installed to shoot an image containing roadside white lines, taking black and white check cloth as an example, a camera of the same model is used to shoot the prepared black and white check cloth, and the obtained image with lens distortion is shown in fig. 2; five distortion coefficients of lens distortion are determined through the opencv camera calibration algorithm, lens distortion is removed through the five distortion coefficients and a subsequent distortion removal algorithm, and an image with distortion removed is shown in fig. 3.
Specifically, the determining the homography transformation matrix according to the image coordinates and the actual physical coordinates of each identification point in the preset identifier in step S101 includes:
acquiring an image by using a camera, wherein the image comprises each identification point in a preset identifier;
establishing an image coordinate system according to the positions of the identification points in the image, and determining the image coordinates of the identification points in the preset identifier;
establishing an actual physical coordinate system according to the position of each identification point in the actual physical world, and determining the actual physical coordinate of each identification point in the preset identifier;
and (3) enabling the image coordinates of each identification point to correspond to the actual physical coordinates one by one, and determining a homography transformation matrix through opencv.
After the distortion coefficient of the camera is determined and the camera is installed, the camera must be installed on the right side of the vehicle because the side parking is to calculate the distance from the right side of the vehicle to the roadside white line; after the camera is installed, the homography transformation matrix is determined, and the homography transformation matrix is determined after the camera is installed and fixed because the homography transformation matrix is related to the specific position and angle of the camera.
In the embodiment of the present invention, before the image including the roadside white line is captured, similarly taking the preset marker as the black and white checked cloth as an example, regarding the image after the distortion is removed as shown in fig. 3, all the interior points in the black and white checked cloth are taken as the identification points, and the identification points are marked and connected in sequence, as shown in fig. 4, where there are 54 interior points, that is, 54 identification points in total.
And finding out the positions of all the inner points in the black-and-white lattice cloth, taking the inner point at the leftmost upper corner as an origin, and taking the inner point to the right as the positive direction of the x axis and the negative direction as the positive direction of the y axis to construct a black-and-white lattice cloth coordinate system shown in figure 5.
Constructing the actual physical coordinates of the 54 interior points according to the actual physical size of each black and white grid in the black and white grid cloth, setting the actual physical size of each black and white grid to be 30mm x30 mm by taking millimeter (mm) as a unit, and marking the actual physical coordinates of the 54 interior points as the following actual physical coordinate array of 9 rows and 6 columns (for saving space, wherein 6 rows are not repeated and replaced by ellipses):
(0，0)，(30，0)，(60，0)，(90，0)，(120，0)，(150，0)
(0，30)，(30，30)，(60，30)，(90，30)，(120，30)，(150，30)
……
(0，240)，(30，240)，(60，240)，(90，240)，(120，240)，(150，240)
in the actual physical coordinate system, sequentially connecting all the inner points of the black-and-white grid cloth from left to right and from top to bottom to obtain an actual physical coordinate schematic diagram of all the inner points as shown in fig. 6;
according to the image after distortion removal shown in fig. 3 and the actual physical coordinate schematic diagram of all the interior points shown in fig. 6, the image coordinates of 54 interior points in the image in fig. 3 correspond to the actual physical coordinates of 54 interior points in the actual physical coordinate schematic diagram in fig. 6 one by one, and a homography transformation matrix of the image coordinates and the actual physical coordinates is calculated through a findhomographic function of opencv;
taking a black-and-white lattice as an example, the image with distortion removed shown in fig. 3 of the black-and-white lattice is subjected to a homography transformation matrix, and then an image after back projection transformation can be obtained, as shown in fig. 7; fig. 7 shows the image after the back projection transformation, which is equivalent to the result of directly performing parallel projection on the black and white scrim, without any perspective distortion, so that the resolution of the whole image (the resolution is the actual physical distance represented by one pixel, and the unit is mm/pixel) is uniform, and the physical distance corresponding to each pixel in the image is equal; since the size of each black and white cell in the image is 30 pixels x30 pixels, while the corresponding true physical size is 30mm x30 mm, it can be derived that the actual resolution of fig. 7 is 1 mm/pixel.
Step S102: when the vehicle is in a side-by-side parking state, acquiring an image acquired by a camera, detecting a roadside white line in the acquired image by adopting a semantic segmentation algorithm based on deep learning, and outputting a binary image; the method has the advantages that the roadside white lines in the collected images are detected by the aid of the semantic segmentation algorithm based on deep learning, the white line positions can be accurately detected at night, in rainy days and on partially worn road surfaces of the white lines, the white line position detection accuracy is improved, and meanwhile, practical application scenes are improved.
The basic definition of semantic segmentation is as follows: inputting an image, calculating a classification label for each pixel by an algorithm, namely judging which type of object each pixel belongs to; for the embodiment of the invention, which pixels belong to the white line and which pixels belong to other contents except the white line are found out from the image collected by the camera. For example, an image acquired by a camera is acquired, and after a roadside white line in the image is detected by adopting a semantic segmentation algorithm based on deep learning, the semantic segmentation algorithm outputs a binary image as shown in fig. 8, wherein the position of the roadside white line is marked by a white pixel, and other contents except the white line are filtered out as irrelevant contents.
Specifically, the detecting the white line on the roadside in the acquired image by using the semantic segmentation algorithm based on the deep learning in step S102, and outputting the binary image includes:
the semantic segmentation neural network model adopted by the semantic segmentation algorithm is a deplab semantic segmentation model based on expansion convolution; by using a deplab semantic segmentation model based on expansion convolution, the receptive field of the network can be improved, and the effect of integrating the large-range image features is achieved;
adjusting the number of layers and the number of characteristic channels of the neural network in the deplab semantic segmentation model to meet the requirement of the running speed of the vehicle when the vehicle stops at the side; adjusting parameters through multiple deep learning, including adjusting the number of layers and the number of characteristic channels of a neural network, and determining a neural network structure so as to meet the requirement of the running speed of the vehicle when the vehicle stops near the side;
detecting a roadside white line in an image acquired by the camera by using the adjusted deplab semantic segmentation model;
and displaying the position of the white line by using a white pixel, filtering other contents except the white line in the acquired image, and outputting a binary image.
Among them, the semantic segmentation neural network model is divided into two main categories: an encoder-decoder based semantic segmentation model and a deplab based semantic segmentation model based on dilation convolution.
For an encoder-decoder-based semantic segmentation model, as shown in fig. 9, the model is composed of two parts, firstly, an encoder branch is provided, so that the size of a feature map (image) is greatly reduced, and a large-range image feature can be integrated for integral object segmentation; then, a decoder branch is arranged to gradually restore the size (2x up) of the feature map, and the small-range detail features of the shallow network are utilized through transverse connection while the large-range image features of the deep network are utilized; the comprehensive feature map integrating the features of the large range and the small range is obtained finally, and the large object and the small object in the image can be segmented.
For the deplab semantic segmentation model based on the dilation convolution, as shown in fig. 10, in order to retain enough edge details of an object, the feature map is usually not reduced to a small resolution (small resolution) by a large margin, but after the network reaches a certain depth, a dilation convolution (also called hole convolution) is used to improve the receptive field of the network, so as to integrate a large range of image features.
After a comparison experiment is carried out on the performance based on the road test instrument, the deplab semantic segmentation model based on the expansion convolution is adopted, only one calculation branch is provided, and the calculation complexity is lower. The number of layers and the number of characteristic channels of the neural network are adjusted through multiple times of deep learning and parameter adjustment, the neural network structure is determined, the operation can be carried out on a road reference instrument at a frame rate of 15fps, and the operation speed requirement of parking at the side is met.
Step S103: removing lens distortion in the binary image based on all distortion coefficients and a distortion removal algorithm to obtain a distortion removal image; according to a binary image of an image containing a white line acquired by an installed camera, removing lens distortion in the binary image by a distortion removing algorithm according to five preloaded distortion coefficients to obtain a distortion removed image;
the above process is similar to the process of obtaining a undistorted image by undistorting a distorted image of a black and white check, which is a preset marker, and the difference is that the undistorted process of the black and white check image aims to obtain five distortion coefficients in advance so as to be used for obtaining a undistorted image of a binary image containing a white line image.
Step S104: carrying out back projection transformation on the distortion-removed image through the homography transformation matrix to obtain an image with uniform resolution; on the basis of the distortion removal, the purpose of the back projection transformation is to back-transform the image obtained by perspective projection transformation, and back-project the points in the two-dimensional plane image into a three-dimensional space.
Specifically, the step S104 includes:
carrying out back projection transformation on each identification point in the distortion-removed image to obtain the coordinate of each identification point in a three-dimensional space;
wherein, each identification point in the three-dimensional space is positioned on the plane of the road ground, and the Z-axis coordinate of each identification point in the three-dimensional space is constantly 0;
and determining the X-axis coordinate and the Y-axis coordinate of each identification point in the three-dimensional space according to each identification point in the distortion-removed image and the homography transformation matrix to obtain an image with uniform resolution.
In the embodiment of the invention, the back projection transformation is to back project points in a two-dimensional plane image into a three-dimensional space, and in contrast, the projection transformation is to project three-dimensional space coordinates onto a two-dimensional plane, and corresponding to an imaging process, one mathematical modeling of camera imaging is a pinhole imaging principle.
The principle of projection transformation provided by the embodiment of the present invention is shown in fig. 11, wherein center of projection is the origin of coordinates, image plane is the image plane, optical axis is the optical axis, the coordinates of a point Q in the three-dimensional space are (X, Y, Z), and the projection of the point Q on the two-dimensional plane (X, Y, Z) is calculated according to the following formula setscreen，yscreen)：
Wherein f isxIs the integrated parameter, f, after the focal length of the lens multiplied by the lateral resolution of the imageryIs the integral parameter of the focal length of the lens multiplied by the longitudinal resolution of the imager, cxIs a shift parameter in the transverse direction of the imaging plane, cyIs a shift parameter in the longitudinal direction of the imaging plane; in combination, the calculation of the above set of equations can be expressed as a matrix multiplication as follows:
wherein the content of the first and second substances,
is the homogeneous coordinate obtained by the projection,
is a matrix of projections of the image data,
is a coordinate point in three-dimensional space.
The projection transformation is that three-dimensional space coordinates are projected onto a two-dimensional plane, corresponding to the imaging process; and the back projection transformation is the inverse transformation of the two-dimensional plane coordinates, so that points in the three-dimensional space are obtained.
In the embodiment of the invention, the point in the three-dimensional space is always assumed to be positioned on the plane of the road ground, and the Z-axis coordinate is constantly 0, so that the point according to the two-dimensional plane can be inversely transformed back to obtain the point in the three-dimensional space; the Z-axis coordinate is known as 0, and the X-axis coordinate and the Y-axis coordinate are calculated only according to each identification point and the homography transformation matrix in the undistorted image to obtain an image with uniform resolution. Because the image after the back projection transformation is equivalent to the result of directly performing parallel projection, and there is no perspective deformation, the resolution of the whole image is uniform, and the physical distances corresponding to different pixels in the image are all equal, i.e. the obtained image has uniform resolution.
Step S105: and measuring the distance between the vehicle and the roadside white line when the vehicle is in a side-by-side parking state according to the resolution uniform image.
Specifically, the step S105 includes:
extracting the left edge of the white line in the resolution uniform image by using a gradient operator in the horizontal direction, and performing straight line fitting; the gradient operator in the horizontal direction may be a sobel operator;
and measuring the actual physical distance between the vehicle and the roadside white line according to the image resolution of the image with uniform resolution and the pixel number between the middle point of the fitted line segment and the left edge of the image after straight line fitting.
For a resolution uniform image obtained by subjecting a binary image to distortion removal and back projection conversion, extracting the left edge of a white line in the resolution uniform image by using a gradient operator in the horizontal direction, performing straight line fitting as shown in FIG. 12, determining the number of pixels between the middle point of a fitted line segment and the left edge of the image, and finally multiplying the number of pixels by the resolution of the image to obtain the actual physical distance between a vehicle and the roadside white line;
the fitted line segment can be not parallel to the left edge, and the distance from the midpoint of the line segment to the left edge of the image is taken; the vehicle body is parallel to the image edge in the image shot by the camera mounting position, and the image on the left side of the vehicle body is cut off, so that the left edge of the image, namely the vehicle body, is ensured, namely the left edge of the image is the position of the vehicle, and the distance from the midpoint of the line segment to the left edge after straight line fitting is the actual physical distance between the vehicle and the white line of the road edge.
From the above description, it can be seen that the present invention achieves the following technical effects:
firstly, the distortion coefficient and the homography transformation matrix of the lens are predetermined and are subsequently applied to the acquired image, so that the image processing efficiency is improved;
secondly, by adopting a semantic segmentation technology based on deep learning, compared with the traditional method based on image processing, the method can process more complex illumination, weather and road surface conditions, can accurately detect the white line position of the road surface in rainy days and at night, improves the accuracy of white line position detection, and simultaneously improves the practical application scene;
thirdly, lens distortion existing in camera imaging can be calculated and removed by adopting a camera calibration algorithm and a distortion removal algorithm;
fourthly, carrying out back projection transformation through the homography transformation matrix to obtain an image with uniform resolution, calculating the actual physical distance between any two points in the image, wherein the calculated actual distance has a low error, and the error is in centimeter level, so that accurate feedback can be provided for the trainee in the driving process.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than presented herein.
An embodiment of the present invention further provides a vehicle parking distance measuring apparatus based on deep learning, for implementing the vehicle parking distance measuring method based on deep learning, as shown in fig. 13, the apparatus includes:
the determining unit 131 is used for determining all distortion coefficients of a camera lens in advance through an opencv camera calibration algorithm, and determining a homography transformation matrix according to image coordinates and actual physical coordinates of each identification point in a preset identifier based on the installed camera;
the detection unit 132 is used for acquiring an image acquired by the camera when the vehicle is in a side-by-side parking state, detecting a white line on the roadside in the acquired image by adopting a semantic segmentation algorithm based on deep learning, and outputting a binary image;
a distortion removing unit 133, configured to remove lens distortion in the binary image based on all the distortion coefficients and a distortion removing algorithm, so as to obtain a distortion removed image;
a back projection transformation unit 134, configured to perform back projection transformation on the distortion-removed image through the homography transformation matrix to obtain an image with uniform resolution;
a measuring unit 135 for measuring a distance between the vehicle and the roadside white line when the vehicle is in a side parking state according to the resolution uniform image
An embodiment of the present invention further provides an electronic device, as shown in fig. 14, where the electronic device includes one or more processors 141 and a memory 142, and one processor 141 is taken as an example in fig. 14.
The controller may further include: an input device 143 and an output device 144.
The processor 141, the memory 142, the input device 143 and the output device 144 may be connected by a bus or other means, and fig. 3 illustrates the connection by a bus as an example.
The Processor 141 may be a Central Processing Unit (CPU), the Processor 141 may also be other general-purpose processors, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, or any combination thereof, and the general-purpose Processor may be a microprocessor or any conventional Processor.
The memory 142, which is a non-transitory computer readable storage medium, may be used for storing non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the control method in the embodiments of the present invention. The processor 141 executes various functional applications of the server and data processing by running non-transitory software programs, instructions and modules stored in the memory 142, that is, implements the deep learning-based vehicle parking ranging method of the above-described method embodiment.
The memory 142 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of a processing device operated by the server, and the like. Further, the memory 142 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 142 optionally includes memory located remotely from processor 141, which may be connected to a network connection device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 143 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the processing device of the server. The output device 144 may include a display device such as a display screen.
One or more modules are stored in memory 142 and, when executed by the one or more processors 141, perform the method illustrated in fig. 1.
Those skilled in the art will appreciate that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and the processes of the embodiments of the motor control methods described above can be included when the computer program is executed. The storage medium may be a magnetic disk, an optical disk, a Read-only memory (ROM), a Random Access Memory (RAM), a Flash Memory (FM), a hard disk (hard disk drive, HDD), or a Solid State Drive (SSD); the storage medium may also comprise a combination of memories of the kind described above.
Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope defined by the appended claims.
Claims (10)
1. A vehicle parking distance measurement method based on deep learning is characterized by comprising the following steps:
determining all distortion coefficients of a camera lens in advance through an opencv camera calibration algorithm, and determining a homography transformation matrix according to image coordinates and actual physical coordinates of each identification point in a preset identifier on the basis of an installed camera;
when the vehicle is in a side-by-side parking state, acquiring an image acquired by a camera, detecting a roadside white line in the acquired image by adopting a semantic segmentation algorithm based on deep learning, and outputting a binary image;
removing lens distortion in the binary image based on all distortion coefficients and a distortion removal algorithm to obtain a distortion removal image;
carrying out back projection transformation on the distortion-removed image through the homography transformation matrix to obtain an image with uniform resolution;
and measuring the distance between the vehicle and the roadside white line when the vehicle is in a side-by-side parking state according to the resolution uniform image.
2. The method according to claim 1, wherein the determining all distortion coefficients of the camera lens in advance through a camera calibration algorithm of opencv comprises:
the lens distortion caused by the camera comprises radial distortion and tangential distortion;
before the camera is installed, determining all distortion coefficients in the radial distortion and the tangential distortion in advance through a camera calibration algorithm of opencv;
all the determined distortion coefficients are saved as preload data.
3. The method of claim 2, wherein the radial distortion is described by the following system of equations:
xdistort＝x(1+k1r2+k2r4+k3r6)
ydistort＝y(1+k1r2+k2r4+k3r6)
the tangential distortion is described by the following system of equations:
xdistort2＝x+[2p1xy+p2(r2+2x2)]
Ydistort2＝y+[p1(r2+2y2)+2p2xy]
wherein x isdistortIs a transverse distortion of radial distortion, ydistortIs the longitudinal distortion, x, of the radial distortionsdistort2Is a transverse distortion in tangential distortion, ydistort2Is the longitudinal distortion in tangential distortion, x is the abscissa without distortion, y is the ordinate without distortion, r is the distance between each point on the image and the center of the image without distortion, k1、k2、k3、p1And p2Is five distortion coefficients in the lens distortion;
the five distortion coefficients are determined by the camera calibration algorithm of opencv.
4. The method according to claim 1, wherein the determining the homography transformation matrix according to the image coordinates and the actual physical coordinates of each identification point in the preset identifier comprises:
acquiring an image by using a camera, wherein the image comprises each identification point in a preset identifier;
establishing an image coordinate system according to the positions of the identification points in the image, and determining the image coordinates of the identification points in the preset identifier;
establishing an actual physical coordinate system according to the position of each identification point in the actual physical world, and determining the actual physical coordinate of each identification point in the preset identifier;
and (3) enabling the image coordinates of each identification point to correspond to the actual physical coordinates one by one, and determining a homography transformation matrix through opencv.
5. The method of claim 1, wherein the detecting the white line at the roadside in the captured image by adopting the semantic segmentation algorithm based on the deep learning and outputting the binary image comprises:
the semantic segmentation neural network model adopted by the semantic segmentation algorithm is a deplab semantic segmentation model based on expansion convolution;
adjusting the number of layers and the number of characteristic channels of the neural network in the deplab semantic segmentation model to meet the requirement of the running speed of the vehicle when the vehicle stops at the side;
detecting a roadside white line in an image acquired by the camera by using the adjusted deplab semantic segmentation model;
and displaying the position of the white line by using a white pixel, filtering other contents except the white line in the acquired image, and outputting a binary image.
6. The method of claim 1, wherein the back-projectively transforming the undistorted image with the homography transformation matrix to obtain a uniform-resolution image comprises:
carrying out back projection transformation on each identification point in the distortion-removed image to obtain the coordinate of each identification point in a three-dimensional space;
wherein, each identification point in the three-dimensional space is positioned on the plane of the road ground, and the Z-axis coordinate of each identification point in the three-dimensional space is constantly 0;
and determining the X-axis coordinate and the Y-axis coordinate of each identification point in the three-dimensional space according to each identification point in the distortion-removed image and the homography transformation matrix to obtain an image with uniform resolution.
7. The method of claim 1, wherein measuring a distance between the vehicle and a roadside white line when the vehicle is in a side-by-side parking state according to the resolution uniform image comprises:
extracting the left edge of the self line in the resolution uniform image by using a gradient operator in the horizontal direction, and performing straight line fitting;
and measuring the actual physical distance between the vehicle and the roadside white line according to the image resolution of the image with uniform resolution and the pixel number between the middle point of the fitted line segment and the left edge of the image after straight line fitting.
8. The utility model provides a vehicle parking range unit based on deep learning which characterized in that includes:
the determining unit is used for determining all distortion coefficients of a camera lens in advance through an opencv camera calibration algorithm, and determining a homography transformation matrix according to image coordinates and actual physical coordinates of each identification point in a preset identifier based on the installed camera;
the detection unit is used for acquiring an image acquired by the camera when the vehicle is in a side-by-side parking state, detecting a white line on the roadside in the acquired image by adopting a semantic segmentation algorithm based on deep learning, and outputting a binary image;
the distortion removing unit is used for removing lens distortion in the binary image based on all the distortion coefficients and a distortion removing algorithm to obtain a distortion removed image;
the back projection transformation unit is used for carrying out back projection transformation on the distortion-removed image through the homography transformation matrix to obtain an image with uniform resolution;
and the measuring unit is used for measuring the distance between the vehicle and the roadside white line when the vehicle is in a side parking state according to the resolution uniform image.
9. A computer-readable storage medium storing computer instructions for causing a computer to perform the deep learning-based vehicle parking ranging method of any one of claims 1 to 7.
10. An electronic device, characterized in that the electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores a computer program executable by the at least one processor to cause the at least one processor to perform the deep learning-based vehicle parking ranging method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110360717.7A CN113034583A (en) | 2021-04-01 | 2021-04-01 | Vehicle parking distance measuring method and device based on deep learning and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110360717.7A CN113034583A (en) | 2021-04-01 | 2021-04-01 | Vehicle parking distance measuring method and device based on deep learning and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113034583A true CN113034583A (en) | 2021-06-25 |
Family
ID=76453825
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110360717.7A Withdrawn CN113034583A (en) | 2021-04-01 | 2021-04-01 | Vehicle parking distance measuring method and device based on deep learning and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113034583A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113449648A (en) * | 2021-06-30 | 2021-09-28 | 北京纵目安驰智能科技有限公司 | Method, system, equipment and computer readable storage medium for detecting indicator line |
| CN116758063A (en) * | 2023-08-11 | 2023-09-15 | 南京航空航天大学 | Workpiece size detection method based on image semantic segmentation |
-
2021
- 2021-04-01 CN CN202110360717.7A patent/CN113034583A/en not_active Withdrawn
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113449648A (en) * | 2021-06-30 | 2021-09-28 | 北京纵目安驰智能科技有限公司 | Method, system, equipment and computer readable storage medium for detecting indicator line |
| CN116758063A (en) * | 2023-08-11 | 2023-09-15 | 南京航空航天大学 | Workpiece size detection method based on image semantic segmentation |
| CN116758063B (en) * | 2023-08-11 | 2023-11-07 | 南京航空航天大学 | Workpiece size detection method based on image semantic segmentation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110285793B (en) | Intelligent vehicle track measuring method based on binocular stereo vision system | |
| CN110322702B (en) | Intelligent vehicle speed measuring method based on binocular stereo vision system | |
| CN112270713B (en) | Calibration method and device, storage medium and electronic device | |
| CN107133985B (en) | Automatic calibration method for vehicle-mounted camera based on lane line vanishing point | |
| CN108399403B (en) | Vehicle distance detection method based on license plate size calculation | |
| CN105718872B (en) | Auxiliary method and system for rapidly positioning lanes on two sides and detecting vehicle deflection angle | |
| CN112902874B (en) | Image acquisition device and method, image processing method and device and image processing system | |
| CN111209780A (en) | Lane line attribute detection method and device, electronic device and readable storage medium | |
| CN111141311B (en) | Evaluation method and system of high-precision map positioning module | |
| CN109741241B (en) | Fisheye image processing method, device, equipment and storage medium | |
| CN104021588A (en) | System and method for recovering three-dimensional true vehicle model in real time | |
| CN113034583A (en) | Vehicle parking distance measuring method and device based on deep learning and electronic equipment | |
| CN111738033B (en) | Vehicle driving information determination method and device based on plane segmentation and vehicle-mounted terminal | |
| CN112330755B (en) | Calibration evaluation method and device of all-round system, storage medium and terminal | |
| CN111243034A (en) | Panoramic auxiliary parking calibration method, device, equipment and storage medium | |
| CN106803073B (en) | Auxiliary driving system and method based on stereoscopic vision target | |
| CN112115737B (en) | Vehicle orientation determining method and device and vehicle-mounted terminal | |
| CN114724119B (en) | Lane line extraction method, lane line detection device, and storage medium | |
| CN115546216A (en) | Tray detection method, device, equipment and storage medium | |
| CN113012239B (en) | Quantitative calculation method for focal length change of vehicle-road cooperative roadside perception camera | |
| CN114428259A (en) | Automatic vehicle extraction method in laser point cloud of ground library based on map vehicle acquisition | |
| CN115294211A (en) | Vehicle-mounted camera installation external parameter calibration method, system, device and storage medium | |
| CN112184605A (en) | Method, equipment and system for enhancing vehicle driving visual field | |
| CN111553181A (en) | Vehicle-mounted camera semantic recognition method, system and device | |
| CN113643374A (en) | Multi-view camera calibration method, device, equipment and medium based on road characteristics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210625 |