CN111860349A - Intelligent vehicle lane line semantic segmentation method and system - Google Patents
Intelligent vehicle lane line semantic segmentation method and system Download PDFInfo
- Publication number
- CN111860349A CN111860349A CN202010716801.3A CN202010716801A CN111860349A CN 111860349 A CN111860349 A CN 111860349A CN 202010716801 A CN202010716801 A CN 202010716801A CN 111860349 A CN111860349 A CN 111860349A
- Authority
- CN
- China
- Prior art keywords
- feature extraction
- image
- convolution
- layer
- intelligent vehicle
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
Abstract
The invention provides a method and a system for semantic segmentation of lane lines of an intelligent vehicle, wherein the method comprises the following steps: selecting a frame of picture in a video acquired by an intelligent vehicle in the driving process as an input picture; inputting the input picture into a feature extraction network for feature extraction to obtain a first layer of feature extraction picture and a last layer of feature extraction picture of the feature extraction network; performing preset times of upsampling operation on the last layer of feature extraction image, combining the upsampling operation with the first layer of feature extraction image and performing convolution operation to obtain convoluted image information; performing preset times of upsampling operation on the feature extraction image output by the last layer, combining the feature extraction image of the first layer and the image information after convolution, and performing convolution operation to obtain the image information after the second convolution; and performing preset-time up-sampling operation on the image data after the second convolution, and outputting a semantic segmentation image of the lane line. The method fully utilizes the existing hardware resources, ensures the prediction accuracy of the image pixel points, and does not increase the hardware cost.
Description
Technical Field
The invention relates to the field of computer vision, in particular to a method and a system for intelligently segmenting lane line semantics of a vehicle, and more particularly to a method for achieving lane line semantics segmentation based on redundant feature extraction information.
Background
The semantic segmentation of the lane lines is the application of a computer vision technology in the field of intelligent vehicles, and assists the intelligent vehicles in identifying the lane lines and constructing and updating high-precision maps in the driving process.
The core of the lane line semantic segmentation technology is to realize classification of the category of each pixel point on a picture for one frame of picture acquired by an intelligent vehicle in the driving process.
The success of convolution operation in the aspect of feature extraction promotes the application of a convolution neural network in the field of semantic segmentation. The neural network of the current coding-decoding architecture is widely applied to the image pixel point classification problem, wherein the coding part extracts important characteristic information in a picture through a series of convolution operations, and the decoding part constructs a final semantic segmentation output picture from the important characteristic information through a series of upsampling operations.
Although the neural network of the encoding-decoding architecture can solve the problem of image pixel point classification to a certain extent, the problem that image pixel point classification errors are troubling the semantic segmentation technology to be better applied to the field of intelligent vehicles is solved.
Patent document CN109766878A (application number: 201910287099.0) discloses a method and device for detecting lane lines, which relate to the technical field of automatic driving and are used for solving the problems of low accuracy and poor robustness of current lane line detection, and patent document CN109766878A discloses that the maximum height value, the average reflection intensity and the point cloud number statistical density of a grid in a bird's-eye view feature map are used as the input of darknet for feature extraction; determining the characteristic information of the lane line points by fusing the high-resolution of the low-level characteristics and the high-semantic information of the high-level characteristics through the FPN; determining lane line points corresponding to the lane line points in the aerial view feature map in the point cloud map according to the feature information; and taking the lane line points with the reflection intensity larger than the average reflection intensity threshold value in the lane line points in the point cloud map as lane line characteristic points, and performing geometric model fitting according to the lane line characteristic points in the point cloud map to determine lane lines.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a method and a system for intelligently segmenting lane line semantics of a vehicle.
The method for semantically segmenting the lane lines of the intelligent vehicle, provided by the invention, comprises the following steps:
step M1: selecting a frame of picture in a video acquired by an intelligent vehicle in the driving process as an input picture;
step M2: inputting the input picture into a feature extraction network for feature extraction, and acquiring a first-layer feature extraction picture and a last-layer feature extraction picture of the feature extraction network;
step M3: performing preset times of upsampling operation on the last layer of feature extraction image, combining the upsampling operation with the first layer of feature extraction image and performing convolution operation to obtain convoluted image information;
step M4: performing preset times of upsampling operation on the feature extraction image output by the last layer, combining the feature extraction image of the first layer and the image information after convolution, and performing convolution operation to obtain the image information after the second convolution;
step M5: and performing preset-time up-sampling operation on the image data after the second convolution, and outputting a semantic segmentation image of the lane line.
Preferably, the feature extraction network comprises x layers of convolution layers and progressively extracts the information features between the pixels in the image.
Preferably, the feature extraction information output from the first layer and the last layer of the feature extraction network respectively performs convolution operations with convolution kernels of 1 × 1.
Preferably, the preset-times upsampling operation comprises a bilinear interpolation operation.
Preferably, the merging includes stitching the feature data tensors based on the number of channels.
Preferably, the convolution operation does not change the height and width of the image data.
The invention provides a system for semantically segmenting a lane line of an intelligent vehicle, which comprises the following steps:
module M1: selecting a frame of picture in a video acquired by an intelligent vehicle in the driving process as an input picture;
module M2: inputting the input picture into a feature extraction network for feature extraction, and acquiring a first-layer feature extraction picture and a last-layer feature extraction picture of the feature extraction network;
module M3: performing preset times of upsampling operation on the last layer of feature extraction image, combining the upsampling operation with the first layer of feature extraction image and performing convolution operation to obtain convoluted image information;
module M4: performing preset times of upsampling operation on the feature extraction image output by the last layer, combining the feature extraction image of the first layer and the image information after convolution, and performing convolution operation to obtain the image information after the second convolution;
module M5: and performing preset-time up-sampling operation on the image data after the second convolution, and outputting a semantic segmentation image of the lane line.
Preferably, the feature extraction network comprises x layers of convolution layers and progressively extracts the information features between the pixels in the image.
Preferably, the preset-times upsampling operation comprises a bilinear interpolation operation.
Preferably, the merging includes stitching the feature data tensors by using the number of channels as a standard;
the convolution operation does not change the height and width of the image data.
Compared with the prior art, the invention has the following beneficial effects:
1. the invention fully utilizes the existing hardware resources, ensures the prediction accuracy of the image pixel points and does not generate the prediction error
The overhead of hardware is increased;
2. the invention fully utilizes the existing software resources, and the difficulty of software development is not additionally increased;
3. compared with the prior art, the method has the advantages that the prediction accuracy of the image pixel points is improved, and meanwhile, the calculation cost is reduced;
4. is beneficial to falling to the ground and being commercialized.
5. The method fully considers the importance of the information contained in the first layer characteristic diagram and the last quadruple up-sampling characteristic diagram, and improves the semantic segmentation accuracy of the image through the redundant operation of characteristic diagram combination.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
FIG. 1 is a flow chart of the method of the present invention
FIG. 2 is a network framework diagram of the present invention
FIG. 3 is a perspective transformed image of the present invention
FIG. 4 is a semantic segmentation output image of the present invention
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that it would be obvious to those skilled in the art that various changes and modifications can be made without departing from the spirit of the invention. All falling within the scope of the present invention.
In order to improve the accuracy of image pixel point classification, the invention provides a method for realizing lane line semantic segmentation based on redundant feature extraction information, which is used for applying the image feature information extracted by a neural network coding part to a decoding part of a neural network for multiple times, so that the accuracy of image pixel point classification can be effectively improved, and the requirements of intelligent vehicle driving and high-precision map construction and updating are met.
Example 1
The method for semantically segmenting the lane lines of the intelligent vehicle, provided by the invention, comprises the following steps:
step M1: selecting a frame of picture in a video acquired by an intelligent vehicle in the driving process as an input picture; carrying out corresponding perspective transformation on the input image to obtain a road surface picture at a proper angle;
step M2: inputting the road surface picture after perspective transformation into a feature extraction network for feature extraction, and acquiring a first layer feature extraction picture and a last layer feature extraction picture of the feature extraction network;
step M3: performing quadruple up-sampling operation on the last layer of feature extraction image, combining the last layer of feature extraction image with the first layer of feature extraction image, and performing convolution operation to obtain convoluted image information;
step M4: performing quadruple up-sampling operation on the feature extraction image output by the last layer, combining the feature extraction image output by the last layer with the feature extraction image output by the first layer and the convolved image information, and performing convolution operation to obtain the image information after the second convolution;
step M5: and performing quadruple up-sampling operation on the image data subjected to the second convolution, and outputting a semantic segmentation image of the lane line.
The feature extraction network is a convolutional neural network composed of convolution operation, ReLU, batch standardization and the like and used for extracting feature information in the picture.
Specifically, the feature extraction network comprises x layers of convolution layers and progressively extracts information features among pixels in the image.
Specifically, the feature extraction information output by the first layer and the last layer of the feature extraction network respectively performs convolution operation with a convolution kernel of 1 × 1, so that the workload of calculation is reduced, and the memory space overhead of the memory is reduced.
In particular, the quadruple upsampling operation comprises a bilinear interpolation operation or other alternative quadruple upsampling operation.
Specifically, the merging includes stitching the feature data tensors using the number of channels as a standard.
Specifically, the convolution operation does not change the height and width of the image data.
The invention provides a system for semantically segmenting a lane line of an intelligent vehicle, which comprises the following steps:
module M1: selecting a frame of picture in a video acquired by an intelligent vehicle in the driving process as an input picture; carrying out corresponding perspective transformation on the input image to obtain a road surface picture at a proper angle;
module M2: inputting the road surface picture after perspective transformation into a feature extraction network for feature extraction, and acquiring a first layer feature extraction picture and a last layer feature extraction picture of the feature extraction network;
module M3: performing quadruple up-sampling operation on the last layer of feature extraction image, combining the last layer of feature extraction image with the first layer of feature extraction image, and performing convolution operation to obtain convoluted image information;
module M4: performing quadruple up-sampling operation on the feature extraction image output by the last layer, combining the feature extraction image output by the last layer with the feature extraction image output by the first layer and the convolved image information, and performing convolution operation to obtain the image information after the second convolution;
module M5: and performing quadruple up-sampling operation on the image data subjected to the second convolution, and outputting a semantic segmentation image of the lane line.
The feature extraction network is a convolutional neural network composed of convolution operation, ReLU, batch standardization and the like and used for extracting feature information in the picture.
Specifically, the feature extraction network comprises x layers of convolution layers and progressively extracts information features among pixels in the image.
Specifically, the feature extraction information output by the first layer and the last layer of the feature extraction network respectively performs convolution operation with a convolution kernel of 1 × 1, so that the workload of calculation is reduced, and the memory space overhead of the memory is reduced.
In particular, the quadruple upsampling operation comprises a bilinear interpolation operation or other alternative quadruple upsampling operation.
Specifically, the merging includes stitching the feature data tensors using the number of channels as a standard.
Specifically, the convolution operation does not change the height and width of the image data.
Example 2
Example 2 is a modification of example 1
The invention provides a method for intelligently segmenting the lane line semantics of a vehicle, which fully utilizes the importance and the redundancy of the feature information of the first layer and the last layer of a feature extraction network to realize the improvement of the lane line semantics segmentation accuracy. In the present embodiment, the semantic segmentation of a single image is taken as an example, and as shown in fig. 1 to 4, the semantic segmentation method includes the following steps:
step 1: one frame of picture collected in a video in the driving process of the intelligent vehicle is used as an original image, and perspective transformation is carried out on the original image to obtain an input image of the concerned road information. And in the process of carrying out perspective transformation processing on the original image, adjusting the size of the image.
Step 2: inputting the perspective transformation image obtained in the step 1 into a feature extraction network; the feature extraction network is composed of six convolutional layers, and extraction of important information among pixel points in the image is achieved.
And step 3: and (3) acquiring image data of the first layer and the last layer of the feature extraction network based on the step (2).
And 4, step 4: the size of the image data of the last layer after performing the convolution operation is 256 × 32, and the data size after performing the four times up-sampling operation is 256 × 128.
And 5: the data size after the convolution operation is performed on the image data of the first layer is 48 × 128.
Step 6: and (5) splicing the image data of the step 5 and the image data of the step 6 to obtain the data size of 304 x 128.
And 7: the data size after performing the convolution operation on the image data of step 6 is 256 × 128.
And 8: and (5) splicing the image data of the step 7 with the image data of the step 4 and the step 5 to obtain the data size of 560 × 128.
And step 9: the data size after performing the convolution operation on the image data of step 8 is 256 × 128.
Step 10: the data size after performing the convolution operation on the image data of step 8 is 9 × 128.
Step 11: the data size after performing the four times up-sampling operation on the image data of step 10 is 9 x 512.
Step 12: a semantically segmented output picture is obtained from the image data of step 11.
In the description of the present application, it is to be understood that the terms "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like indicate orientations or positional relationships based on those shown in the drawings, and are only for convenience in describing the present application and simplifying the description, but do not indicate or imply that the referred device or element must have a specific orientation, be constructed in a specific orientation, and be operated, and thus, should not be construed as limiting the present application.
Those skilled in the art will appreciate that, in addition to implementing the systems, apparatus, and various modules thereof provided by the present invention in purely computer readable program code, the same procedures can be implemented entirely by logically programming method steps such that the systems, apparatus, and various modules thereof are provided in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system, the device and the modules thereof provided by the present invention can be considered as a hardware component, and the modules included in the system, the device and the modules thereof for implementing various programs can also be considered as structures in the hardware component; modules for performing various functions may also be considered to be both software programs for performing the methods and structures within hardware components.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes or modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The embodiments and features of the embodiments of the present application may be combined with each other arbitrarily without conflict.
Claims (10)
1. A method for semantic segmentation of lane lines of an intelligent vehicle is characterized by comprising the following steps:
step M1: selecting a frame of picture in a video acquired by an intelligent vehicle in the driving process as an input picture;
step M2: inputting the input picture into a feature extraction network for feature extraction, and acquiring a first-layer feature extraction picture and a last-layer feature extraction picture of the feature extraction network;
step M3: performing preset times of upsampling operation on the last layer of feature extraction image, combining the upsampling operation with the first layer of feature extraction image and performing convolution operation to obtain convoluted image information;
step M4: performing preset times of upsampling operation on the feature extraction image output by the last layer, combining the feature extraction image of the first layer and the image information after convolution, and performing convolution operation to obtain the image information after the second convolution;
step M5: and performing preset-time up-sampling operation on the image data after the second convolution, and outputting a semantic segmentation image of the lane line.
2. The method for intelligent vehicle lane line semantic segmentation according to claim 1, wherein the feature extraction network comprises x layers of convolutional layers, and the inter-pixel information features in the image are extracted progressively.
3. The method for intelligent vehicle lane line semantic segmentation as claimed in claim 1, wherein the feature extraction information output by the first layer and the last layer of the feature extraction network respectively performs convolution operation with convolution kernel of 1x 1.
4. The method of intelligent vehicle lane line semantic segmentation as claimed in claim 1, wherein the preset-multiple upsampling operation comprises a bilinear interpolation operation.
5. The method for semantically segmenting the lane lines of the intelligent vehicle according to claim 1, wherein the performing the pre-doubled upsampling operation on the feature extraction map of the last layer, merging with the feature extraction map of the first layer, performing the merging in the convolution operation, and performing the pre-doubled upsampling operation on the feature extraction map output by the last layer, merging with the feature extraction map of the first layer and the image information after the convolution, and performing the merging in the convolution operation includes stitching feature data tensors with the number of channels as a standard.
6. The method of intelligent vehicle lane line semantic segmentation as claimed in claim 1 wherein the convolution operation does not change the height and width of the image data.
7. A system for intelligent vehicle lane line semantic segmentation, comprising:
module M1: selecting a frame of picture in a video acquired by an intelligent vehicle in the driving process as an input picture;
module M2: inputting the input picture into a feature extraction network for feature extraction, and acquiring a first-layer feature extraction picture and a last-layer feature extraction picture of the feature extraction network;
module M3: performing preset times of upsampling operation on the last layer of feature extraction image, combining the upsampling operation with the first layer of feature extraction image and performing convolution operation to obtain convoluted image information;
module M4: performing preset times of upsampling operation on the feature extraction image output by the last layer, combining the feature extraction image of the first layer and the image information after convolution, and performing convolution operation to obtain the image information after the second convolution;
module M5: and performing preset-time up-sampling operation on the image data after the second convolution, and outputting a semantic segmentation image of the lane line.
8. The system for intelligent vehicle lane line semantic segmentation according to claim 7, wherein the feature extraction network comprises x layers of convolutional layers, and the inter-pixel information features in the image are extracted progressively.
9. The system for intelligent vehicle lane line semantic segmentation as claimed in claim 7, wherein the preset-multiple upsampling operation comprises a bilinear interpolation operation.
10. The system for semantically segmenting the lane lines of the intelligent vehicle according to claim 7, wherein the performing the pre-doubled upsampling operation on the feature extraction map of the last layer, merging with the feature extraction map of the first layer, performing the merging in the convolution operation, and performing the pre-doubled upsampling operation on the feature extraction map output by the last layer, merging with the feature extraction map of the first layer and the image information after the convolution, and performing the merging in the convolution operation includes stitching feature data tensors with the number of channels as a standard;
the convolution operation does not change the height and width of the image data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010716801.3A CN111860349A (en) | 2020-07-23 | 2020-07-23 | Intelligent vehicle lane line semantic segmentation method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010716801.3A CN111860349A (en) | 2020-07-23 | 2020-07-23 | Intelligent vehicle lane line semantic segmentation method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111860349A true CN111860349A (en) | 2020-10-30 |
Family
ID=72949344
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010716801.3A Pending CN111860349A (en) | 2020-07-23 | 2020-07-23 | Intelligent vehicle lane line semantic segmentation method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111860349A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112785610A (en) * | 2021-01-14 | 2021-05-11 | 华南理工大学 | Lane line semantic segmentation method fusing low-level features |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109034162A (en) * | 2018-07-13 | 2018-12-18 | 南京邮电大学 | A kind of image, semantic dividing method |
| CN110188768A (en) * | 2019-05-09 | 2019-08-30 | 南京邮电大学 | Realtime graphic semantic segmentation method and system |
-
2020
- 2020-07-23 CN CN202010716801.3A patent/CN111860349A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109034162A (en) * | 2018-07-13 | 2018-12-18 | 南京邮电大学 | A kind of image, semantic dividing method |
| CN110188768A (en) * | 2019-05-09 | 2019-08-30 | 南京邮电大学 | Realtime graphic semantic segmentation method and system |
Non-Patent Citations (1)
| Title |
|---|
| 徐国晟;张伟伟;吴训成;苏金亚;郭增高;: "基于卷积神经网络的车道线语义分割算法", 电子测量与仪器学报, no. 07 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112785610A (en) * | 2021-01-14 | 2021-05-11 | 华南理工大学 | Lane line semantic segmentation method fusing low-level features |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021093435A1 (en) | Semantic segmentation network structure generation method and apparatus, device, and storage medium | |
| CN111259758B (en) | Two-stage remote sensing image target detection method for dense area | |
| CN106971155B (en) | Unmanned vehicle lane scene segmentation method based on height information | |
| CN111524150B (en) | Image processing method and device | |
| CN110781980B (en) | Training method of target detection model, target detection method and device | |
| CN110706269B (en) | Binocular vision SLAM-based dynamic scene dense modeling method | |
| CN111027539A (en) | License plate character segmentation method based on spatial position information | |
| CN111563516B (en) | Method, terminal and storage medium for fusion display of pedestrian mask and three-dimensional scene | |
| CN111914698A (en) | Method and system for segmenting human body in image, electronic device and storage medium | |
| WO2023082453A1 (en) | Image processing method and device | |
| CN113343778A (en) | Lane line detection method and system based on LaneSegNet | |
| CN116453121B (en) | Training method and device for lane line recognition model | |
| CN114820679B (en) | Image labeling method and device electronic device and storage medium | |
| CN114529773A (en) | Form identification method, system, terminal and medium based on structural unit | |
| CN116129291A (en) | Unmanned aerial vehicle animal husbandry-oriented image target recognition method and device | |
| CN115376089A (en) | Deep learning-based lane line detection method | |
| Dong et al. | EGFNet: Edge-aware guidance fusion network for RGB–thermal urban scene parsing | |
| CN116433903A (en) | Instance segmentation model construction method, system, electronic equipment and storage medium | |
| CN111860349A (en) | Intelligent vehicle lane line semantic segmentation method and system | |
| CN114419058A (en) | Image semantic segmentation model training method for traffic road scene | |
| CN114283343A (en) | Map updating method, training method and equipment based on remote sensing satellite image | |
| CN113762396A (en) | Two-dimensional image semantic segmentation method | |
| CN114120284A (en) | Deep learning highway lane structuring method, storage medium and device | |
| CN115578721A (en) | Streetscape text real-time detection method based on attention feature fusion | |
| CN112488115B (en) | Semantic segmentation method based on two-stream architecture |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |