WO2024121880A1

WO2024121880A1 - Information processing device, information processing method, and computer-readable medium

Info

Publication number: WO2024121880A1
Application number: PCT/JP2022/044666
Authority: WO
Inventors: 匡孝西田
Original assignee: 日本電気株式会社
Priority date: 2022-12-05
Filing date: 2022-12-05
Publication date: 2024-06-13

Abstract

Provided is an information processing device (10) comprising: a recognition unit (11) that recognizes an area of a lane line of a road from a first image in which the road around a vehicle is captured; a generation unit (12) that generates a second image in which the recognized area of the lane line of the road is represented on a coordinate plane having the longitudinal direction and horizontal direction of the vehicle as two axes; and an estimation unit (13) that estimates a graphic of a line representing the lane line of the road on the basis of the second image.

Description

Information processing device, information processing method, and computer-readable medium

The present disclosure relates to an information processing device, an information processing method, and a non-transitory computer-readable medium on which a program is stored.

Patent Document 1 discloses a technology that, when there are multiple lane candidates that are line candidates and whose brightness is determined to be equal to or greater than a brightness threshold, recognizes the lane candidate that is closest to the vehicle among the multiple lane candidates as the lane marking line. This makes it possible to address the issue that, when multiple line candidates with different brightness such as white lines and yellow lines are detected, there is a risk that only the white lines will always be recognized.

JP 2019-20957 A

However, the technology described in Patent Document 1 may not be able to properly detect lane markings depending on their shape, for example, when the lane marking (e.g., center line, lane boundary line) indicating one side of the lane in which the vehicle is traveling is dashed.

In view of the above-mentioned problems, the objective of the present disclosure is to provide an information processing device, an information processing method, and a non-transitory computer-readable medium on which a program is stored that can properly detect lane markings.

In a first aspect of the present disclosure, an information processing device is provided that includes a recognition unit that recognizes an area of the road's dividing line from a first image of the road around a vehicle, a generation unit that generates a second image in which the recognized area of the road's dividing line is represented on a coordinate plane having the longitudinal and lateral directions of the vehicle as two axes, and an estimation unit that estimates linear figures representing the road's dividing line based on the second image.

In addition, a second aspect of the present disclosure provides an information processing method that recognizes an area of the road's dividing line from a first image of the road around a vehicle, generates a second image that represents the recognized area of the road's dividing line on a coordinate plane having the longitudinal and lateral directions of the vehicle as two axes, and estimates a linear figure representing the road's dividing line based on the second image.

In addition, in a third aspect of the present disclosure, a non-transitory computer-readable medium is provided that stores a program for causing a computer to execute a process of recognizing an area of road dividing lines from a first image of a road around a vehicle, generating a second image in which the recognized area of road dividing lines is represented on a coordinate plane having the longitudinal and lateral directions of the vehicle as two axes, and estimating linear figures representing the road dividing lines based on the second image.

In one aspect, lane markings can be properly detected.

FIG. 1 is a diagram illustrating an example of a configuration of an information processing device according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device according to an embodiment. 10 is a flowchart illustrating an example of processing of the information processing device according to the embodiment. FIG. 2 is a diagram showing an example of a captured image according to the embodiment; FIG. 13 is a diagram showing an example of a captured image after threshold processing according to the embodiment. 11 is a diagram illustrating an example of a binarized image in which a dashed line portion of a demarcation line is detected according to the embodiment; FIG. FIG. 13 is a diagram showing an example of a dashed demarcation line on a bird's-eye view according to an embodiment; FIG. 2 is a diagram illustrating an example of lanes on a bird's-eye view according to the embodiment; FIG. 4 is a diagram illustrating an example of an output image according to the embodiment.

The principles of the present disclosure are described with reference to some exemplary embodiments. It should be understood that these embodiments are set forth for illustrative purposes only, to aid those skilled in the art in understanding and practicing the present disclosure, without implying any limitation on the scope of the present disclosure. The disclosure described herein may be implemented in various ways other than those described below.
In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.
Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

(Embodiment 1)
<Configuration>
The configuration of an information processing device 10 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the information processing device 10 according to an embodiment. The information processing device 10 has a recognition unit 11, a generation unit 12, and an estimation unit 13. Each of these units may be realized by cooperation between one or more programs installed in the information processing device 10 and hardware such as a processor and a memory of the information processing device 10.

The recognition unit 11 recognizes the area of the road dividing lines from a first image of the road around the vehicle. The generation unit 12 generates a second image in which the area of the road dividing lines recognized by the recognition unit 11 is represented on a coordinate plane with the vehicle's vertical direction (e.g., forward direction) and horizontal direction as two axes. The estimation unit 13 estimates linear figures representing the road dividing lines based on the second image.

(Embodiment 2)
<Hardware Configuration>
Fig. 2 is a diagram showing an example of a hardware configuration of an information processing device 10 according to an embodiment. In the example of Fig. 2, the information processing device 10 (computer 100) includes a processor 101, a memory 102, and a communication interface 103. These units may be connected by a bus or the like. The memory 102 stores at least a part of a program 104. The communication interface 103 includes an interface required for communication with other network elements.

When the program 104 is executed by the processor 101, the memory 102, etc. in cooperation with each other, the computer 100 performs at least some of the processing of the embodiments of the present disclosure. The memory 102 may be of any type. The memory 102 may be, as a non-limiting example, a non-transitory computer-readable storage medium. The memory 102 may also be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. Although only one memory 102 is shown in the computer 100, there may be several physically different memory modules in the computer 100. The processor 101 may be of any type. The processor 101 may include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture, as a non-limiting example. The computer 100 may have multiple processors, such as application-specific integrated circuit chips that are time-slaved to a clock that synchronizes the main processor.

Embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device.

The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, that execute on a target real or virtual processor device to perform the processes or methods of the present disclosure. Program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or divided among program modules as desired in various embodiments. The machine-executable instructions of the program modules may be executed in local or distributed devices. In a distributed device, the program modules may be located in both local and remote storage media.

Program codes for carrying out the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes are provided to a processor or controller of a general purpose computer, a special purpose computer, or other programmable data processing apparatus. When the program code is executed by the processor or controller, the functions/operations in the flowcharts and/or implementing block diagrams are performed. The program code may be executed entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine, or entirely on a remote machine or server.

The program may be stored and supplied to the computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible recording media. Examples of non-transitory computer-readable media include magnetic recording media, magneto-optical recording media, optical disk media, semiconductor memory, etc. Magnetic recording media include, for example, flexible disks, magnetic tapes, hard disk drives, etc. Magneto-optical recording media include, for example, magneto-optical disks, etc. Optical disk media include, for example, Blu-ray disks, CD (Compact Disc)-ROM (Read Only Memory), CD-R (Recordable), CD-RW (ReWritable), etc. Semiconductor memories include, for example, solid-state drives, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (random access memory), etc. The program may also be supplied to the computer by various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can provide the program to the computer via a wired communication path, such as an electric wire or optical fiber, or via a wireless communication path.

<Processing>
Next, an example of processing of the information processing device 10 according to the embodiment will be described with reference to Figs. 3 to 9. Fig. 3 is a flowchart showing an example of processing of the information processing device 10 according to the embodiment. Fig. 4 is a diagram showing an example of a captured image according to the embodiment. Fig. 5 is a diagram showing an example of a captured image after threshold processing according to the embodiment. Fig. 6 is a diagram showing an example of a binarized image in which dashed line portions of a demarcation line according to the embodiment are detected. Fig. 7 is a diagram showing an example of dashed lines of a demarcation line on a bird's-eye view according to the embodiment. Fig. 8 is a diagram showing an example of lanes on a bird's-eye view according to the embodiment. Fig. 9 is a diagram showing an example of an output image according to the embodiment.

In step S101, the recognition unit 11 acquires a still image (first image) captured by a camera. Here, the recognition unit 11 may acquire, for example, one frame of a video captured by a monocular camera mounted on a vehicle. In the example of FIG. 4, the recognition unit 11 acquires a captured image 401 of the road in the forward direction (the direction of travel when traveling straight) of the vehicle (hereinafter also referred to as "own vehicle" as appropriate) on which the camera is mounted.

Then, the recognition unit 11 recognizes dashed areas that are dividing lines (e.g., center lines, lane boundaries) that indicate one side of the road lanes within the area in the first image in which the road ahead of the vehicle is photographed (step S102). Here, the recognition unit 11 may perform threshold processing based on the brightness of each pixel, for example. In this case, the recognition unit 11 may convert pixel values in a specific range of brightness into a specific color, for example. This makes it possible to delete unnecessary background information and emphasize contours, for example. The recognition unit 11 may then recognize dashed areas that are dividing lines, for example, based on the contours of areas in the image.

In the example of Fig. 5, image 501 is generated by subjecting captured image 401 of Fig. 4 to threshold processing, in which the contours of the dashed line areas that are demarcation lines are emphasized. In the example of Fig. 6, a binarized image 601 is generated based on image 501 of Fig. 5, in which the dashed line areas of the demarcation lines are distinguished from other areas.

Then, based on the recognition result by the recognition unit 11, the generation unit 12 generates a bird's-eye view (bird's-eye view; "second image") in which the recognized dashed lines are mapped along two axes, the forward direction and the horizontal direction of the vehicle (step S103). Here, the generation unit 12 generates a bird's-eye view in which the values of the two axes correspond to the distance from the vehicle in each of the directions of the two axes in real space. The bird's-eye view may be, for example, a view looking down vertically, or a view looking down diagonally from above the vehicle.

Here, the generation unit 12 may generate the bird's-eye view by, for example, converting the coordinates in the captured image of the pixels in the dashed area of the demarcation line into coordinates in the bird's-eye view. In this case, for example, when the operator installs the camera on the vehicle, the camera may be attached so that the optical axis direction of the camera (center of the image) coincides with the forward direction of the vehicle. In addition, information on the shooting conditions such as the camera's angle of view may be registered in the information processing device by the operator. Then, the generation unit 12 may determine a mathematical formula for converting the coordinates of each pixel in the captured image into the forward position and horizontal position of the vehicle in real space based on the information on the shooting conditions such as the camera's angle of view.

Furthermore, when the system is installed on the vehicle by the operator, the forward and horizontal positions of the vehicle in real space may be registered for each of the coordinates of multiple points in the captured image. In this case, the multiple points may be, for example, the four corner points of a trapezoid whose upper base is shorter than its lower base when the far side in the forward direction of the vehicle is taken as the upper base and the near side as the lower base. The generation unit 12 may then determine a mathematical formula for converting the coordinates of each pixel in the captured image into the forward and horizontal positions of the vehicle in real space, based on information that associates the coordinates of each point in the captured image with the forward and horizontal positions of the vehicle in real space.

For example, when detecting a lane line segment using coordinates in a captured image as shown in FIG. 6, a lane line segment that is relatively far from the vehicle (e.g., line segment 611) is captured as a shape close to a point. Therefore, when detecting lines using coordinates in a captured image, for example, by a Hough transform, problems may arise with the detection accuracy of lane lines that are relatively far from the vehicle. On the other hand, according to the present disclosure, the coordinates in the captured image are converted to coordinates in a bird's-eye view, and then the lines are detected using a Hough transform, so that, for example, the detection accuracy of lane lines that are relatively far from the vehicle can be improved.

Then, the estimation unit 13 divides the bird's-eye view into a plurality of areas according to the distance in the forward direction of the vehicle (step S104). Here, the estimation unit 13 may divide the bird's-eye view into, for example, a first area including a range less than a specific distance in the forward direction of the vehicle, and a second area including a range equal to or greater than the specific distance in the forward direction of the vehicle.

In the example of FIG. 7, the estimation unit 13 divides the bird's-eye view 701 into

areas

711, 712, 713, 714, and 715 in order of proximity to the vehicle, at specific distances (e.g., 10 m) in accordance with the distance ahead of the vehicle.

Then, the generating unit 12 detects the line segments of the demarcation lines for each area of the bird's-eye view (step S105). Here, the generating unit 12 may detect the line segments of the demarcation lines for each area, for example, by performing a Hough transform for each area. Note that the generating unit 12 is not limited to using the Hough transform, and may detect the line segments using other known methods.

Then, the estimation unit 13 estimates a linear figure representing the road dividing line based on the line segments detected in each area (step S106). Here, the estimation unit 13 may estimate, for example, a line passing through a position on a first line segment detected in the first area and a position on a second line segment detected in the second area as one side of a lane on the road. This can improve the detection accuracy of a linear figure representing a curved dividing line, for example, even when the road ahead of the vehicle is curved.

In this case, the estimation unit 13 may detect, for example, a straight line passing through a specific point on the line segment detected in each area as one side of the lane on the road. The specific point may be, for example, a point on the line segment at a position closest to the midpoint in the forward direction of the vehicle in each area. In this case, in the example of FIG. 7, straight lines passing through

points

721, 722, 723, 724, and 725 on the line segment at positions closest to the midpoint in the forward direction of the vehicle (vertical direction in FIG. 7) in each of

areas

711, 712, 713, 714, and 715 are detected as one side of the lane on the road. As a result, for example, a left edge line 811 and a right edge line 812 of the lane in which the vehicle is traveling are detected, as shown in the bird's-eye view 801 of FIG. 8.

The estimation unit 13 may estimate a linear figure representing a lane marking of a road based on a line segment that satisfies a predetermined angle condition among the multiple detected line segments. In this case, the estimation unit 13 may determine, among the multiple line segments detected in the second area, a line segment that satisfies a predetermined angle condition from the direction in which the first line segment detected in the first area extends toward the second area, as a second line segment that forms one side of the same lane as the first line segment. This can improve the detection accuracy of the curved lane marking even when, for example, the road ahead of the vehicle is curved. In this case, the estimation unit 13 may determine, among the multiple line segments detected in the second area, a line segment that exists within a specific angle (for example, 15°) centered on the direction in which the first line segment extends toward the second area from the end of the first line segment that is closer to the second area, as the second line segment. In addition, for example, when another line segment that satisfies a predetermined angle condition is detected for a specific line segment, the estimation unit 13 may determine the reliability of the estimation for the specific line segment to be higher.

In addition, when the width of the detected lane of the road is less than a threshold value, the estimation unit 13 may detect (correct, modify) a line of a specific width from one side of the lane based on the line segment on both sides of the lane that has a higher reliability of line segment detection by Hough transformation as the line on the other side of the lane. This can improve the detection accuracy of lanes at positions relatively far from the vehicle, for example. In this case, the estimation unit 13 may, for example, in the Hough transformation, first project (coordinate transformation) the coordinates (x, y) of each pixel on the line segment of the division line in the bird's-eye view to a point (ρ, θ) in a two-dimensional polar coordinate space of distance ρ and angle θ. Here, the distance ρ may be, for example, the length of a perpendicular line drawn from the origin to a straight line passing through the coordinates (x, y). In addition, the angle θ may be, for example, the angle between the x-axis and a perpendicular line drawn from the origin to a straight line passing through the coordinates (x, y). The estimation unit 13 may then detect line segments in the xy coordinate system in the bird's-eye view based on the projected points (ρ, θ), for example. The estimation unit 13 may then determine that the greater the number of projected points (ρ, θ), the higher the reliability of the line segments in the xy coordinate system in the bird's-eye view that are detected based on the points (ρ, θ).

The estimation unit 13 may also calculate the reliability of the estimation for the linear figure representing the estimated lane markings of the road, and identify the linear figures representing the two lane markings that form both ends of the lane markings of the road based on the calculated reliability. For example, when a camera mounted on a vehicle continuously captures images, the estimation unit 13 may determine the reliability of the linear figure estimated in the current frame to be higher the closer the distance between the linear figure representing the lane markings estimated in the past frame and the linear figure estimated in the current frame is. The estimation unit 13 may also calculate the width between the linear figures representing the estimated lane markings of the road, and calculate the reliability based on a comparison between the calculated width and a predetermined reference value representing the width (width) of the lane markings of the road. In this case, for example, when another parallel line segment is detected at a position away from the detected specific line segment by a distance of the predetermined reference value, the estimation unit 13 may determine the reliability of the specific line segment to be higher. The estimation unit 13 may estimate the dividing line that forms the left end of the lane of the road based on the linear figure that has the highest reliability among the linear figures that are within a first lateral distance (e.g., within 3 m to the left of the center of the vehicle) in each area. Similarly, the estimation unit 13 may estimate the dividing line that forms the right end of the lane of the road based on the linear figure that has the highest reliability among the linear figures that are within a second lateral distance (e.g., within 3 m to the right of the center of the vehicle) in each area.

Then, the estimation unit 13 outputs information based on the estimated road dividing lines (step S107). Here, the estimation unit 13 may output, for example, information indicating the distance from one end of the lane in which the vehicle is traveling to the vehicle itself. The estimation unit 13 may also output, for example, an image in which the lines on both sides of the lane in which the vehicle is traveling are superimposed on the captured image, as shown in FIG. 9. In the example of FIG. 9, the left end line 811 and the right end line 812 of the lane in which the vehicle is traveling in FIG. 8 are converted from the coordinates of the bird's-eye view to the coordinates of the captured image, and

lines

911 and 912 are superimposed on the captured image 401 in FIG. 4 to output an image 901.

This can, for example, assist in the generation of simulation data for the development of an autonomous driving system or a driving assistance system that mimics the swaying of a vehicle actually driven. It can also, for example, assist in the verification of the operation of a vehicle equipped with an autonomous driving system or a driving assistance system when it is actually driven. It can also, for example, detect the lane in which a vehicle equipped with an autonomous driving system or a driving assistance system is traveling in real time when the vehicle is actually driven.

<Modification>
The information processing device 10 may be a device contained in one housing, but the information processing device 10 of the present disclosure is not limited to this. Each unit of the information processing device 10 may be realized by cloud computing configured by one or more computers, for example. Such an information processing device 10 is also included in an example of the "information processing device" of the present disclosure.

Note that this disclosure is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the disclosure.

A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes.
(Appendix 1)
a recognition unit that recognizes an area of a lane marking of a road from a first image obtained by capturing a road around a vehicle;
a generating unit that generates a second image in which the recognized lane marking area of the road is represented on a coordinate plane having two axes that are the longitudinal direction and the lateral direction of the vehicle;
an estimation unit that estimates a linear figure representing a dividing line of the road based on the second image;
An information processing device comprising:
(Appendix 2)
the estimation unit detects a plurality of line segments based on an area of the lane markings of the road represented in the second image, and estimates a linear figure representing the lane markings of the road based on the detected plurality of line segments.
2. The information processing device according to claim 1.
(Appendix 3)
the estimation unit divides the second image into a first area including a range less than a predetermined distance in the vertical direction of the vehicle and a second area including a range equal to or greater than the predetermined distance in the vertical direction of the vehicle, and estimates a linear figure representing a dividing line of the road based on a first line segment detected in the first area and a second line segment detected in the second area.
3. The information processing device according to claim 2.
(Appendix 4)
The information processing device according to claim 2 or 3, wherein the estimation unit estimates a linear figure representing a dividing line of the road based on a line segment among the detected plurality of line segments that satisfies a predetermined angle condition.
(Appendix 5)
the estimation unit determines, based on a direction in which the first line segment extends toward the second area, a second line segment that forms a linear figure representing the same lane marking as the first line segment, among the multiple line segments detected in the second area;
4. The information processing device according to claim 3.
(Appendix 6)
the estimation unit calculates a reliability of estimation for a linear figure representing the estimated lane marking of the road, and identifies linear figures representing two lane markings forming both ends of a lane of the road based on the calculated reliability.
4. The information processing device according to claim 2 or 3.
(Appendix 7)
the estimation unit calculates a width between linear figures representing the estimated lane lines of the road, and calculates the reliability based on a comparison between the calculated width and a predetermined reference value representing a width of a lane of the road;
7. The information processing device according to claim 6.
(Appendix 8)
Recognizing an area of a road division line from a first image obtained by capturing a road around a vehicle;
generating a second image in which the recognized lane marking area of the road is represented on a coordinate plane having two axes corresponding to the longitudinal direction and the lateral direction of the vehicle;
estimating a linear figure representing a dividing line of the road based on the second image;
Information processing methods.
(Appendix 9)
Recognizing an area of a road division line from a first image obtained by capturing a road around a vehicle;
generating a second image in which the recognized lane marking area of the road is represented on a coordinate plane having two axes corresponding to the longitudinal direction and the lateral direction of the vehicle;
A non-transitory computer-readable medium storing a program for causing a computer to execute a process of estimating a linear figure representing a dividing line of the road based on the second image.

1 Monitoring system 10 Information processing device 11 Recognition unit 12 Generation unit 13 Estimation unit

Claims

a recognition unit that recognizes an area of a lane marking of a road from a first image obtained by capturing a road around a vehicle;
a generating unit that generates a second image in which the recognized lane marking area of the road is represented on a coordinate plane having two axes that are the longitudinal direction and the lateral direction of the vehicle;
an estimation unit that estimates a linear figure representing a dividing line of the road based on the second image;
An information processing device comprising:
the estimation unit detects a plurality of line segments based on an area of the lane markings of the road represented in the second image, and estimates a linear figure representing the lane markings of the road based on the detected plurality of line segments.
The information processing device according to claim 1 .
the estimation unit divides the second image into a first area including a range less than a predetermined distance in the vertical direction of the vehicle and a second area including a range equal to or greater than the predetermined distance in the vertical direction of the vehicle, and estimates a linear figure representing a dividing line of the road based on a first line segment detected in the first area and a second line segment detected in the second area.
The information processing device according to claim 2 .
The information processing device according to claim 2 , wherein the estimation unit estimates a linear figure representing a lane marking of the road based on a line segment that satisfies a predetermined angle condition among the plurality of detected line segments.
the estimation unit determines, based on a direction in which the first line segment extends toward the second area, a second line segment that forms a linear figure representing the same lane marking as the first line segment, among the multiple line segments detected in the second area;
The information processing device according to claim 3 .
the estimation unit calculates a reliability of estimation for a linear figure representing the estimated lane marking of the road, and identifies linear figures representing two lane markings forming both ends of a lane of the road based on the calculated reliability.
4. The information processing device according to claim 2 or 3.
the estimation unit calculates a width between linear figures representing the estimated lane lines of the road, and calculates the reliability based on a comparison between the calculated width and a predetermined reference value representing a width of a lane of the road;
The information processing device according to claim 6.
Recognizing an area of a road division line from a first image obtained by capturing a road around a vehicle;
generating a second image in which the recognized lane marking area of the road is represented on a coordinate plane having two axes corresponding to the longitudinal direction and the lateral direction of the vehicle;
estimating a linear figure representing a dividing line of the road based on the second image;
Information processing methods.
Recognizing an area of a road division line from a first image obtained by capturing a road around a vehicle;
generating a second image in which the recognized lane marking area of the road is represented on a coordinate plane having two axes corresponding to the longitudinal direction and the lateral direction of the vehicle;
A non-transitory computer-readable medium storing a program for causing a computer to execute a process of estimating a linear figure representing the dividing line of the road based on the second image.