WO2021215199A1

WO2021215199A1 - Information processing device, image capturing system, information processing method, and computer program

Info

Publication number: WO2021215199A1
Application number: PCT/JP2021/013324
Authority: WO
Inventors: 檜垣　欣成
Original assignee: ソニーセミコンダクタソリューションズ株式会社
Priority date: 2020-04-22
Filing date: 2021-03-29
Publication date: 2021-10-28
Also published as: JP2023089311A

Abstract

[Problem] To estimate the slope of the road surface of a roadway through simple processing. [Solution] An information processing device of the present disclosure comprises a detection unit for detecting a first region line which specifies a region of a roadway in a captured image of a space containing the roadway, and an estimation unit for estimating the slope of the road surface corresponding to a first position on the basis of the coordinate of the first position on the first region line and the slope of the first region line at the first position.

Description

Information processing equipment, imaging system, information processing method and computer program

This disclosure relates to an information processing device, an imaging system, an information processing method, and a computer program.

In driving a vehicle such as an automobile, there is a demand to know the slope of the road surface ahead on the road in order to support safe driving (for example, automatic driving). If the slope of the road surface ahead is known, more appropriate driving becomes possible. For example, if the vehicle is uphill in front, it is possible to control acceleration so that the vehicle does not decelerate, or advise the driver to step on the accelerator deeper.

Patent Document 1 below discloses an image processing device that estimates the distant road gradient in which the automobile is traveling by using two cameras (stereo cameras and the like) mounted on the automobile. However, since it is necessary to use two cameras in this image processing device, there is a problem that the amount of processing is large and the amount of power consumption is large. There is also the problem of high costs.

Japanese Unexamined Patent Publication No. 2009-41972

The present disclosure provides an information processing device, an imaging system, an information processing method, and a computer program that estimate the slope of the road surface of a roadway by a simple process.

The information processing device of the present disclosure is
In the image of the space including the roadway, the detection unit that detects the first area line that identifies the area of the roadway, and the detection unit.
An estimation unit that estimates the slope of the road surface corresponding to the first position based on the coordinates of the first position on the first area line and the slope of the first area line at the first position.
To be equipped.

The imaging system of the present disclosure is
An imaging device that captures the space including the roadway,
In the image acquired by the image pickup device, a detection unit that detects a first region line that identifies the region of the roadway, and a detection unit.
An estimation unit that estimates the slope of the road surface corresponding to the first position based on the coordinates of the first position on the first area line and the slope of the first area line at the first position.
To be equipped.

The information processing method of the present disclosure is
In the image of the space including the roadway, the first area line that specifies the area of the roadway is detected.
The slope of the road surface corresponding to the first position is estimated based on the coordinates of the first position on the first area line and the slope of the first area line at the first position.

The computer program of this disclosure is
In the image of the space including the roadway, the step of detecting the first area line that specifies the area of the roadway, and
Have the computer perform a step of estimating the slope of the road surface corresponding to the first position based on the coordinates of the first position on the first area line and the slope of the first area line at the first position. ..

The block diagram of the image pickup system and the vehicle control system which concerns on 1st Embodiment of this disclosure. A diagram showing an example of a road in real space in a plane. The figure which shows the example of the captured image acquired by the imaging apparatus. A side view showing the relationship between a vehicle and a roadway in real space. The figure which shows the example which set the attention area (target area) ROI. A schematic cross-sectional view showing the geometrical relationship between the image pickup device and the road surface. The figure which shows typically the example of the curved road in the xyz coordinate system. The figure which shows the example which calculates the gradient angle of the road surface included in the attention area ROI. The flowchart of an example of the operation of the information processing apparatus which concerns on this embodiment. The figure which shows another example which set the attention area ROI. The block diagram of the image pickup system and the vehicle control system which concerns on 2nd Embodiment. The side view for demonstrating the operation of the estimation part. The figure which shows the example which sets the point where the preceding vehicle and the road surface are in contact with each other as an observation target point. The figure which shows the example which calculates the distance Δv in the captured image. The flowchart of an example of the operation of the information processing apparatus when the estimation unit uses the first estimation example. The flowchart of an example of the operation of the information processing apparatus when the estimation unit uses the second estimation example.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In one or more embodiments set forth in the present disclosure, the elements included in each embodiment can be combined with each other, and the combined deliverables also form part of the embodiments set forth in the present disclosure.

(First Embodiment)
FIG. 1 is a block diagram of an imaging system 100 and a vehicle control system 301 according to the first embodiment of the present disclosure. The imaging system 100 of FIG. 1 is mounted on a vehicle such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, and a bicycle. In addition, the system of FIG. 1 can be mounted on any type of mobile body such as an AGV (Automated Guided Vehicle), personal mobility, a mobile robot, a construction machine, and an agricultural machine (tractor). As long as these moving bodies can travel on the road surface, they also correspond to one form of the vehicle according to the present embodiment.

The imaging system 100 is connected to the vehicle control system 301 by wire or wirelessly.

The vehicle control system 301 controls a vehicle traveling on a roadway. For example, radar, LiDAR, ToF sensor, camera, weather sensor, etc. are used to detect obstacles such as preceding vehicles and pedestrians, and control is performed to avoid collisions with the detected obstacles and pedestrians. For example, it controls the acceleration / deceleration of the vehicle and controls the output of an alarm to a pedestrian. The vehicle control system 301 may feed back the detection result or the control result to the imaging system 100.

The image pickup system 100 includes an image pickup device 201 and an information processing device 101. The information processing device includes an image acquisition unit 110, a detection unit 120, an area setting unit 130, an estimation unit 140, and a storage unit 150. The image acquisition unit 110, the detection unit 120, the area setting unit 130, and the estimation unit 140 are composed of hardware, software (program), or both. Examples of hardware include a processor such as a CPU (Central Processing Unit), a dedicated circuit, a programmable circuit, a memory, and the like. The storage unit 150 includes any recording medium capable of storing information or data, such as a memory, a hard disk, and an optical disk.

The image pickup device 201 is a camera or sensor device that captures a space including a roadway in front of the vehicle. The image pickup device 201 is, for example, a color camera such as an RGB camera, a monochrome camera, an infrared camera, a brightness sensor, or the like. In this embodiment, the image pickup apparatus 201 is an RGB monocular camera. The image pickup device 201 is attached at a position where the front of the vehicle can be imaged, such as the upper part of the windshield in the vehicle interior or the front nose. The posture of the image pickup apparatus 201 is set so that, for example, the direction of the light source (imaging direction) is in the direction (slightly downward) toward the road surface (ground) of the road, or is substantially toward the road surface. ing.

The image pickup device 201 images the space including the roadway in front of the vehicle at regular time intervals while the vehicle is traveling, and provides the captured image to the information processing device 101. The timing of sensing by the image pickup apparatus 201 may be the timing of receiving an instruction from the vehicle control system 301. The vehicle control system 301 may output an imaging instruction to the imaging device 201 when receiving an instruction for estimating the slope of the road surface from a user (passenger) in the vehicle via an input device (not shown). The image pickup device 201 can also take an image while the vehicle is stopped.

The storage unit 150 stores data or information necessary for the processing of the present embodiment, such as parameter information of the image pickup apparatus 201, a gradient calculation function described later, and a learned neural network.

The image acquisition unit 110 is connected to the image pickup device 201 by wire or wirelessly, and acquires the captured image from the image pickup device 201. The image acquisition unit 110 provides the acquired captured image to the detection unit 120.

The detection unit 120 detects a region line that specifies the region of the roadway included in the captured image. As a specific example of the area line, it may be a traffic line drawn on the road surface (outside road line, central line, etc.), or may be a boundary portion between the traffic line and the roadway on the roadway side. Alternatively, it may be a boundary portion of the traffic line with the roadway, or an edge portion of the traffic line on the opposite side of the roadway (the width of the traffic line is constant or substantially constant). Alternatively, the area line may be a side groove on the road surface, a boundary portion between the side groove and the roadway on the roadway side, or an edge portion on the side groove on the road surface opposite to the roadway (the width of the side groove is constant). Or it is assumed to be substantially constant). The traffic line is generally drawn in white or orange. Alternatively, when the roadway is composed of asphalt or gravel road and one or both sides of the roadway are grass areas, the boundary portion between the road and the lawn on the roadway side, or the boundary between the roadway and the roadway on the lawn. The portion may be a region line. Alternatively, when a fence is provided along the roadway, the boundary portion between the fence and the roadway on the roadway side or the boundary portion on the wall side may be used as the area line. The shape of the area line can be straight, curved, or a mixture of straight and curved. In the present embodiment, the traffic line drawn on the road (white line, etc.) is described as not being a part of the roadway, but the traffic line can also be defined as a part of the roadway.

FIG. 2 is a plan showing an example of a road in real space. FIG. 2A shows an example of a road in which outside road lines (white lines) 10A and 10B are provided on both sides of the roadway 10C. The

outside lines

10A and 10B are straight lines. The opposite side of the

outer roadway lines

10A and 10B from the roadway 10C is, for example, a road other than the roadway (walking road, etc.).

FIG. 2B shows an example in which the outside road lines (white lines) 11A and 11B provided on both sides of the roadway 11C are curved lines.

FIG. 2C shows an example of a road having grooves (side grooves) 12A and 12B on both sides of the roadway 12C.

In FIGS. 2 (A) to 2 (C), white lines or gutters are present on both sides of the roadway, but they may be present on only one side.

FIG. 3 shows a specific example of the roadway and the area line in the captured image acquired by the imaging device 201. FIG. 3A shows an example in which there is no gradient difference between the road surface on which the own vehicle is present and the road surface in front of the road on a straight road. FIG. 3B shows an example of a straight road where there is a gradient difference between the road surface on which the own vehicle is present and the road surface in front (when the front is an uphill).

In FIG. 3A, for example, the boundary portion of the roadway 22 in contact with the white line (outside the roadway) is schematically shown as

area lines

21A and 21B. Similarly, in FIG. 3B, the boundary portion of the roadway 25 in contact with the white line (outside the roadway) is formally shown as the

area lines

24A and 24B. In addition, in FIGS. 3 (A) and 3 (B), the illustration of an object other than the roadway, such as a white line, is omitted.

In the captured image 20 when there is no gradient difference in FIG. 3A, the

area lines

21A and 21B are linear (the white lines adjacent to the

area lines

21A and 21B are also linear). In the captured image 23 when there is a gradient difference in FIG. 3B, the

area lines

24A and 24B have a shape that is slightly bent inward in the middle in the traveling direction.

The position of each pixel in the captured images of FIGS. 3 (A) and 3 (B) can be represented by an uv coordinate system with the horizontal axis as the u axis and the vertical axis as the v axis. The ui coordinate system corresponds to the so-called image coordinate system.

4 (A) and 4 (B) are side views showing the relationship between the vehicle and the roadway in the real space when the captured images of FIGS. 3 (A) and 3 (B) are acquired. The traveling direction of the vehicle is the x-axis, the height direction is the z-axis, and the direction perpendicular to the paper surface (the width direction of the road) is the y-axis. The xyz coordinate system corresponds to the so-called world coordinate system. The position in real space can be represented by xyz coordinates.

As shown in FIG. 4A, there is no gradient difference between the road surface on which the vehicle is present and the road surface in front of the vehicle on the roadway 26 on which the vehicle travels. As shown in FIG. 4B, there is a gradient difference between the road surface on which the vehicle is present and the road surface in front of the vehicle on the roadway 27 on which the vehicle travels.

When the front is imaged on a straight road with no gradient difference as shown in FIG. 4 (A), an image in which the area line of the road is narrowed with a constant inclination as shown in FIG. 3 (A) can be obtained. .. Further, when the front of the vehicle is imaged on a straight road with a gradient difference as shown in FIG. 4 (B), the area line of the road is inward along the traveling direction as shown in FIG. 3 (B). A bent image is obtained. The relationship between the real space of FIG. 4 (A) and the image of FIG. 3 (A) and the relationship between the real space of FIG. 4 (B) and the image of FIG. 3 (B) are the world coordinate system to which the real space belongs and the image. It is represented by a commonly known investment projection transformation using the image coordinate system to which it belongs.

For example, semantic segmentation can be used as a method for the detection unit 120 to detect the area line of the roadway from the captured image. In semantic segmentation, the captured image is divided into a plurality of segments (objects) belonging to any of a plurality of predetermined classes. Examples of classes include roadways, traffic lines (white lines, etc.), or ditches. In this case, for example, the boundary portion (boundary line) of the object having the roadway class with the adjacent object can be detected as the area line of the roadway. Alternatively, the boundary portion of the adjacent object with the object having the roadway class can be detected as the area line of the roadway. Alternatively, the edge portion of the adjacent object on the opposite side of the object having the roadway class can be detected as a roadway area line (for example, when the adjacent object is a traffic line object). The image segmentation method may be a method other than semantic segmentation, such as a clustering method based on color information. Further, the region line of the roadway may be detected by using the region line detection method of detecting the contour from the image instead of the segmentation of the image. The following is an example of the operation when segmentation is performed using semantic segmentation.

For example, semantic segmentation of the captured image is performed using a trained neural network. The trained neural network is stored in the storage unit 150. Semantic segmentation is a method of classifying the class (type) of each pixel of an image on a pixel-by-pixel basis. The class is determined for each pixel, and a label indicating the determined class is output for each pixel. For example, a plurality of classes such as a roadway, a traffic line (white line, etc.), and a gutter are defined in advance. The class is determined for each pixel. As a result of semantic segmentation, a class value corresponding to each pixel is obtained. A group of consecutive pixels with the same class value corresponds to one segment (object). When the trained neural network is used, the captured image is used as the input of the neural network, and the class value corresponding to each pixel is obtained as the output. An image (semantic segmentation image) in which pixels having the same class value have the same color may be generated. Objects of the same type are shown in the same color, so the segmentation results can be displayed in a visually easy-to-understand manner.

The area setting unit 130 sets an area of interest (target area) that includes at least a part of the road surface that is the target of slope estimation on the roadway with respect to the captured image. The shape of the region of interest may be rectangular, trapezoidal, circular, or any other shape, and is not limited to a specific shape. Further, the region of interest may be a straight line (for example, a line having a width of one pixel). Note that setting the region of interest for the captured image includes setting the region of interest directly for the captured image and setting the region of interest for the segmentation image.

The estimation unit 140 has a road surface gradient corresponding to the selected position on the road surface of the roadway based on the position (selected position) selected on the area line of the roadway detected by the detection unit 120 and the inclination of the area line at the selected position. To estimate. When two area lines are detected, the selected selected position on one area line corresponds to the first position, and the selected selected position selected on the other area line corresponds to the second position. However, only one area line may be detected, and in this case, the selected position selected from the area line corresponds to the first position.

Specifically, first, among the area lines of the roadway detected by the detection unit 120, the area line of the part included in the area of interest set by the area setting unit 130 (hereinafter referred to as a partial line) is detected. The estimation unit 140 calculates the inclination at an arbitrary position (selected position) on the detected partial line.

The slope of the partial line is the slope in the uv coordinate system, and the selected position in the partial line is represented by the uv coordinate.

The slope of the partial line may be the slope of the tangent line at the selected position on the partial line, or the slope of a straight line that approximates the partial line.

The selected position may be the position of the part (contact point) of the partial line that is in contact with the tangent line, the center of gravity of the partial line (the average value of the coordinates), or the position of an arbitrary selected coordinate from the coordinates included in the partial line. It may be.
The partial line may have a pixel width of 1 or 2 or more.

FIG. 5 shows an example in which the region of interest (target region) ROI is set for the captured image 23 of FIG. 3 (B). A rectangular attention region ROI that crosses the roadway 25 is set slightly at the center of the captured image 23 in the v-coordinate direction. The region of interest ROI includes at least part of the road surface of the roadway 25. Of the area line 24A of the roadway 25, the portion of the area line included in the region of interest ROI is specified as the partial line 31A. Similarly, of the region line 24B facing the region line 24A via the roadway 25, the portion of the region line included in the region of interest ROI is specified as the partial line 31B. Only one of the two sublines may be specified.

The method of setting the attention area ROI may be arbitrary. For example, when the vehicle control system 301 detects a vehicle in front, a range including at least a part of the detected vehicle may be specified as a region of interest ROI. Alternatively, a predetermined coordinate range in the captured image may be used as the region of interest ROI. Alternatively, when the inclination of the region line changes by a certain amount or more by analyzing the time-series images, the range including the changed portion may be set as the region of interest. Alternatively, the captured image may be displayed on a display device in the vehicle, and the user may specify the area of interest using an input device (touch panel or the like).

Inclination alpha _L of partial lines 31A in the image coordinate system (uv coordinate system), the inclination alpha _R of partial lines 31B is shown. The slope α _L may be the slope of the tangent line at the selected position on the partial line 31A, or the slope of a straight line that approximates the partial line 31A. The position L (uL, vL) (see FIG. 8 described later), which is the selected position on the partial line 31A, may be, for example, a contact point between the partial line 31A and its tangent line, or the center of gravity of the partial line 31A (for example, the center of the u coordinate). It may be the center of the v-coordinate) or the position determined by another method.

Similarly, the slope α _R may be the slope of the tangent line at the selected position on the partial line 31B, or the slope of a straight line that approximates the partial line 31B. The position R (uR, vR), which is the selected position on the partial line 31B, may be, for example, a contact point between the partial line 31B and its tangent line, or the center of gravity of the partial line 31B (for example, the center of the u coordinate and the center of the v coordinate). However, the position may be determined by another method.

The position L and the position R may be determined on the condition that the v coordinates of the position L (uL, vL) and the position R (uR, vR) are the same. The details of the process of calculating the position and inclination of the partial line will be described later.

The estimation unit 140 estimates the slope of the road surface portion included in the region of interest based on the slope of the partial line and the selected position. Hereinafter, the reason why the slope of the road surface portion included in the attention region ROI can be estimated based on the inclination of the partial line included in the attention region ROI and the selected position will be described in detail.

FIG. 6 is a schematic cross-sectional view showing the geometrical relationship between the image pickup device 201 and the road surface. In FIG. 6, as in the case shown in FIG. 4B, a case where there is a gradient difference in the front on a straight roadway is taken as an example. The image pickup device 201 (camera) 35 in the real space, the road surface of the roadway 27, and the road surface angle (road surface gradient angle) θ0 at the observation target point P (target position) are shown. The traveling direction of the vehicle is the x-axis, and the height direction is the z-axis. The y-axis is the width direction of the vehicle. The xyz coordinate system corresponds to the world coordinate system.

The vehicle image pickup device 201 exists at a height H from the origin of the xyz coordinate system. That is, the image pickup apparatus 201 exists at a position at a height H from the plane 38 on which the vehicle exists. The coordinates of the camera light source are (0, 0, H). The direction (angle) of the optical axis 36 of the camera 35 is θ. The observation target point P is a position parallel to the optical axis of the camera, and corresponds to, for example, a selected position on a partial line. The road surface gradient angle θ0 is the inclination of the tangent line 37 at the observation target point P on the road surface of the roadway. The intersection of the tangent line 37 and the x-axis is x0.

As described above, the relationship between the world coordinates and the image coordinates is expressed by the perspective projection transformation, and this relationship is expressed by the following equation.

s is a scaling factor and corresponds to the reciprocal of the depth from the camera to the observation target point.
fy is the horizontal focal length of the camera (unit is pixel), and fz is the vertical focal length of the camera (unit is pixel).
(Cy, cz) is the center of the camera (principal point), that is, the center of the uv coordinate system.
θ is the angle in the optical axis direction.
H is the height of the camera.
(X, y, z) is the coordinates of the observation target point P, which is an unknown value. The observation target point P corresponds to the position of the partial line in the captured image.
On the right side of equation (1), the first matrix corresponds to the internal parameters of the camera and the second matrix corresponds to the external parameters of the camera. The internal parameters and external parameters are measured in advance by calibration and stored in the storage unit 150. Internal parameters and external parameters are collectively called camera parameter information.

The following equations (2), (3), and (4) are derived from the equation (1). z'= z-H.

The slope α of the partial line ( _{representing α L} or α _R ) can be expressed as tan α = dv / du based on the uv coordinate system. The dv / du (differential coefficient of v with respect to u) is analytically derived below. First, equation (5) is obtained from the formula for differentiation.

From the relationship shown in FIG. 6, the following equation (6) can be obtained. x and z are the x and z coordinates of the observation target point P. θ0 is the angle (road surface gradient angle) at which the tangent line of the observation target point P and the x-axis intersect.

From equations (3) and (6), v is expressed as a function of only z among x, y, and z. Therefore, the first term and the second term of the equation (5) become 0, and the following equation (7) is obtained.

Hereinafter, equation (7) is specifically derived. First, the first differential coefficient on the right side is derived.
From the formula (3) and the formula (6), the following formula (8) can be obtained.

From the formula (3), the following formula (9) can be obtained.

From the formula (4), the following formula (10) can be obtained.

From the formulas (8) to (10), the following formula (11) can be obtained.

Next, the second differential coefficient (δz / δu) on the right side of the equation (7) is derived.
From the formula (4) and the formula (6), the following formula (12) can be obtained.

From the formula (2), the following formula (13) can be obtained.

From the formula (12) and the formula (13), the following formula (14) can be obtained.

From the formula (14), the following formula (15) can be obtained.

From the formulas (7), (11), and (15), the following formula (16) can be obtained.

From the formulas (5) and (16), the following formula (17) can be obtained.

In equation (17), the direction θ of the optical axis 36 of the camera, the focal length fz in the vertical direction, and the camera center (cy, cz) are known from the parameter information of the camera. Therefore, from the angle α of the partial line and the position (u, v) which is the selected position on the partial line, the target position (observation target point) on the road surface corresponding to the position (u, v) is determined by the equation (17). The road surface gradient angle θ0 is calculated. That is, the slope of the road surface at the target position can be estimated based on the angle α of the partial line and the coordinates (u, v) of the selected position on the partial line.

The function of equation (17) is stored in advance in the storage unit 150 as a gradient calculation function. The slope calculation function is a function that corresponds to an input variable in which the slope (angle) of a partial line is input, an input variable in which the coordinates of the selected position on the partial line are input, and an output variable that outputs the road surface slope angle. ..

In the derivation of the gradient calculation function described above, a straight roadway is assumed (that is, when the value of y is the same in the xyz coordinate system), but even in the case of a curved roadway as shown in FIG. 2 (B). , The gradient calculation function can be derived in the same way by expressing the relationship between x, y, and z with a polynomial.

FIG. 7 schematically shows an example of a curved roadway in the xyz coordinate system. White lines 51L and 52R are provided on both sides of the roadway 50. For example, the relationship between x, y, and z can be expressed by the following equation (18) and the above equation (6).

Since y now depends on x and z, the second term on the right side of the above equation (7) becomes non-zero, and the following equation (19) is obtained.

Based on the equation (19), the following equation may be derived in the same manner as above.

The estimation unit 140 estimates the slope angle (road surface slope angle) of the road surface included in the region of interest ROI based on the angle of the partial line and the coordinates of the selected position using the slope calculation function.

FIG. 8 shows an example of calculating the slope angle of the road surface included in the region of interest ROI in the same captured image 23 as in FIG. Here, an example is shown in the case where the ROI of interest is a straight line. When the region of interest ROI is a straight line, the partial line (see partial line 31A in FIG. 5) corresponds to the intersection L (point L) of the region of interest ROI (straight line) and the region line 24A. The coordinates of this point L also correspond to the coordinates of the selected position selected on the partial line. The angle αL of the partial line is, for example, the angle of the tangent line 39L at the point L. From the angle αL of the partial line and the position L (uL, vL), the road surface gradient angle θ0L at the position (target position) in the real space corresponding to the position L (uL, vL) is calculated by the above-mentioned gradient calculation function. ..

Similarly, the other partial line (see partial line 31B in FIG. 5) corresponds to the intersection R (point R) between the region of interest ROI and the region line. The coordinates of this point R also correspond to the coordinates of the selected position selected on the partial line. The angle αR of the partial line is, for example, the angle of the tangent line 39R at the point R. From the angle αR of the partial line and the position R (uR, vR), the estimation unit 140 uses the above-mentioned gradient calculation function to determine the road surface gradient at the position (target position) in the real space corresponding to the position L (uR, vR). The angle θ0R is calculated.

The estimation unit 140 estimates the average of the two calculated road surface gradient angles θ0L and θ0R as the road surface gradient angle of the road surface included in the region of interest. The average may be a weighted average. For example, when the road surface is straight, the weight may be set to 1: 1, and when the road surface is a left curve or a right curve, different weights may be set on the left and right. The weight may be adjusted according to the curvature of the curve. Alternatively, either one (maximum value or minimum value) of the two road surface gradient angles θ0L and θ0R may be estimated as the road surface gradient angle of the road surface included in the region of interest.

The estimation unit 140 outputs information representing the slope angle of the road surface estimated by the ROI of interest to the vehicle control system 301. The vehicle control system 301 controls the vehicle by using the information of the road surface gradient angle. For example, acceleration / deceleration control is performed as driving support for the driver to avoid collisions with obstacles, pedestrians, and the like. Specifically, when the road surface gradient angle indicates an uphill, acceleration is performed, and when it indicates a downhill, deceleration is performed.

FIG. 9 is a flowchart of an example of the operation of the information processing device according to the present embodiment. The image acquisition unit 110 acquires an captured image from the imaging device 201 (S101). The detection unit 120 detects a region line that specifies the region of the roadway in the captured image (S102). The region setting unit 130 sets a region of interest (target region) including at least a part of the road surface for which the gradient angle is to be estimated in the captured image (S103). The estimation unit 140 calculates the inclination at the selected position (first position or second position) of the partial line which is a part of the area line included in the region of interest (S104). The estimation unit 140 estimates the slope angle of the road surface included in the region of interest by calculating a slope calculation function that uses the slope of the partial line and the coordinates of the selected position as input variables (S105).

The order of the steps in the flowchart of FIG. 9 is an example, and the order of some steps may be changed. For example, step S103 may be executed before step 102.

As described above, according to the first embodiment, the road surface included in the attention area is based on the inclination at the selection position of the partial line which is the area line of the roadway included in the attention area ROI set in the captured image and the coordinates of the selection position. Estimate the tilt angle of. As a result, the road surface inclination angle can be estimated at high speed with a small amount of processing. Further, according to the first embodiment, estimation can be performed using an image captured by a monocular camera, and since two or more cameras such as a stereo camera are unnecessary, it can be realized at low cost. Further, by performing the estimation using a plurality of partial lines, the accuracy and robustness of the estimation can be improved.

(Modification example 1)
In the above-described embodiment, an example is shown in which the boundary portion on the roadway side of the roadway and the traffic line (white line, etc.) is used as the region line for specifying the region of the roadway, but in this modification, the boundary portion on the white line side, etc. Is shown as an example in which is a region line.

FIG. 10 shows an example of a captured image according to this modified example.

White lines

24A and 24B are provided on both sides of the roadway 40. A rectangular attention area ROI that crosses the

white lines

24A and 24B in the u coordinate direction is set. The

white lines

24A and 24B have a certain width (two or more pixels).

As an example, the boundary portion of the white line 24A in contact with the roadway 40 can be used as the area line 31. The portion of the region line 31 included in the region of interest ROI corresponds to the partial line 31A. On the partial line 31A, the point 45 can be selected as an example of the selection position. In this case, as the slope αL at the point 45, the slope of the tangent line at the point 45 can be used.
Alternatively, the slope of a straight line (not shown) connecting the

intersections

41 and 42 inside the white line 24A and the region of interest ROI may be calculated as the slope of the partial line 31A. In this case, the selected position may be a position selected from the partial line 31A or a position arbitrarily selected from the straight line such as the midpoint on the straight line. As another example, the slope of the straight line connecting the

outer intersections

46 and 47 of the region line 24A and the region of interest ROI may be calculated as the slope of the partial line 31A. In this case, the selected position may be a position selected from the partial line 31A, or a position arbitrarily selected from the straight line such as a midpoint on the straight line. Alternatively, the slope of the straight line connecting the midpoint 43 of the

intersection

41 and 46 and the midpoint 47 of the intersection 42 and the intersection 44 may be calculated as the slope of the partial line 31A. In this case, the selected position may be a position selected from the partial line 31A, or a position arbitrarily selected from the straight line such as a midpoint on the straight line.

As another example, the edge portion of the white line 24A on the opposite side of the roadway 40 can be used as the area line 32. The portion of the region line 31 included in the region of interest ROI corresponds to the partial line 32A. The slope of the partial line 32A and the selected position on the partial line 32A can be calculated in the same manner as in the case of the area line 31.

Although an example of calculating the area line, the inclination and the selected position for the white line 24A has been described, the area line, the inclination and the selected position can be calculated in the same manner for the white line 24B.

Using the slopes and selected positions calculated for the

white lines

24A and 24B, respectively, the slope angle of the road surface included in the region of interest ROI can be estimated from the above slope calculation function.

The method of calculating the slope of the partial line (the slope of the region line included in the region of interest) by the linear approximation described in FIG. 10 can also be applied to the above-described embodiment.

(Modification 2)
Although the region of interest is set in the above-described embodiment, the slope of the road surface may be estimated without setting the region of interest. For example, an arbitrary position on the area line is selected as the selection position, and the slope of the road surface corresponding to the selection position is obtained from the above-mentioned gradient calculation function based on the coordinates of the selected selection position and the inclination of the area line at the selection position. You may estimate. In this case, the same estimation result as in the case where the region of interest of the straight line passing through the selected position is set in the above-described embodiment and the slope of the road surface is estimated using the inclination of one side (left side or right side) and the coordinates of the selected position. Can be obtained.

(Second Embodiment)
In the second embodiment, the road surface gradient angle is estimated as in the first embodiment, but the configuration for estimating is different.

FIG. 11 is a block diagram of the imaging system 500 and the vehicle control system 301 according to the second embodiment. The same elements as those in the block diagram of FIG. 1 of the first embodiment are designated by the same reference numerals, and the description thereof will be omitted as appropriate.

The image pickup system 500 includes an image pickup device 201, a depth sensor 401, and an information processing device 501. The image pickup apparatus 201 is the same as that of the first embodiment.

The depth sensor 401 is a sensor device that detects the depth information of the space including the front of the vehicle. As the depth information, for example, a depth image in which the depth value is stored in each pixel is acquired. The depth sensor 401 is, for example, a LiDAR, a laser, a stereo camera, a ToF sensor, or the like. The posture of the depth sensor 401 is set so that, for example, the light source direction (imaging direction) is closer to the road surface (ground) (slightly downward), or is substantially oriented toward the road surface. There is. The parameter information of the depth sensor 401 is stored in the storage unit 150. Information on the positional relationship between the depth sensor 401 and the imaging device 201 is also stored in the storage unit 150.

The depth sensor 401 acquires the depth information of the space including the front at regular time intervals while the vehicle is running. The depth sensor 401 provides the acquired depth information to the information processing device 101. The depth sensor 401 operates in synchronization with, for example, the image pickup apparatus 201. For example, the depth sensor 401 and the image pickup apparatus 201 operate according to the same synchronization signal. Alternatively, the timing of sensing by the depth sensor 401 may be the timing of receiving an instruction from the vehicle control system 301. When the vehicle control system 301 receives an instruction for estimating the slope of the road surface from a user (passenger) in the vehicle via an input device (not shown), the vehicle control system 301 may output a sensing instruction to the image pickup device 201. The depth sensor 401 may be sensed while the vehicle is stopped.

The depth information acquisition unit 210 of the information processing device 101 is connected to the depth sensor 401 by wire or wirelessly, and acquires the depth information from the depth sensor 401. The depth information acquisition unit 210 provides the acquired depth information to the estimation unit 240.

The image acquisition unit 110 acquires an image captured from the image pickup device 201 and provides the acquired image to the estimation unit 240.

The estimation unit 240 specifies the depth value of the observation target point (third position) on the roadway based on the depth information acquired from the depth information acquisition unit 210. Further, in the captured image, the position (fourth position) corresponding to the observation target point is specified. The slope of the observation target point on the road surface is estimated based on the depth value of the observation target point and the coordinates of the position corresponding to the observation target point. Below, two examples of estimating the gradient are shown.

(First estimation example)
FIG. 12 is a side view for explaining the operation of the estimation unit 240. The traveling direction of the vehicle is the x-axis, the height direction is the z-axis, and the direction perpendicular to the paper surface (the width direction of the road) is the y-axis. The observation target point P1 is a position in real space and is a point on the road surface from which the gradient is to be estimated. The observation target point P1 is, for example, a position where the preceding vehicle exists when the preceding vehicle exists. As the position where the preceding vehicle exists, the position at the lower end of the preceding vehicle or the position where the preceding vehicle and the road surface are in contact with each other may be used. The preceding vehicle is detected by the estimation unit 240 based on the depth information acquired by the depth sensor 401.

FIG. 13 shows an example in which the position of the lower end of the rear part of the preceding vehicle is set as the observation target point P1.

The distance from the vehicle (more specifically, the depth sensor) to the observation target point P1 is the distance X. The point at which the observation target point P1 is projected onto the plane on which the own vehicle exists is defined as the projection point P2 (second point). The distance from the vehicle (more specifically, the depth sensor) to the projection point P2 is defined as the distance X'. Let Z be the distance between the observation target point P1 and the projection point P2. Let θ be the road surface gradient angle at the observation target point P1. The road surface gradient angle θ is the difference in the slope angle of the road surface when the road surface on which the own vehicle exists is used as a reference. Strictly speaking, the observation target point P1 may not be a point on the road surface, but in this case as well, since it is considered that the observation target point P1 and the road surface are extremely close to each other, there is no problem in treating it as an error range. ..

The estimation unit 240 calculates the distance (pixel distance) Δv on the captured image corresponding to the distance Z in the real space. As a premise, based on the principle of triangulation, epipolar constraint, parameter information of the imaging device 201, parameter information of the depth sensor 401, and the positional relationship between the imaging device 201 and the depth sensor 401, the position of the pixel in the depth information and the captured image. The correspondence with the pixel position is uniquely determined.

FIG. 14 shows an example of calculating the distance Δv in the captured image captured by the imaging device. In the captured image, the principal point (origin) in the image coordinate system (uv coordinate system) obtained in advance from the parameter information of the imaging device 201 is shown as a point O. Position on the captured image corresponding to the observation object point P1 is a point S1, the distance v coordinate direction at the point S1 with respect to the principal point O is v _1. Position on the captured image corresponding to the projection point P2 is S2, the distance v coordinate direction at the point S2 with respect to the principal point O is v _2. Therefore, the distance Δv corresponding to the distance Z is a value obtained by subtracting _{v 1} from v _2.

At this time, the road surface gradient angle θ is calculated by the following formula. fz is the focal length of the image pickup apparatus 201 in the vertical direction.

In this way, the estimation unit 240 specifies the position (fourth position) on the captured image corresponding to the observation target point (third position) on the roadway. In addition, the position (sixth position) on the captured image corresponding to the position (fifth position) in which the observation target point is projected on the plane on which the own vehicle exists is specified. The road surface gradient angle at the observation target point is calculated from the equation (20) based on the difference Δv of the v-coordinates of the two specified positions (fourth position and sixth position).

FIG. 15 is a flowchart of an example of the operation of the information processing device when the estimation unit 240 uses the first estimation example. The image acquisition unit 110 acquires the captured image from the image pickup apparatus 201, and the depth information acquisition unit 210 acquires the depth information from the depth sensor 401 (S201). The estimation unit 240 detects the preceding vehicle based on the depth information (S202), and identifies the lower end position of the rear portion of the preceding vehicle as an observation target point (S203). The estimation unit 240 identifies a position on the captured image corresponding to the observation target point and a position on the captured image corresponding to the projection point where the observation target point is projected on the plane on which the own vehicle exists (S204). Then, the difference (distance Δv) of the v-coordinates of these two specified positions is calculated (S204). The estimation unit 240 estimates the road surface gradient angle at the observation target point based on Δv and the equation (20) (S205).

(Second estimation example)
The second estimation example will be described with reference to FIGS. 12 and 14 used in the description of the first estimation example.
In the second estimation example, the road surface gradient angle (gradient difference) θ is estimated by applying the principle of monocular distance measurement. The principle of monocular ranging is disclosed, for example, in the following paper.
Gideon P. Stein et al., “Vision-based ACC with a Single Camera: Bounds on Range and Range Rate Accuracy” (2003)

The following equation (21) holds from the principle of monocular distance measurement. fz is the focal length of the image pickup apparatus 201 in the vertical direction. As _{described with reference to FIG. 14, v 2} corresponds to the distance in the v coordinate direction from the principal point O of the position S2 when the position on the captured image corresponding to the projection point P2 is S2.

From the relationship shown in the equation (21) and FIG. 12, the following equations (22) and (23) are established. As _{described with reference to FIG. 14, v 1} corresponds to the distance in the v coordinate direction from the principal point O of the position S1 when the position on the captured image corresponding to the observation target point P1 is S1. fz is the focal length of the image pickup apparatus 201 in the vertical direction. H is the height of the image pickup apparatus 201. X is the distance (depth value) to the observation target point P1.

The estimation unit 240 is based on the distance X to the observation target point P1 and the distance v1 in the v coordinate direction from the principal point O of the position S1 on the captured image corresponding to the observation target point P1 from the equation (22). Calculate the gradient angle θ.

The estimation unit 240 provides the vehicle control system 301 with information on the estimated road surface gradient angle θ. The operation of the vehicle control system 301 is the same as that of the first embodiment.

FIG. 16 is a flowchart of an example of the operation of the information processing apparatus when the estimation unit 240 uses the second estimation example. The image acquisition unit 110 acquires the captured image from the image pickup apparatus 201, and the depth information acquisition unit 210 acquires the depth information from the depth sensor 401 (S301). The estimation unit 240 detects the preceding vehicle based on the depth information (S302), and identifies the lower end position of the rear portion of the preceding vehicle as an observation target point (S303). The estimation unit 240 specifies a position on the captured image corresponding to the observation target point (S304). The v-coordinate of the specified position (distance v1 in the v-coordinate direction from the principal point O in the captured image) is specified. Estimation unit 240, _{v 1} and, based on the equation (22) estimates the road surface slope angle at the observation target point (S305).

As described above, according to the second embodiment, the road surface gradient angle can be estimated at high speed with a low calculation amount. Further, according to the present embodiment, the road surface gradient angle can be estimated with high accuracy.

Note that the above-described embodiment shows an example for embodying the present disclosure, and the present disclosure can be implemented in various other forms. For example, various modifications, substitutions, omissions, or combinations thereof are possible without departing from the gist of the present disclosure. The forms in which such modifications, substitutions, omissions, etc. are made are also included in the scope of the invention described in the claims and the equivalent scope thereof, as are included in the scope of the present disclosure.

Further, the effects of the present disclosure described in the present specification are merely examples, and other effects may be obtained.

The present disclosure may also have the following structure.
[Item 1]
In the image of the space including the roadway, the detection unit that detects the first area line that identifies the area of the roadway, and the detection unit.
An estimation unit that estimates the slope of the road surface corresponding to the first position based on the coordinates of the first position on the first area line and the slope of the first area line at the first position.
Information processing device equipped with.
[Item 2]
An area setting unit for setting a target area including at least a part of the road surface of the roadway is provided for the image.
The detection unit has a slope of the road surface included in the target region based on the coordinates of the first position on the first region line in the target region and the inclination of the first region line in the first position. The information processing apparatus according to item 1, further comprising estimating.
[Item 3]
The detection unit detects the second region line facing the first region line, and
The estimation unit is described in item 2 for estimating the gradient based on the coordinates of the second position on the second region line in the target region and the slope of the second region line at the second position. Information processing equipment.
[Item 4]
A segment portion for segmenting the image is included.
The detection unit
The boundary portion of the segment of the roadway where the segment of the roadway is adjacent to the other segment, the boundary portion of the other segment where the other segment is adjacent to the segment of the roadway, or
The information processing apparatus according to any one of items 1 to 3, wherein the edge portion of the other segment on the side opposite to the object on the roadway adjacent to the other object is the first area line.
[Item 5]
The information processing device according to item 4, wherein the other segment is a traffic line segment.
[Item 6]
The information processing apparatus according to any one of items 1 to 5, wherein the inclination at the first position is an inclination of a tangent line with the first area line at the first position.
[Item 7]
The information processing device according to any one of items 1 to 6, wherein the estimation unit further estimates the gradient based on an angle in the optical axis direction of the image pickup device that has captured the image.
[Item 8]
The estimation unit is based on a function in which a first input variable having the slope as an input, a second input variable having the first position as an input, and an output variable having the slope as an output correspond to each other. The information processing apparatus according to any one of items 1 to 7.
[Item 9]
The image includes the preceding vehicle,
The information processing device according to item 2, wherein the target area is an area including at least a part of the preceding vehicle.
[Item 10]
The information processing device according to any one of items 1 to 9, further comprising an acquisition unit for acquiring the image from an image pickup device mounted on a moving body.
[Item 11]
The information processing device according to item 10, wherein the image pickup device is a monocular camera.
[Item 12]
The estimation unit identifies a fourth position on the image corresponding to the third position on the roadway based on the depth information of the roadway, and based on the coordinates of the fourth position, the slope of the road surface at the third position. The information processing apparatus according to any one of items 1 to 11.
[Item 13]
The image is an image acquired by an imaging device arranged at a predetermined height from the first plane.
The estimation unit is the coordinates of the fourth position in the image corresponding to the third position on the roadway and the coordinates of the sixth position in the image corresponding to the fifth position where the third position is projected onto the first plane. The information processing apparatus according to any one of items 1 to 12, which estimates the slope of the road surface at the third position based on the above.
[Item 14]
The information processing device according to item 12, wherein the third position is a position on the roadway where a preceding vehicle exists.
[Item 15]
An imaging device that captures the space including the roadway,
In the image acquired by the image pickup device, a detection unit that detects a first region line that identifies the region of the roadway, and a detection unit.
An estimation unit that estimates the slope of the road surface corresponding to the first position based on the coordinates of the first position on the first area line and the slope of the first area line at the first position.
Imaging system equipped with.
[Item 16]
In the image of the space including the roadway, the first area line that specifies the area of the roadway is detected.
An information processing method for estimating the slope of the road surface corresponding to the first position based on the coordinates of the first position on the first area line and the slope of the first area line at the first position.
[Item 17]
In the image of the space including the roadway, the step of detecting the first area line that specifies the area of the roadway, and
Have the computer perform a step of estimating the slope of the road surface corresponding to the first position based on the coordinates of the first position on the first area line and the inclination of the first area line at the first position. Computer program for.

100, 500: Imaging system 101, 501: Information processing device 110: Image acquisition unit 120: Detection unit 130: Area setting unit 140, 240: Estimating unit 150: Storage unit 201: Imaging device 210: Depth information acquisition unit 301: Vehicle Control system 401: Depth sensor

Claims

In the image of the space including the roadway, the detection unit that detects the first area line that identifies the area of the roadway, and the detection unit.
An estimation unit that estimates the slope of the road surface corresponding to the first position based on the coordinates of the first position on the first area line and the slope of the first area line at the first position.
Information processing device equipped with.
An area setting unit for setting a target area including at least a part of the road surface of the roadway is provided for the image.
The estimation unit has a slope of the road surface included in the target region based on the coordinates of the first position on the first region line in the target region and the inclination of the first region line in the first position. The information processing apparatus according to claim 1.
The detection unit detects the second region line facing the first region line, and
According to claim 2, the estimation unit estimates the gradient based on the coordinates of the second position on the second region line in the target region and the slope of the second region line at the second position. The information processing device described.
A segment portion for segmenting the image is included.
The detection unit
The boundary portion of the segment of the roadway where the segment of the roadway is adjacent to the other segment, the boundary portion of the other segment where the other segment is adjacent to the segment of the roadway, or
The information processing apparatus according to claim 1, wherein the edge portion of the other segment on the side opposite to the object on the roadway adjacent to the other object is the first area line.
The information processing apparatus according to claim 4, wherein the other segment is a traffic line segment.
The information processing apparatus according to claim 1, wherein the inclination at the first position is an inclination of a tangent line with the first area line at the first position.
The information processing device according to claim 1, wherein the estimation unit further estimates the gradient based on an angle in the optical axis direction of the image pickup device that has captured the image.
The estimation unit is based on a function in which a first input variable having the slope as an input, a second input variable having the first position as an input, and an output variable having the slope as an output correspond to each other. The information processing apparatus according to claim 1.
The image includes the preceding vehicle,
The information processing device according to claim 2, wherein the target area is an area including at least a part of the preceding vehicle.
The information processing device according to claim 1, further comprising an acquisition unit that acquires the image from an image pickup device mounted on a moving body.
The information processing device according to claim 10, wherein the image pickup device is a monocular camera.
The estimation unit identifies a fourth position on the image corresponding to the third position on the roadway based on the depth information of the roadway, and based on the coordinates of the fourth position, the slope of the road surface at the third position. The information processing apparatus according to claim 1.
The image is an image acquired by an imaging device arranged at a predetermined height from the first plane.
The estimation unit is the coordinates of the fourth position in the image corresponding to the third position on the roadway and the coordinates of the sixth position in the image corresponding to the fifth position where the third position is projected onto the first plane. The information processing apparatus according to claim 1, wherein the slope of the road surface at the third position is estimated based on the above.
The information processing device according to claim 12, wherein the third position is a position where a preceding vehicle on the roadway exists.
An imaging device that captures the space including the roadway,
In the image acquired by the image pickup device, a detection unit that detects a first region line that identifies the region of the roadway, and a detection unit.
An estimation unit that estimates the slope of the road surface corresponding to the first position based on the coordinates of the first position on the first area line and the slope of the first area line at the first position.
Imaging system equipped with.
In the image of the space including the roadway, the first area line that specifies the area of the roadway is detected.
An information processing method for estimating the slope of the road surface corresponding to the first position based on the coordinates of the first position on the first area line and the slope of the first area line at the first position.
In the image of the space including the roadway, the step of detecting the first area line that specifies the area of the roadway, and
Have the computer perform a step of estimating the slope of the road surface corresponding to the first position based on the coordinates of the first position on the first area line and the inclination of the first area line at the first position. Computer program for.