WO2020090250A1

WO2020090250A1 - Image processing apparatus, image processing method and program

Info

Publication number: WO2020090250A1
Application number: PCT/JP2019/036083
Authority: WO
Inventors: 之寛斉藤
Original assignee: ソニー株式会社
Priority date: 2018-10-31
Filing date: 2019-09-13
Publication date: 2020-05-07

Abstract

A straight line detection unit 21 uses an image of the vicinity of an imaging position in a captured image generated by an imaging device 10 to carry out a Hough transform, and detects a coordinate position in a ρ-θ space, which is a coordinate position that is in a polar coordinate space and indicates a straight line in the captured image. On the basis of the distance between the coordinate position detected by the straight line detection unit 21 and a prescribed coordinate position, a filter processing unit 22 extracts from the captured image a straight line indicated by the prescribed coordinate position. An extension unit 23 sets a search reference region that has a prescribed size and includes a straight line selected by the filter processing unit 22, sets a boundary search region in a line segment direction of a boundary seed, which is the straight line selected within the search reference region, and detects from the boundary search region, as a boundary extension line, a straight line which is connected to the boundary seed and for which the inclination with respect to the boundary seed is smaller than a threshold value. An estimated boundary line is formed from a line segment comprising the boundary extension line and the boundary seed. Thus, it is possible to detect a boundary with good precision from a captured image.

Description

Image processing apparatus, image processing method, and program

This technology relates to image processing devices, image processing methods, and programs, and accurately detects boundaries using captured images.

Conventionally, straight lines have been detected and, for example, white lines on roads have been recognized based on the detection results. For example, in Patent Document 1, a straight line penetrating the edge point group is calculated for each window, and a white line candidate point selected from the edge point group penetrating the straight line is filtered to calculate the shape of the road white line. Has been done. Further, in Patent Document 2, straight lines are detected by using a method such as Hough transform or RANSAC (Random Sample Consensus), and a plurality of detected straight lines are filtered to detect white lines. ..

Japanese Patent Laid-Open No. 2004-199341 Japanese Patent Laid-Open No. 2018-041315

By the way, since the time series filter is used in the filter processing of Patent Documents 1 and 2, and the time series filter is performed on the XY plane, the filter processing result may diverge. Further, the detection methods of Patent Document 1 and Patent Document 2 are weak against noise in the road and lack robustness, so that it is not possible to accurately detect, for example, a boundary between a white line and a portion other than the white line.

Therefore, the purpose of this technology is to provide an image processing device, an image processing method, and a program that can accurately detect boundaries using captured images.

The first aspect of this technology is
Filtering processing for selecting a straight line corresponding to the straight line indicated by the predetermined coordinate position from the detected straight lines based on the distance between the coordinate position of the polar coordinate space indicating the straight line detected from the captured image and the predetermined coordinate position The image processing apparatus includes a unit.

In this technique, the straight line detection unit performs Hough transform using an image of an image region near the image capturing position in a captured image generated by an image capturing device provided in a moving body, and a polar coordinate space indicating a straight line in the captured image. The coordinate position in the ρ-θ space, which is the coordinate position, is detected.

The filter processing unit extracts the straight line indicated by the predetermined coordinate position from the captured image based on the distance between the coordinate position detected by the straight line detection unit and the predetermined coordinate position. The predetermined coordinate position is a preset coordinate position or a coordinate position indicating a straight line detected from a captured image generated at a time different from the captured image by the filter processing unit. Further, the predetermined coordinate position is a coordinate position indicating the straight line closest to the moving body in the straight line detected by the straight line detection unit. The filter processing unit sets a permissible range for a predetermined coordinate position and uses a coordinate position within the permissible range at the coordinate position detected by the straight line detection unit. The allowable range is set according to the moving state of the moving body or the moving environment. Further, when the straight line corresponding to the straight line indicated by the predetermined coordinate position cannot be selected from the detected straight lines, the filter processing unit outputs the lost determination information indicating that the corresponding straight line cannot be selected.

Further, the decompression unit sets a search reference area of a predetermined size including the straight line selected by the filter processing unit, and sets the boundary search area in the direction of the line segment of the boundary seed that is the selected straight line in the search reference area. Then, a straight line having a gradient with respect to the boundary seed smaller than the threshold value and connected to the boundary seed is extracted from the boundary search region as a boundary extension line. In addition, the decompression unit sets a new boundary search region adjacent to the extracted boundary decompression line in the direction of the line segment of the extracted boundary decompression line according to the fact that the boundary decompression line is extracted from the boundary decompression line. A straight line whose inclination to the line is smaller than the threshold value and which is connected to the boundary extension line is extracted from the new boundary search region as a boundary extension line. In addition, the decompression unit sets the curvature range of the estimated boundary line formed by the boundary seed and the boundary expansion line based on the area size of the boundary search area and the threshold value.

The second aspect of this technology is
Based on the distance between the coordinate position of the polar coordinate space indicating the straight line detected from the captured image and the predetermined coordinate position, a straight line corresponding to the straight line indicated by the predetermined coordinate position is obtained from the detected straight line by the filter processing unit. An image processing method including selecting.

The third aspect of this technology is
A program that causes a computer to perform image processing of a captured image,
A procedure for selecting, from the detected straight lines, a straight line corresponding to the straight line indicated by the predetermined coordinate position based on the distance between the coordinate position of the polar coordinate space indicating the straight line detected from the captured image and the predetermined coordinate position. In the program that is executed by the computer.

Note that the program of the present technology is, for example, a storage medium or a communication medium provided in a computer-readable format to a general-purpose computer capable of executing various program codes, for example, a storage medium such as an optical disk, a magnetic disk, or a semiconductor memory. It is a program that can be provided by a medium or a communication medium such as a network. By providing such a program in a computer-readable format, processing according to the program is realized on the computer.

It is the figure which illustrated the composition of the image processing device. It is a figure for explaining Hough transform. It is a figure showing an operation example of Hough conversion. It is a figure for demonstrating the detection operation of the straight line based on an edge point. It is a figure showing an example of operation of a filter processing part. It is a figure for demonstrating operation | movement of an expansion part. It is a figure for demonstrating operation | movement of an expansion part. 6 is a flowchart illustrating the operation of the image processing apparatus. It is a flowchart which shows a filter process. It is a flow chart which shows decompression processing. 2 is a block diagram showing a schematic configuration example of a vehicle control system 100. FIG.

Hereinafter, modes for carrying out the present technology will be described. The description will be given in the following order.
1. Configuration of image processing apparatus 2. Operation of image processing apparatus 3. Application example

<1. Configuration of image processing device>
FIG. 1 illustrates the configuration of the image processing apparatus. The image processing device 20 has a straight line detection unit 21, a filter processing unit 22, and an expansion unit 23.

The straight line detection unit 21 detects the coordinate position in the polar coordinate space indicating the straight line from the captured image generated by the image capturing apparatus 10, and detects the coordinate position of the straight line that is a candidate for the boundary. The straight line detection unit 21 detects a straight line using, for example, Hough transform, and outputs the coordinate position in the ρ-θ space indicating the detected straight line (also referred to as a candidate straight line) to the filter processing unit 22 as candidate straight line information. Further, the straight line detection unit 21 suppresses erroneous detection of a candidate straight line by setting the straight line detection area for detecting the straight line in the vicinity of the neighborhood and the range where the boundary appears in the detection of the straight line.

The filter processing unit 22 detects a straight line corresponding to the straight line indicated by the predetermined coordinate position based on the distance between the coordinate position in the polar coordinate space indicating the straight line detected by the straight line detection unit 21 from the captured image and the predetermined coordinate position. Select from the straight lines. The filter processing unit 22 uses a time series filter such as a Kalman filter, an extended Kalman filter, a sigma point Kalman filter, or a particle filter. The filter processing unit 22 filters the candidate straight line indicated by the candidate straight line information supplied from the straight line detection unit 21, and selects the candidate straight line corresponding to the straight line indicating the boundary indicated by the predetermined coordinate position. In the filter processing unit 22, the coordinate position of the straight line indicating the boundary in the polar coordinate space is set in advance as a predetermined coordinate position (also referred to as a reference position). The filter processing unit 22 performs a time-series filter process to determine a coordinate position of the candidate straight line detected by the straight line detection unit 21 from the coordinate position to the reference position that is the shortest and is shorter than a preset threshold value. Select a straight line. In addition, the filter processing unit 22 sets the coordinate position of the selected straight line as a new reference position, and determines the coordinate position closest to the reference position from the coordinate position indicating the candidate straight line detected by the straight line detection unit 21 from the subsequent captured image. Select the straight line as described above. The filter processing unit 22 outputs selected straight line information indicating a straight line (also referred to as a selected straight line) selected from the candidate straight lines to the decompression unit 23. Further, when the straight line corresponding to the straight line indicated by the reference position cannot be selected from the candidate straight lines, the filter processing unit 22 outputs the lost determination information indicating that the straight line corresponding to the straight line indicated by the reference position cannot be selected. To do.

The expansion unit 23 sets a search reference area of a predetermined size including the selected straight line indicated by the selected straight line information from the filter processing unit 22. Further, the decompressing unit 23 sets a boundary search area in the direction of the line segment of the boundary seed with the portion included in the search reference area in the selected straight line as a boundary seed, and the inclination from the boundary search area to the boundary seed is smaller than the threshold value. A straight line connected to the boundary seed is extracted as a boundary extension line. Further, in response to the extraction of the boundary extension line, the expansion unit 23 newly sets a boundary search area adjacent to the boundary extension line in the direction of the line segment. The decompression unit 23 extracts a straight line that has a smaller inclination with respect to the boundary extension line than the threshold value and that is connected to the boundary extension line from the newly set boundary search area, and sets it as a new boundary extension line. Similarly, the above processing is repeated until the boundary extension line cannot be extracted or until the boundary extension line is continuously extracted a predetermined number of times. The decompression unit 23 outputs the information indicating the estimated boundary line to the outside, with the boundary seed extracted by the filter processing unit 22 and the boundary expansion line connected to the boundary seed as the estimated boundary line. Further, the decompression unit 23 may output a captured image on which an image indicating the estimated boundary line is superimposed, as the output of the information indicating the estimated boundary line.

<2. Operation of image processing device>
Next, the operation of the image processing apparatus will be described. The straight line detection unit 21 performs straight line detection using, for example, Hough transform, and converts a straight line passing through a point on the xy plane into a coordinate position in a ρ-θ space which is a polar coordinate space. FIG. 2 is a diagram for explaining the Hough transform. FIG. 2A illustrates a straight line L on the xy plane, and the straight line passing through the point (xi, yi) is the distance between the origin and the straight line as a parameter ρ, and the angle between the normal line of the straight line and the x axis. Is a parameter θ, it can be expressed by the relationship of the equation (1). The relationship between the parameter ρ and the parameter θ of the straight line L passing through the point (xi, yi) has the characteristic shown in (b) of FIG.
ρ = xicos θ + yisin θ (1)

Fig. 3 shows an operation example of Hough conversion. For example, when the captured image shown in FIG. 3A is input, the straight line detection unit 21 performs a filter process for removing noise without blurring the contour of the captured image, for example, a filter process using a bilateral filter or the like. Then, the binarized edge image shown in FIG. 3B is generated from the captured image after the filtering process. Further, the straight line detection unit 21 projects the edge points of the binarized edge image in the ρ-θ space, and the parameters ρ and θ of the points where the characteristic curves of the respective edge points overlap correspond to the characteristic curve in which the overlapping occurs. Let ρ and θ be the parameters of the straight line that passes through each edge point. Further, the straight line detection unit 21 sets the straight line detection area AS in the vicinity of the neighborhood and in the range where the boundary appears, as shown in FIG. For example, the imaging device 10 is provided in a moving body (for example, a vehicle), and the straight line detection unit 21 uses a captured image obtained by imaging the front of the imaging device. In this case, the boundary line extending in front of the vehicle is displayed as a straight line in which the boundary line having the same length is longer on the captured image in the near region than in the distant region. For this reason, when performing with a predetermined length (section length) for determining whether or not the straight line is the boundary line, there is a possibility that the straight line cannot be detected with the accuracy of the section length when the image of the distant area is used. Therefore, the straight line detection unit 21 can determine whether or not the straight line is the boundary line with the accuracy of the section length by setting the straight line detection area AS in the neighborhood area, and the erroneous detection of the candidate straight line is performed. Can be suppressed. Further, the calculation cost of the straight line detection can be reduced as compared with the case where the straight line detection area AS is not set.

FIG. 4 is a diagram for explaining a straight line detection operation based on edge points. For example, assume that edge points (x1, y1) (x2, y2) (x3, y3) are detected on the xy plane as shown in FIG. Further, as shown in FIG. 4B, characteristic curves for each edge point in the ρ-θ space are curves CLa, CLb, CLc. In this case, since the curves CLa, CLb, and CLc overlap at the coordinate position (ρs, θs), the straight line corresponding to the coordinate position (ρs, θs) is an edge point (indicated by a broken line in FIG. 4A). The straight line SL passes through x1, y1) (x2, y2) (x3, y3). Therefore, the straight lines La1 to Lan in the straight line detection area AS can be detected as shown in FIG. 3 (c) based on the edge points detected in the straight line detection area AS shown in FIG. 3 (b).

The filter processing unit 22 performs time-series filter processing on the candidate straight line information in the ρ-θ space, and selects a candidate straight line close to a predetermined preset initial value.

For example, on the xy plane, a straight line is defined as shown in equation (2) using the slope a and the intercept b.
y = ax + b (2)

Thus, when a straight line is defined by the slope a and the intercept b, the candidate straight line of the boundary does not have continuity in the slope a and the intercept b with respect to time, and the candidate straight line indicating the boundary is accurately selected by the time series filtering process. Difficult to do. When the candidate straight line is horizontal to the y-axis, the slope a diverges to infinity, which makes it impossible to perform the filter calculation. However, in the ρ-θ space, a line that is close to the boundary candidate line with respect to time has a characteristic that it has continuity and does not change abruptly. Therefore, it is possible to select a boundary straight line from the candidate lines.

The filter processing unit 22 performs time-series filter processing using the candidate straight line information, and the distance from the coordinate position of the candidate straight line detected by the straight line detection unit 21 to the reference position is the shortest and shorter than a preset threshold value. Select the straight line at the coordinate position that is the distance. As the distance, for example, the Euclidean distance in the ρ-θ space may be used, or the Mahalanobis distance using the covariance matrix of the Kalman filter may be used. The filter processing unit 22 selects the straight line having the shortest calculated distance from the candidate straight lines.

Here, when the set of candidate straight lines detected by the straight line detection unit 21 is LF = {Lf1, Lf2, ... Lfn} and the straight line corresponding to the reference position is the straight line Ltg, the distance is calculated based on Expression (3). A candidate straight line Lfdmin having the shortest dist (Lfi, Ltg) is selected.
Lfdmin st min (dist (Lfi, Ltg)), LfiεLF
... (3)

Further, the filter processing unit 22 sets the coordinate position indicating the selected straight line to a new reference position, and filters the candidate straight lines detected from the captured image thereafter.

FIG. 5 shows an operation example of the filter processing unit. FIG. 5A illustrates the candidate straight lines detected from the captured image at time t0 in the ρ-θ space. In addition, FIG. 5B illustrates the candidate straight lines detected from the captured image at time t1 after time t0 in the ρ-θ space. The black circles represent candidate straight lines.

Here, as shown in (a) of FIG. 5, when the reference position is previously set to the coordinate position PSr (white circle position), the filter processing unit 22 has the shortest distance from the coordinate position PSr and is set in advance. The straight line indicated by the coordinate position PSLt0, which is a distance shorter than the threshold value, is set as the selected straight line. Further, with the coordinate position PSLt0 as the reference position at the time t1, as shown in FIG. 5B, the coordinate position PSLt0 has the shortest distance from the coordinate position PSLt0 and is indicated by the coordinate position PSLt1 which is shorter than the preset threshold value. Select the straight line as the selected straight line. Similarly, a selected straight line is tracked in the time direction by sequentially selecting a straight line whose coordinate position is closest to the reference position and has a distance shorter than a preset threshold value. The filter processing unit 22 outputs the selected straight line information indicating the selected straight line to the decompression unit 23.

In addition, when the filter processing unit 22 cannot select a straight line corresponding to the straight line indicated by the reference position from the candidate straight lines, that is, a straight line at a coordinate position whose distance from the reference position is shorter than a preset threshold value is a candidate. If the straight line cannot be selected, the lost determination information indicating that the straight line corresponding to the straight line indicated by the reference position cannot be selected is output. For example, if a candidate straight line whose distance is shorter than a preset threshold value is not found continuously for a predetermined period, lost determination information indicating that the selected straight line (or boundary seed) is not detected is generated and output to the outside.

Further, the filter processing unit 22 sets the coordinate position indicating the straight line selected from the candidate straight lines based on the predetermined rule to the reference position and starts the time-series filter processing, not only when the reference position is set in advance. Good.

Also, the filter processing unit 22 may adjust the threshold value according to a user operation, a captured image acquisition status, or the like. In this case, since the filtering process is performed in the ρ-θ space, the threshold can be set by the parameter ρ and the parameter θ. For example, the filter processing unit 22 can set the permissible range when the boundary included in the captured image is displaced by adjusting the threshold value of the parameter ρ according to the user operation or the acquisition status of the captured image. Further, the filter processing unit 22 can set the allowable range of continuity (smoothness) of the boundary included in the captured image by adjusting the threshold value of the parameter θ according to the user operation or the acquisition status of the captured image. .. In this way, the filter characteristic of the filter processing unit 22 can be adjusted by the parameter ρ and the parameter θ. Moreover, since the threshold value can be adjusted by the parameter ρ and the parameter θ, the user can easily adjust the filter characteristic to a desired characteristic with the parameter that is easy to understand. Further, since the straight line can be specified by the parameter ρ and the parameter θ, the setting of the selection process becomes easier as compared with the case of selecting the straight line on the xy plane.

The decompression unit 23 performs viewpoint conversion of the captured image. In the viewpoint conversion, the vertical direction is the viewpoint direction with respect to the plane where the selected straight line indicated by the selected straight line information is located. The viewpoint conversion may be performed using camera parameters that are acquired in advance for the imaging device 10 that generated the captured image. By performing the viewpoint conversion in this way, the boundary included in the captured image can be represented by a two-dimensional plane, and the boundary extension line can be easily detected.

The decompressing unit 23 sets the search reference area so as to include the selection straight line with the position of the selection straight line indicated by the selection straight line information from the filter processing unit 22 as a reference, and uses the straight line in the search reference region as a boundary seed. To do. Further, the decompression unit 23 sets a boundary search area (search segment) in the direction of the line segment of the boundary seed and performs Hough transform of the boundary search area, and the inclination with respect to the boundary seed extracted from the boundary search area by the filter processing unit 22 is A straight line smaller than the threshold value and connected to the boundary seed is extracted as a boundary extension line. The boundary search area and the threshold value are set in advance so that a boundary line having a curvature larger than the curvature specified by the manufacturer of the moving body or the user can be detected. For example, in the search reference area and the boundary search area, the length of the straight line in the direction of the line segment is a section length for determining whether or not the straight line is the boundary line. In the boundary search area, the length of the straight line in the direction orthogonal to the line segment direction is set so as to include a line segment whose inclination with respect to the line segment of the boundary seed or the boundary extension line is a threshold value.

6 and 7 are diagrams for explaining the operation of the decompression unit, and the imaging device 10 is provided in, for example, a moving body. 6A illustrates the captured image at time tb1, and FIG. 7A illustrates the captured image at time tb2, which is later than time tb1. 6B shows a part of the bird's-eye view of the captured image at time tb1, and FIG. 7B shows a part of the bird's-eye view of the captured image at time tb2 for reference.

When the imaging device 10 is provided on the moving body, the straight line detecting unit 21 detects a candidate straight line from the straight line detecting area provided in the vicinity position as described above, for example, the straight line detecting area near the front of the moving body. There is. 6C to 6G are bird's-eye views showing the moving body OBM and the front area in the captured image at time tb1, and FIGS. 7C to 7G show the moving body OBM in the captured image at time tb2. The bird's-eye view which shows a front area is illustrated. Further, at the time tb1, the curve area EG-c of the boundary EG is separated from the moving body OBM, and at the time tb2, the curve area EG-c of the boundary EG is close to the moving body OBM.

(D) of FIG. 6 shows the search reference area AEs including the selected straight line detected from the captured image at the time tb1 and selected by the filter processing unit 22. The boundary seed LEs is a selected straight line portion included in the search reference area AEs. The decompressing unit 23 sets the boundary search area AEa adjacent to the search reference area AEs in the line segment direction of the boundary seed LEs with the position of the boundary seed LEs as a reference, as shown in (e) of FIG. In addition, the decompression unit 23 performs the Hough transform on the boundary search area AEa, and as shown in (f) of FIG. 6, a straight line whose inclination with respect to the boundary seed LEs is smaller than the threshold value and which is connected to the boundary seed LEs from the boundary search area AEa. The boundary extension line LEa is detected. Further, the decompression unit 23 newly sets the boundary search area AEa in the line segment direction based on the position of the boundary extension line LEa in response to the extraction of the boundary extension line LEa. The decompression unit 23 extracts a straight line whose inclination with respect to the boundary decompression line extracted immediately before is smaller than a threshold value and is connected to the boundary decompression line from the newly set boundary search area AEa, and sets it as a new boundary decompression line LEa. Similarly, the decompression unit 23 repeats the above process until the boundary decompression line cannot be extracted or until the boundary decompression line is continuously extracted a predetermined number of times. As shown in (g) of FIG. 6, the decompression unit 23 uses the boundary seed LEs extracted by the filter processing unit 22 and the boundary decompression line LEa sequentially extracted as the estimated boundary line LE to output information indicating the estimated boundary line to the outside. Output to. Further, the decompression unit 23 may output a captured image on which an image indicating the estimated boundary line is superimposed, as the output of the information indicating the estimated boundary line.

FIG. 7D shows the search reference area AEs including the selected straight line detected by the captured image at the time tb2 and selected by the filter processing unit 22. The boundary seed LEs is a selected straight line portion included in the search reference area AEs. The decompressing unit 23 sets the boundary search area AEa adjacent to the search reference area AEs in the line segment direction of the boundary seed LEs with the position of the boundary seed LEs as a reference, as shown in (e) of FIG. 7. Further, the decompression unit 23 performs the Hough transform on the boundary search area AEa, and as shown in (f) of FIG. 7, a straight line having a smaller inclination with respect to the boundary seed LEs than the threshold value and connected to the boundary seed LEs is extracted from the boundary search area AEa. The boundary extension line LEa is detected. Therefore, a straight line having an inclination smaller than the threshold value with respect to the boundary seed LEs is also extracted as the boundary extension line LEa. Further, the decompression unit 23 newly sets the boundary search area AEa in the line segment direction based on the position of the boundary extension line LEa in response to the extraction of the boundary extension line LEa. The decompression unit 23 extracts a straight line whose inclination with respect to the boundary decompression line extracted immediately before is smaller than a threshold value and is connected to the boundary decompression line from the newly set boundary search area AEa, and sets it as a new boundary decompression line LEa. Similarly, the decompression unit 23 repeats the above process until the boundary decompression line cannot be extracted or until the boundary decompression line is continuously extracted a predetermined number of times. As shown in (g) of FIG. 7, the decompression unit 23 uses the boundary seed LEs extracted by the filter processing unit 22 and the boundary decompression line LEa sequentially extracted as the estimated boundary line LE to estimate the estimated boundary line. Output information to the outside.

As described above, the decompression unit 23 extracts the straight line extending from the boundary seed selected by the filter processing unit 22 for each boundary search region and connects the straight line as an estimated boundary line, and the estimated boundary line information indicating the estimated boundary line. Can be generated.

Further, the decompression unit 23 may set the curvature range of the estimated boundary line according to the area size of the boundary search area and the threshold value. For example, the decompression unit 23 can limit the estimated boundary line to a line segment having a small curvature by increasing the size of the boundary seed or the direction of the boundary expansion line in the boundary search region or decreasing the threshold value. In addition, the expansion unit 23 can also make a line segment having a large curvature an estimated boundary line by increasing the size in the direction of the boundary expansion line or decreasing the threshold value. Therefore, the extension unit 23 may preset the detectable curvature range of the boundary according to the region size and the threshold value.

Furthermore, the decompression unit 23 may output a captured image on which an image indicating the estimated boundary line is superimposed, as the output of the information indicating the estimated boundary line. In this way, by superimposing the image showing the estimated boundary line on the captured image, it becomes possible to easily recognize which position of the captured image is detected as the boundary line.

FIG. 8 is a flowchart illustrating the operation of the image processing apparatus. In step ST1, the image processing device starts acquisition of a captured image. The image processing device 20 starts acquisition of the captured image generated by the imaging device 10 and proceeds to step ST2.

The image processing apparatus performs straight line detection processing in step ST2. The straight line detection unit 21 of the image processing device 20 detects a straight line in the ρ-θ space from the captured image acquired from the imaging device 10, and proceeds to step ST3.

At step ST3, the image processing apparatus performs filter processing. The filter processing unit 22 of the image processing device 20 selects a straight line indicating a boundary from the straight lines detected in step ST2. FIG. 9 is a flowchart showing the filtering process.

In step ST11, the filter processing unit detects a straight line that minimizes the distance. The filter processing unit 22 detects, from the candidate straight lines detected by the straight line detection unit 21, the straight line having the minimum distance based on the equation (3). When first performing the filter processing, the filter processing unit 22 determines the distance between the coordinate position of the straight line detected by the straight line detection unit in the ρ-θ space and the preset predetermined coordinate position (reference position) for each candidate straight line. Then, the straight line with the minimum distance is detected. Further, when the predetermined coordinate position is updated in the setting update process of step ST13 described later, the updated coordinate position is used to detect the straight line having the smallest distance from the candidate straight lines detected after the update. The distance between the coordinate position of the detected candidate straight line Lfdmin and the coordinate position of the straight line Ltg corresponding to the reference position is the minimum distance dist (Lfdmin, Ltg). The filter processing unit 22 performs the time-series filter processing in this way, detects the straight line with the minimum distance, and proceeds to step ST12.

In step ST12, the filter processing unit determines whether the minimum distance is smaller than the threshold value. If the minimum distance dist (Lfdmin, Ltg), which is the distance between the coordinate position of the straight line detected in step ST11 and the predetermined coordinate position, is smaller than the threshold value, the filter processing unit 22 proceeds to step ST13, and the minimum distance dist (Lfdmin , Ltg) is greater than or equal to the threshold value, the process proceeds to step ST14.

In step ST13, the filter processing unit performs setting update processing. The filter processing unit 22 sets the straight line detected in step ST11 as the selected straight line, and sets the coordinate position indicating this selected straight line as the predetermined coordinate position.

In step ST14, the filter processing unit outputs the lost judgment information. Since the minimum distance dist (Lfdmin, Ltg) is determined to be greater than or equal to the threshold, the filter processing unit 22 indicates that a straight line close to a predetermined coordinate position, that is, a straight line that can be regarded as a boundary line cannot be detected. Outputs lost judgment information.

Returning to FIG. 8, when the process proceeds from step ST3 to step ST4, the image processing apparatus performs decompression processing. The decompression unit 23 of the image processing device 20 decompresses the line segment indicating the boundary by detecting a straight line following the straight line selected by the filtering process. FIG. 10 is a flowchart showing the decompression process.

At step ST21, the decompression unit performs viewpoint conversion. The decompression unit 23 performs viewpoint conversion of the captured image with respect to the plane where the straight line selected by the filtering process is located, with the vertical direction being the viewpoint direction, and proceeds to step ST22.

In step ST22, the extension unit sets the boundary seed. The decompression unit 23 sets the search reference area so as to include the selected straight line, with the position of the straight line selected by the filter processing unit 22 as a reference. Further, the decompression unit 23 performs boundary seeding on the straight line in the search reference area and proceeds to step ST23.

In step ST23, the decompression unit sets the boundary search area. The extension unit 23 sets the boundary search area adjacent to the search reference area in the direction of the line segment of the boundary seed, with the position of the boundary seed as a reference. In addition, when the boundary extension line is detected as a straight line, the extension unit 23 sets the boundary search area on the basis of the position of the boundary extension line so as to be adjacent to the boundary search area where the boundary extension line is detected as a straight line. In this way, the decompression unit 23 sets the boundary search area and proceeds to step ST24.

In step ST24, the decompression unit detects a straight line in the boundary search area. The decompression unit 23 detects a straight line included in the boundary search area set in step ST23 and proceeds to step ST25.

At step ST25, the extension unit determines whether or not a straight line with a close inclination can be detected. The decompression unit 23 calculates the slope with respect to the straight line detected in step ST24 and the boundary seed or the boundary expansion line, and if the straight line having the smallest slope and smaller than the threshold is detected in step ST24, the slope is the smallest and A straight line smaller than the threshold is set as a boundary extension line and the process returns to step ST23. In addition, the decompression unit 23 ends the decompression process when a straight line whose inclination is smaller than the threshold value cannot be detected.

Returning to FIG. 8, when the process proceeds from step ST4 to step ST5, the image processing device performs boundary line information output processing. The image processing device 20 outputs the estimated boundary line information indicating the estimated boundary line detected by the processes of step ST3 and step ST4 to the external device. In addition, the image processing apparatus 20 outputs the lost determination information to the external device when a straight line whose minimum distance is smaller than the threshold value is not detected in the filter process of step ST3.

According to the present technology as described above, since the straight line indicating the boundary is selected by the time series filter processing in the ρ-θ space, the boundary can be detected with higher accuracy than in the case of detecting the straight line indicating the boundary on the xy plane. it can.

Further, since the straight line is specified in the ρ-θ space, the filter characteristic of the filter processing unit, that is, the tracking characteristic of the selected straight line can be made a desired characteristic by adjusting the threshold value in the ρ-θ space, for example. Become. For example, the above-mentioned parameter ρ can specify the allowable range of the positional deviation of the straight line. Further, the allowable range of continuity (smoothness) of the straight line can be designated by the parameter θ.

Furthermore, since the boundary search area is set and the boundary seed is expanded, the processing is easier than in the case of estimating the boundary line by function approximation. Further, since only the straight lines in the boundary search area are candidates and the boundary lines are estimated based on the continuity of the straight lines, robustness against noise and the like can be improved. Furthermore, since the boundary line is estimated based on the continuity of straight lines on the bird's-eye view, it is possible to deal with the case where the boundary line has a curvature when turning right or left.

Note that, in the above-described image processing device, the case where the straight line is detected by using the Hough transform is illustrated, but the straight line may be detected by another method. For example, the straight line detection unit 21 detects a straight line using RANSAC (Random Sample Consensus) or the least squares method, and the filter processing unit 22 performs filter processing using the coordinate position of the polar coordinate space indicating the detected straight line. You may do it.

<3. Application example>
The technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure is applicable to any type of vehicle, electric vehicle, hybrid electric vehicle, motorcycle, bicycle, personal mobility, carrier vehicle used in factories, robots, construction machines, agricultural machines (tractors), and the like. May be realized as a device mounted on the moving body.

FIG. 11 is a block diagram showing a schematic configuration example of a vehicle control system 100 which is an example of a mobile body control system to which the technology according to the present disclosure can be applied.

Note that, hereinafter, when distinguishing a vehicle provided with the vehicle control system 100 from other vehicles, the vehicle is referred to as the own vehicle or the own vehicle.

The vehicle control system 100 includes an input unit 101, a data acquisition unit 102, a communication unit 103, an in-vehicle device 104, an output control unit 105, an output unit 106, a drive system control unit 107, a drive system system 108, a body system control unit 109, and a body. The system 110, the storage unit 111, and the automatic operation control unit 112 are provided. The input unit 101, the data acquisition unit 102, the communication unit 103, the output control unit 105, the drive system control unit 107, the body system control unit 109, the storage unit 111, and the automatic driving control unit 112, via the communication network 121, Connected to each other. The communication network 121 is, for example, an in-vehicle communication network or a bus compliant with any standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), or FlexRay (registered trademark). Become. In addition, each part of the vehicle control system 100 may be directly connected without going through the communication network 121.

Note that, hereinafter, when each unit of the vehicle control system 100 communicates via the communication network 121, the description of the communication network 121 is omitted. For example, when the input unit 101 and the automatic driving control unit 112 communicate with each other via the communication network 121, it is simply described that the input unit 101 and the automatic driving control unit 112 communicate with each other.

The input unit 101 includes a device used by the passenger to input various data and instructions. For example, the input unit 101 includes an operation device such as a touch panel, a button, a microphone, a switch, and a lever, and an operation device that can be input by a method other than a manual operation such as voice or gesture. Further, for example, the input unit 101 may be a remote control device that uses infrared rays or other radio waves, or an externally connected device such as a mobile device or a wearable device that corresponds to the operation of the vehicle control system 100. The input unit 101 generates an input signal based on the data and instructions input by the passenger, and supplies the input signal to each unit of the vehicle control system 100.

The data acquisition unit 102 includes various sensors that acquire data used for processing of the vehicle control system 100, and supplies the acquired data to each unit of the vehicle control system 100.

For example, the data acquisition unit 102 includes various sensors for detecting the state of the own vehicle and the like. Specifically, for example, the data acquisition unit 102 includes a gyro sensor, an acceleration sensor, an inertial measurement unit (IMU), an accelerator pedal operation amount, a brake pedal operation amount, a steering wheel steering angle, and an engine speed. It is provided with a sensor or the like for detecting the number of rotations of the motor or the rotation speed of the wheels.

Further, for example, the data acquisition unit 102 includes various sensors for detecting information outside the vehicle. Specifically, for example, the data acquisition unit 102 includes an imaging device such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. In addition, for example, the data acquisition unit 102 includes an environment sensor for detecting weather or weather, and an ambient information detection sensor for detecting an object around the vehicle. The environment sensor includes, for example, a raindrop sensor, a fog sensor, a sunshine sensor, a snow sensor, and the like. The ambient information detection sensor includes, for example, an ultrasonic sensor, a radar, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), a sonar, and the like.

Further, for example, the data acquisition unit 102 includes various sensors for detecting the current position of the vehicle. Specifically, for example, the data acquisition unit 102 includes a GNSS receiver that receives a GNSS signal from a GNSS (Global Navigation Satellite System) satellite.

Further, for example, the data acquisition unit 102 includes various sensors for detecting information inside the vehicle. Specifically, for example, the data acquisition unit 102 includes an imaging device that images the driver, a biometric sensor that detects biometric information of the driver, a microphone that collects sound in the vehicle interior, and the like. The biometric sensor is provided on, for example, a seat surface or a steering wheel, and detects biometric information of an occupant sitting on a seat or a driver who holds the steering wheel.

The communication unit 103 communicates with the in-vehicle device 104 and various devices outside the vehicle, a server, a base station, etc., and transmits data supplied from each unit of the vehicle control system 100 or receives received data from the vehicle control system. It is supplied to each part of 100. The communication protocol supported by the communication unit 103 is not particularly limited, and the communication unit 103 can also support a plurality of types of communication protocols. For example, the communication unit 103 uses a wireless LAN. , Bluetooth (registered trademark), NFC (Near Field Communication), WUSB (Wireless USB), or the like to perform wireless communication with the in-vehicle device 104. Further, for example, the communication unit 103 uses a USB (Universal Serial Bus), HDMI (registered trademark) (High-Definition Multimedia Interface), or MHL (via a connection terminal (and a cable, if necessary), not shown. Mobile High-definition Link) is used to perform wired communication with the in-vehicle device 104.

Further, for example, the communication unit 103 communicates with a device (for example, an application server or a control server) existing on an external network (for example, the Internet, a cloud network, or a network unique to a business operator) via a base station or an access point. Communicate. In addition, for example, the communication unit 103 uses a P2P (PeerToPeer) technology to communicate with a terminal (for example, a pedestrian or a shop terminal, or an MTC (MachineType Communication) terminal) that exists near the vehicle. Communicate. Furthermore, for example, the communication unit 103 may communicate between the vehicle and the vehicle, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication. ) Perform V2X communication such as communication. In addition, for example, the communication unit 103 includes a beacon receiving unit, receives radio waves or electromagnetic waves transmitted from a wireless station installed on the road, and obtains information such as the current position, traffic congestion, traffic regulation, and required time. To do.

The in-vehicle device 104 includes, for example, a mobile device or a wearable device that the passenger has, an information device that is carried in or attached to the vehicle, a navigation device that searches for a route to an arbitrary destination, and the like.

The output control unit 105 controls the output of various information to the passengers of the own vehicle or the outside of the vehicle. For example, the output control unit 105 generates an output signal including at least one of visual information (for example, image data) and auditory information (for example, audio data), and supplies the output signal to the output unit 106 to output the output unit. The output of visual information and auditory information from 106 is controlled. Specifically, for example, the output control unit 105 synthesizes image data captured by different imaging devices of the data acquisition unit 102 to generate a bird's-eye view image or a panoramic image, and outputs an output signal including the generated image. It is supplied to the output unit 106. Further, for example, the output control unit 105 generates voice data including a warning sound or a warning message for a danger such as collision, contact, or entry into a dangerous zone, and outputs an output signal including the generated voice data to the output unit 106. Supply.

The output unit 106 includes a device capable of outputting visual information or auditory information to a passenger of the vehicle or outside the vehicle. For example, the output unit 106 includes a display device, an instrument panel, an audio speaker, headphones, a wearable device such as a glasses-type display worn by a passenger, a projector, a lamp, and the like. The display device included in the output unit 106 includes visual information in the driver's visual field, such as a device having a head-up display, a transmissive display, and an AR (Augmented Reality) display function, in addition to a device having a normal display. It may be a display device.

The drive system control unit 107 controls the drive system system 108 by generating various control signals and supplying them to the drive system system 108. Further, the drive system control unit 107 supplies a control signal to each unit other than the drive system system 108 as necessary to notify the control state of the drive system system 108 and the like.

The drive system 108 includes various devices related to the drive system of the vehicle. For example, the drive system system 108 includes a drive force generation device for generating a drive force of an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to wheels, a steering mechanism for adjusting a steering angle, It is equipped with a braking device that generates a braking force, an ABS (Antilock Brake System), an ESC (Electronic Stability Control), an electric power steering device, and the like.

The body system control unit 109 controls the body system 110 by generating various control signals and supplying them to the body system 110. Further, the body system control unit 109 supplies a control signal to each unit other than the body system system 110 as necessary to notify the control state of the body system system 110 and the like.

The body system 110 includes various body-related devices mounted on the vehicle body. For example, the body system 110 includes a keyless entry system, a smart key system, a power window device, a power seat, a steering wheel, an air conditioner, and various lamps (for example, headlights, backlights, brake lights, blinkers, fog lights, etc.). And so on.

The storage unit 111 includes, for example, a magnetic storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an HDD (Hard Disc Drive), a semiconductor storage device, an optical storage device, and a magneto-optical storage device. .. The storage unit 111 stores various programs and data used by each unit of the vehicle control system 100. For example, the storage unit 111 stores map data such as a three-dimensional high-precision map such as a dynamic map, a global map having a lower accuracy than the high-precision map and covering a wide area, and a local map including information around the vehicle. Memorize

The automatic driving control unit 112 performs control related to autonomous driving such as autonomous driving or driving support. Specifically, for example, the automatic driving control unit 112 may perform collision avoidance or impact mitigation of the own vehicle, follow-up traveling based on inter-vehicle distance, vehicle speed maintenance traveling, collision warning of the own vehicle, lane departure warning of the own vehicle, or the like. Coordinated control for the purpose of realizing the functions of ADAS (Advanced Driver Assistance System) including Further, for example, the automatic driving control unit 112 performs cooperative control for the purpose of autonomous driving that autonomously travels without depending on the operation of the driver. The automatic driving control unit 112 includes a detection unit 131, a self-position estimation unit 132, a situation analysis unit 133, a planning unit 134, and an operation control unit 135.

The detection unit 131 detects various kinds of information necessary for controlling automatic driving. The detection unit 131 includes a vehicle exterior information detection unit 141, a vehicle interior information detection unit 142, and a vehicle state detection unit 143.

The outside-vehicle information detection unit 141 performs detection processing of information outside the own vehicle based on data or signals from each unit of the vehicle control system 100. For example, the vehicle exterior information detection unit 141 performs detection processing, recognition processing, and tracking processing of an object around the vehicle, and detection processing of a distance to the object. Objects to be detected include vehicles, people, obstacles, structures, roads, traffic lights, traffic signs, road markings, and the like. Further, for example, the vehicle exterior information detection unit 141 performs a process of detecting the environment around the vehicle. The surrounding environment to be detected includes, for example, weather, temperature, humidity, brightness, and road surface condition. The vehicle exterior information detection unit 141 uses the data indicating the result of the detection process to obtain the self-position estimation unit 132, the map analysis unit 151 of the situation analysis unit 133, the traffic rule recognition unit 152, the situation recognition unit 153, and the operation control unit 135. To the emergency avoidance unit 171 and the like.

The in-vehicle information detection unit 142 performs in-vehicle information detection processing based on data or signals from each unit of the vehicle control system 100. For example, the in-vehicle information detection unit 142 performs driver authentication processing and recognition processing, driver state detection processing, passenger detection processing, and in-vehicle environment detection processing. The driver's state to be detected includes, for example, physical condition, arousal level, concentration level, fatigue level, line-of-sight direction, and the like. The environment inside the vehicle to be detected includes, for example, temperature, humidity, brightness, odor, and the like. The in-vehicle information detection unit 142 supplies the data indicating the result of the detection processing to the situation recognition unit 153 of the situation analysis unit 133, the emergency situation avoidance unit 171 of the operation control unit 135, and the like.

The vehicle state detection unit 143 performs detection processing of the state of the vehicle based on data or signals from each unit of the vehicle control system 100. The state of the vehicle to be detected includes, for example, speed, acceleration, steering angle, presence / absence of abnormality, content of driving operation, position and inclination of power seat, state of door lock, and other in-vehicle devices. State etc. are included. The vehicle state detection unit 143 supplies the data indicating the result of the detection process to the situation recognition unit 153 of the situation analysis unit 133, the emergency situation avoidance unit 171 of the operation control unit 135, and the like.

The self-position estimating unit 132 estimates the position and attitude of the own vehicle based on data or signals from each unit of the vehicle control system 100 such as the vehicle exterior information detecting unit 141 and the situation recognizing unit 153 of the situation analyzing unit 133. Perform processing. The self-position estimation unit 132 also generates a local map (hereinafter, referred to as a self-position estimation map) used for estimating the self-position, if necessary. The self-position estimation map is, for example, a high-precision map using a technology such as SLAM (Simultaneous Localization and Mapping). The self-position estimation unit 132 supplies data indicating the result of the estimation process to the map analysis unit 151, the traffic rule recognition unit 152, the situation recognition unit 153, and the like of the situation analysis unit 133. Further, the self-position estimation unit 132 stores the self-position estimation map in the storage unit 111.

The situation analysis unit 133 analyzes the situation of the vehicle and surroundings. The situation analysis unit 133 includes a map analysis unit 151, a traffic rule recognition unit 152, a situation recognition unit 153, and a situation prediction unit 154.

The map analysis unit 151 uses data or signals from each unit of the vehicle control system 100, such as the self-position estimation unit 132 and the vehicle exterior information detection unit 141, as necessary, and stores various maps stored in the storage unit 111. Performs analysis processing and builds a map containing information necessary for automatic driving processing. The map analysis unit 151 uses the constructed map as a traffic rule recognition unit 152, a situation recognition unit 153, a situation prediction unit 154, a route planning unit 161, a behavior planning unit 162, and a motion planning unit 163 of the planning unit 134. Supply to.

The traffic rule recognition unit 152 determines the traffic rules around the vehicle based on data or signals from the vehicle position control unit 100 such as the self-position estimation unit 132, the vehicle exterior information detection unit 141, and the map analysis unit 151. Perform recognition processing. By this recognition processing, for example, the position and state of the signal around the own vehicle, the content of traffic regulation around the own vehicle, the lane in which the vehicle can travel, and the like are recognized. The traffic rule recognition unit 152 supplies data indicating the result of the recognition process to the situation prediction unit 154 and the like.

The situation recognition unit 153 converts data or signals from the vehicle position control unit 100 such as the self-position estimation unit 132, the vehicle exterior information detection unit 141, the vehicle interior information detection unit 142, the vehicle state detection unit 143, and the map analysis unit 151. Based on this, recognition processing of the situation regarding the own vehicle is performed. For example, the situation recognition unit 153 performs recognition processing of the situation of the own vehicle, the situation around the own vehicle, the situation of the driver of the own vehicle, and the like. The situation recognition unit 153 also generates a local map (hereinafter, referred to as a situation recognition map) used for recognizing the situation around the own vehicle, as necessary. The situation recognition map is, for example, an occupancy grid map (Occupancy Grid Map).

The situation of the subject vehicle to be recognized includes, for example, the position, posture, movement (for example, speed, acceleration, moving direction, etc.) of the subject vehicle, and the presence / absence and content of an abnormality. The situation around the subject vehicle to be recognized is, for example, the type and position of a stationary object in the surroundings, the type and position of a moving object in the surroundings, position and movement (for example, speed, acceleration, moving direction, etc.), and surrounding roads. The configuration and the condition of the road surface, and the surrounding weather, temperature, humidity, and brightness are included. The driver's state to be recognized includes, for example, physical condition, arousal level, concentration level, fatigue level, line-of-sight movement, and driving operation.

The situation recognition unit 153 supplies data indicating the result of the recognition process (including a situation recognition map, if necessary) to the self-position estimation unit 132, the situation prediction unit 154, and the like. The situation recognition unit 153 also stores the situation recognition map in the storage unit 111.

The situation predicting unit 154 performs a process of predicting the situation regarding the own vehicle based on data or signals from each unit of the vehicle control system 100 such as the map analyzing unit 151, the traffic rule recognizing unit 152, and the situation recognizing unit 153. For example, the situation prediction unit 154 performs a prediction process of the situation of the own vehicle, the situation around the own vehicle, the situation of the driver, and the like.

The situation of the subject vehicle to be predicted includes, for example, the behavior of the subject vehicle, the occurrence of abnormality, and the mileage that can be traveled. The situation around the subject vehicle to be predicted includes, for example, the behavior of a moving object around the subject vehicle, a change in the signal state, and a change in the environment such as the weather. The driver's situation to be predicted includes, for example, the driver's behavior and physical condition.

The situation prediction unit 154, together with the data from the traffic rule recognition unit 152 and the situation recognition unit 153, data indicating the result of the prediction process, the route planning unit 161, the action planning unit 162, and the operation planning unit 163 of the planning unit 134. Etc.

The route planning unit 161 plans a route to a destination based on data or signals from each part of the vehicle control system 100 such as the map analysis unit 151 and the situation prediction unit 154. For example, the route planning unit 161 sets a route from the current position to the designated destination based on the global map. Further, for example, the route planning unit 161 appropriately changes the route based on traffic jams, accidents, traffic restrictions, construction conditions, and the physical condition of the driver. The route planning unit 161 supplies data indicating the planned route to the action planning unit 162 and the like.

The action planning unit 162 safely operates the route planned by the route planning unit 161 within the planned time on the basis of data or signals from each unit of the vehicle control system 100 such as the map analysis unit 151 and the situation prediction unit 154. Plan your vehicle's behavior to drive. For example, the action planning unit 162 makes a plan such as start, stop, traveling direction (for example, forward, backward, left turn, right turn, direction change, etc.), lane, traveling speed, and overtaking. The action planning unit 162 supplies data indicating the planned action of the own vehicle to the action planning unit 163 and the like. The action planning unit 163 receives data from each unit of the vehicle control system 100, such as the map analysis unit 151 and the situation prediction unit 154. Alternatively, based on the signal, the action planning unit 162 plans the action of the own vehicle for realizing the action planned. For example, the operation planning unit 163 makes a plan such as acceleration, deceleration, and traveling track. The operation planning unit 163 supplies data indicating the planned operation of the own vehicle to the acceleration / deceleration control unit 172, the direction control unit 173, and the like of the operation control unit 135.

The operation control unit 135 controls the operation of the own vehicle. The operation control unit 135 includes an emergency avoidance unit 171, an acceleration / deceleration control unit 172, and a direction control unit 173.

The emergency avoidance unit 171 is based on the detection results of the vehicle exterior information detection unit 141, the vehicle interior information detection unit 142, and the vehicle state detection unit 143, and collides, touches, enters a dangerous zone, the driver's abnormality, and the vehicle Performs detection processing for emergencies such as abnormalities. When the occurrence of an emergency is detected, the emergency avoidance unit 171 plans the operation of the own vehicle for avoiding an emergency such as a sudden stop or a sharp turn. The emergency avoidance unit 171 supplies data indicating the planned operation of the own vehicle to the acceleration / deceleration control unit 172, the direction control unit 173, and the like.

The acceleration / deceleration control unit 172 performs acceleration / deceleration control for realizing the operation of the vehicle planned by the operation planning unit 163 or the emergency situation avoiding unit 171. For example, the acceleration / deceleration control unit 172 calculates the control target value of the driving force generation device or the braking device for realizing the planned acceleration, deceleration, or sudden stop, and drives the control command indicating the calculated control target value. It is supplied to the system control unit 107.

The direction control unit 173 performs direction control for realizing the operation of the vehicle planned by the operation planning unit 163 or the emergency avoidance unit 171. For example, the direction control unit 173 calculates the control target value of the steering mechanism for realizing the planned traveling track or steep turn by the operation planning unit 163 or the emergency situation avoidance unit 171 and performs control indicating the calculated control target value. The command is supplied to the drive system control unit 107.

In the vehicle control system 100 having such a configuration, the image processing device 20 according to an embodiment of the present technology is provided in, for example, the vehicle exterior information detection unit 141 and uses, for example, a captured image of the front of the vehicle acquired by the data acquisition unit 102 to drive the traveling lane. And a boundary line of another area, for example, a boundary line such as a boundary with a road shoulder or a white line portion indicating a section from another traveling lane is detected. The vehicle exterior information detection unit 141 outputs estimated boundary line information indicating the detected estimated boundary line to the situation recognition unit 153 and the emergency situation avoidance unit 171.

In this case, the initial value used in the filter processing unit 22 is not limited to the preset reference position, but the coordinate position indicating the straight line selected from the candidate straight lines based on the predetermined rule is set as the reference position to start the filter processing. Good. For example, the coordinate position indicating the straight line closest to the vehicle side surface from the straight line detected by the straight line detection unit at a predetermined timing (timing at the start of traveling or timing when the traveling position reaches a predetermined position) is set as the reference position. You may set and start a filter process.

The situation recognition unit 153 performs a situation recognition process for the vehicle based on the estimated boundary line information. For example, the situation recognition unit 153 determines the relative positional relationship between the own vehicle and the white line or the shoulder of the road, and supplies the determination result to the direction control unit 173 of the operation control unit 135 so that the own vehicle travels in the lane. The direction is controlled so that it does not come off.

The emergency avoidance unit 171 also performs an emergency detection process such as deviation from the driving lane based on the estimated boundary line information. When the occurrence of an emergency is detected, the emergency avoidance unit 171 plans the operation of the own vehicle for avoiding an emergency such as a sudden stop or a sharp turn. The emergency avoidance unit 171 supplies data indicating the planned operation of the own vehicle to the direction control unit 173 and the like, and performs emergency avoidance control.

In this way, if the present technology is applied to the vehicle control system 100, the boundary can be detected with high accuracy, so safer driving is possible. Further, the filter processing unit 22 of the image processing device 20 may change the filter characteristic according to the moving state or moving environment of the vehicle. Specifically, the filter characteristic of the filter processing unit 22 is changed according to the route planned by the route planning unit 161. For example, on expressways, white lines and the like are maintained compared to ordinary roads, and continuity is secured. Further, it is desirable to be able to accurately detect boundaries such as road shoulders and white lines when traveling at high speed. Therefore, the filter processing unit 22 makes the permissible ranges of the parameters ρ and the parameters θ stricter than when traveling on a general road when traveling on a highway or during high-speed traveling so that the boundary can be correctly detected. In addition, there are many places where there are no white lines on city roads, and in many cases there are discontinuities. Further, when traveling at a low speed, it is possible to easily drive according to the situation even when the detection accuracy of the boundary such as the road shoulder or the white line is low. Therefore, the filter processing unit 22 may make the permissible range of the parameter ρ or the parameter θ more lenient when traveling on an urban road or at low speed than when traveling on a highway or the like so as to easily detect a boundary. .. Further, the allowable ranges of the parameter ρ and the parameter θ may be set according to the motion model of the own vehicle. For example, the position of the road shoulder or the white line that moves according to the movement of the vehicle may be estimated, and the allowable range of the parameter ρ or the parameter θ may be set based on the estimated position. By setting the allowable range in this way, the estimated boundary line information can be generated in consideration of the movement of the own vehicle.

In addition, if the operation control is switched based on the lost determination information output from the image processing device, even if the boundary is not detected, information from other sensors is used to perform driving assistance or automatic driving. be able to. Further, it is possible to notify the driver of the lost judgment information to call attention, or to instruct the driver to switch from automatic operation to manual operation.

When the present technology is applied to the vehicle control system 100, the extension unit 23 sets the length of the search reference area and the boundary search area (the length of the boundary seed in the line segment direction) to several tens of centimeters to 1 meter (preferably). Is about 80 cm), and the width (length in the direction orthogonal to the line segment direction of the boundary seed) is about several tens of centimeters (preferably about 50 cm), the boundary indicating the road shoulder or the white line. The extension line can be detected accurately.

The series of processes described in the specification can be executed by hardware, software, or a composite configuration of both. When executing the processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed. Alternatively, the program can be installed and executed in a general-purpose computer that can execute various processes.

For example, the program can be recorded in advance in a hard disk, SSD (Solid State Drive), or ROM (Read Only Memory) as a recording medium. Alternatively, the program is a flexible disk, CD-ROM (Compact Disc Read Only Memory), MO (Magneto optical) disc, DVD (Digital Versatile Disc), BD (Blu-Ray Disc (registered trademark)), magnetic disc, semiconductor memory card. It can be temporarily (or permanently) stored (recorded) in a removable recording medium such as. Such a removable recording medium can be provided as so-called package software.

In addition to installing the program from the removable recording medium to the computer, the program may be wirelessly or wired transferred from the download site to the computer via a network such as a LAN (Local Area Network) or the Internet. In the computer, the program thus transferred can be received and installed in a recording medium such as a built-in hard disk.

Note that the effects described in this specification are merely examples and are not limited, and additional effects not described may be present. Further, the present technology should not be construed as being limited to the above-described embodiments of the technology. The embodiments of this technology disclose the present technology in the form of exemplification, and it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of the present technology. That is, in order to judge the gist of the present technology, the claims should be taken into consideration.

Further, the image processing device of the present technology can also have the following configurations.
(1) A straight line corresponding to the straight line indicated by the predetermined coordinate position is selected from the detected straight lines based on the distance between the coordinate position in the polar coordinate space indicating the straight line detected from the captured image and the predetermined coordinate position. An image processing apparatus including a filter processing unit.
(2) The predetermined coordinate position is a preset coordinate position or a coordinate position indicating a straight line detected and selected from a captured image generated at a time different from the captured image by the filter processing unit (1 ) The image processing device described in (1).
(3) An imaging device that acquires the captured image is provided on the moving body,
The image processing device according to (2), wherein the predetermined coordinate position is a coordinate position indicating a straight line closest to the moving body.
(4) The filter processing unit sets an allowable range for the predetermined coordinate position and uses the coordinate position within the allowable range indicating a straight line detected from the captured image (1) to (3). The image processing device according to any one of 1.
(5) An imaging device that acquires the captured image is provided on the moving body,
The image processing device according to (4), wherein the filter processing unit sets the allowable range according to a moving state or a moving environment of the moving body.
(6) When the straight line corresponding to the straight line indicated by the predetermined coordinate position cannot be selected from the detected straight lines, the filter processing unit outputs the lost determination information indicating that the corresponding straight line cannot be selected ( The image processing device according to any one of 1) to (5).
(7) The image processing device according to any one of (1) to (6), further including a straight line detection unit that detects a coordinate position in a polar coordinate space indicating a straight line from the captured image.
(8) The image processing device according to (7), wherein the straight line detection unit detects the coordinate position indicating a straight line from an image region near the image pickup position in the captured image.
(9) The image according to (7) or (8), wherein the straight line detection unit performs Hough transform using the captured image to detect a coordinate position indicating a straight line in the ρ-θ space that is the polar coordinate space. Processing equipment.
(10) A search reference area having a predetermined size including the straight line selected by the filter processing unit is set, and a boundary search area is set in a line segment direction of a boundary seed that is the selected straight line in the search reference area. The image according to any one of (1) to (9), further including a decompression unit that detects, as a boundary decompression line, a straight line that has a slope with respect to the boundary seed that is smaller than a threshold value and that is connected to the boundary seed from the boundary search region. Processing equipment.
(11) When the extension unit detects the boundary extension line from the boundary search region, the extension unit newly sets the boundary search region adjacent in the direction of a line segment of the detected boundary extension line, and detects the boundary extension line. The image processing apparatus according to (10), wherein a straight line having a slope with respect to the boundary extension line smaller than a threshold value and connected to the boundary extension line is detected as the boundary extension line by detecting from the new boundary search area.
(12) The decompression unit sets a curvature range of an estimated boundary line formed by the boundary seed and the boundary expansion line based on the area size of the boundary search area and the threshold value. (10) or (11) Image processing device.

10 ... Imaging device 20 ... Image processing device 21 ... Straight line detection unit 22 ... Filter processing unit 23 ... Expansion unit

Claims

Filtering processing for selecting a straight line corresponding to the straight line indicated by the predetermined coordinate position from the detected straight lines based on the distance between the coordinate position of the polar coordinate space indicating the straight line detected from the captured image and the predetermined coordinate position Image processing apparatus including a unit.
The predetermined coordinate position is a preset coordinate position or a coordinate position indicating a straight line detected and selected from a captured image generated at a time different from the captured image by the filter processing unit. Image processing device.
An imaging device that acquires the captured image is provided on the moving body,
The image processing apparatus according to claim 2, wherein the predetermined coordinate position is a coordinate position indicating a straight line closest to the moving body.
The image processing apparatus according to claim 1, wherein the filter processing unit sets an allowable range for the predetermined coordinate position and uses a coordinate position within the allowable range indicating a straight line detected from the captured image.
An imaging device that acquires the captured image is provided on the moving body,
The image processing apparatus according to claim 4, wherein the filter processing unit sets the allowable range according to a moving state or a moving environment of the moving body.
If the straight line corresponding to the straight line indicated by the predetermined coordinate position cannot be selected from the detected straight lines, the filter processing unit outputs lost determination information indicating that the corresponding straight line cannot be selected. The image processing device described.
The image processing apparatus according to claim 1, further comprising a straight line detection unit that detects a coordinate position in a polar coordinate space that indicates a straight line from the captured image.
The image processing apparatus according to claim 7, wherein the straight line detection unit detects the coordinate position indicating a straight line from an image area near the image capturing position in the captured image.
The image processing apparatus according to claim 7, wherein the straight line detection unit performs Hough transform using the captured image to detect a coordinate position indicating a straight line in the ρ-θ space that is the polar coordinate space.
By setting a search reference area of a predetermined size including the straight line selected by the filter processing unit, set the boundary search area in the line segment direction of the boundary seed that is the selected straight line in the search reference area, The image processing apparatus according to claim 1, further comprising a decompressing unit that extracts a straight line that has a slope with respect to the boundary seed smaller than a threshold value and that is connected to the boundary seed from the boundary search region and that is a boundary expansion line.
The decompression unit newly sets the boundary search regions adjacent to each other in the line segment direction of the detected boundary decompression line in response to being able to extract the boundary decompression line from the boundary search region, and performs the detection. The image processing device according to claim 10, wherein a straight line having a slope with respect to the boundary extension line smaller than a threshold value and connected to the boundary extension line is extracted as the boundary extension line from the new boundary search area.
The image processing apparatus according to claim 10, wherein the decompression unit sets a curvature range of an estimated boundary line formed by the boundary seed and the boundary decompression line based on the area size of the boundary search area and the threshold value.
Based on the distance between the coordinate position of the polar coordinate space indicating the straight line detected from the captured image and the predetermined coordinate position, a straight line corresponding to the straight line indicated by the predetermined coordinate position is obtained from the detected straight line by the filter processing unit. An image processing method including selecting.
A program that causes a computer to perform image processing of a captured image,
A procedure for selecting, from the detected straight lines, a straight line corresponding to the straight line indicated by the predetermined coordinate position based on the distance between the coordinate position of the polar coordinate space indicating the straight line detected from the captured image and the predetermined coordinate position. A program that causes the computer to execute.