JP7166096B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing device and an image processing method.

2. Description of the Related Art Conventionally, there is known a device mounted on a vehicle and recognizing a parking closing line from an image obtained by photographing with a camera. For example, the arithmetic processing unit of the in-vehicle parking frame recognition device disclosed in Patent Document 1 includes a line extraction unit, a curve determination unit, and a parking lot determination unit. The line extracting unit extracts a marking line drawn on the running surface of the vehicle based on the image captured by the camera. The curve determination unit determines whether or not the marking line is an outside roadway line by determining whether or not the marking line extracted by the line extracting unit has a curved portion that satisfies a predetermined condition. The parking lot determination unit determines whether or not the vehicle is traveling in the parking lot based on the determination result of the curve determination unit. output the recognition result.

In addition, the marking lines include various road marking lines drawn on the road (roadway center line, lane boundary line, roadway outer line, crosswalk, etc.) and parking frame lines drawn in the parking lot. .

JP 2014-106740 A

When extracting a marking line such as a parking closing line drawn on a road surface from a photographed image, first, a plurality of edge lines, which are candidates for the outline of the marking line, are detected from the photographed image. Then, using a plurality of detected edge lines, a marker line is extracted under a condition such as a combination of two edge lines arranged parallel to each other with a predetermined interval, for example.

Conventionally, when the detected edge line includes a curved line, the direction in which the edge line extends may be erroneously recognized, and an inappropriate parking frame line or the like may be extracted. For example, if the parking frame line is not properly extracted, the parking frame may be misidentified or the parking frame may not be recognized.

SUMMARY OF THE INVENTION In view of the above problems, an object of the present invention is to provide a technique that can appropriately capture the shape of an edge line detected from a captured image.

In order to achieve the above object, the image processing apparatus of the present invention comprises: an edge line detection unit for detecting an edge line from a photographed image; and an extraction unit for extracting the straight line segment based on the dividing point (first configuration).

In the first configuration described above, it is preferable that the dividing section obtains the dividing point at a position where the curvature of the edge line is equal to or greater than a predetermined value (second configuration).

In the image processing apparatus having the first or second configuration, the dividing unit obtains the dividing point based on direction information obtained for each division obtained by dividing the edge line into a plurality of divisions at a predetermined position (third configuration).

In the image processing apparatus having the first or second configuration, the dividing unit obtains the dividing point based on the edge line and a straight line connecting two end points of the edge line (fourth configuration). It's okay.

In the image processing apparatus having the third or fourth configuration, the dividing unit determines whether or not to further divide the edge line based on the provisional straight line and the edge line set according to the division points obtained in advance. It is preferable that the configuration (fifth configuration) determines whether the

The image processing device having any one of the first to fifth configurations above further includes a marking line detection unit that detects a marking line provided on the road surface based on the straight line segment extracted by the extraction unit (first 6) is preferable.

In the image processing device of the sixth configuration, the captured image is captured by a camera that captures the surroundings of the vehicle, and the marking line is a partition line forming a parking frame (seventh configuration). you can

To achieve the above object, an image processing method of the present invention is an image processing method executed by an image processing apparatus for processing a photographed image, comprising an edge line detection step of detecting an edge line from the photographed image; a dividing step of determining dividing points on an edge line that enable acquisition of a plurality of straight line segments forming a set of approximation lines approximating the edge line; and extracting the straight line segments based on the dividing points. and an extraction step (eighth configuration).

According to the present invention, it is possible to appropriately capture the shape of edge lines detected from a captured image.

1 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the present invention; FIG. Flowchart showing a rough flow of straight line segment extraction processing Flowchart showing an example of the flow of the edge line detection process Diagram for explaining edge line detection Flowchart showing an example of the flow of the division process Flowchart illustrating the flow of the first division process Diagram for explaining the first division process Flowchart illustrating the flow of the second division process Diagram for explaining the second division process FIG. 1 shows a configuration example of a parking assistance system to which the image processing device of the present invention is applied; Schematic diagram for explaining the operation of the parking assistance system

Exemplary embodiments of the invention are described in detail below with reference to the drawings.

<1. Image processing device>
FIG. 1 is a block diagram showing the configuration of an image processing apparatus 1 according to an embodiment of the invention. An image captured by a camera is input to the image processing apparatus 1 . In this embodiment, the camera is a camera mounted on a mobile body and capturing an image of the surroundings of the mobile body. The camera captures, for example, the front, sides, rear, etc. of the moving object. The moving body is, for example, a vehicle, a robot, or the like. The image processing apparatus 1 is mounted on a moving body.

As shown in FIG. 1 , the image processing apparatus 1 includes a control section 11 and a storage section 12 . The control unit 11 controls the entire image processing apparatus 1 . The control unit 11 is configured by a computer including, for example, a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory). The storage unit 12 nonvolatilely stores computer programs and data necessary for the operation of the control unit 11 . As the storage unit 12, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory), a flash memory, or the like can be used.

The image acquisition unit 111, the edge line detection unit 112, the division unit 113, and the extraction unit 114 shown in FIG. In other words, the image processing device 1 includes an image acquisition unit 111 , an edge line detection unit 112 , a division unit 113 and an extraction unit 114 .

At least one of the functional units 111 to 114 included in the control unit 11 may be configured by hardware such as ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). Each functional unit 111 to 114 included in the control unit 11 is a conceptual component. A function performed by one component may be distributed to a plurality of components, or a function possessed by a plurality of components may be integrated into one component.

The video acquisition unit 111 temporally continuously acquires analog or digital captured images from the camera at a predetermined cycle (for example, 1/30 second cycle). Then, when the acquired temporally continuous captured images (acquired video) are analog, the video acquisition unit 111 converts the analog captured images into digital captured images (A/D conversion). In this embodiment, the video acquisition unit 111 converts the captured image into a grayscale image (grayscale image).

The edge line detection unit 112 detects edge lines from the captured image. Specifically, the edge line detection unit 112 detects edge lines from the captured image processed by the video acquisition unit 111 . An edge line refers to a boundary line between light and dark in a captured image, and the contour line of a subject appearing in the captured image corresponds to the edge line. In this embodiment, the edge line includes the outline of the marking line on the road surface in the captured image. The marking lines include various road marking lines drawn on the road (roadway center line, lane boundary line, roadway outer line, pedestrian crossing, etc.), parking frame lines drawn in the parking lot, and the like. The edge line detection range may be the entire range of the captured image, or may be a partial range of the captured image.

The dividing unit 113 obtains dividing points on the edge line that enable acquisition of a plurality of straight line segments forming a set of approximate lines that approximate the edge line. In this embodiment, the dividing unit 113 obtains dividing points at positions where the curvature of the edge line is equal to or greater than a predetermined value. A position where the curvature of the edge line is equal to or greater than a predetermined value is a position included in a section (curved section) where the amount of change in the slope of the edge line is greater than the predetermined value. As a result, the curved section and the straight section can be divided, and the straight section can be detected with high accuracy. The dividing unit 113 obtains at least one dividing point when the edge line includes a curved line. The dividing unit 113 does not obtain a dividing point when the edge line is composed only of straight lines.

The extraction unit 114 extracts straight line segments based on the division points obtained by the division unit 113 . An approximation line approximating the edge line is obtained by a plurality of straight line segments extracted based on at least one dividing point on the edge line. The approximation line is composed of a combination of straight line segments and does not include curved lines. In this embodiment, the plurality of straight line segments extracted by the extraction unit 114 are stored in the storage unit 12 and used to detect the marking line. However, this is an example, and the plurality of straight line segments extracted by the extracting unit 114 may be used, for example, for detecting a three-dimensional object.

According to the present embodiment, when an edge line has a curved portion, the shape of the edge line can be captured more appropriately than in the case where an approximate line of the edge line is obtained by connecting the endpoints of the edge line with straight lines. . Moreover, according to the present embodiment, when the edge line has a curved portion, the shape of the edge line can be properly captured as compared with the case where the approximation line of the edge line is obtained by the method of least squares. These effects will be made clearer by the contents described later.

Note that the image processing apparatus 1 may further include a marking line detection unit that detects marking lines provided on the road surface based on the straight line segments extracted by the extraction unit 114 . In such an image processing apparatus, since the shape of the edge line can be properly captured and the marking line can be detected, the marking line can be detected with high accuracy.

FIG. 2 is a flow chart showing a rough flow of straight line segment extraction processing executed by the image processing apparatus 1 of the present invention. In this embodiment, straight line segment extraction processing is performed to extract a straight line segment group that is used to detect a marking line from a captured image.

First, the edge line detection unit 112 performs an edge line detection step of detecting edge lines from a photographed image acquired by a single camera (step S1). Note that in the present embodiment, the edge line detection unit 112 detects edge lines from the captured image converted into a grayscale image. A plurality of edge lines are usually detected. However, it is possible that no edge line is detected or only one edge line is detected. Details of the edge line detection process will be described later.

Next, the dividing unit 113 performs a dividing step of obtaining dividing points on the edge lines detected in the edge line detecting step (step S2). The dividing points are obtained in order to obtain a plurality of straight line portions forming pairs of approximate lines that approximate edge lines. As described above, if the edge line does not contain a curve (if it is a straight line), no dividing point is found. A division process is performed for each edge line detected by the edge line detection unit 112 . Details of the dividing step will be described later.

Finally, the extraction unit 114 performs an extraction step of extracting straight line segments based on the division points obtained in the division step. A straight line segment constitutes an approximation line that approximates an edge line. An edge line for which no dividing point has been found is estimated to be a straight line, so the edge line itself is extracted as a straight line segment. The extracted straight line segments are stored in the storage unit 12, for example.

In addition, in the present embodiment, the straight line segments extracted in the extraction step are sorted. Of the extracted straight line segments, straight line segments excluding straight line segments extending in a direction perpendicular to the road surface are selected as a straight line segment group used for detecting marking lines.

FIG. 3 is a flow chart showing an example of the flow of the edge line detection process. By obtaining a photographed image for edge line detection, the same processing is repeatedly performed for each pixel included in a preset processing range. In other words, loop processing is performed to repeat the same processing for each pixel. By starting loop processing for each pixel (step S11), it is confirmed whether or not an edge point is extracted from the pixel to be processed (step S12). A change in the brightness of the image is obtained by obtaining the difference in density gradation between the pixel to be processed and the adjacent pixel, and points of change in the difference in brightness above a certain level are extracted as edge points.

If an edge point is extracted from the pixel to be processed (Yes in step S12), it is checked whether or not there is an extracted edge point in the adjacent pixel (step S13). If there is an edge point in the adjacent pixel (Yes in step S13), processing is performed to connect the edge point of the pixel to be processed and the edge point of the adjacent pixel, and the line connected by the connection processing is registered as an edge line. (step S14). Note that the connection of adjacent pixels with edge points may be restricted based on predetermined conditions such as the direction of the luminance gradient. Further, when the edge points of adjacent pixels form already registered edge lines, the already registered edge line data is updated to new data by connecting the edge points of the pixels to be processed. .

When the edge line registration processing is performed, it is confirmed whether or not the loop processing for each pixel can be terminated (step S15). When the loop processing is completed for all pixels included in the processing range, it is determined that the loop processing is finished (Yes in step S15), and the edge line detection process is finished. On the other hand, if the loop processing has not been completed for all the pixels included in the processing range, it is determined that the loop processing should not end (No in step S15), and the loop processing is continued by returning to step S12. At this time, the pixel to be processed is changed.

During loop processing for each pixel, if no edge point is extracted from the pixel to be processed (No in step S12) or if there is no edge point in the adjacent pixel (No in step S13), an edge line is formed. Can not. Therefore, the process of step S15 is performed without performing edge line registration.

FIG. 4 is a diagram for explaining edge line detection. For example, it is assumed that a U-shaped marking line ML as shown in FIG. 4A appears in the processing range in the captured image. In this case, the edge line detection process executed by the edge line detection unit 112 detects a substantially L-shaped edge line EL having one side end point a and the other side end point b, as shown in FIG. 4B, for example. can be detected.

Note that the edge line EL shown in FIG. 4B is merely an example of the edge line detected from the marking line ML. In addition to the edge line EL shown in FIG. 4(b), a plurality of edge lines are normally detected from the marking line ML shown in FIG. 4(a). Further, an edge line having a shape (for example, U-shaped) different from the edge line EL shown in FIG. 4B may be detected from the marking line ML shown in FIG. 4A.

FIG. 5 is a flow chart showing an example of the flow of the division process. In the dividing step, similar processing is repeatedly performed on each edge line detected in the edge line detecting step. In other words, loop processing is performed to repeat the same processing for each edge line. By starting the loop processing for each edge line (step S21), the first division processing (step S22) and the second division processing (step S23) are executed in this order.

When the second division processing is finished, it is checked whether or not the loop processing for each edge line can be finished (step S24). When the loop processing is completed for all the edge lines detected in the edge line detection step, it is determined that the loop processing is finished (Yes in step S24), and the dividing step is finished. On the other hand, if the loop processing has not been completed for all the edge lines detected in the edge line detection step, it is determined that the loop processing is not to end (No in step S24), and the process returns to step S22 to complete the loop processing. can continue. At this time, the edge line to be processed is changed.

FIG. 6 is a flowchart illustrating the flow of the first division process. FIG. 7 is a diagram for explaining the first division processing. When explaining the flowchart shown in FIG. 6, the explanation will be made with appropriate reference to the schematic example shown in FIG. 7 in order to facilitate understanding.

As shown in FIG. 6, in the first division process, edge lines are first segmented (step S221). Description will be made with reference to FIG. FIG. 7A shows an example of edge lines EL detected in the edge line detection process. It is assumed that the edge line EL shown in FIG. 7(a) is the same as the edge line EL shown in FIG. 4(b). Suppose segmentation is performed for the edge line EL shown in FIG. 7(a). As shown in FIG. 7(b), the edge line EL is divided into a plurality of sections from the one end point a of the edge line EL at a predetermined interval d. In the example shown in FIG. 7, it is divided into 10 sections from section S1 to section S10. Note that the division of the edge lines EL described above is merely an example. For example, the division of the edge line EL may be performed based on the other side end point b instead of the one side end point a.

Returning to FIG. 6, once the edge lines are segmented, the direction of each segment is calculated (step S222). The direction of each section is obtained, for example, as the direction of a straight line that connects the end point of the edge line on one side of the section and the end point of the other side of the section. In FIG. 7(c), the direction of each section is indicated by an arrow. Note that the direction for each section may be a direction obtained by another method. For example, the direction of each section may be the direction of the luminance gradient obtained from the captured image in each section.

After obtaining the direction for each section, a representative direction is calculated (step S223). The representative direction is, for example, the mode of a histogram obtained by statistically processing the directions of each section. The histogram is obtained by classifying the direction of each segment into one of multiple direction classes with a constant angular width. The direction class with the highest frequency is the mode and becomes the representative direction. In the example shown in FIG. 7, sections S1 to S8 are classified into the same directional class, and sections S9 and S10 are classified into different directional classes. For this reason, the direction class into which the sections S1 to S8 are classified has the highest frequency and becomes the representative direction.

After obtaining the representative direction, the same processing is repeated for each segment. In other words, loop processing is performed to repeat the same processing for each segment. By starting the loop processing for each section (step S224), direction comparison between the direction of the section to be processed and the representative direction is performed (step S225). In the present embodiment, it is checked whether or not the direction of the section to be processed matches the representative direction by comparison. Specifically, since the representative direction has a certain angular width, it is determined that the direction of the processing target section matches the representative direction when the direction of the processing target section is within the angle range indicating the representative direction. If the direction of the section to be processed is outside the angle range indicating the representative direction, it is determined that the direction of the section to be processed does not match the representative direction.

As a result of direction comparison, it is determined whether or not the relationship with the representative direction has changed compared to the comparison result of the previously (immediately) processed segment (step S226). For example, if it is determined that the direction of the previously processed section matches the representative direction, and if it is determined that the direction of the section to be processed matches the representative direction, it is determined that the relationship with the representative direction does not change. . On the other hand, if it is determined that the direction of the previously processed section matches the representative direction and the direction of the section to be processed does not match the representative direction, it is determined that the relationship with the representative direction has changed. be.

If it is determined that the relationship with the representative direction has changed (Yes in step S226), the edge line is divided in the processing target section (step S227). Specifically, a dividing point is set at the boundary between the previously (immediately) determined section and the processing target section. On the other hand, if it is determined that the relationship with the representative direction does not change (No in step S226), the process proceeds to step S228, which will be described later, without setting a division point.

When the dividing point is set, or when it is determined that the dividing point is not set, it is confirmed whether or not the loop processing for each segment can be terminated (step S228). When the loop processing is completed for all segments, it is determined that the loop processing is to be terminated (Yes in step S228), and the first division processing is terminated. On the other hand, if the loop processing has not been completed for all segments, it is determined that the loop processing should not be terminated (No in step S228), and the loop processing is continued by returning to step S225. At this time, the classification to be processed is changed.

In the example shown in FIG. 7, the directions of sections S1 to S8 are determined to match the representative direction. For this reason, it is not determined that the relationship with the representative direction has changed during the processing of these divisions, and the dividing point P is not set. On the other hand, in section S9, it is determined that the direction does not match the representative direction. For this reason, in the processing of section S9, it is determined that the relationship with the representative direction has changed, and a dividing point P is set at the boundary between section S8 and section S9 as shown in FIG. 7(d). Note that the direction of section S10 is determined not to match the representative direction. Therefore, in the processing of section S10, it is determined that the relationship with the representative direction has not changed, and the division point P is not set.

As described above, the dividing unit 113 obtains the dividing point P based on the direction information obtained for each section obtained by dividing the edge line EL into a plurality of sections at predetermined positions. According to this, for example, for an edge line EL including a straight line and a curved line, a dividing point can be set near the boundary between the straight line and the curved line, and only the straight line can be extracted from the edge line. Become. Further, by finely setting dividing points for the edge line according to the direction information, it is possible to appropriately obtain an approximation line that approximates the edge line by combining straight line segments.

In the above description, it is determined only whether or not the directions match in the comparison between the direction of the section to be processed and the representative direction (step S225), but this is an example. In the comparison, the magnitude of the orientation deviation may also be determined in addition to whether the orientations match. In this case, in the next step S226, it is possible to determine a change in the relationship with the representative direction by considering the magnitude of the deviation of the direction, and it becomes possible to set the dividing points more finely. For example, in the example shown in FIG. 7, it is possible to set the division point P not only at the boundary between the sections S8 and S9, but also at the boundary between the sections S9 and S10.

In the above description, the configuration for comparing the relationship from section S1 to the representative direction has been described, but this is merely an example. For example, the configuration may be such that the relationship from section S10 to the representative direction is compared.

FIG. 8 is a flowchart illustrating the flow of the second division process. FIG. 9 is a diagram for explaining the second division processing. When explaining the flowchart shown in FIG. 8, the explanation will be made with appropriate reference to the schematic example shown in FIG. 9 in order to facilitate understanding.

As shown in FIG. 8, in the second division processing, similar processing is repeatedly performed for each division point. In other words, loop processing is performed to repeat the same processing for each division point. Note that, in the present embodiment, not only the division points obtained by the first division processing, but also the division points obtained during the second division processing are subjected to the loop processing.

By starting loop processing for each division point (step S230), provisional straight lines are set based on the division points to be processed (step S231). A provisional straight line is a straight line that connects a dividing point to be processed and a predetermined point adjacent to the dividing point to be processed. The predetermined point is the end point or division point of the edge line. Two temporary straight lines are set.

In FIG. 9A, the solid line is the edge line EL. The edge line EL shown in FIG. 9(a) is assumed to be the same as the edge line EL shown in FIGS. 4(b) and 7(a). A division point P is a division point determined by the flow chart shown in FIG. In the example shown in FIG. 9, two provisional straight lines VL are set based on the dividing point P, as indicated by broken lines in FIG. 9(a). One is a first provisional straight line VL1 connecting the dividing point P and one end point a of the edge line EL, and the other is a second provisional straight line VL2 connecting the dividing point P and the other end point b of the edge line EL.

Once the provisional lines are set, similar processing is repeatedly performed for each provisional line, as shown in FIG. In other words, loop processing is performed to repeat the same processing for each provisional line. When the loop processing for each provisional line is started (step S232), a process of searching for the maximum perpendicular length from a point on the edge line to the provisional line is performed (step S233).

An example of search processing will be described with reference to FIG. In the search process, a perpendicular line PL is set with respect to the temporary straight line VL. The position of this perpendicular PL is moved along the provisional straight line VL. In accordance with the movement of the position of the perpendicular PL, the distance between the intersection of the temporary straight line VL and the perpendicular PL and the intersection of the edge line EL and the perpendicular PL is obtained as appropriate. Then, the maximum value of the distance is obtained using a technique such as a hill-climbing method.

After obtaining the maximum perpendicular length, it is checked whether the maximum perpendicular length is equal to or greater than a preset first threshold (step S234). The first threshold is set as one of the indices for determining whether it is necessary to set the dividing point on the edge line. If the maximum perpendicular length is smaller than the first threshold (No in step S234), it is determined that setting of division points is unnecessary, and the process proceeds to step S238, which will be described later. On the other hand, if the maximum perpendicular length is greater than or equal to the first threshold (Yes in step S234), the edge line is segmented at the position of the maximum perpendicular length (step S235). That is, the edge lines are divided at the intersection of the perpendicular line having the maximum length of the perpendicular line and the edge line.

In the example shown in FIG. 9, the first temporary straight line VL1 overlaps the edge line EL, and the maximum perpendicular length is smaller than the first threshold. For this reason, the edge line EL is not divided by processing the first provisional straight line VL1. On the other hand, the maximum perpendicular length of the second provisional straight line VL2 is greater than or equal to the first threshold. For this reason, as shown in FIG. 9(b), the edge line EL is divided at the intersection point CP between the perpendicular line PL having the maximum length of the perpendicular line and the edge line EL.

After the edge line is segmented, it is checked whether or not the angle formed between the two segmented segments is equal to or less than a second threshold (step S236). One of the two segments is an edge line segment sandwiched between the edge line segmentation point and one end point of the provisional straight line. The other of the two segments is the segment of the edge line sandwiched between the point at which the edge line was segmented and the other end point of the provisional straight line. The angle formed between two sections is the angle formed by a straight line connecting two end points of one side section and a straight line connecting two end points of the other side section. In the example shown in FIG. 9, it is assumed that the temporary straight line to be processed is the second temporary straight line VL2. In this case, the straight line connecting the two end points of the one-side section is the straight line connecting the intersection point CP and the division point P. Also, the straight line that connects the two end points of the other-side section is the straight line that connects the intersection point CP and the other-side end point b. It is checked whether the angle formed by these two straight lines is less than or equal to a second threshold.

Like the first threshold, the second threshold is also set as one of the indices for determining whether or not it is necessary to set the dividing point on the edge line. However, the second threshold is set from a viewpoint different from that of the first threshold, and is a value different from that of the first threshold. If the angle formed between the two sections is greater than the second threshold (No in step S236), it is determined that setting of division points is unnecessary, and the process proceeds to step S238, which will be described later. On the other hand, if the angle formed between the two sections is equal to or less than the second threshold (Yes in step S236), a division point is set at the division position to divide the edge line (step S237). Note that step S236 may be omitted.

When the dividing point is set, or when it is determined that the dividing point is not set, it is confirmed whether or not the loop processing for each provisional straight line can be terminated (step S238). When the loop processing is completed for all provisional straight lines, it is determined that the loop processing is finished. On the other hand, if the loop processing has not been completed for all provisional lines, it is determined that the loop processing is not to end (No in step S238), and the process returns to step S233 to continue the loop processing for each provisional line. At this time, the provisional straight line to be processed is changed.

If it is determined that the loop processing for each provisional line should be terminated (Yes in step S238), it is confirmed whether or not the loop processing for each division point may be terminated (step S238). When the loop processing is completed for all the set division points, it is determined that the loop processing is finished (Yes in step S239), and the second division processing is finished. Note that if a division point is added in step S237, the division point is also subject to loop processing. On the other hand, if the loop processing has not been completed for all division points, it is determined that the loop processing should not end (No in step S239), and the process returns to step S231 to continue the loop processing for each division point. At this time, the division point to be processed is changed.

In the example shown in FIG. 9, if the target provisional line is the first provisional line, the division point P is not set because it is determined in step S234 in FIG. 8 that the maximum perpendicular length is smaller than the first threshold. On the other hand, when the target provisional straight line is the second provisional straight line VL2, a new dividing point P is set. That is, as shown in FIG. 9(c), a first dividing point P1 and a second dividing point P2 are set on the edge line EL. By setting the second dividing point P2, a new provisional straight line is set, and processing for further setting of dividing points is performed. be done.

By setting the two dividing points P1 and P2, as shown in FIG. 9(d), three straight line segments LS are extracted as a set of approximation lines that approximate the edge line EL. Specifically, a first straight line segment LS1 connecting one end point a of the edge line EL and the first dividing point P1, and a second straight line segment LS2 connecting the first dividing point P1 and the second dividing point P2. , and a third straight line segment LS3 connecting the second dividing point P2 and the end point b on the other side of the edge line EL are extracted. The three extracted straight line segments LS1 to LS3 are used to detect the marking line.

As described above, the dividing unit 113 determines whether or not to further divide the edge line EL based on the provisional straight line and the edge line EL set according to the dividing point P previously obtained. According to this, it is possible to make the approximation line that approximates the edge line EL composed of a plurality of straight line segments closer to the shape of the edge line. For this reason, it is possible to improve the detection accuracy of the indicating line using the straight line segment.

<2. Parking assistance system>
Next, a description will be given of a parking assistance system to which the image processing device of the present invention is applied. In the following description, the straight traveling direction of the vehicle, which is the direction from the driver's seat toward the steering wheel, is referred to as the "forward direction." Also, the direction in which the vehicle travels straight ahead and is directed from the steering wheel toward the driver's seat is called the "rearward direction." Also, the direction perpendicular to the straight traveling direction and the vertical line of the vehicle and directed from the right side to the left side of the driver who is facing forward is referred to as the "left direction." Also, the direction perpendicular to the straight traveling direction and the vertical line of the vehicle and from the left side to the right side of the driver who is facing forward is referred to as the "right direction." A vehicle equipped with a parking assistance system including the image processing device of the present invention is referred to as "own vehicle".

FIG. 10 is a diagram showing one configuration example of a parking assistance system to which the image processing device of the present invention is applied. As shown in FIG. 10, the parking assistance system includes an image processing device 1A, an imaging unit 2, a parking control ECU (Electronic Control Unit) 3, an EPS (Electric Power Steering)-ECU 4, and an in-vehicle network 5. Prepare.

The image processing device 1A is connected to the photographing unit 2, and is also connected to the parking control ECU 3 and the EPS-ECU 4 via an in-vehicle network 5 such as a CAN (Controller Area Network). A display device such as a liquid crystal display is preferably connected to the image processing device 1A. The image processing device 1A preferably generates a display image in which an index indicating the target parking position is superimposed on the captured image, and displays the image on the display device.

The photographing unit 2 includes a camera for photographing the surroundings of the own vehicle. The camera lens is, for example, a fisheye lens. The imaging unit 2 includes a side camera 20 provided on the side of the own vehicle. The side camera 20 is provided, for example, on a door mirror of the own vehicle. The side cameras 20 preferably include a left side camera and a right side camera. The left side camera has an optical axis along the left-right direction of the own vehicle, and photographs the left direction of the own vehicle. The right side camera has an optical axis along the left-right direction of the own vehicle, and photographs the right direction of the own vehicle. In addition to the side cameras 20, the photographing unit 2 preferably includes a front camera for photographing the front of the vehicle and a back camera for photographing the rear of the vehicle.

The image processing device 1A processes an image captured by a camera that captures the surroundings of the own vehicle. The image processing apparatus 1A has substantially the same configuration as the image processing apparatus 1 (see FIG. 1) described above. However, a marking line detection unit 115 , a recognition unit 116 , and a management unit 117 are added as functions realized by the control unit 11 executing a computer program stored in the storage unit 12 .

The partition line detection unit 115 detects the partition line (parking frame line) that forms the parking frame based on the straight line segment extracted by the extraction unit 114 . Demarcation lines are drawn in white or yellow lines on the paved surface of the parking lot to demarcate the parking space. A partition line is an example of a marking line. The lane marking detection unit 115 is an example of a marking line detection unit. The marking line detection unit 115 detects, as a marking line, an area sandwiched by two straight line segments arranged parallel to each other with a predetermined interval, for example. The predetermined interval is appropriately determined, for example, based on information on lane markings used in standard parking lots for general public use.

The recognition unit 116 recognizes the parking frame based on the lane marking information detected by the lane marking detection unit 115 . The recognition unit 116 obtains the parallelism, width, etc. between the two lane markings, and recognizes the parking frame based on these results. For example, the recognition unit 116 recognizes an area between two lane markings arranged in parallel with a predetermined interval as a parking frame. The predetermined interval is, for example, the width of a standard parking area for general public use, which is stipulated by laws and regulations related to parking lots.

The management unit 117 manages information regarding the parking frame recognized by the recognition unit 116 . Information about parking frames is managed, for example, as a combination of a plurality of vertex coordinates. The management unit 117 stores information about the parking frame in the storage unit 12 as frame management information, and updates the frame management information as appropriate. The management unit 117 updates the parking space management information based on, for example, observation information about the parking space acquired from the captured image and estimated information about the parking space estimated from the amount of movement of the own vehicle. The amount of movement of the own vehicle can be obtained from information such as the steering angle, vehicle speed, and shift of the own vehicle V acquired from the in-vehicle network 5 . In addition, the management unit 117 calculates the target parking position, for example, as a place where the vehicle should be parked in an empty parking space that is closest to the vehicle when the vehicle stops.

FIG. 11 is a schematic diagram for explaining the operation of the parking assistance system of this embodiment. FIG. 11 shows a state in which the own vehicle V has entered the parking lot. 1 A of image processing apparatuses start the process for recognizing parking frame PS, for example, when the vehicle speed of the own vehicle V becomes below predetermined speed. The image processing device 1A acquires captured images obtained from the side camera 20 at predetermined time intervals.

The image processing device 1A extracts straight line segments from each captured image based on detection of edge lines, and detects lane markings SL. In the example shown in FIG. 11, the marking line SL can be detected from the captured image obtained from the left side camera 20L. The image processing device 1A recognizes the parking frame PS based on the information of the detected lane marking SL, and manages the information of the recognized parking frame PS. In the example shown in FIG. 11, it is recognized that a plurality of parking frames PS exist on the left side of the own vehicle V, and the position information of the parking frames PS is updated as appropriate according to the movement of the position of the own vehicle V.

The image processing device 1A calculates the target parking position as a place to park the vacant parking frame PS closest to the vehicle V when the vehicle V stops. Target parking position information is transmitted to parking control ECU3. The parking control ECU 3 calculates a steering amount based on the target parking position received from the image processing device 1 and the output of a clearance sonar sensor (not shown), and transmits information on the steering amount to the EPS-ECU 4 . The EPS-ECU 4 performs automatic steering when the own vehicle V is parked based on the information about the steering amount received from the parking control ECU 3 . In addition, in this embodiment, the driver is in charge of the accelerator operation and the brake operation.

In this embodiment, even when edge lines detected from a captured image include curved portions, it is possible to detect lane markings by extracting only straight line segments that appropriately reflect the shape of the edge lines. For this reason, it is possible to prevent erroneous recognition of the shape of the parking frame or failure to detect the parking frame due to inappropriate detection of the lane marking. As a result, during parking assistance, it is possible to estimate the target parking position with high accuracy without deviation. In addition, during parking assistance, it is possible to prevent the parking frame displayed on the screen of the display device installed in the vehicle from deviating from the actual parking frame, thereby preventing the user from feeling uneasy.

<3. Notes>
Various modifications can be made to the various technical features disclosed in this specification without departing from the gist of the technical creation in addition to the above-described embodiments. That is, the above embodiments should be considered as examples in all respects and not restrictive, and the technical scope of the present invention is not defined by the description of the above embodiments, but by the scope of claims. All changes that come within the meaning and range of equivalency of the claims are to be understood. In addition, multiple embodiments and modifications shown in this specification may be implemented in combination to the extent possible.

In the above description, the dividing unit 113 is configured to perform the first dividing process and the second dividing process, but this is an example. For example, the dividing unit 113 may be configured to perform only the first dividing process.

Moreover, the dividing unit 113 may be configured to perform only the second dividing process. In this case, first, the dividing unit 113 obtains a dividing point based on the edge line and a straight line connecting two endpoints of the edge line detected by the edge line detecting unit 112 . As for the detailed method of obtaining the dividing points, the same processing as in steps S233 to S237 shown in FIG. 8 may be performed. After obtaining the first division point in this manner, the same processing as that shown in FIG. 8 may be performed. Even in this case, at least one dividing point can be set on the edge line, and an approximation line close to the shape of the edge line can be formed by a set of a plurality of straight line segments.

1, 1A... Image processing device 20... Side camera (camera)
112... Edge line detection unit 113... Division unit 114... Extraction unit 115... Marking line detection unit (marking line detection unit)
EL ... edge line LS ... straight line segment ML ... marking line P ... division point PS ... parking frame SL ... lane marking (marking line)
V: Own vehicle (vehicle)
VL・・・temporary straight line

Claims (8)

An image processing device comprising a control unit that detects a marking line provided on a road surface,
The control unit
Detecting an edge line that is a candidate for the contour line of the marking line from the captured image,
segmenting the edge line into a plurality of segments,
obtaining a first division point on the edge line based on direction information obtained for each section ;
An image processing device that divides the edge line into a straight section and a curved line section based on the first division point. 2. The image processing apparatus according to claim 1, wherein said control unit obtains said first division point at a position where curvature of said edge line is equal to or greater than a predetermined value. The control unit
the direction information obtained for each of the segments ;
a representative direction obtained by statistically processing the direction information for each of the sections;
3. The image processing apparatus according to claim 1, wherein said first division point is obtained based on . The image processing apparatus according to any one of claims 1 to 3 , wherein the control section obtains second division points for approximating the curved section with a straight line . The control unit further divides the curve section based on the provisional straight line set according to the first division point or the second division point obtained previously and the edge lines forming the curve section. 5. The image processing apparatus according to claim 4 , which determines whether or not. The control unit controls the marking based on the approximation line of the edge line composed of a straight line segment forming the straight section and a straight line segment set in the curved section according to the second dividing point. 6. The image processing device according to claim 4 , which detects lines. The captured image is captured by a camera that captures the surroundings of the vehicle,
The image processing device according to any one of claims 1 to 6, wherein the marking line is a marking line forming a parking frame. An image processing method executed by an image processing device that processes a captured image,
detecting an edge line from the captured image;
segmenting the edge line into a plurality of segments ,
obtaining dividing points on the edge line based on direction information obtained for each section ;
An image processing method, wherein the edge line is divided into a straight section and a curved section based on the dividing point.

Images