JP7453008B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing device and an image processing method that estimate parking spaces and driving lanes provided on a road surface based on image signals output from an imaging device that images the road surface around a vehicle.

2. Description of the Related Art Recently, vehicle parking support devices have been put into practical use that automatically detect a target parking space and park the vehicle automatically when the vehicle is parked in a predetermined parking lot. When detecting a parking space, parking lot lines are detected based on an image taken around the vehicle using a camera mounted on the vehicle, and an area surrounded by the parking lot lines is detected as a parking space. In addition, a driving support device has also been put into practical use that automatically detects the lane boundary lines that separate driving lanes based on images taken of the surrounding area of the vehicle while the vehicle is driving, and allows the vehicle to drive automatically. has been done.

Technologies for detecting white lines such as parking lot lines and lane boundary lines include white line detection devices and white line detection methods that detect edges from images taken around the vehicle and detect white lines based on the detected edges. It has been disclosed (for example, see Patent Documents 1 and 2).

Japanese Patent Application Publication No. 8-167023 Japanese Patent Application Publication No. 2007-179386

However, in the above-mentioned conventional technology, parking lot lines and the like may not be detected due to relatively large shadows of the own vehicle, other vehicles, walls, fences, etc., sunlight filtering through trees, reflection of light from road lights, etc. In particular, because the brightness and brightness difference of the parking lot line differ between the shadow and the area other than the shadow, the edge of the parking lot line may not be detected. Therefore, in order to improve the accuracy of detecting parking lot lines and the like, as well as the accuracy of detecting parking spaces and driving lanes, it has been desired to develop a technology that can suppress erroneous edge detection and non-detection.

SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide an image processing device and an image processing method that can detect markers drawn on a road surface, such as parking lot lines and lane boundary lines, with high precision.

In order to achieve the above object, an image processing device of the present invention is an image processing device that detects markers drawn on a road surface, and includes an image signal output from an imaging device that captures an image of the road surface around a vehicle. an edge detection unit that scans a base image in a predetermined direction and detects a plurality of edges; and for each of the plurality of edges, detects an edge having the shortest distance from the edge, and When the distance between the end points of the edge with the shortest distance to the edge is less than or equal to a threshold, the marker is detected based on the edge connection part where two edges are connected to form one edge, and the plurality of edges. The present invention is characterized by comprising a marker detection section that performs the following steps.

The image processing device of the present invention configured as described above detects the edge having the shortest distance from a plurality of edges detected from a captured image, and determines the distance between the edges. When the distance between the end points of the shortest edge is less than or equal to a threshold, the two edges are connected to form one edge. As a result, discontinuation of the edge of the marker due to shadows, illumination, etc. can be suppressed, and the end point of the edge and the angle of the edge can be detected with higher precision.

This makes it possible to detect markers drawn on the road surface, such as parking lot lines and lane boundary lines, with high precision. As a result, it becomes possible to detect parking spaces provided on the road surface of a parking lot or the like, detect driving lanes divided by lane boundary lines, etc. with high precision.

1 is a block diagram showing a schematic configuration of a parking assistance device to which an image processing device according to an embodiment of the present invention is applied. It is a figure showing an example of the arrangement position of the imaging device of the parking assistance device which is an embodiment. 1 is a functional block diagram showing a schematic configuration of an image processing device according to an embodiment. FIG. 3 is a flowchart for explaining an example of the operation of the image processing apparatus according to the embodiment. 7 is a flowchart for explaining an example of the operation of the edge connection section. FIG. 2 is a diagram for explaining an example of the operation of the image processing device according to the embodiment, and shows an example of a vehicle and a parking lot line drawn on the road surface of a parking lot. FIG. 2 is a diagram for explaining an example of the operation of the image processing device according to the embodiment, and is a diagram schematically showing an overhead view image and edges detected from the overhead view image. FIG. 2 is a diagram for explaining an example of the operation of the image processing apparatus according to the embodiment, and shows an image in which edges interrupted by shadows are connected. FIG. 4 is a diagram for explaining an example of the operation of the image processing device according to the embodiment, in which (a) shows an edge of a parking lot line interrupted by a shadow, and (b) shows an edge detected due to the influence of sunlight filtering through trees. show. 2 is a diagram for explaining an example of the operation of the image processing device according to the embodiment, in which (a) is a schematic diagram of edge B extracted within a predetermined distance from edge A; FIG. FIG. 3 is a diagram for explaining a scanning procedure between the end points of two edges A and B. FIG. 2A and 2B are diagrams for explaining an example of the operation of the image processing device according to the embodiment, in which (a) is a diagram showing a set of a short edge A1 and a long edge B1, and (b) is a diagram showing a set of a long edge A2 and a short edge B1. It is a figure which shows a set with edge B2. 2A and 2B are diagrams for explaining an example of the operation of the image processing apparatus according to the embodiment, in which (a) is a diagram showing a set of a long edge A3 and a short edge B3, and (b) is a diagram showing a set of a short edge A4 and a long edge B3. It is a figure showing a set with edge B4.

(Schematic configuration of parking assist device)
Embodiments of the present invention will be described below based on the drawings. FIG. 1 is a block diagram showing a schematic configuration of a parking assistance device to which an image processing device according to an embodiment of the present invention is applied. FIG. 2 is a diagram showing an example of the arrangement position of the imaging device of the parking assist device. Note that although a parking assistance device will be described below, the device to which the image processing device according to the embodiment of the present invention is applied is not limited to the parking assistance device; It can also be applied to driving support devices and the like.

As shown in FIG. 1, the parking assistance device 1 is mounted on a vehicle V (see FIG. 2) and performs a parking assistance operation. Specifically, the parking support device 1 recognizes a parking slot in which this vehicle V can park. Then, the parking support device 1 controls the vehicle V to park the vehicle V in the recognized parking slot.

As shown in FIG. 2, a plurality of small cameras (imaging devices) are provided at the front, rear, left and right sides of the vehicle V.

Specifically, a front camera 20a is mounted on the front bumper or front grill of the vehicle V so as to face the front of the vehicle V. A rear camera 20b is attached to the rear bumper or rear garnish of the vehicle V, facing toward the rear of the vehicle V. A left side camera 20c is mounted on the left door mirror of the vehicle V to face the left side of the vehicle V. A right side camera 20d is mounted on the right door mirror of the vehicle V, facing toward the right side of the vehicle V.

The front camera 20a, the rear camera 20b, the left side camera 20c, and the right side camera 20d are each equipped with a wide-angle lens or a fisheye lens that can observe a wide range. It is now possible to observe the entire area including the road surface. These cameras 20a to 20d constitute an imaging device that images the road surface around the vehicle V. In the following description, when the individual cameras (imaging devices) 20a to 20d are described without distinction, they will be simply referred to as the camera 20.

Returning to FIG. 1, the parking assist device 1 includes a front camera 20a, a rear camera 20b, a left camera 20c, a right camera 20d, a camera ECU 21, a navigation device 30, a wheel speed sensor 32, and a steering angle sensor 33. and has.

The camera ECU 21 is mainly composed of a microcomputer including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like. The camera ECU 21 controls the camera 20, and uses information detected by the camera 20 to perform processes for generating an overhead image, detection processing for detecting a parking frame, and determining whether the vehicle V can be parked in the detected parking frame. Performs judgment processing etc. to judge.

The navigation device (display device) 30 has a monitor 31 having an image display function. The navigation device 30 has a storage unit that stores map data for route guidance and the like. The navigation device 30 provides route guidance to the target point set by the operator of the navigation device 30 based on the map data and the current position of the vehicle V detected by a GPS device (not shown) or the like. Various images during the route guidance operation are displayed on the monitor 31.

The wheel speed sensor 32 is a sensor that detects the wheel speed of the vehicle V. Detection information (wheel speed) detected by the wheel speed sensor 32 is input to the vehicle control ECU 40.

The steering angle sensor 33 detects the steering angle of the steering wheel of the vehicle V. The steering angle when the vehicle V runs straight is set as a neutral position (0 degrees), and the rotation angle from the neutral position is output as the steering angle. Detection information (steering angle) detected by the steering angle sensor 33 is input to the vehicle control ECU 40.

Further, the parking assist device 1 includes a vehicle control ECU 40, a steering control unit 50, a throttle control unit 60, and a brake control unit 70.

The vehicle control ECU 40 is mainly composed of a microcomputer including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like. The vehicle control ECU 40 executes various processes to support parking of the vehicle V based on each detection information input from the camera ECU 21, the wheel speed sensor 32, and the steering angle sensor 33.

That is, for example, when the driver turns on an automatic parking start switch (not shown) to activate the parking support device 1, the vehicle control ECU 40 automatically parks the vehicle V in the parking slot that the camera ECU 21 determines is available for parking. Executes automatic parking processing.

The steering control unit 50 controls the steering angle of the vehicle V by driving the power steering actuator 51 based on vehicle control information determined by the vehicle control ECU 40.

The throttle control unit 60 controls the throttle of the vehicle V by driving the throttle actuator 61 based on vehicle control information determined by the vehicle control ECU 40.

The brake control unit 70 controls the brakes of the vehicle V by driving the brake actuator 71 based on vehicle control information determined by the vehicle control ECU 40.

The camera ECU 21, wheel speed sensor 32, and steering angle sensor 33 are connected to the vehicle control ECU 40 by a sensor information CAN (Controller Area Network) 80, which is an in-vehicle LAN (Local Area Network) ("CAN"). is a registered trademark).

Furthermore, the steering control unit 50, throttle control unit 60, and brake control unit 70 are connected to the vehicle control ECU 40 by a vehicle information CAN 81, which is an in-vehicle LAN.

In the parking assist device 1 having the above configuration, the image processing device 100 of this embodiment is mainly configured by the camera ECU 21.

(Functional configuration of image processing device)
FIG. 3 is a functional block diagram showing a schematic configuration of image processing apparatus 100 according to this embodiment. Image processing apparatus 100 according to this embodiment includes a control section 110 and a storage section 120. The control unit 110 is mainly composed of the CPU of the camera ECU 21, and the storage unit 120 is mainly composed of the ROM, RAM, flash memory, etc. of the camera ECU 21.

The control unit 110 controls the entire image processing apparatus 100. In addition, the control unit 110 is configured based on parking lot lines and parking spaces that demarcate parking spaces detected and estimated by an edge detection unit 111, an edge connection unit 112, a marker detection unit 113, and a parking space detection unit 114, which will be described later. In order to have the vehicle control ECU 40 execute automatic parking processing in which the vehicle V is automatically parked in a parking space determined to be available for parking, information necessary for the automatic parking processing (parking space, position of the parking space, shape, etc.) is sent to this vehicle control ECU 40.

The vehicle control ECU 40 controls the power steering based on the information provided from the control unit 110 and further based on the detection information detected by the wheel speed sensor 32 and the steering angle sensor 33 (only sensors are shown in FIG. 3). The actuator 51, the throttle actuator 61, and the brake actuator 71 (only the actuator is shown in FIG. 3) are drive-controlled.

The control unit 110 includes arithmetic elements such as a CPU, a programmable logic device such as an FPGA, and an integrated circuit such as an ASIC.

A control program (not shown) is stored in the storage unit 120 of the image processing device 100, and this control program is executed by the control unit 110 when the image processing device 100 is started up, and the image processing device 100 operates as shown in FIG. It will have the functional configuration as shown. In particular, since the image processing apparatus 100 of this embodiment performs high-speed image processing as described later, it is preferable to include an arithmetic element capable of high-speed calculation, such as an FPGA.

As shown in FIG. 3, the control section 110 includes an edge detection section 111, an edge connection section 112, a marker detection section 113, a parking frame detection section 114, and a display control section 115. Note that when the image processing device 100 and the image processing method are applied to a driving support device and a driving support method, a driving lane detection unit can be used instead of the parking frame detection unit 114 in FIG. An image processing device and an image processing method for detecting are obtained.

The edge detection unit 111 detects edges of parking lot lines on the road surface of a parking lot or the like by edge detection based on an image signal output from the camera 20 that images the road surface R around the vehicle V. Furthermore, when applied to a driving support device, the edge detection unit 111 detects edges of markers such as lane boundary lines on the road surface. The marker here refers to a boundary line such as a parking lot line or a driving lane. A parking lot line is a line drawn as a boundary line (straight line) that mainly divides a parking area provided on the road surface R. FIG. 5 shows an example of a vehicle V and a parking lot line 200 drawn on the road surface R of the parking lot P where the vehicle V is attempting to park. Between the parking lot lines 200 is a parking frame 201 representing a parking space.

In addition, when detecting the driving lane, the lane boundary line is mainly drawn as a boundary line that divides the driving lane (driving lane) provided on the road surface, and is a continuous solid line (straight line or curved line) or a predetermined line. A broken line (straight line or curved line) is a series of short line segments. Note that markers are not limited to parking lot lines or driving lanes, and include, for example, diagonal lines indicating areas where parking is prohibited, crosswalks, wheelchair marks, etc.

Parking lane lines and lane lane lines are generally indicated by white lines, but may also be drawn by lines of a color other than white, such as yellow lines, for example. Therefore, the parking lot line or lane boundary line detected by the edge detection unit 111 is not limited to a "white line," but in general, a parking lot line or lane boundary line that has contrast with the road surface is used. It is sufficient to detect it as a line.

The edge detection unit 111 scans the image in a predetermined direction and detects pixels in which the luminance value or color parameter information (for example, RGB, RGBA, etc.) included in the image signal changes more than a threshold value. , a portion where the detected pixel arrangement has a length longer than a predetermined length is detected as an edge. Scanning here refers to selecting pixels one by one in a predetermined direction and comparing brightness or color parameters between adjacent pixels.

Note that the scanning direction is preferably set in a direction perpendicular to the parking lot line drawn on the road surface R. That is, as shown in FIG. 5, when the parking lot line 200 extends in a direction perpendicular to the traveling direction of the vehicle V, it is desirable to scan the overhead image G along the traveling direction. On the other hand, when the parking lot line 200 extends along the traveling direction of the vehicle V, it is desirable to scan the overhead image G in a direction perpendicular to the traveling direction. Generally, since the direction in which the parking lot line 200 extends is unknown, the edge detection unit 111 detects the direction in which the parking lot line 200 extends in two steps, respectively along the traveling direction of the vehicle V and the direction perpendicular thereto on the overhead image G. It is desirable to scan.

When extracting edges based on brightness values, the edge detection unit 111 detects a change from a dark pixel with low brightness (for example, a black pixel) to a bright pixel with high brightness (for example, a white pixel) with a difference greater than a threshold value. An edge where the luminance difference between adjacent pixels becomes greater than a predetermined value in the positive direction is detected as a plus edge (also referred to as a "rising edge"). Detection of this plus edge indicates that the scanning position has been switched from the road surface R to what is estimated to be the parking lot line 200.

The edge detection unit 111 also detects an edge where a pixel with high brightness changes to a dark pixel with low brightness with a difference larger than a threshold value, that is, a brightness difference between adjacent pixels is larger than a predetermined value in the negative direction. This edge is detected as a negative edge (also referred to as a "falling edge"). Detection of this negative edge indicates that the scanning position has been switched from what is estimated to be the parking lot line 200 to the road surface R.

On the other hand, when extracting edges based on color parameters, the color parameters of the road surface R and the color parameters of the parking lot line 200 are compared. The edge detection unit 111 detects as a negative edge the arrangement of pixels in which the value of the color parameter changes in the direction of increasing (changes in the negative direction), and detects as a negative edge the arrangement of pixels in which the value of the color parameter changes in the direction of decreasing (changes in the positive direction). ) is detected as a plus edge. Furthermore, when the brightness of the parking lot line is lower than that of the road surface (or the color parameter is larger), the changes in the brightness value and color parameter are reversed. In either case, a plus edge and a minus edge are detected on both sides of a marker such as a parking lot line, so it is possible to extract pairs as described below.

By repeating the above scanning for a plurality of lines (rows), a line segment (pixel column) consisting of continuous plus edges in a direction intersecting the scanning direction is extracted as a plus edge line segment. Furthermore, a line segment (pixel row) composed of continuous negative edges is extracted as a line segment of negative edges.

FIG. 6A schematically shows the bird's-eye view image G and edges (thick straight lines and thick broken lines) detected from the bird's-eye view image G. In the example of FIG. 6A, the X axis (here, the direction along the traveling direction of the vehicle V and orthogonal to the extending direction of the parking lot line 200) of the overhead image G is set in the left-right direction in the figure, and the Y The axis (in this case, the extending direction of the parking lot line 200) is set in the vertical direction in the figure. The edge detection unit 111 scans the overhead image G in a direction perpendicular to the traveling direction of the vehicle V, from left to right in the figure (X-axis positive direction), and detects positive edges and negative edges. go. Note that when pixels are scanned from right to left in the figure, that is, in the negative direction of the X axis, the plus edges and minus edges are reversed. Further, plus edges and minus edges may be detected based on information on color parameters (for example, RGB, RGBA, etc.) included in the image signal. In this case, these are detected based on changes in the size (gradation) of predetermined colors.

By the way, if a large shadow such as the shadow of the vehicle V overlaps with the parking lot line 200, when the edge is detected on the overhead image G, the edge of the first parking lot line will overlap the road surface R and the shadow, as shown in FIG. 6A. It may be cut off near the border. This is due to the fact that the brightness of the road surface is different between the sunny part that is not in the shadow and the shaded part that is the shaded part, and the brightness of the parking lot line is also different from each other.

For this reason, the values of the brightness difference between the road surface and the parking lot line in the sunny and shaded areas and the angles (directions) of the edges are detected differently, and there is no continuity between the edges of the sunny and shaded areas. are detected as separate edges. Furthermore, edges near the boundary between sunlight and shade may not be detected. As a result, the edges of parking lot lines are detected as being interrupted, which may affect the accuracy of parking frame detection and lane detection. In addition, when a part of the parking lot line is strongly illuminated by road lights or headlights, the luminance difference value of the road surface or parking lot line in the image will be different between this strongly illuminated part and the non-illuminated part. A similar phenomenon may occur, as it may cause overexposure or overexposure.

In order to avoid such an influence on detection accuracy, the edge connection unit 112 performs a process of connecting disconnected edges. In this embodiment, the edge connection unit 112 detects the edge having the shortest distance from each of the plurality of edges detected by the edge detection unit 111, and connects the edge with the edge. Edges with the shortest distance are detected as a set of edges that are candidates for connection. Further, the edge refers to an edge that is selected one by one from a plurality of edges and is a target for detecting other edges within a predetermined distance.

For each set of detected edges, the edge connection unit 112 connects two edges into one when the distance between the end points of the edges where the distance between the two edges is the shortest is equal to or less than a threshold value. edge (hereinafter sometimes referred to as "connection edge"). The edge connection unit 112 scans between the end points of the edge and the edge with the shortest distance in a direction corresponding to the direction in which the edge extends, and finds the edge with the shortest distance between the edges. It is determined whether information that can be determined as a marker is included between the edge and the edge. If the edge connecting unit 112 determines that information that can be determined as a marker is included, it connects the edge to the edge with the shortest distance to the edge, but if it determines that the edge is not included, Do not connect these.

The processing in the edge connection unit 112 of this embodiment will be described in more detail. In this specification, of the two endpoints of the line segment of each edge, the endpoint detected first by edge detection (the side whose Y coordinate is closer to the origin) is referred to as the "starting point," and the endpoint detected later (the side whose Y coordinate is closer to the origin) is referred to as the "starting point." The side farthest from the origin) is called the "end point."

A procedure for detecting an edge having the shortest distance to the edge will be described with reference to FIG. 7. 7A shows an edge of a parking lot line interrupted by a shadow in the overhead image G of FIG. 6A, and FIG. 7B shows an edge detected due to the influence of sunlight filtering through trees. As shown in FIG. 7B and FIG. 6A, a plurality of edges (noise) may be detected densely due to the influence of sunlight filtering through trees and the like. If such noise-causing edges are connected, lines other than parking lot lines will be detected as parking lot lines, which will affect detection efficiency and accuracy.

In order to suppress this, the edge connection unit 112 counts, for each of the plurality of edges, the number of other edges within a predetermined distance from the edge. When the count is greater than or equal to a predetermined number (threshold), the edge is not connected to other edges.

For example, in the example shown in FIG. 7(a), when searching for other plus edges within a predetermined distance (range indicated by a circle in the figure) from the end point E1 of the line segment A of the plus edge, A line segment B with a plus edge is detected. In this case, the count is 1 (less than the threshold), and the plus edge line segment A and the plus edge line segment B form a pair of edges that are connection candidates. Note that if multiple edge line segments (but less than the threshold) are detected, the edge line segment B that has the shortest distance from the plus edge line segment A becomes the connected edge line segment. .

On the other hand, if a noise edge is detected due to shadows (shades of buildings, sunlight filtering through trees, etc.), as shown in FIG. When searching for other edges within a predetermined distance from ', a large number of edges are detected, and the count becomes greater than or equal to the predetermined number (threshold). In other words, the line segment A' of the edge can also be determined to be one of the noises, and the line segment A' of the edge is not connected to other edges. This prevents edges of noise due to light reflection, dust, etc. from being unexpectedly connected, and the noise is removed by subsequent filtering, etc., and marker detection efficiency and detection accuracy are improved.

Furthermore, the edge connection unit 112 calculates the distances between the line segment A of the edge and all the start points and end points of the edge component B that are within a predetermined distance from the extracted line segment A of the edge. The calculation procedure will be explained with reference to FIG. 8(a). FIG. 8(a) shows line segment A of the edge (hereinafter referred to as "edge A") and edge component B (hereinafter referred to as "edge B") extracted within a predetermined distance. It is a schematic diagram. The starting point of edge A is Sa, the ending point is Ea, and the starting point of edge B is Sb, and the ending point is Eb.

Then, an equation for a straight line L including edge A (a straight line extending edge A) is calculated, and based on this equation and the position coordinates of the starting point Sb and ending point Eb of edge B on the image, the equation for straight line L and edge B is calculated. The distance Ds from the starting point Sb of the edge B, and the distance De between the straight line L and the ending point Eb of the edge B are calculated. It is determined whether the distances Ds and De from the straight line L are equal to or less than a predetermined value (threshold value). If it is determined that the threshold value is exceeded, this edge B may be noise and is therefore discarded. The above distance calculation and determination are performed for all edges B extracted for the edge A.

Then, if there is an edge B whose distances Ds and De are equal to or less than a predetermined value (threshold value), the edge B whose distance to edge A is the shortest is obtained from among the edges B that satisfy this condition, and the edge It is considered as a candidate for connection with A.

Next, a scanning procedure between mutually opposing end points of a pair of edge A and edge B having the shortest distance from edge A will be described with reference to FIG. 8(b). FIG. 8(b) is a diagram for explaining the scanning procedure between the end points of two edges A and B.

A predetermined distance between the end point Ea of edge A and the start point Sb of edge B is scanned in a direction intersecting the direction in which edges A and B extend. By setting this "predetermined length" to the length of the width of the parking lot line (or the number of pixels) + a threshold value, the edge of the interrupted part of the parking lot line can be detected with higher accuracy.

At this time, it is desirable to always scan the pixel line including the end point Ea and the start point Sb. However, between the end point Ea1 and the start point Sb1, it is not necessary to scan all the pixel lines, and it is sufficient to scan a plurality of lines at predetermined intervals in the Y-axis direction. The brightness difference and direction thresholds in this scan may be smaller than the brightness differences and direction thresholds used when the edge detection unit 111 detects edges.

When a rising edge (plus edge) and a falling edge (minus edge) of luminance are detected through this scanning, it is determined that information that can be determined as a marker has been detected. In FIG. 8(b), the lines and directions to be scanned are shown by arrows. Further, detected positive edges are shown as white circles, and detected negative edges are shown as black circles.

Here, if the rise and fall of the brightness are also detected in the pixel line including the end point Ea and the start point Sb, respectively, and all other lines, it is determined that a parking lot line has been detected. Note that even if the rise and fall of luminance are not detected in about one line among the other lines, it is determined that a parking lot line has been detected. If neither condition is satisfied, it is determined that no parking lot line is detected. This improves the accuracy of parking lot line detection.

If a parking lot line is detected between the end points of the two edges and the distance between the two edges is less than or equal to the threshold, the two edges are connected to form one edge (connected edge). That is, in the example shown in FIG. 8B, information about connected edges in which the starting point of edge A is the starting point of the connecting edge and the ending point of edge B is the ending point of the connecting edge is stored in the storage unit 120. Therefore, the information on edge A and edge B is deleted from the storage unit 120.

On the other hand, if no line edge is detected between the end points of the two edges, or if the distance between the two edges exceeds the threshold, the edges are detected from different objects or are originally separate edges. , no connection is made. Therefore, the information on edge A and edge B (coordinate information on the starting point and ending point) is stored in the storage unit 120 as is.

By the way, since the pair of edges that are connection candidates are the edges of the same parking lot line, the extending directions (angles) of the two edges are basically the same, but the effects of distortion correction of the captured image and conversion processing to an overhead image may be affected. There may be a difference in the angle. If this angular difference is large, the detection accuracy of the parking lot line will be affected when the two edges are connected, so it may be better not to connect them.

For this reason, in this embodiment, the distance between the edge and the end point of the edge with the shortest distance of a pair of edges that are considered connection candidates is calculated, and whether the distance is less than or equal to a threshold value is calculated. Based on this determination result, it is determined whether or not to connect both edges. The greater the angular difference between both edges, the greater these distances.

That is, the edge connection unit 112 regards the longer edge as the long edge and the shorter edge as the short edge among the two edges of the set of edges that are candidates for connection, and calculates the equation of the straight line formed by the long edges. Then, when either of the following conditions (1) or (2) is satisfied, the long edge and the short edge are connected to form one edge.

(1) The distance between the straight line composed of long edges and the start point of the short edge is less than or equal to the first threshold, and the distance between this straight line and the end point of the short edge is equal to or less than the second threshold (however, (threshold value<second threshold value) or less.
(2) The distance between the straight line of the long edge and the start point of the short edge is less than or equal to the second threshold, and the distance between this straight line and the end point of the short edge is less than or equal to the first threshold.

FIG. 9A is an example of a case where the above condition (1) is satisfied in a pair of short edge A1 and long edge B1. The distance D1 between the straight line (broken line) L1 that is an extension of the long edge B1 and the starting point Sa1 of the short edge A1 is less than or equal to the first threshold (severe threshold), and the straight line L1 and the ending point Ea1 of the short edge A1 The distance D1 is less than or equal to the second threshold (relaxed threshold).

FIG. 9(b) is an example of a case where the above condition (2) is satisfied in a pair of short edge A2 and long edge B2. The distance D3 between the straight line (broken line) L2 that is an extension of the long edge B2 and the starting point Sa2 of the short edge A2 is less than or equal to the second threshold (loose threshold), and the straight line L2 and the end point Ea2 of the short edge A2 The distance D4 is less than or equal to the first threshold (severe threshold).

Furthermore, when connecting two edges that satisfy the above condition (1) or (2), in order to bring the angle closer to the edge of the actual parking lot line, the edge connecting portion 112 of this embodiment is connected to the edge of the actual parking lot line. The end point of the edge on the opposite side from the end point closest to the edge with the shortest distance is set as the coordinates of the starting point of one edge. In addition, if the distance between the straight line that is an extension of the edge and the x-coordinate of the end point on the opposite side of the end point of the edge that is close to the edge and has the shortest distance to the edge is within the threshold, the distance to the edge The end point of the edge where is the shortest is the coordinate of the end point of one edge. On the other hand, if the distance between the straight line that is an extension of the edge and the x-coordinate of the end point of the edge with the shortest distance to the edge is greater than the threshold, then the end point of the edge with the shortest distance to the edge The point of intersection perpendicular to the straight line extending the edge from the x-coordinate of is the coordinate of the end point of one edge. More specifically, the following processing is performed.

(3) If the end point of the long edge and the start point of the short edge are opposite, the edge connection unit 112 sets the coordinates of the start point of the long edge to the coordinates of the start point of one connected edge. The coordinates of the end point are set according to the following conditions.

(3-1) The distance (difference) between the straight line that is an extension of the long edge and the x-coordinate of the end point of the short edge is less than or equal to the third threshold (stricter threshold), and the distance between the straight line and the end point of the short edge is less than or equal to the fourth threshold (relaxed threshold), the coordinates of the end point of the short edge are set as the coordinates of the end point of one edge.
(3-2) If the above condition (3-1) is not satisfied, the coordinates of the intersection perpendicular to the straight line from the x-coordinate of the end point of the short edge are set as the coordinates of the end point of one edge.

On the other hand, (4) if the start point of the long edge and the end point of the short edge are opposite, the end point coordinates of the long edge are the end point coordinates of one edge, and the start point coordinates of the short edge are the end point coordinates of one edge. Use the start point coordinates of the edge.

A specific example is shown in FIG. FIG. 10(a) is a specific example that satisfies the condition (3) above. The edge A3 is the long edge, and the edge B3 having the shortest distance from the edge A3 is the short edge. The end point Ea3 of the long edge A3 and the start point Sb3 of the short edge B3 are opposite to each other. In this case, the difference in the X coordinates of the straight line L3 formed by the long edge A3 and the end point Eb3 of the short edge B3 is defined as m, and the distance between the straight line L3 and the end point Eb3 of the short edge B3 is defined as D5.

The coordinates of the starting point of one edge (connection edge) connecting the long edge A3 and the short edge B3 are the coordinates (x ₃ , y ₃ ) of the starting point Sa3 of the long edge A3. Further, if the difference m and the distance D5 satisfy the conditions (3-1): difference m≦third threshold and distance D5≦4th threshold, the coordinates (x ₄ , y ₄ ) is the coordinate of the end point of the connected edge. On the other hand, in case (3-2), the coordinates (x ₅ , y ₅ ) of the intersection of the perpendicular h from the end point Eb3 of the short edge B3 and the straight line L3 are the coordinates of the end point of the connecting edge, and the position of the end point is is corrected.

FIG. 10B is a specific example that satisfies the condition (4) above, in which the edge A4 is a short edge, and the edge B is a long edge. The end point Ea4 of the short edge A4 and the start point Sb4 of the long edge B4 (the edge with the shortest distance) are opposite to each other. In this case, the coordinates (x ₆ , y ₆ ) of the starting point Sa4 of the short edge A4 are the coordinates of the starting point of the connecting edge. Further, the coordinates (x ₉ , y ₉ ) of the end point Eb4 of the long edge B4 are the coordinates of the end point of the connecting edge.

In this way, the coordinates of the starting point and end point of the connecting edge are set according to the difference in length between the relevant edge and the edge with the shortest distance to the relevant edge, and the distance between the straight line of the long edge and the end point of the short edge. . Furthermore, if any distance exceeds the threshold, the corrected coordinates are set at the end point of the connected edge. This allows the end points of the connection edges to be set at more accurate positions.

By the above-described operation of the edge connecting section 112, the interrupted edges are connected. FIG. 6B shows an image in which interrupted edges are connected by the edge connection section 112.

The marker detection unit 113 detects a parking lot line as a marker based on the edge detected by the edge detection unit 111 and the edge connected by the edge connection unit 112. More specifically, first, plus-edge line segments and minus-edge line segments that have a length equal to or greater than a predetermined reference length and extend in a predetermined direction (angle) are extracted.

The reference length can be, for example, the length of the vehicle V (for example, 5 m), but if the parking lot line is short, the reference length is set to be shorter than the vehicle length. The angle is an angle that takes into account the running direction of the vehicle V, the direction of the camera 20 that took the image, and the like. In the case of FIG. 6, the parking lot line is a straight line extending substantially perpendicularly toward the parking space with respect to the driving direction, so the angle is set to 90°±threshold value.

Next, the positions (coordinates) of the start and end points of each of the extracted plus edge line segments and minus edge line segments are calculated, and based on these positions, the positions (coordinates) of the plus edge line segments adjacent to each other at a predetermined interval are calculated. Line segments with negative edges are extracted and determined to be a pair of edges forming a parking lot line. For example, when the distance between the plus edge and the minus edge is within the range of, for example, the line width of the parking lot line±threshold value, these are determined to be a pair. On the other hand, edges shorter than the reference length, long edge line segments extending in a direction other than the vertical direction, and edge line segments for which no pair is found are discarded as noise. In the example shown in FIG. 6B, parking lot lines K1, K2, K3, K4, K5, and K6 are detected.

The parking frame detection unit 114 detects a parking frame and a parking space based on the parking lot line detected by the marker detection unit 113, and stores parking frame registration data 121 in the storage unit 120.

More specifically, the parking frame detection unit 114 first configures a parking space among the pairs of plus edge line segments and minus edge line segments of the plurality of parking lot lines detected by the marker detection unit 113. Select two opposing lines that have the potential to The two lines selected here are the lines that constitute the left and right ends of a pair of parking lot lines that separate the parking spaces, and are the negative edge line segments of a predetermined parking lot line (for example, K3 shown in FIG. 6B). and a plus edge line segment of the parking lot line (K4 shown in FIG. 6B) facing this.

Then, the parking frame detection unit 114 calculates the distance between the line segments of the two selected edges (inner dimensions of adjacent parking lot lines) based on the coordinates of the end points of each edge. It is determined whether the calculated distance is within a predetermined range (for example, within a range of parking space width ± threshold value). If this distance is within a predetermined range, the area partitioned by the line segments of the two edges is detected as a parking space. The width of the parking space is preferably 2m to 3m if it is for a regular car or small cargo vehicle. If it is a parking space for a large freight vehicle or bus, a height of 3.3 m or more is desirable.

The parking frame 201 is a rectangular frame whose long side is a line along the opposite positive edge line segment and negative edge line segment that constitute the detected parking space, and whose short side is a line connecting the opposing ends, respectively. (see FIG. 5), and it can be estimated that the inside thereof is a parking space.

Edges are detected with high accuracy by the edge connection unit 112, and parking lot lines are determined with high accuracy through subsequent processing by the marker detection unit 113, so parking spaces are also detected with high accuracy by the parking frame detection unit 114. be able to. In other words, the end points of parking lot lines interrupted by shadows are corrected to more accurate positions, and the shapes and positions of parking frames and parking spaces can also be detected more accurately.

Then, the parking frame detection unit 114 calculates the coordinate values of the end points of the opposing plus-edge line segments and minus-edge line segments that constitute the parking frame 201 that partitions the parking space, and stores them as parking frame registration data 121. Register at 120. At this time, by registering the coordinate values of at least the two end points of the parking frame 201 on the side closer to the vehicle V, the parking frame 201 can be identified while minimizing the storage capacity. However, if the coordinate values of the four points are registered, Good too. Further, the angle (extending direction) of the parking lot line 200 and other information necessary for automatic parking processing can also be added to the parking space registration data 121.

Moreover, the parking frame detection unit 114 can also perform a determination process or the like to determine whether or not the vehicle V can be parked in the detected parking frame. For example, if there are other vehicles or obstacles in the detected parking space, it is determined that parking is not possible and the parking space is not registered in the storage unit 120 as the parking space registration data 121. Moreover, the parking frame detection unit 114 can also determine a parking frame that is close to the vehicle V or a parking frame that is easy to park as a parking frame that can be parked, and register it in the storage unit 120 as the parking frame registration data 121.

The display control unit 115 displays the road surface image around the vehicle V captured by the camera 20 and the image indicating the parking space detected and estimated by the parking space detection unit 114, as appropriate, overlapping with this road surface image or singly. A display control signal to be displayed on the monitor 31 of the navigation device (display device) 30 is sent to the navigation device 30.

The storage unit 120 includes a storage medium such as a large capacity storage medium such as a hard disk drive, or a semiconductor storage medium such as ROM or RAM. The storage unit 120 temporarily or non-temporarily stores various data used during various operations in the control unit 110.

Further, as described above, the storage unit 120 stores parking space registration data 121 and parameter data 122. The parameter data 122 includes a threshold for detecting an edge, a threshold for the distance between edges when detecting the edge with the shortest distance to the edge, thresholds for various distances between the straight line of the edge, the start point of the edge, and the end point. 1, 2nd, 3rd, and 4th thresholds), count thresholds, standard lengths of boundary lines as markers such as parking lot lines and lane boundary lines, parking space widths, and their thresholds. Furthermore, various parameters used by the image processing apparatus 100, such as the width of the boundary line and the angle of the extending direction, can be stored. In addition, a plurality of parameters are stored corresponding to the country and region where the parking assist device 1 is used, the shape and size of the parking space (parking frame), the distance between driving lanes, the shape of the lane boundary line, etc. It may also be configured to select appropriate parameters.

(Operation of image processing device)
Next, an example of the operation of the image processing apparatus 100 according to this embodiment will be described with reference to the flowcharts of FIGS. 4A and 4B.

FIG. 4 is a flowchart for explaining the operation of the image processing apparatus 100, and FIG. 4B is a flowchart for explaining an example of the operation of the edge connection unit 112. The operations shown in the flowcharts of FIGS. 4A and 4B are started when the driver inputs an instruction to start automatic parking by operating an automatic parking start switch (not shown).

In step S1 shown in FIG. 4A, the control unit 110 of the image processing device 100 acquires an image signal of the road surface R around the vehicle V captured by the camera 20.

In step S2, based on the image signal acquired in step S1, the control unit 110 generates a signal by combining these image signals. The signal that is synthesized and generated in step S2 is a signal that causes the navigation device 30 to display an image (overhead image) as if a camera was installed above the vehicle V and looked down directly below. Techniques for generating signals for displaying such bird's-eye images are well known, and examples include techniques disclosed in Japanese Patent Application Laid-Open Nos. 3-99952 and 2003-118522.

FIG. 6 is an example of an overhead image G based on signals generated by combining. This bird's-eye image G is converted into an image signal for displaying bird's-eye images g1, g2, g3, and g4 looking down on the vehicle V from directly above, respectively, based on the image signals taken by the cameras 20a to 20d. Furthermore, it is an image displayed based on a signal generated by combining each image signal. In the center of the bird's-eye view image G, an icon I indicating a state in which the vehicle V is viewed from directly above is displayed.

Note that the signal synthesis work for displaying the bird's-eye view image G may not be performed in step S2, or the signal synthesis work in step S2 may be performed after the extraction of plus edges and minus edges in the next step S3. However, the processing load on the image processing apparatus 100 can be reduced by performing the signal synthesis work and then performing the positive edge and negative edge extraction work.

In step S3 (edge detection step), as described above, the edge detection unit 111 scans the overhead image G synthesized in step S2 in a predetermined direction, and detects positive edges and negative edges in the image based on the brightness value of the image signal. Detect edges (see FIG. 6A).

In the next step S4 (edge connection step), the edge connection portion 112 connects the interrupted edges. Details of the edge connection process by the edge connection section 112 will be explained with reference to the flowchart of FIG. 4B.

In the edge connection process, as shown in FIG. 4B, a detection loop process is executed for the edge B in the vicinity (meaning "within a predetermined distance") of steps S401 to S405, and the edge B detected by the edge detection unit 111 is detected. Edges existing in the vicinity of the plurality of edges (edge A) are extracted. This loop process of steps S401 to S405 is repeated until the process for all edges A is completed.

First, in step S402, one edge is selected from among a plurality of edges, and this is defined as the edge (edge A). An edge (edge B) within a predetermined distance around the end point of edge A is extracted. After extracting edge B, count up the number of edge points.

In step S403, it is determined whether an edge B within a predetermined distance has been detected, and when it is determined that it has been detected (number of edge points>0) (YES), the number of edge points and information on the detected edge B are It is registered in the storage unit 120 in correspondence with the information on edge A. On the other hand, if it is determined that no edge points are detected (number of edge points=0) (NO), the process of step S404 is skipped, and the process returns to S402 to process the next edge A.

When the processing for all edges A is completed and the detection loop processing for the nearby edge B is completed, the process advances to step S406. Then, edge connection loop processing in steps S406 to SS416 is executed to connect edges interrupted by shadows, illumination, or the like. This loop processing of steps S406 to S416 is repeated until processing for all combinations of edges A and B is completed.

First, in step S407, it is determined whether or not the number of edge points is equal to or less than a threshold value for a set of combinations sequentially selected from among the combinations of edge A and one or more edges B. If it is determined that the number of edge points is less than or equal to the threshold (YES), it is assumed that there is an edge B that has little noise and can be connected to edge A, and the process advances to step S408.

On the other hand, if it is determined that the number of edge points exceeds the threshold (NO), it is estimated that edge B is a lot of noise due to sunlight filtering through the trees, etc., and the edge A is not connected to the edge and the steps from step S408 onward are performed. Skip processing. If the next combination of edge A and edge B exists, the process returns to step S407, and if it does not exist, the edge connection loop ends.

In the next step S408, the distances between the straight line of edge A and the start and end points of all edges B are calculated (see FIG. 8(a)). In the next step S409, it is determined whether there is a straight line of edge A and the start point and end point of edge B for which the distance is less than or equal to a threshold value. If it is determined that there is a value below the threshold (YES), the process advances to step S410. On the other hand, if it is determined that there is no one below the threshold value (NO), the processing from step S410 onward is skipped. If the next combination of edge A and edge B exists, the process returns to step S407, and if it does not exist, the edge connection loop ends.

In step S410, edge B having the shortest distance to edge A is extracted based on the distance calculated in step S408. Next, in step S411, a scan is performed between the end point of edge A and the start point of edge B, and positive edges and negative edges (that is, information that can be determined as parking lot lines) are detected. If it is determined in step S412 that a parking lot line has been detected (YES), the process advances to step S413. As described above, if a plus edge and a minus edge are detected by scanning between end points, it is determined that a parking lot line has been detected. On the other hand, if a parking lot line is not detected, that is, if an edge is not detected (NO), edge A and edge B are edges detected from different objects, so the connection target In order to exclude it from the process, the process from step S413 onward is skipped. If the next combination of edge A and edge B exists, the process returns to step S407, and if it does not exist, the edge connection loop ends.

In step S413, edge A and edge B are compared, and the longer one is determined to be a long edge, and the shorter one is determined to be a short edge. Then, as explained above using FIGS. 9(a) and 9(b), the equation of the long edge straight line is calculated, and the distance between the long edge straight line and the start and end points of the short edge is calculated. do.

Then, in step S414, it is determined whether each calculated distance is less than or equal to a threshold value. More specifically, the distance between a straight line composed of long edges and the starting point of the short edge is less than or equal to a first threshold, and the distance between this straight line and the end point of the short edge is equal to or less than the second threshold (however, (first threshold<second threshold), or the distance between the straight line and the start point of the short edge is less than or equal to the second threshold, and the distance between the straight line and the end point of the short edge is less than or equal to the first Determine whether it is below the threshold.

If it is determined in this step S414 that any of the above conditions is satisfied (YES), the process advances to step S415, where the long edge and short edge are connected and registered in the storage unit 120 as one edge (connected edge). do. Specifically, as explained above using conditions (3) and (4) and FIGS. 10(a) and 10(b), the starting point of one edge is set as the starting point of the connecting edge according to the conditions. , the position coordinates of each edge are registered in the storage unit 120, with the end point of the other edge as the end point of the connected edge. If the next combination of edge A and edge B exists, the process returns to step S407, and if it does not exist, the edge connection loop ends.

On the other hand, if it is determined in step S414 that none of the above conditions are satisfied (NO), the process in step S415 is skipped and the edges are not connected. If the next combination of edge A and edge B exists, the process returns to step S407, and if it does not exist, the edge connection loop ends.

As described above, edges interrupted by shadows, lighting, etc. are appropriately connected. Further, even when the edge is divided into three or more parts, these parts are appropriately connected. That is, when edge connection processing is performed on the first edge, the first and second edges are connected to generate a connected edge. Thereafter, edge connection processing is performed on the third edge, thereby connecting the connection edge to the third edge and generating a new connection edge. Similarly, the fourth and subsequent edges are connected to the connecting edge, and the edge divided into a plurality of edges finally becomes one edge.

When the edge connecting process of step S4 is completed, the process returns to FIG. 4A and proceeds to the next step S5. In this step S5 (marker (parking lane line) detection step), based on the plurality of positive edges and negative edges detected by the edge detection unit 111 and the positive edges and negative edges connected by the edge connection unit 112, The marker detection unit 113 detects a parking lot line as a marker using the procedure described above. That is, a pair of a plus-edge line segment and a minus-edge line segment having a length equal to or longer than a predetermined reference length and adjacent to each other at a predetermined interval is detected and determined to be a pair forming a parking lot line. On the other hand, noise edges detected due to sunlight filtering through trees, garbage, etc., or edges for which no pair is found, are not extracted as parking lot line edges and are discarded.

Next, in step S6, the marker detection unit 113 registers the coordinate information of the start and end points of the plus-edge line segment and the minus-edge line segment in the storage unit 120 as parking lot line data.

After that, the process advances to step S7, and the parking frame detection unit 114 performs parking according to the procedure described above based on the parking lot lines registered in the storage unit 120 (in the example of FIG. 6B, parking lot lines K1 to K6). Detect frames and parking spaces.

In the next step S8, the parking frame detection unit 114 calculates the coordinate values of the end points of the opposing plus-edge line segments and minus-edge line segments that constitute each parking frame 201, and stores them as parking frame registration data 121. Register at 120. All detected parking space registration data 121 may be registered in the storage unit 120, or only parking space registration data 121 suitable for parking may be registered.

With the above steps, the processing by the image processing apparatus 100 ends. Parking slot registration data 121 detected by image processing device 100 is sent to vehicle control ECU 40, and various processes to support parking of vehicle V are executed.

(Effect of image processing device)
In the image processing apparatus 100 of this embodiment configured as described above, the edge detection section 111 detects a plurality of edges, and the edge connection section 112 connects edges interrupted by shadows, illumination, etc. Then, the marker detection unit 113 detects a parking lot line as a marker based on the plurality of detected edges and connected edges. Thereby, the end point of each edge is detected at a more accurate position. Furthermore, the direction (angle) in which the edge extends can also be set to a value closer to the true value. Therefore, it is possible to provide an image processing device 100 and an image processing method that can detect parking lot lines as markers drawn on a road surface with high precision.

Further, the edge connection unit 112 scans between the end points of the edge and the edge with the shortest distance in a direction according to the direction in which the edge extends, and calculates the distance between the edge and the edge. It is determined whether or not information that can be determined as a marker is included between the edge and the edge where is the shortest. As a result, when information that can be determined to be a marker is included, the edge is connected to the edge with the shortest distance to the edge, making it possible to reliably connect the disconnected marker and make the end point coordinates of the marker more accurate. The value can be close to the value. Therefore, markers can be detected with higher accuracy.

Furthermore, the edge connection unit 112 is configured to count the number of other edges within a predetermined distance from the edge, and when the count is equal to or greater than the predetermined number, does not connect the edge to the other edge. As a result, noise edges detected due to shadows (shadows of buildings, sunlight filtering through trees, etc.) and reflection of light can be excluded from connection targets, and processing efficiency and accuracy can be improved.

Furthermore, the edge connection unit 112 sets the coordinates of the starting point of one edge to the end point of the edge on the opposite side of the end point closest to the edge where the distance to the edge is the shortest. In addition, if the distance between the straight line that is an extension of the edge and the x-coordinate of the end point on the opposite side of the end point of the edge that is close to the edge and has the shortest distance to the edge is within the threshold, the distance to the edge The end point of the edge where is the shortest is the coordinate of the end point of one edge. On the other hand, if the distance between the straight line that is an extension of the edge and the x-coordinate of the end point of the edge with the shortest distance to the edge is greater than the threshold, then the end point of the edge with the shortest distance to the edge The point of intersection perpendicular to the straight line extending the edge from the x-coordinate of is the coordinate of the end point of one edge. With this configuration, the end point of one edge can be detected more accurately, and the direction in which one edge extends can be brought closer to the direction in which the actual marker extends, thereby improving marker detection accuracy.

In addition, when the marker is a parking lot line that divides a parking space, the configuration includes a parking frame detection unit 114 that detects a parking frame based on the marker detected by the marker detection unit 113. It becomes possible to detect frames with higher precision. Therefore, by applying the image processing device 100 and the image processing method to a parking support device and a parking support method, the accuracy of the final parking position can be improved, and more appropriate parking support can be provided.

In addition, when the marker is a lane boundary line that divides a driving lane, the driving lane can be determined by a configuration including a driving lane detection unit that detects the driving lane based on the marker detected by the marker detection unit 113. can be detected with higher precision. Therefore, the image processing device 100 and the image processing method can be suitably used as a driving support device and a driving support method, and more appropriate driving support can be provided.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configuration is not limited to these embodiments and examples, and design changes can be made to the extent that they do not depart from the gist of the present invention. are included in the present invention.

For example, in the image processing apparatus 100 according to the embodiment described above, the magnitude and direction of change (positive direction or negative direction) of information on brightness and color parameters (for example, RGB, RGBA, etc.) included in the image signal are ), but the present invention is not limited to these, and edges may be detected based on the magnitude and direction of change in other information included in the image signal.

20 Camera (imaging device) 20a Front camera (imaging device)
20b Rear camera (imaging device) 20c Left side camera (imaging device)
20d Right side camera (imaging device) 100 Image processing device 111 Edge detection unit 112 Edge connection unit 113 Marker detection unit 114 Parking frame detection unit 200 Parking lane line (marker) 201 Parking frame A Edge (approximate edge) B Edge (distance (the shortest edge)
E End point G Overhead view image (image)
L Straight line with extended edge R Road surface S Starting point V Vehicle

Claims (8)

An image processing device that detects markers drawn on a road surface,
an edge detection unit that detects a plurality of edges by scanning an image in a predetermined direction based on an image signal output from an imaging device that images the road surface around the vehicle;
For each of the plurality of edges, detect the edge with the shortest distance to the edge, and when the distance between the end points of the edge with the shortest distance between the edge and the edge is less than or equal to a threshold; an edge connection part that connects two edges to form one edge;
a marker detection unit that detects the marker based on the plurality of edges ,
The edge connection part is defined by the distance between a straight line extending the edge and the starting point of the edge where the distance between the edge is the shortest, and the distance between the line extending the edge and the end point of the edge closest to the edge. An image processing apparatus characterized in that when the distances of the edges are each equal to or less than a threshold value, the edge and the edge having the shortest distance from the edge are connected to form one edge . The edge connection portion is such that a distance between a straight line extending the edge and a starting point of the edge having the shortest distance to the edge is equal to or less than a first threshold, and the straight line extending the edge , the distance to the end point of the edge with the shortest distance to the edge is less than or equal to a second threshold that is greater than the first threshold, and the distance between the straight line extending the edge and the edge The distance to the starting point of the edge with the shortest distance is less than or equal to the second threshold, and the distance between the straight line extending the edge and the end point of the edge with the shortest distance is the second threshold. A first threshold value or less, when any one of the following is satisfied, the edge and the edge having the shortest distance to the edge are connected to form one edge. 1. The image processing device according to 1. The edge connection section scans between end points of the edge and an edge having the shortest distance to the edge in a direction corresponding to the direction in which the edge extends, and connects the edge to the edge. 3. The image processing apparatus according to claim 1, wherein the image processing apparatus determines whether information that can be determined as the marker is included between the edge and the edge having the shortest distance. The edge connection section counts the number of other edges within a predetermined distance from the edge, and when the count is equal to or greater than a predetermined number, does not connect the edge to the other edge. The image processing device according to any one of claims 1 to 3 . The edge connection part is a straight line extending the edge, with the coordinates of the starting point of the one edge being the end point of the edge opposite to the end point closest to the edge where the distance to the edge is the shortest. and the x-coordinate of the end point on the opposite side of the end point of the edge that is close to the edge and is the shortest distance to the edge, and the distance between the edge and the edge is within a threshold, the edge that has the shortest distance to the edge. is the coordinate of the end point of the one edge, and when the distance between the straight line extending the edge and the x-coordinate of the end point of the edge with the shortest distance to the edge is greater than the threshold , The coordinates of the end point of the one edge are the intersection of a straight line extending the edge and a perpendicular line that passes through the end point of the edge that has the shortest distance to the edge and intersects perpendicularly to the straight line extending the edge. The image processing device according to any one of claims 1 to 4 , characterized in that: The marker is a parking lot line that demarcates a parking space,
The image processing device according to any one of claims 1 to 5 , further comprising a parking frame detection unit that detects a parking frame based on the marker detected by the marker detection unit. The marker is a lane boundary line that divides the driving lane,
The image processing device according to any one of claims 1 to 5 , further comprising a driving lane detection unit that detects the driving lane based on the marker detected by the marker detection unit. An image processing method for detecting markers drawn on a road surface, the method comprising:
an edge detection step of detecting a plurality of edges by scanning an image in a predetermined direction based on an image signal output from an imaging device that images the road surface around the vehicle;
For each of the plurality of edges, detect the edge with the shortest distance to the edge, and when the distance between the end points of the edge with the shortest distance between the edge and the edge is less than or equal to a threshold; an edge connecting step of connecting two edges to form one edge;
a marker detection step of detecting the marker based on the plurality of edges ,
The edge connecting step includes determining the distance between a straight line extending the edge and the starting point of the edge where the distance to the edge is the shortest, and the distance between the line extending the edge and the end point of the edge closest to the edge. An image processing method characterized in that, when the distances of the edges are each equal to or less than a threshold value, the edge and the edge having the shortest distance to the edge are connected to form one edge .