JP7446851B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing device and an image processing method that estimate parking spaces and driving lanes provided on a road surface based on image signals output from an imaging device that images the road surface around a vehicle.

2. Description of the Related Art A parking support device has been put into practical use that automatically detects a parking slot as a parking target when parking a vehicle in a predetermined parking lot, and automatically parks the vehicle (for example, see Patent Document 1). In the technology disclosed in Patent Document 1, parking lot lines are detected from an image captured by an imaging unit, and an area surrounded by a pair of parking lot lines is recognized as a parking target position where parking is possible, that is, a parking space. ing.

Japanese Patent Application Publication No. 2014-166834

However, with the above conventional technology, parking lot lines may not be detected due to relatively large shadows from vehicles, walls, fences, etc., sunlight filtering through trees, reflection of light from road lights, blurring, presence of obstacles, etc. be. In particular, because the brightness and brightness difference of parking lot lines differ between shadows and non-shadow areas, the edges of parking lot lines may not be detected properly, resulting in parts of parking lot lines not being detected or The line may be detected as being divided, and the position of the end point of the parking frame may be misaligned.

Therefore, an object of the present invention is to provide an image processing device and an image processing method that can detect parking spaces with high precision.

In order to achieve the above object, the image processing device of the present invention includes a parking lot line detection unit that detects parking lot lines from an image based on an image signal output from an imaging device that images the surrounding situation of the vehicle; Based on the extending direction of the parking lot line, it is determined from a position a predetermined distance away whether there is a continuation parking lot line on an extension line of the parking lot line, and the continuation parking lot line is detected. and a correction unit that connects the parking lot line detected first and the continuous parking line detected next to form one parking lot line; The present invention is characterized by comprising a parking frame setting unit that sets a parking frame using the parking lot line of the parking lot line.

In the image processing device of the present invention configured as described above, even if a part of the parking lot line is not detected or is detected as being divided due to the influence of shadows, etc., the end point of the parking space can be determined by the true value. It can be corrected to a closer appropriate position. Therefore, it becomes possible to detect parking spaces with high precision. As a result, highly accurate parking assistance using the detected parking spaces becomes possible.

1 is a block diagram showing a schematic configuration of a parking assistance device to which an image processing device according to an embodiment of the present invention is applied. It is a figure showing an example of the arrangement position of the imaging device of the parking assistance device which is an embodiment. 1 is a functional block diagram showing a schematic configuration of an image processing device according to an embodiment. FIG. 3 is a flowchart for explaining an example of the operation of the image processing apparatus according to the embodiment. FIG. 2 is a diagram for explaining an example of the operation of the image processing device according to the embodiment, and shows an example of a vehicle and a parking lot line drawn on the road surface of a parking lot. It is a figure for explaining an example of operation of an image processing device which is an embodiment, and is a figure showing typically a bird's-eye view image, an edge detected from a bird's-eye view image, and a parking frame. FIG. 2 is a diagram for explaining an example of the operation of the image processing device according to the embodiment, and is a diagram schematically showing an edge whose end point has been appropriately corrected and a parking frame. FIG. 4 is a diagram for explaining an example of the operation of the image processing device according to the embodiment, and (a) is a diagram for explaining the correction processing procedure when there are divided parking lot lines; b) is a diagram for explaining the correction processing procedure when there is no divided parking lot line. FIG. 3 is a diagram for explaining an example of the operation of the image processing apparatus according to the embodiment, and is a diagram for explaining a correction processing procedure when there are divided parking lot lines. FIG. 3 is a diagram for explaining an example of the operation of the image processing apparatus according to the embodiment, and is a diagram for explaining a correction processing procedure when there are divided parking lot lines.

(Schematic configuration of parking assist device)
Embodiments of the present invention will be described below based on the drawings. FIG. 1 is a block diagram showing a schematic configuration of a parking assistance device to which an image processing device according to an embodiment of the present invention is applied. FIG. 2 is a diagram showing an example of the arrangement position of the imaging device of the parking assist device. Note that although a parking assistance device will be described below, the device to which the image processing device according to the embodiment of the present invention is applied is not limited to the parking assistance device; It can also be applied to driving support devices and the like.

As shown in FIG. 1, the parking assistance device 1 is mounted on a vehicle V (see FIG. 2) and performs a parking assistance operation. Specifically, the parking support device 1 recognizes a parking slot in which this vehicle V can park. Then, the parking support device 1 controls the vehicle V to park the vehicle V in the recognized parking slot.

As shown in FIG. 2, a plurality of small cameras (imaging devices) are provided at the front, rear, left and right sides of the vehicle V. Specifically, a front camera 20a is mounted on the front bumper or front grill of the vehicle V so as to face the front of the vehicle V. A rear camera 20b is attached to the rear bumper or rear garnish of the vehicle V, facing toward the rear of the vehicle V. A left side camera 20c is mounted on the left door mirror of the vehicle V to face the left side of the vehicle V. A right side camera 20d is mounted on the right door mirror of the vehicle V, facing toward the right side of the vehicle V.

The front camera 20a, the rear camera 20b, the left side camera 20c, and the right side camera 20d are each equipped with a wide-angle lens or a fisheye lens that can observe a wide range. It is now possible to observe the entire area including the road surface. These cameras 20a to 20d constitute an imaging device that images the surroundings of the vehicle V (in this embodiment, the road surface R around the vehicle V). In the following description, when the individual cameras (imaging devices) 20a to 20d are described without distinction, they will be simply referred to as the camera 20.

Returning to FIG. 1, the parking assist device 1 includes a front camera 20a, a rear camera 20b, a left camera 20c, a right camera 20d, a camera ECU 21, a navigation device 30, a wheel speed sensor 32, and a steering angle sensor 33. and has.

The camera ECU 21 is mainly composed of a microcomputer including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like. The camera ECU 21 controls the camera 20, and uses information detected by the camera 20 to perform processes for generating an overhead image, detection processing for detecting a parking frame, and determining whether the vehicle V can be parked in the detected parking frame. Performs judgment processing etc. to judge.

The navigation device (display device) 30 has a monitor 31 having an image display function. The navigation device 30 has a storage unit that stores map data for route guidance and the like. The navigation device 30 provides route guidance to the target point set by the operator of the navigation device 30 based on this map data and the current position of the vehicle V detected by a GPS device (not shown) or the like. Various images during the route guidance operation are displayed on the monitor 31.

The wheel speed sensor 32 is a sensor that detects the wheel speed of the vehicle V. Detection information (wheel speed) detected by the wheel speed sensor 32 is input to the vehicle control ECU 40.

The steering angle sensor 33 detects the steering angle of the steering wheel of the vehicle V. The steering angle when the vehicle V runs straight is set as a neutral position (0 degrees), and the rotation angle from the neutral position is output as the steering angle. Detection information (steering angle) detected by the steering angle sensor 33 is input to the vehicle control ECU 40.

Further, the parking assist device 1 includes a vehicle control ECU 40, a steering control unit 50, a throttle control unit 60, and a brake control unit 70.

The vehicle control ECU 40 is mainly composed of a microcomputer including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like. The vehicle control ECU 40 executes various processes to support parking of the vehicle V based on each detection information input from the camera ECU 21, the wheel speed sensor 32, and the steering angle sensor 33.

That is, for example, when the driver turns on an automatic parking start switch (not shown) to activate the parking support device 1, the vehicle control ECU 40 automatically parks the vehicle V in the parking slot that the camera ECU 21 determines is available for parking. Executes automatic parking processing.

The steering control unit 50 controls the steering angle of the vehicle V by driving the power steering actuator 51 based on vehicle control information determined by the vehicle control ECU 40.

The throttle control unit 60 controls the throttle of the vehicle V by driving the throttle actuator 61 based on vehicle control information determined by the vehicle control ECU 40.

The brake control unit 70 controls the brakes of the vehicle V by driving the brake actuator 71 based on vehicle control information determined by the vehicle control ECU 40.

The camera ECU 21, wheel speed sensor 32, and steering angle sensor 33 are connected to the vehicle control ECU 40 by a sensor information CAN (Controller Area Network) 80, which is an in-vehicle LAN (Local Area Network) ("CAN"). is a registered trademark).

Furthermore, the steering control unit 50, throttle control unit 60, and brake control unit 70 are connected to the vehicle control ECU 40 by a vehicle information CAN 81, which is an in-vehicle LAN.

In the parking assist device 1 having the above configuration, the image processing device 100 of this embodiment includes a camera ECU 21.

(Functional configuration of image processing device)
FIG. 3 is a functional block diagram showing a schematic configuration of image processing apparatus 100 according to this embodiment. Image processing apparatus 100 according to this embodiment includes a control section 110 and a storage section 120. The control unit 110 is mainly composed of the CPU of the camera ECU 21, and the storage unit 120 is mainly composed of the ROM, RAM, flash memory, etc. of the camera ECU 21.

The control unit 110 controls the entire image processing apparatus 100. In addition, the control unit 110 controls the parking lot lines and parking spaces that delimit the parking spaces detected and set by the edge detection unit 111, the parking lot line detection unit 112, the parking frame setting unit 113, and the correction unit 114, which will be described later. In order to cause the vehicle control ECU 40 to automatically park the vehicle V in the parking slot determined to be available for parking based on the information required for the automatic parking process (parking space, position of the parking slot) , shape, etc.) are sent to this vehicle control ECU 40.

The vehicle control ECU 40 adjusts the power based on the information provided from the control unit 110 and further based on the detection information detected by the wheel speed sensor 32 and the steering angle sensor 33 (only sensors are shown in FIG. 3). The steering actuator 51, the throttle actuator 61, and the brake actuator 71 (only actuators are shown in FIG. 3) are drive-controlled.

The control unit 110 includes arithmetic elements such as a CPU, a programmable logic device such as an FPGA, and an integrated circuit such as an ASIC.

A control program (not shown) is stored in the storage unit 120 of the image processing device 100, and this control program is executed by the control unit 110 when the image processing device 100 is started up, and the image processing device 100 operates as shown in FIG. It will have the functional configuration as shown. In particular, since the image processing apparatus 100 of this embodiment performs high-speed image processing as described later, it is preferable to include an arithmetic element capable of high-speed calculation, such as an FPGA.

As shown in FIG. 3, the control section 110 includes an edge detection section 111, a parking lot line detection section 112, a parking frame setting section 113, a correction section 114, and a display control section 115.

The edge detection unit 111 detects the edges of markers such as parking lot lines on the road surface R of the parking lot P etc. by edge detection based on the image signal output from the camera 20 that images the road surface R around the vehicle V. do. The marker here refers to a boundary line such as a parking lot line or a driving lane. A parking lot line is a line drawn as a boundary line (straight line) that mainly divides a parking area provided on the road surface R. FIG. 5 shows an example of a vehicle V and a parking lot line 200 drawn on the road surface R of the parking lot P where the vehicle V is attempting to park. Between the parking lot lines 200 is a parking frame 201 representing a parking space.

In this specification, the side closer to the vehicle V is referred to as the "front", "front", etc. of the parking frame 201, and the opposite side is referred to as the "back", "rear", etc. Then, among the four vertices (end points) that make up the rectangle of the parking frame, that is, the two end points on each of the two sides of the parking frame along the parking lot line, the end point on the side closer to the vehicle V is designated as the "starting point". Okay, the rear end point is called the "end point."

Parking lot lines are generally indicated by white lines, but may also be drawn by lines of a color other than white, such as yellow lines. Therefore, the parking lot line detected by the edge detection unit 111 is not limited to a "white line", and in general, a boundary line that has a contrast with the road surface may be detected as a parking lot line.

The edge detection unit 111 scans the image in a predetermined direction and detects pixels in which the luminance value or color parameter information (for example, RGB, RGBA, etc.) included in the image signal changes more than a threshold value. , a portion where the detected pixel arrangement has a length longer than a predetermined length is detected as an edge. Scanning here refers to selecting pixels one by one in a predetermined direction and comparing brightness or color parameters between adjacent pixels. The detected edge is defined as a plus edge or a minus edge depending on the direction (trend) of change in the brightness or color parameter.

Note that the scanning direction is preferably set in a direction perpendicular to the parking lot line drawn on the road surface. That is, as shown in FIG. 5, when the parking lot line 200 extends in a direction perpendicular to the traveling direction of the vehicle V (see the arrow in FIG. 5), the traveling direction is It is desirable to scan along the On the other hand, when the parking lot line 200 extends along the traveling direction of the vehicle V, it is desirable to scan the overhead image G in a direction perpendicular to the traveling direction. Generally, since the direction in which the parking lot line 200 extends is unknown, the edge detection unit 111 detects the direction in which the parking lot line 200 extends in two steps, respectively along the traveling direction of the vehicle V and the direction perpendicular thereto on the overhead image G. It is desirable to scan.

When extracting edges based on brightness values, the edge detection unit 111 detects a change from a dark pixel with low brightness (for example, a black pixel) to a bright pixel with high brightness (for example, a white pixel) with a difference greater than a threshold value. An edge where the luminance difference between adjacent pixels becomes greater than a predetermined value in the positive direction is detected as a plus edge (also referred to as a "rising edge"). Detection of this plus edge indicates that the scanning position has been switched from the road surface R to what is estimated to be the parking lot line 200.

The edge detection unit 111 also detects an edge where a pixel with high brightness changes to a dark pixel with low brightness with a difference larger than a threshold value, that is, a brightness difference between adjacent pixels is larger than a predetermined value in the negative direction. This edge is detected as a negative edge (also referred to as a "falling edge"). Detection of this negative edge indicates that the scanning position has been switched from what is estimated to be the parking lot line 200 to the road surface R.

On the other hand, when extracting edges based on color parameters, the color parameters of the road surface R and the color parameters of the parking lot line 200 are compared. The edge detection unit 111 detects as a negative edge the arrangement of pixels in which the value of the color parameter changes in the direction of increasing (changes in the negative direction), and detects as a negative edge the arrangement of pixels in which the value of the color parameter changes in the direction of decreasing (changes in the positive direction). ) is detected as a plus edge. Furthermore, when the brightness of the parking lot line is lower than that of the road surface (or the color parameter is larger), the changes in the brightness value and color parameter are reversed. In either case, a plus edge and a minus edge are detected on both sides of a marker such as a parking lot line, so it is possible to extract pairs as described below.

By repeating the above scanning for a plurality of lines (rows), a line segment (pixel column) consisting of continuous plus edges in a direction intersecting the scanning direction is extracted as a plus edge line segment. Furthermore, a line segment (pixel row) composed of continuous negative edges is extracted as a line segment of negative edges.

In the example of FIG. 6A, the X axis (here, the direction along the traveling direction of the vehicle V and orthogonal to the extending direction of the parking lot line 200) of the overhead image G is set in the left-right direction in the figure, and the Y The axis (in this case, the extending direction of the parking lot line 200) is set in the vertical direction in the figure. The edge detection unit 111 scans the overhead image G in a direction perpendicular to the traveling direction of the vehicle V, from left to right in the figure (X-axis positive direction), and detects positive edges and negative edges. go. Note that when pixels are scanned from right to left in the figure, that is, in the negative direction of the X axis, the plus edges and minus edges are reversed. Further, plus edges and minus edges may be detected based on information on color parameters (for example, RGB, RGBA, etc.) included in the image signal. In this case, these are detected based on changes in the size (gradation) of predetermined colors.

The parking lot line detection unit 112 detects a parking lot line based on the edge detected by the edge detection unit 111. More specifically, the parking lot line detection unit 112 detects that the detected plus-edge line segments and minus-edge line segments have a length equal to or greater than a predetermined reference length, and have a predetermined length. A plus edge line segment and a minus edge line segment extending in a direction (angle) are extracted. On the other hand, edges that are shorter than the reference length or line segments of edges that extend in a direction other than the vertical direction are not extracted. This eliminates the reflection of light on the road surface and the edges of noise caused by dust, dirt, and the like.

The reference length can be, for example, the length of the vehicle V (for example, 5 m), but if the parking lot line is short, the reference length is set to be shorter than the vehicle length. The angle is an angle that takes into account the running direction of the vehicle V, the direction of the camera 20 that took the image, and the like. In the case of FIG. 6, the parking lot line is a straight line extending substantially perpendicularly toward the parking space with respect to the driving direction, so the angle is set to 90°±threshold value.

The parking lot line detection unit 112 calculates the positions (coordinate values) of the start and end points of each of the extracted plus edge line segments and minus edge line segments, and based on these positions, the parking lot line detection unit 112 A line segment with a plus edge and a line segment with a minus edge that match are extracted and determined to be a pair of edges constituting a parking lot line. For example, when the distance between the plus edge and the minus edge is within the range of the line width of the parking lot line±threshold value, these are determined to be a pair.

On the overhead image G of FIG. 6A, a plus edge line segment Ep (thick solid line) that is longer than the extracted reference length and extends at a predetermined angle, and a minus edge line segment Em (thick broken line) are schematically shown. show. In the example of FIG. 6A, pairs of plus-edge line segments Ep1-Ep6 and minus-edge line segments Em1-Em6 are placed on both side edges of parking lot lines (actually, parking lot line images) K1 to K6, respectively. was extracted.

Here, regarding the parking lot lines K4 and K5, the edge line segment is divided by the shadow S, and in addition to the pair of plus edge line segments Ep4 and Ep5 and minus edge line segments Em4 and Em5, which are longer than the reference length, Plus-edge line segments Ep7 and Ep8 and plus-edge line segments Em7 and Em8 that are shorter than the reference length are detected. Furthermore, another edge may be detected between the divided edges, such as between line segments Ep4 and Ep7. These edge line segments Ep7, Ep8, Em7, Em8, etc. are discarded by filtering if their length is less than the reference length. However, if the length is longer than the reference length, the pair of edge segments Ep4 and Em4 and the pair of edge segments Ep5 and Em5 may be extracted as a different pair of edges. Regarding the parking lot line K6, a pair of a plus-edge line segment Ep6 and a minus-edge line segment Em6, which are shorter than the actual length of the parking lot line K6, is extracted by the shadow S.

This phenomenon occurs because the brightness of the road surface is different between the sunny part that is not in the shade and the shaded part that is in the shade, and the brightness of the parking lot line is also different from each other. In other words, the difference in brightness between the road surface and the parking lot line in the sunny and shaded areas and the edge angle (direction) are detected differently, and there is no continuity between the edge of the sunny area and the shaded area. Edges may be detected as separate edges. Furthermore, edges near the boundary between sunlight and shade may not be detected. In addition, when a part of the parking lot line is strongly illuminated by road lights or headlights, the luminance difference value of the road surface or parking lot line in the image will be different between this strongly illuminated part and the non-illuminated part. A similar phenomenon may occur, as it may cause overexposure or overexposure.

As a result, the length of the edge of a parking lot line may be detected as being different from the actual length, or multiple parking spaces may be assigned to one parking space based on a parking lot line that is detected as being divided into multiple parts. settings, which may affect the accuracy of parking slot detection. Therefore, in this embodiment, the correction unit 114 corrects the end point positions of the parking lot lines that constitute the parking slot. In this embodiment, after the parking lot line detection unit 112 detects a parking lot line and the parking frame setting unit 113 sets a parking frame based on the detected parking lot line, the correction unit 114 performs correction. .

The parking frame setting unit 113 sets a parking frame based on the pair of edges forming the parking line detected by the parking line detecting unit 112. More specifically, the parking frame setting unit 113 selects two adjacent edge line segments that may constitute a parking space from among a plurality of pairs of plus edge line segments and minus edge line segments. select. The two edge line segments selected here are lines that constitute the left and right sides of a pair of parking lot lines that partition the parking space. For example, there are a minus-edge line segment Em1 of the parking lot line K1 shown in FIG. 6A, and a plus-edge line segment Ep2 of the adjacent parking lot line K2.

The parking frame setting unit 113 calculates the distance between the line segments of the two selected edges (inner dimensions of adjacent parking lot lines) based on the coordinate values of the end points of each edge. It is determined whether the calculated distance is within a predetermined range (for example, within a range of parking space width ± threshold value). If this distance is within a predetermined range, the area partitioned by the line segments of the two edges is detected as a parking space. The width of the parking space is preferably 2m to 3m if it is for a regular car or small cargo vehicle. If it is a parking space for a large freight vehicle or bus, a height of 3.3 m or more is desirable.

A rectangular frame with the long side (side) being the line along the adjacent positive edge line segment and negative edge line segment that make up the detected parking space, and the short side being the line connecting the opposing ends. It is a parking frame, and it can be estimated that the inside thereof is a parking space. The parking frame setting unit 113 sets the two end points of the line segment with the plus edge and the two end points of the line segment with the minus edge as the four end points of the parking frame, and calculates their positions (coordinate values) in order to pass them to the correction unit 114. is registered in the work area of the storage unit 120.

Moreover, the parking frame setting unit 113 can also perform a determination process or the like to determine whether or not the vehicle V can be parked in the set parking frame. For example, if there are other vehicles or obstacles in the detected parking space, it is determined that parking is not possible, and this parking space is not registered in the storage unit 120. Moreover, the parking frame setting unit 113 can also determine a parking frame that is close to the vehicle V or a parking frame that is easy to park as a parking frame in which parking is possible, and register it in the storage unit 120.

In the example of FIG. 6A, three parking spaces A1, A2, and A3 are set between each parking lot line K1 to K6. However, since the edges of the parking lot lines K4 to K6 are detected to be short and the end point positions are shifted, the parking frames A2 and A3 are detected not as rectangles but as trapezoids or parallelograms.

In order to correct this, the correction unit 114 corrects all the parking spaces set by the parking space setting unit 113 from a position a predetermined distance away, based on the direction in which the parking lot lines constituting the parking space extend. It is determined whether there is a continuation parking lot line on the extension line of the line. That is, it is determined whether or not there is another parking lot line on the extension line of the detected parking lot line (beyond the parking lot line). When a continuation parking lot line is detected, the correction unit 14 connects the first detected parking lot line and the next detected continuation parking lot line to form one parking lot line. Thereafter, the parking frame setting unit 113 sets (corrects) a parking frame based on the one parking lot line connected by the correction unit 114.

More specifically, the correction unit 114 determines whether there is a shadow or the like in order to determine whether the edge of the parking lot line that constitutes the parking frame is divided by a shadow or the like. To do this, based on the end of the parking lot line detected by the parking lot line detection unit 112, the correction unit 114 moves the edge of the parking lot line toward a predetermined direction from the end point of the edge of the parking lot line (in the case of a positive edge, to the scanning source side; In the case of an edge, a pixel at a position offset by a predetermined amount (referred to as "shadow search offset amount α") to the scanning destination side) is sandwiched between the pixels and a predetermined length ("shadow search length β") along the direction in which the parking lot line extends. ), the bird's-eye view image G is scanned, and the brightness difference between adjacent pixels is calculated. As a result of scanning, it is determined whether the absolute value of the brightness difference between adjacent pixels is greater than or equal to a predetermined value, and if it is greater than or equal to the predetermined value, it is determined that the parking lot line is divided by shadows, blur, etc. If it is determined that the absolute value of the brightness difference is less than the predetermined value, it is determined that the parking lot line is not divided. At this time, it is only necessary to recognize that there is a brightness difference (contrast) between adjacent pixels, so when the correction unit 114 scans with the edge of the extracted parking lot line as a reference, the edge detection unit 111 detects the edge. It is sufficient to compare the brightness differences using a threshold value smaller than the threshold value of the brightness difference at the time of detection, and there is no need to compare the angles (directions).

When determining that the parking lot line is divided, the correction unit 114 moves the parking lot line to a position that is a predetermined length away from the parking lot line (referred to as "re-search start position offset amount γ") in the direction in which the parking lot line extends. Determine whether there is a parking lot line to be connected to. Specifically, a predetermined length (referred to as "research success determination length δ" ) to scan the overhead image G and identify edges where the brightness difference between adjacent pixels is larger than a predetermined value in the positive direction and edges where the brightness difference between adjacent pixels is larger than the predetermined value in the negative direction. Determine whether a pair exists. In this case as well, the brightness difference may be compared using a threshold value smaller than the brightness difference threshold value used when the edge detection unit 111 detects an edge, and there is no need to compare the angles (directions).

If a pair of plus and minus edges is detected, the correction unit 114 determines that a divided parking lot line (continuing parking lot line) that should be connected to the parking lot line in question has been detected, and continues scanning. Then, the ends (end points) of the divided parking lot lines are detected. Then, one end point of the edge of the parking lot line and the end point on the scan destination side of the edge of the divided parking lot line are defined as one and the other end points of the parking space, and the positions (coordinate values) of these end points are determined. , to correct (update) parking slot information by registering it in the work area. The parking space setting unit 113 registers the updated parking space information on the work area in the storage unit 120 as parking space registration data 121.

On the other hand, when the above-mentioned edge pair is not detected even if the bird's-eye view image G is scanned by the re-search success determination length δ, or when the length of the edge pair is less than δ, the correction The unit 114 stops scanning. In this case, regarding the parking lot line, it is determined that there is no divided parking lot line, and the correction unit 114 does not correct (update) the work area. As a result, the parking space setting unit 113 stores the positions (coordinate values) of one and the other end points of the edge line segment of the parking lot line (first detected parking lot line) as the parking space registration data 121. 120.

Search for shadows, etc. and search for edges of divided parking lot lines are performed on one end point side and the other end point side of adjacent plus edge line segments and negative edge line segments that make up the parking space. It is desirable to do so. Thereby, the positions (coordinate values) of the four end points of the parking frame can be corrected to appropriate positions closer to the true values.

The values of α to δ above are not particularly limited, and may vary depending on the function used for image processing, edge detection method, image resolution, shooting range, environment of parking lot P (buildings, fences, lighting, etc.). An appropriate value can be set depending on the presence/absence, size, etc. Specifically, for example, if the shadow search offset amount α is set to 5 pixels and the shadow search length β is set to 15 pixels, the presence or absence of a shadow etc. can be appropriately determined.

Furthermore, if the re-search start position offset amount γ is 10 pixels and the re-search success determination length δ is 5 pixels, a divided parking lot line with a final length of 15 pixels or more will be detected and the parking lot line will Connected. On the other hand, a divided parking lot line with a final length of less than 15 pixels is discarded without being connected to the parking lot line. In this embodiment, 100 pixels of the image corresponds to 3 meters. Therefore, when the parking lot line is divided by a length of 45 cm or more, the end points of the parking lot line and the parking frame are corrected, and as a result, the length is appropriately corrected. On the other hand, when the divided length is less than 45 cm, the parking lot line and the end point of the parking frame are not corrected, and the length is also not corrected.

However, if the error in the end point of the parking space is within 15 pixels (45 cm), the error is within the allowable range that can be corrected by image processing, etc., and a mark indicating the uncorrected parking space is displayed on the display image of the monitor 31. Even when displayed, the user does not feel uncomfortable, and the influence on parking assistance and the like can be suppressed. When the standard length of the parking lot line is 5 m, γ is 10 pixels = 30 cm, and the re-search success determination length δ is 5 pixels = 15 cm, which are 6% and 3% of the length of the parking lot line, respectively. . From this, the re-search start position offset amount γ is set to 6% or less of the length of the parking lot line, and the re-search success determination length δ is set to 4% or less of the length of the parking lot line. Parking lane lines can be detected with high precision and high speed.

FIG. 6B schematically shows edges whose end points have been appropriately corrected by the correction process by the correction unit 114 and parking frames A2 and A3. As shown in FIG. 6(b), the starting points of the plus edge line segments Ep4 and Ep5 and the minus edge line segments Em4 and Em5 (on the side closer to icon I), the plus edge line segment Ep6 and the minus edge line segment The position of the end point (farthest from icon I) of line segment Em6 has been appropriately corrected to a position closer to the true value. As a result, the positions of the end points of the parking frames A2 and A3 are also corrected to appropriate positions, and the detection accuracy of the parking frames can be improved.

Hereinafter, the procedure for correcting parking spaces by the correction unit 114 will be explained with reference to FIGS. 7 and 8. FIG. 7(a) is a diagram for explaining the correction processing procedure when there is a divided parking lot line (continuation of the parking lot line), and FIG. FIG. 12 is a diagram for explaining a correction processing procedure when a continuation parking lot line does not exist. FIGS. 8 and 9 are diagrams for explaining the correction processing procedure when there is a parking lot line (continuation of the parking lot line) divided in a state different from that in FIG. 7(a). In these figures, following FIG. 6A, the direction perpendicular to the parking lot line K is assumed to be the X-axis, and the direction in which the parking lot line extends is assumed to be the Y-axis. In addition, among the ends of the divided parking lot lines to be detected, one end of the illustrated parking lot line K is referred to as the "tip", and the opposite end (not shown) is referred to as the "rear end". explain.

FIG. 7A shows an example in which the leading end side of the parking lot line K that constitutes the parking frame is divided by a shadow S and is not detected. In this case, in order to determine whether or not it is divided by the shadow S, from the end point d of the edge line segment (here, a line segment Em of a negative edge is explained as an example), ), the image is scanned by the shadow search length β in the direction in which the negative edge line segment Em extends (Y-axis negative direction), sandwiching the pixel at the position offset by the shadow search offset amount α, and the adjacent pixels are Calculate the brightness difference between Through this scanning, it is determined whether the absolute value of the luminance difference between adjacent pixels is greater than or equal to a predetermined value. In the example of FIG. 7A, since the scanning area switches from the road surface R to the shadow S, the absolute value of the brightness difference between adjacent pixels at the boundary of this switching becomes greater than or equal to a predetermined value. Therefore, it is determined that the edge of the parking lot line is divided by the shadow S.

Next, scanning is performed in the positive direction of the X-axis from a position offset by the re-search start position offset amount γ on the extension line from the end point d of the line segment Em of the negative edge. At this time, scanning is performed one pixel (one line) at a time in the Y-axis direction with the width (length) of the parking lot line plus the threshold value in the X-axis direction. After scanning from the offset position for the re-search success determination length δ, when a pair of a plus edge and a minus edge is detected, it is determined that there is a divided parking lot line K'. In the example of FIG. 7A, the plus edge (indicated by a black circle) and the minus edge (indicated by a white circle) in the shaded portion are detected with lengths greater than or equal to δ. Further, this scanning is continued until no plus edge or minus edge is detected, that is, until the tip of the divided parking lot line K' is reached. Then, the parking frame setting unit 113 sets the end point on the leading end side of the minus edge of the divided parking lot line K' as the starting point (or end point) of one side of the parking frame, and sets the end point on the front end side of the minus edge of this divided parking lot line K' as the starting point (or end point) of one side of the parking lot line, and The end point of is defined as the end point (or starting point) on one side of the parking frame, and each position (coordinate value) is registered in the storage unit 120 as parking frame registration data 121. By performing the above processing also on the plus edge that is adjacent to this minus edge and constitutes the other side of the parking frame, the positions of the four end points of the parking frame can be appropriately corrected. The same applies to subsequent examples.

On the other hand, as shown in FIG. 7(b), if the parking lot line K is not divided by a shadow or the like, if the shadow search offset amount α is used to offset the parking lot line K and the shadow search length β is scanned, It is determined that the absolute value of the brightness difference between adjacent pixels is less than a predetermined value. Therefore, it is determined that there is no divided parking lot line to be connected to the parking lot line K. The parking frame setting unit 113 stores the positions (coordinate values) of the start and end points of the negative edge of the parking lot line K in the storage unit 120 as the positions of the start and end points on one side of the parking frame (parking frame registration data 121). register.

Moreover, FIGS. 8 and 9 show an example in which a part of the parking lot line becomes a shadow and is divided into two parts. In this case, the leading edge may be discarded as noise or detected as separate parking lot lines. Also, in the examples shown in FIGS. 8 and 9, by scanning for the shadow search length β at a position offset by the shadow search offset amount α from the negative edge line segment Em, the boundary between the shadow S and the road surface R is It is determined that the absolute value of the brightness difference between adjacent pixels is equal to or greater than a predetermined value.

In the example of FIG. 8, when scanning is performed in the positive direction of the X-axis from a position offset by the re-search start position offset amount γ from the end point d, a positive edge and a negative edge that are longer than the re-search success determination length δ are detected, A divided parking lot line K' is detected. In this case, the parking frame setting unit 113 determines the position (coordinate value) of the end point of the minus edge of the newly detected divided parking lot line K' and the position (coordinate value) of the other end point of the minus edge of the parking lot line K'. The coordinate values) are registered in the storage unit 120 as the positions (parking frame registration data 121) of the starting point and end point on one side of the parking frame.

On the other hand, in the example of FIG. 9, when scanning is performed in the positive direction of the X-axis from a position offset by the re-search start position offset amount γ from the end point d, positive edges and negative edges are detected, but the detection length is less than the re-search success determination length δ. In this case, there is no need to correct the parking frame, and it is determined that there is no divided parking lot line. The parking frame setting unit 113 registers the positions (coordinate values) of both end points of the negative edge of the parking lot line K in the storage unit 120 as the positions of the starting and ending points on one side of the parking frame (parking frame registration data 121). .

As described above, the parking frame setting unit 113 determines the coordinate values of the end points of the adjacent plus-edge line segments and minus-edge line segments that constitute the parking frame 201 (if corrected by the correction unit 114, the corrected coordinates). value) is registered in the storage unit 120 as the parking space registration data 121 so that the vehicle control ECU and the like can refer to it. At this time, by registering the coordinate values of at least the two end points (starting points) of the parking frame 201 on the side closer to the vehicle V, the parking frame 201 can be specified while minimizing the storage capacity. On the other hand, the coordinate values of the four end points (starting point and ending point) may be registered, and appropriate parking assistance can be provided for both forward parking and backward parking. Furthermore, the angle (extending direction) of the parking lot line 200 and other information necessary for automatic parking processing can be added to the parking space registration data 121, making it possible to provide more appropriate parking support.

The display control unit 115 appropriately overlaps the road surface image around the vehicle V captured by the camera 20 and the image showing the parking space set by the parking space setting unit 113 and corrected by the correction unit 114 on this road surface image. A display control signal for displaying on the monitor 31 of the navigation device (display device) 30 is sent to the navigation device 30 either alone or separately.

The storage unit 120 includes a storage medium such as a large capacity storage medium such as a hard disk drive, or a semiconductor storage medium such as ROM or RAM. The storage unit 120 temporarily or non-temporarily stores various data used during various operations in the control unit 110.

Furthermore, as described above, the storage unit 120 stores parking space registration data 121 and parameter data 122. The parameter data 122 includes threshold values for edge detection, various setting values (α to δ) used for determining whether parking lot lines are divided or detecting divided parking lot lines, and boundary lines such as parking lot lines and lane boundary lines. The standard length, parking space width, its threshold value, etc. can be stored. Furthermore, various parameters used by the image processing apparatus 100, such as the width of the boundary line and the angle of the extending direction, can be stored. In addition, a plurality of parameters are stored corresponding to the country and region where the parking assistance device 1 is used, the shape and size of the parking space (parking frame), the distance between driving lanes, the shape of the lane boundary line, etc. It may also be configured to select appropriate parameters.

(Operation of image processing device)
Next, an example of the operation of the image processing apparatus 100 according to this embodiment will be described with reference to the flowchart of FIG. 4 and FIGS. 6A and 6B.

FIG. 4 is a flowchart for explaining the operation of the image processing apparatus 100. The operation shown in the flowchart of FIG. 4 starts when the driver inputs an instruction to start automatic parking by operating an automatic parking start switch (not shown).

In step S1, the control unit 110 of the image processing device 100 acquires an image signal of the road surface R around the vehicle V captured by the camera 20.

In step S2, based on the image signal acquired in step S1, the control unit 110 generates a signal by combining these image signals. The signals synthesized and generated in step S2 cause the navigation device 30 to display an image as if a camera was installed above the vehicle V and looked down directly below (see "overhead image G" in FIG. 6A etc.). This is a signal for Techniques for generating signals for displaying such bird's-eye images are well known, and examples include techniques disclosed in Japanese Patent Application Laid-Open Nos. 3-99952 and 2003-118522.

Note that the signal synthesis work for displaying the bird's-eye view image G may not be performed in step S2, or the image synthesis work in step S2 may be performed after the extraction of plus edges and minus edges in the next step S3. However, the processing load on the image processing apparatus 100 can be reduced by performing the processing in step S3 after performing the signal synthesis work.

In step S3 (edge detection step), as described above, the edge detection unit 111 scans the overhead image G synthesized in step S2 in a predetermined direction, and detects the positive Extract edges and negative edges.

In the next step S4, the parking lane line detection unit 112 filters the plus edges and minus edges detected in step S3 based on the reference length as described above. This eliminates short edges that cause noise due to light reflection on the road surface, dust, dirt, etc. Although this filtering can be performed after pair extraction in the next step S5, image processing can be speeded up by performing it before pair extraction to remove noise. In the example shown in FIG. 6A, plus edge line segments Ep1 to Ep8 and minus edge line segments Em1 to Em8 are extracted.

In the next step S5 (parking lot line detection step), the parking lot line detection unit 112 extracts a pair of adjacent plus edge line segments and minus edge line segments from the plurality of edge line segments detected in step S4. do. At this time, the distance between adjacent plus edges and minus edges on the road surface is calculated based on the overhead image G, and if this distance is within the range of the predetermined line width ± threshold value, the pair of edges forming the parking lot line are judge.

In the next step S6 (parking frame setting step), the parking frame setting unit 113 sets a parking frame and a parking space according to the procedure described above based on the pair of edges forming the parking lot line determined in step S4. do. The information (coordinate values) of the set parking slot is temporarily stored in the storage unit 120. In the example shown in FIG. 6A, as shown by the imaginary line, the parking slot A1 is appropriately set, and the parking slots A2 and A3 are set in a deformed manner.

Through the correction loop process (correction process) of the next steps S7 to S14, the correction unit 114 determines whether or not the parking lot line forming the parking frame is divided by a shadow or the like, and if it is divided, , correct the position of the end point of the parking frame to an appropriate position. The processing in steps S7 to S14 ends when it is determined that all the parking slots set in step S6 have been processed.

First, in step S8, the correction unit 114 determines the position (coordinate value ) to obtain. In step S9, a predetermined length in the direction in which the parking lot line (edge) extends, sandwiching a pixel at a position offset by a predetermined amount (α) in a predetermined direction from one end point of the line segment of each edge of the parking lot line. (β) to scan the image.

In the next step S10, as a result of scanning, it is determined whether the absolute value of the brightness difference between adjacent pixels is greater than or equal to a predetermined value, and if it is greater than or equal to the predetermined value (YES), the parking lot line is divided by a shadow, etc. Since it is estimated that there is, the process advances to step S11.

On the other hand, if it is determined that the absolute value of the brightness difference is less than the predetermined value (NO), it is assumed that there is no division of the parking lot line due to shadows, etc., and it is necessary to correct the position of the end point of the parking space. Therefore, steps S11 to S14 are skipped and the process proceeds to step S15.

In step S11, in order to search for a divided parking lot line, from a position a predetermined length (γ) away from the parking lot line in the direction in which the parking lot line extends, connect the parking lot line by a predetermined length (δ). The overhead image G is scanned in the orthogonal direction (X-axis direction).

In step S12, it is determined whether a plus edge line segment and a minus edge line segment having a predetermined length (δ) or more are detected as a result of scanning. If it is determined that it has been detected (YES), that is, if it is determined that a divided parking lot line exists, the process advances to step S12.

On the other hand, if it is determined in step S12 that it is not detected (NO), that is, if it is determined that there is no divided parking lot line, there is no need to correct the position of the end point of the parking space, so step Skip S13 to S14 and proceed to step S15.

In step S13, scanning is continued to detect the end of the divided parking lot line. Next, the process proceeds to step S14, and the position (coordinate value) of one end point of one side of the parking frame, which was temporarily registered in the work area of the storage unit 120 in step S6, is set to the edge of the divided parking lot line. Correct to the position (coordinate value) of the end point.

On the other hand, if it is determined in step S11 that a line segment with a plus edge and a line segment with a minus edge of a predetermined length (δ) or more are not detected (NO), there is no need to correct the position of the end point of the parking frame. , skips steps S12 and S13 and proceeds to step S14.

Note that the processes in steps S8 to S14 are performed on at least the starting points of the adjacent plus-edge line segments and the starting points of the minus-edge line segments that constitute the parking frame, so that the positions of the two starting points of the parking frame can be determined. Can be set appropriately. More preferably, the positions of the four end points (two start points and two end points) of the parking frame can be appropriately set by performing the processes of steps S8 to S13 on each start point and end point of the plus edge and the minus edge. can.

In step S15, it is determined whether there is a parking slot to be processed next. If there is a parking space, the process returns to step S7 and processes for the next parking space are performed. If there is no parking frame to be processed, the loop is ended and the process proceeds to step S16.

In step S16, the parking frame setting unit 113 registers the positions (coordinate values) of the end points (two starting points or four end points) of each parking frame temporarily registered in the work area of the storage unit 120. It is registered in the storage unit 120 as data 121. The position of the end point of each parking frame has been appropriately corrected by the correction processing in steps S8 to S14. As a result, for example, in the example shown in FIG. 6A, the positions of the end points of parking frames A2 and A3, which are set in a shape different from the actual parking frame, are appropriately corrected, and as shown in FIG. Set to a close rectangle.

Then, the parking space registration data 121 registered in the storage unit 120 is sent to the vehicle control ECU 40, and various processes to support parking of the vehicle V are executed.

(Effect of image processing device)
In the image processing device 100 according to the present embodiment configured as described above, the parking lane line detection unit 112 detects a bird's-eye view image G based on an image signal output from an image capturing device that captures an image of the surrounding situation of the vehicle V. , detect parking lot lines. The correction unit 114 determines whether there is a continuation parking lot line on the extension line of the parking lot line from a position separated by a predetermined distance (re-search start position offset amount γ) based on the direction in which the car lot line extends. Then, when a continuation parking lot line is detected, the first detected parking lot line (K) and the next detected continuation parking lot line (K') are connected to form one parking lot. Line. The parking frame setting unit 113 sets a parking frame using one parking lot line connected by the correction unit 114.

As a result, even if the parking lot line is not detected or detected incorrectly due to the effects of shadows, light reflections, the presence of obstacles, blurred lines, lens distortion, etc., and the end points of the parking space are not set appropriately, , the end point of the parking frame can be corrected to an appropriate position closer to the true value. Therefore, it is possible to provide an image processing device 100 and an image processing method that can detect parking spaces with high precision. Further, by including the image processing device 100 or the image processing method, it is possible to provide a parking support device and a parking support method that can perform parking support with high precision.

Furthermore, in this embodiment, the bird's-eye view image G is scanned in a predetermined direction to detect pixels whose brightness included in the image signal changes more than a threshold value, and the detected pixel array has a length longer than a predetermined value. The edge detecting section 111 detects the portion where the edges are formed as an edge. Then, the parking lot line detection unit 112 detects a parking lot line based on the plurality of edges detected by the edge detection unit 111. Therefore, calculation speed is improved, and parking lot lines can be detected with higher accuracy and faster.

In addition, the correction unit 114 corrects adjacent pixels along the extending direction of the parking lot line detected by the parking lot line detection unit 112 based on the end of the parking lot line detected by the parking lot line detection unit 112. Calculate the brightness difference between If the absolute value of the calculated brightness difference is larger than a predetermined value, it is determined that the parking lot line is divided, and the parking lot line is separated by a predetermined length (research start position offset amount γ) in the direction in which the parking lot line extends. From the position where the parking lot line extends, scanning is performed in a direction orthogonal to the direction in which the parking lot line extends, and it is determined whether there is a continuation parking lot line on the extension line of the parking lot line. With this configuration, it is possible to efficiently determine whether there is a continuation parking lot line, and the position of the end point of the parking space can be corrected to an appropriate position closer to the true value, further improving the accuracy of parking space detection. .

In addition, the threshold of the brightness difference when the correction unit 114 scans based on the end of the parking lot line detected by the parking lot line detection unit 112 is the brightness difference when the edge detection unit 111 scans the bird's-eye view image G. It is desirable that the value is smaller than the threshold value of . As a result, the presence of shadows and the like can be appropriately detected, and the edges of shadowed parking lot lines can also be detected with high accuracy. Thereby, parking frames can be detected with high precision.

In addition, the predetermined length (research start position offset amount γ) in the extending direction of the parking lot line when detecting the continuation of the parking lot line is 6% or less of the parking lot line, and the length of the continuation of the parking lot line detected is The predetermined length of the parking lot line (research success determination length δ) is preferably 3% or less of the parking lot line. Thereby, if the error in the position of the end point of the parking frame is within the allowable range, the error can be corrected by image processing, etc., and if it exceeds the allowable range, the end point can be appropriately corrected by the correction unit 114.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configuration is not limited to these embodiments and examples, and design changes can be made to the extent that they do not depart from the gist of the present invention. are included in the present invention.

For example, in the image processing apparatus 100 according to the above-described embodiment, information on brightness and color parameters (for example, RGB, RGBA, etc.) of an image is based on the magnitude of change and the direction of change (positive direction or negative direction). However, the present invention is not limited to this, and edges may be detected based on the magnitude and direction of change in other information included in the image signal.

Further, in the image processing device 100 according to the above-described embodiment, after the parking frame setting unit 113 sets a parking frame based on the parking lot line detected by the parking lot line detection unit 112, the set parking frame is The correction unit 114 corrects the position of the end point and sets the divided parking lot line as one parking lot line, and the parking frame setting unit 113 finally registers the parking space using this one parking lot line. There is. However, the configuration is not limited to this, and after the parking lot line detection section 112 detects the parking lot line, the correction section 114 appropriately corrects the position of the end point with respect to the detected parking lot line. It is also possible to adopt a configuration in which the parking frame setting unit 113 sets a parking frame based on the corrected parking lot line. Also with this configuration, the position of the end point of the parking frame can be appropriately set.

20 Camera (imaging device) 20a Front camera (imaging device)
20b Rear camera (imaging device) 20c Left side camera (imaging device)
20d Right side camera (imaging device) 100 Image processing device 111 Edge detection unit 112 Parking line detection unit 113 Parking frame setting unit 114 Correction unit 200 Parking line 201 Parking frame G Bird's-eye view image (image) R Road surface V Vehicle d End point

Claims (3)

a parking lot line detection unit that detects parking lot lines from an image based on an image signal output from an imaging device that images the surrounding situation of the vehicle;
Based on the direction in which the parking lot line extends, it is determined from a position a predetermined distance away whether there is a continuation parking lot line on an extension line of the parking lot line, and the continuation parking lot line is detected. a correction unit that connects the first detected parking lot line and the next detected continuous parking lot line to form one parking lot line;
a parking frame setting unit that sets a parking frame using the one parking lot line connected by the correction unit ,
The correction unit is configured to adjust adjacent pixels along the direction in which the parking lot line detected by the parking lot line detection unit extends based on the end of the parking lot line detected by the parking line detection unit. If the absolute value of the calculated brightness difference is larger than a predetermined value, it is determined that the parking lot line is divided, and the parking lot line is separated by a predetermined length in the direction in which the parking lot line extends. An image processing device for determining whether or not there is a continuation parking lot line on an extension line of the parking lot line by scanning from a position where the parking lot line extends in a direction orthogonal to the direction in which the parking lot line extends . The brightness difference threshold when the correction unit scans with reference to the end of the parking lot line detected by the parking lot line detection unit is the threshold value of the brightness difference when scanning the image to detect the parking lot line. The image processing device according to claim 1, wherein the brightness difference is smaller than a threshold value. a parking lot line detection step of detecting a parking lot line from an image based on an image signal output from an imaging device that images the surrounding situation of the vehicle;
Based on the extending direction of the parking lot line, it is determined from a position a predetermined distance away whether there is a continuation parking lot line on an extension line of the parking lot line, and the continuation parking lot line is detected. sometimes, a correction step of connecting the first detected parking lot line and the next detected continuous parking lot line to form one parking lot line;
a parking frame setting step of setting a parking frame using the one parking lot line connected in the correction step ,
The correction step includes determining adjacent pixels along the direction in which the parking lot line detected in the parking lot line detection step extends based on the end of the parking lot line detected in the parking lot line detection step. If the absolute value of the calculated brightness difference is larger than a predetermined value, it is determined that the parking lot line is divided, and the parking lot line is separated by a predetermined length in the direction in which the parking lot line extends. An image processing method comprising: scanning in a direction orthogonal to the direction in which the parking lot line extends from a position where the parking lot line extends to determine whether or not there is a continuation parking lot line on an extension line of the parking lot line .