JP7245640B2 - Image processing device and image processing method - Google Patents

Description

The disclosed embodiments relate to an image processing apparatus and an image processing method.

2. Description of the Related Art In recent years, with the development of automatic driving technology, an image processing device that detects a parking frame for parking a vehicle from an image captured around the vehicle is becoming popular. In this type of image processing device, a marking line that defines a parking frame is detected from an image, and the parking frame is detected based on the detected marking line (see Patent Document 1, for example).

When the vehicle is parked by automatic driving, such an image processing device needs to determine whether the parked vehicle exists within the parking frame.

JP 2017-87758 A

However, in the prior art, in order to determine whether or not there is a parked vehicle in the parking frame, for example, it is necessary to use a monocular camera together with other sensors such as a sonar or a stereo camera. Therefore, it is difficult to determine whether or not there is a parked vehicle in the parking frame.

One aspect of the embodiments has been made in view of the above, and provides an image processing device and an image processing method that can determine the presence or absence of a parked vehicle in a parking frame from an image captured by a monocular camera. for the purpose.

An image processing apparatus according to an aspect of an embodiment includes a detection unit and a determination unit. The detection unit detects a partition line that partitions the parking frame from the image. The judging unit determines, based on a change in the length of the pair of demarcation lines sequentially detected by the detection unit from a plurality of the images captured continuously while moving, the image demarcated by the pair of demarcation lines. It is determined whether or not there is a parked vehicle within the parking frame.

An image processing device and an image processing method according to an aspect of an embodiment can determine the presence or absence of a parked vehicle in a parking frame from an image captured by a monocular camera.

FIG. 1A is a diagram showing an example of mounting an image processing device. FIG. 1B is a diagram showing an overview of the image processing method. FIG. 2 is a block diagram of the image processing device. FIG. 3 is a block diagram of a parking frame detection unit. FIG. 4A is an explanatory diagram showing a procedure for determining the presence or absence of a parked vehicle. FIG. 4B is an explanatory diagram showing a procedure for determining the presence or absence of a parked vehicle. FIG. 4C is an explanatory diagram showing a procedure for determining the presence or absence of a parked vehicle. FIG. 5 is a flowchart illustrating an example of processing executed by a marking line detection unit.

An image processing apparatus and an image processing method according to embodiments will be described in detail below with reference to the accompanying drawings. In addition, this invention is not limited by this embodiment.

First, an outline of an image processing apparatus according to an embodiment will be described with reference to FIGS. 1A and 1B. FIG. 1A is a diagram showing an example of mounting an image processing device. FIG. 1B is a diagram showing an overview of the image processing method. This image processing method is executed by the image processing apparatus 1 shown in FIG. 1A.

As shown in FIG. 1A, the image processing apparatus 1 according to the embodiment is mounted on a vehicle (hereinafter referred to as vehicle C) equipped with an on-vehicle camera 10, and an image captured by the on-vehicle camera 10 (hereinafter , simply referred to as an image) to detect the parking frame PS.

The in-vehicle camera 10 is a monocular imaging device that has an imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) and captures an image of the surroundings of the vehicle C, for example. A wide-angle lens such as a fisheye lens is adopted as the lens of the vehicle-mounted camera 10, for example. Thereby, the in-vehicle camera 10 can image the parking frame PS existing in the wide-angle imaging area R as shown in FIG. 1A.

In the example shown in FIG. 1A, the vehicle-mounted camera 10 is a left-side camera that captures the left side of the vehicle C, but the vehicle-mounted camera 10 includes a front camera that captures the front of the vehicle C, A back camera that captures an image behind the vehicle C and a right side camera that captures an image of the right side of the vehicle C are included.

When detecting the parking frame PS, the image processing device 1 first detects, from the image, a lane line candidate that is a candidate for the lane line that divides each parking stall PS, and based on the detected lane line candidate, each parking stall. Detect PS. At this time, for example, when the marking line candidates are detected discretely or discontinuously, the parking frame PS may not be detected.

Therefore, the image processing apparatus 1 according to the embodiment integrates the detected marking line candidates based on a predetermined integration condition. As a result, the image processing device 1 according to the embodiment can improve the detection accuracy of the parking frame.

Specifically, as shown in FIG. 1B, the image processing apparatus 1 first detects marking line candidates Lc from the image I (step S1). For example, the image processing device 1 detects the marking line candidates Lc based on edge lines connecting edge points obtained by performing edge processing on the image I. FIG.

Then, the image processing device 1 detects an edge line corresponding to the boundary between the marking line and the road surface in the image I as a marking line candidate Lc. That is, the marking line candidate Lc is a pair of edge lines corresponding to the left and right ends of the marking line in the width direction.

Subsequently, the image processing apparatus 1 integrates the lane marking candidates Lc detected in step S1 based on a predetermined integration condition, and detects lane markings Li1 and Li2 (step S2). As a result, the image processing device 1 can detect the area sandwiched between the pair of marking lines Li1 and Li2 as the parking frame PS.

When the vehicle C is parked in the parking frame PS thus detected by automatic operation, the image processing device 1 determines whether or not there is a parked vehicle in the parking frame PS, and performs automatic operation control. It is necessary to notify the ECU (Electronic Control Unit).

At this time, a general image processing device cannot determine whether or not a parked vehicle exists within the parking frame PS unless, for example, a monocular camera and other sensors such as a sonar or a stereo camera are used together. was difficult. In other words, it is difficult for a general image processing device to determine whether or not there is a parked vehicle in the parking frame PS only with an image captured by a monocular camera.

Therefore, the image processing device 1 according to the embodiment can determine whether or not there is a parked vehicle in the parking frame PS from an image captured by a monocular camera (for example, the vehicle-mounted camera 10).

Specifically, when the image processing device 1 detects a pair of marking lines Li1 and Li2 from images that are continuously captured while moving in front of a parking frame PS where no parked vehicle exists, each A whole image of the division lines Li1 and Li2 can be detected. In such a case, the image processing apparatus 1 can determine that there is no change in the lengths of the marking lines Li1 and Li2 detected from each image by analyzing the images.

On the other hand, when the image processing device 1 detects a pair of lane markings Li1 and Li2 from images captured continuously while moving in front of the parking frame PS in which the parked vehicle Cx exists, The whole image cannot be detected because part of it is hidden by the parked vehicle Cx.

The lengths of the lane markings Li1 and Li2 that can be detected by the image processing device 1 are the lengths of the vehicle-mounted camera 10 that changes as the vehicle C moves, the parked vehicle Cx, and the pair of lane markings Li1 and Li2. Varies with relative position.

Therefore, the image processing device 1 determines the presence or absence of the parked vehicle Cx based on the change pattern of the length of the pair of marking lines Li1 and Li2 sequentially detected from a plurality of images continuously captured while moving (step S3). ).

Thereby, the image processing device 1 can determine whether or not the parked vehicle Cx exists within the parking frame PS from the image captured by the monocular camera (for example, the vehicle-mounted camera 10). A specific procedure for determining the presence or absence of a parked vehicle according to the embodiment will be described later with reference to FIGS. 4A to 4C.

Next, a configuration example of the image processing apparatus 1 according to the embodiment will be described with reference to FIG. FIG. 2 is a block diagram of the image processing device 1. As shown in FIG. 2 shows a parking assistance system 100 including the image processing device 1. As shown in FIG.

As shown in FIG. 2, the parking assistance system 100 includes an image processing device 1, an on-vehicle camera 10, a sensor group Sc, and a host ECU 50. As shown in FIG. Further, as shown in FIG. 2, the image processing apparatus 1, the sensor group Sc, and the host ECU 50 can communicate with each other via a communication bus B that conforms to a CAN (Control Area Network) communication standard.

The sensor group Sc is various sensors that detect the running state of the vehicle C, and notifies the image processing device 1 of detected sensor values. The sensor group Sc includes a vehicle speed sensor that detects the number of rotations of the wheels of the vehicle C, a steering angle sensor that detects the steering angle of the vehicle C, and the like.

The host ECU 50 is, for example, an ECU that assists automatic parking of the vehicle C, and parks the vehicle C in the parking frame PS based on the parking frame PS detected by the image processing device 1, for example. For example, the host ECU 50 is an EPS (Electric Power Steering)-ECU that controls the steering angle of the vehicle C, and can control the steering angle to the parking frame PS detected by the image processing device 1 . Note that the host ECU 50 may include an ECU that performs accelerator control and brake control.

As shown in FIG. 2 , the image processing device 1 includes a control section 2 and a storage section 3 . The control unit 2 includes a line segment extraction unit 21, an inappropriate area determination unit 22, a lane line candidate detection unit 23, an exclusion determination unit 24, a parking frame detection unit 25, a parking frame management unit 26, and a stop position determination unit. a portion 27;

The control unit 2 includes, for example, a computer having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), input/output ports, and various circuits.

The CPU of the computer, for example, by reading and executing programs stored in the ROM, the line segment extraction unit 21, the unsuitable area determination unit 22, the marking line candidate detection unit 23, the exclusion determination unit 24, the parking It functions as a frame detection unit 25 , a parking frame management unit 26 and a stop position determination unit 27 .

In addition, at least one of the line segment extraction unit 21, the inappropriate area determination unit 22, the marking line candidate detection unit 23, the exclusion determination unit 24, the parking frame detection unit 25, the parking frame management unit 26, and the stop position determination unit 27 of the control unit 2 Alternatively, part or all of it can be configured by hardware such as ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

Also, the storage unit 3 corresponds to, for example, a RAM or an HDD. The RAM and HDD can store various information and information of various programs. Note that the image processing apparatus 1 may acquire the above-described programs and various information via another computer or a portable recording medium connected via a wired or wireless network.

For example, when it is assumed that the vehicle C is traveling in a parking lot (for example, the vehicle speed is within 30 km/h), the control unit 2 may perform a parking frame detection process described later, or , the detection process may be performed during the entire period in which the vehicle C is running.

The line segment extraction unit 21 detects edge lines connecting edge points based on the luminance of each pixel from the image input from the vehicle-mounted camera 10 . Specifically, the line segment extraction unit 21 grayscales the image input from the vehicle-mounted camera 10 to convert it into a grayscale image. A grayscale image is a process of converting each pixel in the image so that it is expressed in each gradation from white to black (for example, 256 gradations) according to the luminance.

Subsequently, the line segment extraction unit 21 can obtain the edge strength and the luminance gradient of each pixel by applying, for example, a Sobel filter to the grayscale image. Subsequently, the line segment extraction unit 21 can extract the edge points by extracting pixels whose edge strength exceeds a predetermined value, and can extract edge lines by connecting adjacent edge points. The line segment extraction unit 21 notifies the unsuitable region determination unit 22 of edge information regarding the extracted edge points and edge lines.

Based on the edge points and edge lines extracted by the line segment extraction unit 21, the unsuitable area determination unit 22 determines whether or not there is an unsuitable area in which it is difficult to detect the lane markings forming the parking frame. For example, the unsuitable area determining unit 22 can determine an unpaved road surface area (e.g., gravel) from which more edge points are extracted than a paved road surface, or a grating area as an unsuitable area.

Specifically, the unsuitable area determination unit 22 can determine an area in which the density of each edge point is equal to or greater than a predetermined value and the luminance gradient of each edge point is uneven as an unsuitable area. Based on the determined inappropriate area, the inappropriate area determination unit 22 removes the edge information related to the inappropriate area from the above edge information, and transfers the edge information to subsequent processing.

The lane marking candidate detection unit 23 detects lane line candidates that are candidates for lane markings that divide the parking frame based on the edge lines extracted by the line segment extraction unit 21 . Specifically, the marking line candidate detection unit 23 detects, as marking line candidates, edge lines that are substantially parallel to each other and whose spacing falls within a predetermined range according to the width of the marking line.

That is, the marking line candidate detection unit 23 detects edge lines corresponding to the left and right ends of each marking line in the width direction as marking line candidates. The marking line candidate detection unit 23 generates marking line information regarding the detected marking line candidates, and notifies the exclusion determination unit 24 of the generated marking line information.

Note that the marking line candidate detection unit 23 can perform the marking line candidate detection process except for the unsuitable area detected by the unsuitable area determination unit 22 . In other words, the marking line candidate detection unit 23 does not perform the marking line candidate detection process on the inappropriate area. This makes it possible to suppress the processing load on the control unit 2 .

The exclusion determination unit 24 determines whether or not there is a no-parking area where the vehicle C is not permitted to park, based on the lane marking candidates detected by the lane marking candidate detection unit 23 . For example, the exclusion determination unit 24 determines whether or not there is a parking-impossible area such as a zebra zone (traffic zone) as the parking-impossible area.

Specifically, when the zebra zone assumes that substantially parallel lane markings are lane markings (referred to as supporting lanes), lane lane candidates that are inclined with respect to the supporting lanes are spaced apart by a predetermined interval. If there are three or more support division lines, the area sandwiched by the support division lines is determined to be a non-parking area.

The exclusion determining unit 24 can also determine whether or not there is a marking line candidate that is unnecessary for detection of a parking frame such as a road sign. For example, the exclusion determining unit 24 can detect each road sign included in the image by performing matching processing between the marking line candidate detected by the marking line candidate detection unit 23 and the template model of each road sign. .

The exclusion determining unit 24 removes unnecessary marking line candidates from the marking line information, adds information about the parking prohibited area to the marking line information, and notifies the parking frame detection unit 25 of the information.

The parking frame detection unit 25 detects a parking frame based on the marking line candidates detected by the marking line candidate detection unit 23 . Specifically, the parking frame detection unit 25 detects lane marking candidates arranged at predetermined intervals as parking frames.

Here, the predetermined interval is the width of a standard parking area for general public use, which is stipulated by laws and regulations concerning parking lots. Also, at this time, the parking frame detection unit 25 can detect the parking frame while avoiding the area determined by the exclusion determination unit 24 to be a non-parking area.

That is, the parking frame can be detected while avoiding the zebra zone or the like. When the parking frame is detected, the parking frame detection unit 25 notifies the parking frame management unit 26 of parking frame information related to the parking frame. In addition, below, the marking line candidate detected by the parking frame detection part 25 as a marking line candidate which divides a parking frame is described as a marking line. Moreover, the vertex coordinates of each lane marking with the vehicle C as a reference are included in the parking space information.

The parking frame detection unit 25 also performs processing for determining whether or not a parked vehicle exists within the detected parking frame. The parking frame detection unit 25 detects the vehicle parked in the parking frame defined by the pair of marking lines based on the change in the length of the pair of marking lines sequentially detected from a plurality of images captured continuously while moving. exists.

Then, the parking frame detection unit 25 notifies the parking frame management unit 26 of the determination result as to whether or not the parked vehicle is present in the parking frame in the parking frame information. A specific example of the configuration and operation of the parking frame detection unit 25 will be described later with reference to FIGS. 3 and 4A to 4C.

The parking frame management unit 26 manages the parking frames detected by the parking frame detection unit 25 in chronological order. The parking space management unit 26 estimates the amount of movement of the vehicle C based on the sensor values input from the sensor group Sc, and calculates the actual vertex coordinates of each lane line based on the past parking space information based on the amount of movement. can be estimated.

The parking space management unit 26 can also update the coordinate information of the lane markings in the past parking space information based on the newly input parking space information. That is, the parking space management unit 26 updates the relative positional relationship between the vehicle C and the parking space as the vehicle C moves.

The parking frame management unit 26 can also set the detection range of the parking frames on the assumption that a plurality of parking frames are arranged in succession. For example, the parking frame management unit 26 uses one parking frame detected by the parking frame detection unit 25 as a reference, and assumes that there are a plurality of parking spaces that are continuous with the parking frame.

Then, the parking frame management unit 26 sets the position of the assumed parking frame as the detection range. As a result, the line segment extraction unit 21 needs to perform edge line detection processing only in the detection range set by the parking frame management unit 26, so the processing load on the control unit 2 can be suppressed. .

The stop position determination unit 27 determines the stop position when the vehicle C is parked in the parking frame based on the edge lines detected by the line segment extraction unit 21 . For example, the stop position determination unit 27 determines the stop position of the vehicle C by detecting wheel chocks, curbs, walls, white lines extending in the vehicle width direction, etc. based on the edge lines detected by the line segment extraction unit 21. do.

When the wheel chock is detected, the stop position determination unit 27 determines the stop position so that the rear wheels of the vehicle C come in front of the wheel chock. The stop position is determined so that the rear end of the vehicle C (for example, the front end of the rear bumper) is positioned in front of the vehicle.

Next, the parking frame detection unit 25 according to the embodiment will be specifically described with reference to FIGS. 3 and 4A to 4C. FIG. 3 is a block diagram of the parking frame detection unit 25. As shown in FIG. 4A to 4C are explanatory diagrams showing the procedure for determining whether or not there is a parked vehicle.

As shown in FIG. 3 , the parking frame detection section 25 includes a detection section 251 and a determination section 252 . Further, the storage unit 3 stores specific change pattern information 31 when the parking frame detection unit 25 determines whether or not a parked vehicle exists in the parking frame.

The specific change pattern information 31 is a specific change indicating a change in the length of a pair of lane markings that occurs when a parked vehicle is present in a parking frame in a plurality of images obtained by continuously capturing the parking frame while moving. This is information indicating characteristics of the pattern.

The detection unit 251 is a processing unit that detects a pair of partition lines that partition the parking frame from the image. Further, the determination unit 252 determines whether the vehicle is positioned within the parking frame demarcated by the pair of demarcation lines based on changes in the length of the pair of demarcation lines sequentially detected by the detection unit 251 from a plurality of images captured while moving. It is a processing unit that determines whether or not there is a parked vehicle.

When the determining unit 252 determines that the pattern of change in the length of the pair of section lines is a specific pattern of change peculiar to the presence of the parked vehicle in the parking frame, based on the specific change pattern information 31. First, it is determined that a parked vehicle exists within the parking frame.

In this way, the determination unit 252 performs a simple procedure of comparing the pattern of change in the length of the pair of marking lines detected by the detection unit 251 with a specific pattern of change from the image captured by the monocular camera. It is possible to determine whether or not there is a parked vehicle in the parking frame.

For example, when the vehicle C is traveling in the position shown in FIG. 4A, the front side of the pair of partition lines Li1 and Li2 that partition the parking frame SP in the moving direction of the vehicle C indicated by the white arrows in the figure. An in-vehicle camera 10 is positioned on the longitudinal extension of the division line Li1.

In this case, the detection unit 251 detects the entire image of the front marking line Li1 (the hatched portion of the marking line Li1) because there is nothing blocking the space between the vehicle-mounted camera 10 and the front marking line Li1. can do.

However, the detection unit 251 does not overlap the parking vehicle Cx (the partition line Li2 Only the hatched part in ) can be detected.

After that, when the vehicle C moves forward to the position shown in FIG. 4B, it is located in front of the parked vehicle Cx. In such a case, the detection unit 251 detects that a portion of the front marking line Li1 overlaps the parked vehicle Cx. shorter than that shown in 4A.

On the other hand, the section line Li2 on the far side has a smaller portion that overlaps with the parked vehicle Cx. For this reason, in the detection unit 251, the length of the detection portion (the hatched portion of the partition line Li2) at the partition line Li2 on the far side is longer than in the case shown in FIG. 4A.

Then, when the vehicle C moves forward to the position shown in FIG. 4C, the vehicle-mounted camera 10 is positioned on the longitudinal extension line of the compartment line Li2 on the far side. In this case, the detection unit 251 detects the entire image of the rear marking line Li2 (the hatched portion of the marking line Li2) because there is nothing blocking the space between the vehicle-mounted camera 10 and the rear marking line Li2. can do.

However, the detection unit 251 determines that the length of the detection portion (the hatched portion of the marking line Li1) of the marking line Li1 on the front side is shorter than that in the case shown in FIG. 4B. .

In this way, when a vehicle is parked within the parking frame PS, when the vehicle C runs in front of the parking frame PS, the length of the demarcation line Li1 on the near side detected by the detection unit 251 gradually becomes shorter, The length of the compartment line Li2 on the far side gradually increases.

Therefore, the determination unit 252 determines that, of the pair of lane markings Li1 and Li2, the length of the lane marking Li1 on the front side in the moving direction of the vehicle C gradually becomes shorter, and the lane marking on the far side becomes shorter. When the length of the line Li2 gradually increases, it is determined to be a specific change pattern.

Then, when the determination unit 252 determines that the change pattern of the length of the pair of partition lines Li1 and Li2 detected by the detection unit 251 is a specific change pattern, the parked vehicle Cx is present in the parking frame PS. Then judge.

Further, when the determination unit 51 determines that the change pattern of the length of the pair of partition lines Li1 and Li2 detected by the detection unit 251 is not a specific change pattern, the parked vehicle Cx does not exist within the parking frame PS. I judge.

Thereby, the determination unit 252 can accurately determine whether or not the parked vehicle Cx exists within the parking frame PS from the image captured by the monocular camera (for example, the vehicle-mounted camera 10).

Next, with reference to FIG. 5, processing executed by the parking frame detection unit 25 will be described. FIG. 5 is a flowchart showing an example of processing executed by the parking frame detection unit 25. As shown in FIG. The parking frame detection unit 25 repeatedly executes the processing shown in FIG. 5, for example, when it is assumed that the vehicle C is traveling in the parking lot (for example, the vehicle speed is within 30 Km/h).

Specifically, as shown in FIG. 5, the parking frame detection unit 25 sequentially detects a pair of marking lines from images captured continuously while moving (step S101), and detects a pair of marking lines for each image. are sequentially calculated (step S102).

Subsequently, the parking frame detection unit 25 determines whether or not the change pattern of the length of the pair of marking lines is a specific change pattern (step S103). And when it determines with it being a specific change pattern (step S103, Yes), the parking frame detection part 25 determines that there is no parked vehicle in a parking frame (step S104). After that, the parking frame detection unit 25 terminates the processing and restarts the processing from step S101.

Moreover, the parking frame detection part 25 determines with no parked vehicle in a parking frame, when it determines with it not being a specific change pattern (step S103, No) (step S105). After that, the parking frame detection unit 25 detects the parking frame determined to have no parked vehicle, ends the process, and starts the process from step S101 again.

It should be noted that the specific change pattern described in the above-described embodiment is an example, and various modifications are possible. For example, as the vehicle C moves, the front section line Li1 shown in FIGS. 4A to 4C gradually becomes longer and then gradually becomes shorter, and the rear section line Li2 gradually becomes longer. It is good also considering a change pattern as a specific change pattern.

Thereby, for example, even when another parked vehicle exists in the parking frame on the right side of the parked vehicle Cx shown in FIG. It is possible to accurately determine whether or not

Further effects and modifications can be easily derived by those skilled in the art. Therefore, the broader aspects of the invention are not limited to the specific details and representative embodiments so shown and described. Accordingly, various changes may be made without departing from the spirit or scope of the general inventive concept defined by the appended claims and equivalents thereof.

REFERENCE SIGNS LIST 100 parking support system 10 in-vehicle camera 1 image processing device 2 control unit 21 line segment extraction unit 22 inappropriate area determination unit 23 lane line candidate detection unit 24 exclusion determination unit 25 parking frame detection unit 26 parking frame management unit 27 stop position determination unit 251 Detection unit 252 Determination unit 3 Storage unit 31 Specific change pattern information 50 Upper ECU
Sc sensor group

Claims (2)

Detecting a pair of demarcation lines demarcating a parking frame from a plurality of captured images captured as the vehicle moves,
Of the pair of detected lane markings, when the length of the lane marking on the front side in the moving direction of the vehicle gradually becomes shorter and the length of the lane marking on the far side gradually becomes longer,
It is determined that a parked vehicle exists within the parking frame partitioned by the pair of partition lines.
An image processing device having a control unit . Detecting a pair of demarcation lines demarcating a parking frame from a plurality of captured images captured as the vehicle moves ,
Of the pair of detected lane markings, when the length of the lane marking on the front side in the moving direction of the vehicle gradually becomes shorter and the length of the lane marking on the far side gradually becomes longer,
It is determined that a parked vehicle exists within the parking frame partitioned by the pair of partition lines.
Image processing method.

Images