WO2018066510A1

WO2018066510A1 - Image processing device

Info

Publication number: WO2018066510A1
Application number: PCT/JP2017/035816
Authority: WO
Inventors: 祐司石田; 謙二岡野
Original assignee: 株式会社デンソー
Priority date: 2016-10-07
Filing date: 2017-10-02
Publication date: 2018-04-12
Also published as: US20200039437A1; JP6607163B2; US10926700B2; DE112017005114T5; CN109844801A; JP2018060459A

Abstract

The image processing device (1) is provided with an image acquisition unit (1, S1), a detection unit (1, S3), and a display processing unit (1, S11-S15, S41-S45, S5). The image acquisition unit is configured so as to acquire captured images from an imaging device (2) for capturing images of the surroundings of a vehicle. The detection unit is configured so as to detect lane markers (30) defining a vehicle traveling path from the captured images acquired by the image acquisition unit. The display processing unit causes a display device (3) to display a superimposed image, that is, a captured image superimposed with marker images, such that marker images (31) for assisting with visual recognition of the lane markers are superimposed over the lane markers detected by the detection unit.

Description

Image processing device

Cross-reference of related applications

This international application claims priority based on Japanese Patent Application No. 2016-199184 filed with the Japan Patent Office on October 7, 2016, and is based on Japanese Patent Application No. 2016-199184. The entire contents are incorporated herein by reference.

The present disclosure relates to an image processing apparatus.

In order to make it easy to grasp the positional relationship between a vehicle and a white line, a device that processes a captured image acquired from an imaging device and displays it on a display device is known. For example, Patent Document 1 discloses the following technique. First, a rear captured image captured by the rear camera is converted into a rear bird's-eye image. Next, the converted rear bird's-eye view image is moved to the front of the vehicle, the positional relationship between the rear bird's-eye view image and the vehicle position is shifted, and an estimated bird's-eye view image is generated by estimating the side and front of the vehicle. To do. That is, it is estimated that the white line included in the rear bird's-eye image continues to the side and the front of the vehicle, and an estimated bird's-eye image is created. By using this technique, particularly when the vehicle is moving forward, the estimated bird's-eye view image is displayed on a monitor or the like together with an image showing the position of the host vehicle, so that the position of the vehicle relative to the white line can be easily grasped.

JP 2003-006621 A

In Patent Document 1, for example, when the white line on the road is missing or the white line is unclear at night, the white line included in the rear bird's-eye view image is also missing or unclear. Therefore, even in the estimated bird's-eye view image created from the rear bird's-eye view image, the white line image is missing or unclear. Therefore, as a result of detailed studies by the inventors, the driver has difficulty in visually recognizing the white line even when looking at the monitor on which the estimated bird's-eye view image is displayed, and it is difficult to grasp the positional relationship between the vehicle and the white line. It was.

One aspect of the present disclosure is to provide an image processing device that makes it easy for a driver to grasp the positional relationship between a vehicle and a lane marker that defines a traveling path of the vehicle.
One aspect of the present disclosure is an image processing apparatus that includes an image acquisition unit, a detection unit, and a display processing unit. An image acquisition part is comprised so that a captured image may be acquired from the imaging device which images the periphery of a vehicle. The detection unit is configured to detect a lane marker that defines the traveling path of the vehicle from the captured image acquired by the image acquisition unit. The display processing unit is configured to display a superimposed image, which is a captured image on which the marker image is superimposed, on the display device so that a marker image for supporting visual recognition of the lane marker is superimposed on the lane marker detected by the detection unit. Is done.

According to the image processing device of the present disclosure, the driver can easily grasp the positional relationship between the vehicle and the lane marker by displaying the superimposed image on the display device. Further, for example, even when the lane marker is partially missing or the lane marker is unclear at night, the lane marker is complemented by the marker image, so that the driver can easily recognize the lane marker.

In addition, the code | symbol in the parenthesis described in the claim shows the correspondence with the specific means as described in embodiment mentioned later as one aspect, Comprising: It does not limit the technical scope of this indication. Absent.

It is a block diagram which shows the structure of an image processing apparatus. It is a flowchart showing the image processing of 1st Embodiment. It is a flowchart showing the superimposition process of 1st Embodiment. It is a bird's-eye view image which shows the state where a part of white line is missing. FIG. 5 is a superimposed image after the image processing of the first embodiment is performed on the bird's-eye view image shown in FIG. 4. It is a figure which shows the state in which a white line is unclear at night. 7 is a superimposed image after the image processing of the first embodiment is performed on the bird's-eye view image shown in FIG. 6. It is a flowchart showing the superimposition process of 2nd Embodiment. It is a figure which shows the state which a part of white line is missing and the own vehicle moves to the right lane. When a part of white line is missing and the own vehicle moves to the right lane, the image processing of the second embodiment is performed. It is a figure which shows the state which a white line is unclear at night and the own vehicle moves to the right lane. It is the figure which performed the image processing of 2nd Embodiment, when the white line is unclear at night and it is the state which shows the state which the own vehicle moves to a right lane.

Hereinafter, embodiments for carrying out the present disclosure will be described with reference to the drawings.
[1. First Embodiment]
[1-1. Configuration of Image Processing Device 1]
The configuration of the image processing apparatus 1 will be described with reference to FIG. The image processing device 1 is an in-vehicle device mounted on a vehicle. Hereinafter, the vehicle on which the image processing apparatus 1 is mounted is referred to as the own vehicle. The image processing device 1 is an ECU, and an imaging device 2 and a display device 3 are connected to the image processing device 1. The ECU is an abbreviation for “Electronic Control Unit”, that is, an abbreviation for an electronic control unit.

The imaging device 2 includes a front camera, a left side camera, a right side camera, and a rear camera. The front camera is installed in the host vehicle so that the road surface ahead of the host vehicle is within the imaging range. The left side camera is installed in the host vehicle so that the road surface on the left side of the host vehicle is within the imaging range. The right side camera is installed in the host vehicle so that the road surface on the right side of the host vehicle is within the imaging range. The rear side camera is installed in the host vehicle so that the road surface behind the host vehicle is in the imaging range. Each camera repeatedly captures images at a preset time interval, for example, at 1/15 second intervals, and outputs the captured images to the image processing apparatus 1.

The display device 3 is a display device having a display screen such as liquid crystal or organic EL. The display device displays an image according to a signal input from the image processing device 1.
The image processing apparatus 1 is mainly configured by a known microcomputer including a CPU, a RAM, a ROM, and a semiconductor memory such as a flash memory. Various functions of the image processing apparatus 1 are realized by the CPU executing a program stored in a non-transitional physical recording medium. In this example, the semiconductor memory corresponds to a non-transitional tangible recording medium storing a program. Also, by executing this program, a method corresponding to the program is executed. The number of microcomputers constituting the image processing apparatus 1 may be one or more.

The image processing apparatus 1 includes, as a functional configuration realized by the CPU executing a program, an image acquisition processing unit 4, a video conversion processing unit 5, a lane marker detection processing unit 6, and a detection result output processing unit 7. . The method for realizing these elements constituting the image processing apparatus 1 is not limited to software, and some or all of the elements may be realized using hardware that combines a logic circuit, an analog circuit, and the like. .

[1-2. Image processing]
The image processing executed by the image processing apparatus 1 will be described with reference to the flowcharts of FIGS. The image processing is executed at a predetermined time interval such as 1/15 second, for example, while the ignition switch of the host vehicle is on.

As shown in FIG. 2, in step 1, the image processing apparatus 1 performs a process of acquiring captured images from the front camera, the left side camera, the right side camera, and the rear camera and converting them into digital signals. The image processing apparatus 1 functions as the image acquisition processing unit 4 by executing the process of step 1.

In subsequent step 2, the image processing apparatus 1 converts the four captured images converted into digital signals into a bird's-eye image viewed from a preset virtual viewpoint, and synthesizes the bird's-eye image that reflects the surroundings of the host vehicle. Generate the process. Specifically, the image processing apparatus 1 performs well-known bird's-eye conversion on the four captured images, thereby converting the photographed image into a bird's-eye image that is a viewpoint image looking down from above the host vehicle. Perform the process. The image processing apparatus 1 functions as the video conversion processing unit 5 by executing the process of step 2.

In subsequent step 3, the image processing apparatus 1 performs processing for detecting the lane marker 30 from the bird's-eye view image generated in step 2. Here, the lane marker 30 defines the traveling path of the host vehicle. In step 3, the lane marker 30 is marked on the road surface, and a white line that defines the traveling road and a road edge that is mainly a boundary line between the roadway and the sidewalk are detected. Examples of white lines include continuous lines and broken lines. Examples of road edges include curbs, gutters, and guardrails. For example, in the bird's-eye view images shown in FIGS. 4 and 6, two white lines are detected as the lane marker 30.

The detection of the lane marker 30 can be performed using a known technique. For example, the detection of the white line can be performed by calculating the luminance in the bird's-eye view image and extracting the edge from the luminance-converted image. The detection of the road edge can be performed by using information such as luminance, color, or shape of the bird's-eye view image as a clue. The image processing apparatus 1 functions as the lane marker detection processing unit 6 by executing the process of step 3.

In subsequent step 4, the image processing apparatus 1 performs a superimposition process for superimposing a marker image 31 for assisting the visual recognition of the lane marker 30 on the lane marker 30 detected in step 3. Specifically, the marker image 31 is a band-like image for clearly showing the lane marker 30 as shown in FIGS. 5 and 7, for example. FIG. 5 is a superimposed image in which two marker images 31 are superimposed on two white lines on the bird's-eye view image shown in FIG. 4. As shown in FIG. 5, the marker image 31 is a band-shaped image that is arranged so as to overlap the lane marker 30 on the bird's-eye view image and extends from end to end on the bird's-eye view image. Therefore, even when the lane marker 30 is missing as shown in FIG. 4, the position of the lane marker 30 can be clearly shown by superimposing such a marker image 31. FIG. 7 is a superimposed image in which two marker images 31 are superimposed on two white lines on the bird's-eye view image shown in FIG. As shown in FIG. 6, even when the lane marker 30 is unclear at night, the position of the lane marker 30 can be clearly shown by superimposing the marker image 31.

Specific processing of superimposition processing will be described with reference to the flowchart of FIG.
In step 11, the image processing apparatus 1 performs processing for setting the initial value of the variable N to 0. N is a natural number.

In subsequent step 12, the image processing apparatus 1 performs a process of determining whether or not the number of detected lane markers 30 detected in step 3 is greater than the variable N. If an affirmative determination is made in step 12, the process proceeds to step 13. In step 13, the image processing apparatus 1 performs a process of adding 1 to the variable N.

In subsequent step 14, the image processing apparatus 1 acquires the distance from the own vehicle to the lane marker 30 and the inclination of the lane marker 30 from the bird's-eye image based on the Nth lane marker 30 detected in step 3. Perform the process. The distance from the host vehicle to the lane marker 30 is a distance from the host vehicle to a line closer to the host vehicle indicating the longitudinal direction of the belt-like lane marker 30.

In subsequent step 15, the image processing apparatus 1 performs a process of determining the coordinate position of the marker image 31 to be superimposed on the Nth lane marker 30 based on the result obtained in step 14. Specifically, the image processing apparatus 1 performs a process of determining the position of the line on the near side of the host vehicle indicating the longitudinal direction of the band-shaped marker image 31. This line extends from end to end on the bird's-eye view image.

In subsequent step 16, the image processing apparatus 1 performs a process of determining whether or not the distance from the host vehicle to the Nth lane marker 30 is equal to or less than a threshold value. If an affirmative determination is made in step 16, the image processing apparatus 1 performs a process of setting the risk level to be strong in step 17. If a negative determination is made in step 16, the image processing apparatus 1 performs a process of setting the risk level to be weak in step 18. In addition, the process of step 16, step 17, and step 18 is called a risk determination process.

After step 17 or step 18, proceed to step 19. In step 19, the image processing apparatus 1 performs a process of setting the width of the marker image 31 according to the risk level set in steps 17 and 18. Specifically, the image processing apparatus 1 sets the display mode of the marker image 31 so that the width of the marker image 31 is wide when the risk level is strong, and the width of the marker image 31 is narrow when the risk level is low. The display mode of the marker image 31 is set so that The image processing apparatus 1 performs a process of setting the width of the marker image 31 by determining the distance from the position of the line of the marker image 31 determined in step 15 to the side far from the host vehicle. .

In subsequent step 20, the image processing apparatus 1 performs a process of determining whether or not the Nth lane marker 30 detected in step 3 is a white line. If an affirmative determination is made in step 20, a process of drawing a linear marker image 31 showing a white line on the bird's-eye view image in step 21 is performed. If a negative determination is made in step 20, a process of drawing a three-dimensional marker image 31 indicating a road edge on the bird's-eye view image in step 22 is performed. In addition, the process of step 20, step 21, and step 22 is called a superimposition drawing process.

After step 21 or step 22, return to step 12. In step 12, the process from step 13 to step 22 is performed while an affirmative determination is made, and the superimposition process ends when a negative determination is made.

After step 4, go to step 5. In step 5, the image processing device 1 performs a process of outputting the superimposed image on which the marker image 31 is superimposed in step 4 to the display device 3. Note that the image processing apparatus 1 functions as the detection result output processing unit 7 by executing the processing of Step 4 and Step 5.

After step 5, the process returns to step 1, and the image processing apparatus 1 performs image processing on the acquired captured image. When the ignition switch is turned off, the image processing apparatus 1 ends the image processing.

[1-3. effect]
According to the first embodiment described in detail above, the following effects can be obtained.
(1a) The image processing apparatus 1 according to the first embodiment performs processing for causing the display device 3 to display a superimposed image on which the marker image 31 is superimposed. Therefore, the position of the lane marker 30 can be clearly shown. Therefore, the driver can easily grasp the positional relationship between the host vehicle and the lane marker 30. For example, even when the lane marker 30 is partially missing or the lane marker 30 is unclear at night, the lane marker 30 is complemented by the marker image 31, so that the driver can easily recognize the lane marker 30.

(1b) The image processing apparatus 1 according to the first embodiment performs a process of determining the display mode of the marker image 31. Therefore, the display mode of the marker image 31 can be changed. Therefore, it is possible to display on the display device 3 a superimposed image on which the marker image 31 having an arbitrary display mode is superimposed.

(1c) The image processing apparatus 1 according to the first embodiment performs a process of changing the width of the marker image according to the degree of risk. Therefore, the driver can recognize the risk level. Therefore, the driver can grasp the position of the host vehicle in the width direction of the travel path.

(1d) The image processing apparatus 1 according to the first embodiment performs a process of increasing the width of the marker image 31 when the degree of danger is high. Therefore, when the degree of danger is strong, the marker image 31 can be highlighted and displayed. Therefore, the driver can easily recognize that the degree of danger is high, that is, the host vehicle is about to depart from the road.

(1e) The image processing apparatus 1 according to the first embodiment performs a process of changing the display mode of the marker image 31 according to a white line or a road edge. Therefore, even if the marker image 31 is superimposed and the lane marker 30 is hidden in the superimposed image, the driver can recognize the type of the lane marker 30.

In the first embodiment, S1 corresponds to processing as an image acquisition unit, S3 corresponds to processing as a detection unit, S11 to S15 and S5 correspond to processing as a display processing unit, and S16 to S22. Corresponds to processing as a determination unit.

[2. Second Embodiment]
[2-1. Difference from the first embodiment]
Since the basic configuration of the second embodiment is the same as that of the first embodiment, differences will be described below. Note that the same reference numerals as those in the first embodiment indicate the same configuration, and the preceding description is referred to.

In the first embodiment described above, the image processing apparatus 1 performs a process of causing the display device 3 to display the marker image 31 that is superimposed on all the detected lane markers 30. In contrast, in the second embodiment, when the image processing apparatus 1 determines that the host vehicle is in a state of changing the vehicle, the marker image is superimposed on the lane marker 30 that is expected to be exceeded by the lane change. The process which does not display 31 is performed.

[2-2. Image processing]
A superimposition process executed by the image processing apparatus 1 according to the second embodiment instead of the superimposition process according to the first embodiment shown in FIG. Note that the processing from step 41 to step 45 in FIG. 8 is the same as the processing from step 11 to step 15 in FIG.

After step 45, proceed to step 46. In step 46, the image processing apparatus 1 performs a risk determination process. The risk determination process is the same as the process of Step 16, Step 17 and Step 18 in FIG.

In subsequent step 47, the image processing apparatus 1 determines whether or not the position of the Nth lane marker 30 detected in step 3 is on the right side of the host vehicle and the right winker is on. Do. If an affirmative determination is made in step 47, the process returns to step 42. For example, as shown in FIGS. 9 and 11, when the position of the white line is on the right side of the host vehicle and the right winker is on, that is, when the host vehicle is moving to the right lane, As shown in FIGS. 10 and 12, the marker image 31 is not superimposed on the white line. If a negative determination is made in step 47, the process proceeds to step 48.

In step 48, the image processing apparatus 1 determines whether or not the position of the Nth lane marker 30 detected in step 3 is on the left side of the host vehicle and the left winker is on. . If an affirmative determination is made in step 48, the process returns to step 42. That is, if it is determined that the host vehicle is moving to the left lane, the marker image 31 is not superimposed on the lane marker 30. If a negative determination is made in step 48, the process proceeds to step 49. Note that the processing of step 47 and step 48 is referred to as non-display determination processing.

In step 49, the image processing apparatus 1 performs a process of setting the width of the marker image 31 in accordance with the risk level set in step 46.
In subsequent step 50, the image processing apparatus 1 performs a superimposing drawing process. The superimposing drawing process is the same as the process of Step 20, Step 21, and Step 22 in FIG.

After step 50, the process returns to step 42. If a negative determination is made in step 42, the superimposition process ends.
[2-3. effect]
According to 2nd Embodiment explained in full detail above, there exists the effect of 1st Embodiment mentioned above, Furthermore, there exist the following effects.

(2a) The image processing apparatus 1 according to the second embodiment performs a non-display determination process. Therefore, the marker image 31 superimposed on the lane marker 30 that is predicted to be exceeded by the host vehicle due to the lane change is not displayed on the display device 3. If the marker image 31 is displayed when the lane is changed, the driver feels that it is difficult to change the lane. However, since the marker image 31 is not displayed, the driver can change the lane without feeling resistance. In addition, if the marker image 31 is displayed when the lane is changed, the marker image 31 indicating that the degree of danger is high in accordance with the lane change is displayed. However, since the marker image 31 is not displayed, information indicating that the degree of danger is high for the driver. Can be actively stopped.

In the second embodiment, S41 to S45 correspond to processing as a display processing unit, S46, S49, and S50 correspond to processing as a determination unit, and S47 and S48 correspond to processing as a determination unit.

[3. Other Embodiments]
As mentioned above, although embodiment of this indication was described, this indication is not limited to the above-mentioned embodiment, and can carry out various modifications.

(3a) In the above embodiment, an example in which image processing is performed on a bird's-eye image as an image processing target has been shown, but the image processing target is not limited to this, and image processing is performed on a captured image. You may go.

(3b) In the above embodiment, the example in which the image processing apparatus 1 performs the process of setting the width of the marker image 31 according to the degree of risk has been described. However, the setting of the display mode of the marker image is not limited to this. Absent. Examples of the display mode of the marker image include the color of the marker image, the transmittance, the blinking speed, and the like. Specifically, when the degree of danger increases, the marker image may gradually become red, the transmittance may decrease, or the blinking speed may increase.

(3c) In the above embodiment, an example of the process of changing the display mode of the marker image 31 depending on whether the lane marker 30 is a white line or a road edge is shown as the overlay drawing process, but the overlay drawing process is limited to this. It is not a thing. The superposition drawing process may be a process of changing the display mode of the marker image according to the type of white line or road edge. For example, when the white line is a broken line, the marker image may be a broken line.

(3d) In the above embodiment, an example in which the risk level is set according to the distance between the host vehicle and the lane marker 30 is shown as the risk level determination process. However, the risk level determination process is not limited to this. . In the risk determination process, the risk may be set according to the type of the lane marker. For example, if the lane change is permitted as shown by a broken line or the like, the danger level is set to be weak, and if the lane change is not allowed as shown by a yellow line or the like, the danger level is set. May be set to strong.

(3e) The image processing apparatus 1 according to the above embodiment may determine the brightness of the bird's-eye view image and change the marker image to an optimum brightness according to the determined brightness. For example, the marker image has the same luminance as the bird's-eye view image. As a result, a superimposed image with a small difference in luminance is displayed on the display device, so that a superimposed image that is gentle to the eyes of the driver can be provided.

(3f) In the above-described embodiment, an example in which image processing is performed on the entire bird's-eye view image is shown as the image processing range, but the image processing range is not limited to this. The range of image processing may be, for example, a portion close to the vehicle of the bird's-eye view image. Thereby, compared with the case where image processing is performed on the entire bird's-eye view image, the processing time is reduced.

(3g) In the above-described embodiment, an example in which the front camera, the left side camera, the right side camera, and the rear camera are used as the imaging device 2 has been described. However, the imaging device is not limited to this. One or more cameras may be used.

(3h) A plurality of functions of one constituent element in the above embodiment may be realized by a plurality of constituent elements, or a single function of one constituent element may be realized by a plurality of constituent elements. . Further, a plurality of functions possessed by a plurality of constituent elements may be realized by one constituent element, or one function realized by a plurality of constituent elements may be realized by one constituent element. Moreover, you may abbreviate | omit a part of structure of the said embodiment. In addition, at least a part of the configuration of the above embodiment may be added to or replaced with the configuration of the other embodiment. In addition, all the aspects included in the technical idea specified from the wording described in the claims are embodiments of the present disclosure.

(3i) In addition to the image processing apparatus 1 described above, a system having the image processing apparatus 1 as a constituent element, a program for causing a computer to execute the image processing described above, and a non-transitional actual situation such as a semiconductor memory storing the program The present disclosure can also be realized in various forms such as a typical recording medium and an image processing method.

Claims

An image processing device (1),
An image acquisition unit (1, S1) configured to acquire a captured image from an imaging device (2) that captures the periphery of the vehicle;
A detection unit (1, S3) configured to detect a lane marker (30) that defines a travel path of the vehicle from the captured image acquired by the image acquisition unit;
A superimposed image, which is the captured image on which the marker image is superimposed, is displayed on the display device (3) so that a marker image (31) for assisting visual recognition of the lane marker is superimposed on the lane marker detected by the detection unit. A display processing unit (1, S11 to S15, S41 to S45, S5) configured to be displayed on
An image processing apparatus.
The image processing apparatus according to claim 1,
A determination unit (1, S16 to S22, S46, S49, S50) configured to determine a display mode of the marker image;
The display processing unit is an image processing device configured to cause the display device to display the superimposed image based on the display mode determined by the determination unit.
The image processing apparatus according to claim 2,
The determination unit (1, S16 to S19, S46, S49) calculates a distance between the vehicle and the lane marker, and is configured to change a display mode of the marker image according to the calculated distance. Processing equipment.
The image processing apparatus according to claim 3,
The determination unit displays the marker image so that a width of the marker image when the calculated distance is equal to or less than a certain value is wider than a width of the marker image when the calculated distance is not equal to or less than a certain value. An image processing apparatus configured to change an aspect.
An image processing apparatus according to any one of claims 2 to 4, wherein
The image processing device, wherein the determination unit (1, S20 to S22, S50) is configured to change a display mode of the marker image according to a type of the lane marker detected by the detection unit.
An image processing apparatus according to any one of claims 1 to 5, wherein
A determination unit (1, S47, S48) for determining whether or not the vehicle is in a state of changing lanes;
The display processing unit does not display the marker image superimposed on the lane marker that is expected to be exceeded by the lane change when the determination unit determines that the vehicle is in a state of changing lanes. An image processing apparatus configured as described above.