WO2020039838A1

WO2020039838A1 - Image processing device and image processing method

Info

Publication number: WO2020039838A1
Application number: PCT/JP2019/029151
Authority: WO
Inventors: 拓紀茂泉; 宏治土井
Original assignee: 日立オートモティブシステムズ株式会社
Priority date: 2018-08-22
Filing date: 2019-07-25
Publication date: 2020-02-27
Also published as: JPWO2020039838A1; JP7139431B2; CN112534473A

Abstract

The present invention realizes an image processing device that can improve preceding-target tracking accuracy by appropriately performing switching processing of a template in accordance with changes in a light source state of a preceding target at night. The present invention performs usual template processing (step S210) without performing template exchanging processing if light spot information of a tracking vehicle does not change (step S204). If the light spot information has changed, confirmation is made as to whether a template image corresponding to present light spot information exists in a template storage unit or not (step S205), a template image according to the light spot is selected when the template image corresponding to present light spot information exists (S206), and template exchange is performed (step S207). In Step S205, when the template corresponding to the light spot does not exist in the template storage unit, a template image corresponding to the light spot is newly created (step S208), and the new template image and the light spot information are associated together and stored in the template storage unit (step S209).

Description

Image processing apparatus and image processing method

The present invention relates to an image processing device and an image processing method.

In recent years, advanced driving assistance systems and automatic driving systems that use external recognition sensors such as in-vehicle cameras and radars have attracted attention.

For example, as a driving support function for improving convenience and comfort, an ACC (Adaptive Cruise Control) that measures a distance and a relative speed to a vehicle traveling ahead of the own vehicle and automatically performs following control is provided. Accordingly, there is an increasing demand for vehicle detection technology for improving the performance of such functions.

車両 In the vehicle detection technology using a vehicle-mounted camera described in Patent Document 1, template matching is performed to search for and track a preceding vehicle from a captured image.

JP 2013-161241 A

(4) When performing template matching, the appearance gradually changes due to a change in the inter-vehicle distance to a preceding vehicle, a change in a lane or a change in a curve radius of a road, and the positional relationship between the own vehicle and the preceding vehicle with the passage of time. The appearance change that occurs with the passage of time may reduce the ease of matching with the template, and it is necessary to update the template to suppress this.

In the template update process in the vehicle tracking process at night, when a part of the body of the preceding vehicle is exposed to strong light by turning on the brake lamp of the preceding vehicle or headlights of the oncoming vehicle, the brightness of the surrounding area is saturated and the appearance is large. Change. This reduces the similarity with the template, makes it difficult to update the template, and may cause template matching to fail.

技術 In the technology described in Patent Document 1, there is no recognition of the phenomenon that the template matching fails due to the lighting of the brake lamp of the preceding vehicle or the like, and no countermeasures are described.

Accordingly, in the prior art, it was difficult to further improve the vehicle tracking accuracy.

An object of the present invention is to realize an image processing apparatus and an image processing method capable of appropriately performing template switching processing according to a change in the light source state of a preceding target object at night to improve the tracking accuracy of the preceding target object.

ため In order to achieve the above object, the present invention is configured as follows.

An image processing apparatus for detecting a vehicle to be tracked by template matching, comprising: an imaging unit for imaging an object; a light spot detection unit for detecting a light spot from an image captured by the imaging unit; a template and light for a plurality of vehicle images A template storage unit for storing point information, and a recognition processing unit for storing information of the changed light point in the template storage unit in association with the template of the vehicle image when the light spot of the image captured by the imaging unit changes. Is provided.

An image processing method for detecting a vehicle to be tracked by template matching, wherein a template and light spot information of a plurality of vehicle images are stored in a template storage unit, and a light spot is detected from an image captured by an imaging unit, and the image capturing is performed. When the light spot of the image captured by the unit changes, information on the changed light spot is stored in the template storage unit in association with the template of the vehicle image.

According to the present invention, it is possible to realize an image processing apparatus and an image processing method capable of appropriately performing template switching processing according to a change in the light source state of a preceding target object at night and improving the tracking accuracy of the preceding target object.

FIG. 1 is a configuration diagram of an image processing apparatus according to a first embodiment of the present invention. FIG. 7 is a diagram illustrating a flow of a template replacement process in the first embodiment. FIG. 8 is a specific explanatory diagram of a template replacement process according to the first embodiment. FIG. 14 is a specific explanatory diagram of a template replacement process according to the second embodiment.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

(Example 1)
FIG. 1 is a configuration diagram of an image processing apparatus 100 according to the first embodiment of the present invention.

In the first embodiment, three-dimensional object detection using a stereo camera is described as a nearby vehicle detection unit. However, for example, a three-dimensional object detection using only a monocular camera or a combination of a monocular camera and radar is performed. Is also feasible.

In FIG. 1, a camera 101 (camera 1) and a camera 102 (camera 2), which are image pickup apparatuses (image pickup units for picking up an object), are installed so that a field of view in front of the vehicle can be obtained.

In the first embodiment, a stereo image required to calculate the parallax information used in the three-dimensional object detection process and a light spot detection image required to calculate the light spot information used in the light spot detection process are captured.

The stereo image is captured by the camera 101 and the camera 102 whose exposure time is set by automatic exposure adjustment in order to detect a non-self-luminous object.

The image for light spot detection is captured by the camera 101 or the camera 102 which has set the exposure time recorded in the exposure time storage unit 117 of the storage area 115. As the light spot detection image, either the image of the camera 101 or the camera 102 may be used, or the image of both the camera 101 and the camera 102 may be used.

The stereo image and the light spot detection image are captured at different timings. For example, an image for light spot detection is captured after capturing a stereo image.

The images captured by the camera 101 and the camera 102 are input to the image signal processing unit 103.

The image signal processing unit 103 calculates parallax information using stereo images input from the

cameras

101 and 102. The image signal processing unit 103 includes an exposure time control unit 104 that controls the exposure time of the camera 101 and the camera 102, an image capturing unit 105 that captures an image by the camera 101 and the camera 102, and an image captured by the image capturing unit 105. And a light spot detection unit 107 that detects a light spot from an image captured by the image capturing unit 105.

The image signal processing 103 is connected to the storage area 115.

The images captured by the

cameras

101 and 102 and the parallax information calculated by the parallax calculating unit 106 are sent to the storage area 115. The captured image is stored in the image information storage unit 116 of the storage area 115, and the disparity information is stored in the disparity information storage unit 118 of the storage area 115.

The disparity information calculated by the disparity calculating unit 106 is sent to the vehicle search unit 109 of the recognition processing unit 108.

The light spot detection unit 107 performs light spot detection using the light spot detection image, and the detected light spot is sent to the light spot information calculation unit 110 of the recognition processing unit 108. The light spot information calculation unit 110 calculates light spot information such as the type and position of the light spot based on the light spot information sent from the light spot detection unit 107.

As an example of the light spot detection method in the light spot detection unit 107, a binary process is performed on the light spot detection image with a brightness value to extract a high brightness pixel. In the case of a headlight light spot, a white high-brightness pixel is extracted, and in the case of a taillight light spot, a red high-brightness pixel is extracted. Grouping processing is performed on the extracted high-luminance pixels, and a group of adjacent high-luminance pixels is set as a light spot. Furthermore, filtering is performed according to the shape, aspect ratio, and area of the light spot to narrow down.

The vehicle search unit 109 of the recognition processing unit 108 performs a vehicle search based on the parallax information, acquires vehicle candidate information, and sends it to the template selection unit 111. Further, the light spot information calculation unit 110 sends the calculated light spot information to the template selection unit 111.

The template selection unit 111 selects a template from the vehicle candidate information sent from the vehicle search unit 109 and the light spot information sent from the light spot information calculation unit 110. A newly detected vehicle is newly registered as a tracking target (preceding target object), and a template is generated by the template generating unit 112. At this time, the light spot information existing on the image area used as the template is associated with the template image of the vehicle. The template image associated with the light spot information is stored in the template image storage unit 122 of the template storage unit 121 in the storage area 115.

場合 If there is a vehicle registered as a tracking target in the past, a template matching search is performed using the template image of the vehicle.

The template image used in the template matching search is selected by the template selection unit 111. The template selection unit 111 acquires light spot information around the search area from the light spot information storage unit 123 of the template storage unit 121, and associates the acquired light spot information with the template image. Then, a plausible template image is selected by comparing with the obtained light spot information.

(4) If the light spot information changes and no plausible template image is found, a new template image is generated by the template generation unit 112 and stored in the template image storage unit 122. The template image storage unit 122 stores templates of a plurality of vehicle images.

(4) The vehicle tracking processing unit 113 performs a tracking process on vehicle information such as a template image generated by the template generating unit 112. That is, a vehicle to be tracked is detected by template matching, and tracking processing is performed.

(4) In order for the vehicle tracking processing unit 113 to pass the vehicle information on which the tracking process has been performed to the control process, the vehicle information is sent to the control signal output unit 114. The control signal output unit 114 outputs a control signal for controlling the vehicle based on the vehicle information from the vehicle tracking processing unit 113.

The storage area 115 includes a three-dimensional object information storage unit 119 and a vehicle characteristic information storage unit 120.

FIG. 2 is a diagram showing the flow of the template replacement process in the first embodiment. The process illustrated in FIG. 2 is performed by the recognition processing unit 108.

In FIG. 2, the vehicle search unit 109 performs a vehicle candidate search based on the parallax information (image information) from the image capturing unit 105, and acquires vehicle candidate information (step S201).

Light spot information calculation section 110 acquires light spot information based on the light spot information (step S202).

Next, vehicle tracking processing (steps S203 to S210) is performed based on the vehicle candidate information from the vehicle search unit 109 and the light spot information from the light spot information calculation unit 110.

First, in step S203, it is checked whether or not there is a vehicle being tracked. If the vehicle is not being tracked (before starting tracking the vehicle), the vehicle indicated by the vehicle candidate information from the vehicle search unit 109 is set as a new tracking target. Start tracking. At this time, a template image of the vehicle image is generated by associating the light spot information present on the image area used as the template (step S208), and registered (saved) in the template saving unit 121 (step S209).

If it is determined in step S203 that there is a vehicle being tracked, the process advances to step S204 to determine whether the light spot of the vehicle being tracked has changed. If the light spot information of the vehicle being tracked has not changed in step S204, the conventional template processing is performed without performing the template replacement processing (step S210).

If it is determined in step S204 that the light spot information has changed, the process advances to step S205 to check whether a template image corresponding to the current light spot information exists in the template storage unit 121.

If a corresponding template exists in step S205, the template selecting unit 111 selects a template image corresponding to the changed light spot (step S206), and performs replacement of the template image (template replacement) (step S207). . Then, the tracking operation is continued.

The vehicle tracking processing unit 113 predicts the position of the tracked vehicle (tracking range of the tracked vehicle) based on the vehicle information searched from the vehicle search unit 109. As described above, when the template image corresponding to the light spot is selected, the template images are replaced. The vehicle tracking processing unit 113 performs the replacement (selected) image of the template and the prediction of the tracking vehicle described above. The similarity with the position (predicted range) is calculated.

車両 Then, the vehicle tracking processing unit 113 corrects the predicted position of the tracked vehicle (tracking range of the tracked vehicle) based on the calculated similarity. By correcting the corrected “predicted position of the tracked vehicle (tracking predicted range of the tracked vehicle), the tracking position deviation of the tracked vehicle can be suppressed, and the matching accuracy can be improved.

If the template corresponding to the light spot does not exist in the template storage unit 121 in step S205, a template image corresponding to the light spot is newly generated (step S208), and the light spot information changed as a new vehicle image template Is registered (stored) in the template storage unit 121 in association with (step S209). Then, the tracking operation is continued.

FIG. 3 is a specific explanatory diagram of the template replacement process in the first embodiment.

In FIG. 3, it is assumed that the preceding vehicle 302 has been searched for in the captured image 301 in the time (tn) frame. If the preceding vehicle 302 has not been registered as a tracked vehicle at this time (time (tn)) (No in step S205), the preceding vehicle 302 is registered as a new tracking target, and the image area surrounding the entire vehicle is registered. Is stored in the template image storage unit 122 of the template storage unit 121 as the template image 314 (steps S208 and S209).
At this time, it is assumed that the

tail lights

303 and 304 of the preceding vehicle have been acquired as light spot information. When registering the template image 314 in the template image storage unit 122, the light point information of the

tail lights

303 and 304 is stored in the light point information storage unit 123 in association with the template image.

Next, it is assumed that the same preceding vehicle 306 as the preceding vehicle 302 is being tracked in the captured image 305 at the time (t). The tracking predicted position of the vehicle 306 in the current frame (frame at time t) is calculated from the moving speed of the preceding vehicle 302 in the previous frame (frame at time (tn)), and a template matching search is performed around the predicted tracking position. .

At this time, the light spot information of the

tail lights

307 and 308 of the preceding vehicle 306 existing around the predicted tracking position is acquired, and it is determined whether the state of the light spot information has changed from the time (tn). (Step S204). If there is no change in the state of the

tail lights

307 and 308, the change in appearance due to the change in the light source is considered to be small. Therefore, the template 314 generated in the (t−n) frame is not replaced, and template matching is performed using the template 314 as it is. Do.

It is assumed that the same preceding vehicle 310 as the preceding

vehicles

302 and 306 is being tracked in the captured image 309 ((t + n) frame image) at time (t + n).

とする At this time, it is assumed that the preceding vehicle 310 applies a brake and the

brake lamps

311, 312, and 313 are turned on. When it is determined whether the light spot information of the preceding vehicle 310 has changed from the state of the light spot information at the time (t) (step S204), at the time (t), the light spot information of the

tail lights

307 and 308 is changed. Although registered, at time (t + n), the

brake lamps

311, 312, and 313 are turned on, and the number and size of light spots are changed.

Therefore, in this case, the appearance change due to the light source change is considered to be large. Therefore, the template corresponding to the light spot is selected, the templates are replaced, and the image area surrounding the vehicle at (t + n) is set as the template image 318 as the template image 318. The image is stored in the image storage unit 122. At this time, the light spot information of the

brake lamps

311, 312, and 313 is stored in association with the template image.

As described above, according to the first embodiment of the present invention, the light spot information of the preceding vehicle is acquired, and if the light spot information has changed, the light spot information has changed if there is a template corresponding to the changed light spot information. When the template corresponding to the changed light spot information is replaced with the template corresponding to the changed light spot information, the tracking operation is performed, and the tracking operation is performed. Is configured.

Therefore, the template switching process is appropriately performed in accordance with the change in the light source state of the preceding object at night, the template matching accuracy for a sudden change in appearance due to the light source change is suppressed, and the preceding object tracking accuracy can be improved. An image processing device and an image processing method can be realized.

(Example 2)
Next, a second embodiment of the present invention will be described.

In the above-described first embodiment, the template is replaced every time the light spot information of the preceding vehicle changes. In the second embodiment, a plurality of templates corresponding to the change of the light spot are stored, and the templates are changed according to the light spot information. In this case, an appropriate template is selected.

The image processing apparatus according to the second embodiment is the same as the example shown in FIG. Further, since the processing shown in FIG. 2 is also applicable to the second embodiment, illustration and detailed description are omitted.

FIG. 4 is a specific explanatory diagram of the template replacement process in the second embodiment.

In FIG. 4, it is assumed that the preceding vehicle 402 is being tracked by the captured image 401 in the time (tn) frame. It is assumed that the image of the vehicle 402 in the captured image 401 is stored in the template image storage unit 122 as a template image 415.

At this time, the light spot information of the

tail lights

403 and 404 of the preceding vehicle 402 is stored in association with the template image. In addition to the template image 415, for example, it is assumed that a template image 416 when the brake lamp is turned on and a template image 417 when the left blinker is turned on are stored in the template image storage unit 122 of the template storage unit 121.

とする Assume that the same preceding vehicle 406 as the preceding vehicle 402 has turned on the blinker 408 in the captured image 405 in the time (t) frame. At the time (tn) frame, the light spot information of the

tail lights

403 and 404 is registered, but at the time (t) frame, the

tail lights

407 and 409 of the preceding vehicle 406 and the blinker 408 are turned on, and The numbers and types are changing.

Therefore, it is searched whether or not a template registered in the template image storage unit 122 has a template corresponding to the light spot information similar to the

tail lights

407 and 409 and the blinker 408 (step S205). Since there is a template image 417 to be executed, the template image 417 is selected and template matching is performed.

At this time, for example, when pitching occurs in the preceding vehicle 406, the tracking predicted position shifts, and light spot information around the shifted tracking predicted position is acquired, so that an erroneous light spot state change is observed. As a result, a template may be erroneously selected.

To avoid erroneous selection when selecting a template based on light spot information, set an allowable range for the amount of change in the tracking predicted position due to pitching, etc. If the tracking predicted position has changed beyond the allowable range Avoid making a selection.

If there is no template corresponding to the same light spot information in the template image storage unit 122, it is registered as a new template image in the template image storage unit 122 (steps S208 and S209).

Similarly, in the captured image 410 at the time (t + n) frame, it is determined whether or not the preceding vehicle 411 has changed the light spot information. In the captured image 410, it is assumed that the preceding vehicle 411 applies a brake and the

brake lamps

413, 412, and 413 are turned on. When it is determined whether the light spot information of the preceding vehicle 411 has changed from the state of the light spot information at the time (t), at the time (t), the light spot information of the

tail lights

407 and 408 is registered. However, at time (t + n), the

brake lamps

413, 412, and 413 are turned on, and the number and size of light spots change.

Therefore, in this case, the appearance change due to the light source change is considered to be large, so the template 416 corresponding to the light spot is selected and template matching is performed.

According to the second embodiment of the present invention, the same effects as those of the first embodiment can be obtained. In addition, a plurality of templates corresponding to the change of the light spot are stored, and an appropriate template is selected according to the light spot information. With this configuration, an appropriate template can be applied more quickly to a change in the state of the light spot.

Here, the application of the above-described embodiment of the present invention to the automatic driving technique will be described.

According to the present invention, since the template and the light spot information are stored in association with each other, it is possible to grasp the behavior of the preceding vehicle such as the intention to change lanes by turning on the brake or blinker.

Using this information, it is possible to determine the intention of the preceding vehicle to change lanes, detect an interrupt, and decelerate the host vehicle, or detect that there is no vehicle ahead and control the vehicle to accelerate. Becomes

100 image processing device, # 101, 102 camera, # 103 image signal processing unit, # 104 exposure time control unit, # 105 image imaging unit, # 106 parallax calculation unit 107: light spot detection unit, # 108: recognition processing unit, # 109: vehicle search unit, # 110: light spot information calculation unit, # 111: template selection unit, # 112: template generation unit , # 113: vehicle tracking processing unit, # 114: control signal output unit, # 115: storage area, # 116: image information storage unit, # 117: exposure time storage unit, # 118: parallax information Storage unit, # 119: three-dimensional object information storage unit, # 120: vehicle feature information storage unit, # 121: template storage unit, # 122: template image storage unit, # 123 ... point information storage unit

Claims

An image processing device that detects a vehicle to be tracked by template matching,
An imaging unit for imaging an object,
A light spot detection unit that detects a light spot from an image captured by the imaging unit,
A template storage unit for storing a plurality of vehicle image templates and light spot information,
When the light spot of the image captured by the imaging unit changes, a recognition processing unit that stores information of the changed light spot in the template storage unit in association with the vehicle image template,
An image processing apparatus comprising:
The image processing apparatus according to claim 1,
The recognition processing unit, when a light spot of an image captured by the imaging unit changes, a template selection unit that selects a template according to the change of the light spot from the template of the plurality of vehicle images from the template storage unit. An image processing apparatus comprising:
The image processing apparatus according to claim 1,
The recognition processing unit is configured to change the light spot of the image when the light spot of the image captured by the image capturing unit changes and the template corresponding to the change of the light spot does not exist in the template storage unit. An image processing apparatus for storing the image information and the image as a template in the template storage unit.
The image processing apparatus according to claim 2,
The recognition processing unit includes a vehicle tracking processing unit that predicts a tracking range of the tracked vehicle. The vehicle tracking processing unit includes an image of the template selected by the template selection unit and a predicted tracking range of the tracking vehicle. An image processing apparatus for calculating a similarity.
The image processing apparatus according to claim 4,
The image processing device, wherein the vehicle tracking processing unit corrects a predicted tracking range of the tracked vehicle based on the calculated similarity.
The image processing apparatus according to claim 1,
The recognition processing unit includes a vehicle search unit that performs a vehicle candidate search based on image information from the imaging unit, and exists in image information based on vehicle image information from the vehicle search unit before vehicle tracking starts. An image processing apparatus, wherein a template image of a vehicle image is generated in association with light spot information and stored in the template storage unit.
An image processing method for detecting a vehicle to be tracked by template matching,
Saving templates and light spot information of a plurality of vehicle images in a template storage unit,
A light spot is detected from the image captured by the imaging unit,
An image processing method, wherein when a light spot of an image captured by the imaging unit changes, information of the changed light spot is stored in a template storage unit in association with a template of the vehicle image.
The image processing method according to claim 7,
An image processing method, wherein when a light spot of an image captured by the imaging unit changes, a template corresponding to the change of the light spot is selected from the templates of the plurality of vehicle images from the template storage unit.
The image processing method according to claim 7,
In the case where the light spot of the image captured by the imaging unit has changed, and when a template corresponding to the change of the light spot does not exist in the template storage unit, the information of the changed light spot of the image and the image are An image processing method comprising storing a template in the template storage unit.
The image processing method according to claim 8,
An image processing method, wherein a tracking range of a tracked vehicle is predicted, and a similarity between the image of the selected template and the predicted tracking range of the tracked vehicle is calculated.
The image processing method according to claim 10,
An image processing method, wherein a predicted tracking range of a tracked vehicle is corrected based on the calculated similarity.
The image processing method according to claim 7,
Before starting the vehicle tracking, a vehicle candidate search is performed based on the image information from the imaging unit, and based on the image information of the searched vehicle, the light spot information present in the image information is associated with the template image of the vehicle image. An image processing method, comprising: generating and storing in the template storage unit.