WO2023089686A1

WO2023089686A1 - External environment recognition device

Info

Publication number: WO2023089686A1
Application number: PCT/JP2021/042199
Authority: WO
Inventors: 将一坂本; 直也多田; 裕史大塚
Original assignee: 日立Astemo株式会社
Priority date: 2021-11-17
Filing date: 2021-11-17
Publication date: 2023-05-25

Abstract

Provided is an external environment recognition device that accurately determines a situation in which a function (driver assistance control) should be temporarily stopped, and determines such situation as a HALT situation and stops the function if image degradation is severe, such as in heavy rain, and does not determine such situation as a HALT situation and does not stop the function if the image degradation is mild, such as in light rain. An image feature value immediately after wiper wiping is saved and the rate of decrease from a current image feature value is calculated, and if the rate of decrease is greater than a standard value and becomes greater than the standard value multiple times cyclically, the function (driver assistance control) of a recognition unit (4) is temporarily stopped.

Description

External recognition device

The present invention relates to an external world recognition device such as an in-vehicle stereo camera, and more particularly to an external world recognition device that accurately determines whether to stop driving support control using an image captured by an in-vehicle camera.

Driving support controls such as adaptive cruise control (ACC) and collision damage mitigation braking (AEB) are well known. For these controls, the vehicle is controlled by recognizing the surrounding environment (outside world) from an image captured by a camera installed in the windshield with the subject in front of the vehicle, for example.

In the above driving support control, if the camera's field of view is obstructed by heavy rain or cloudy windshield, etc., and the image deteriorates, non-detection / false detection will occur, and AEB will not brake, erroneous acceleration during ACC, AEB erroneous braking and erroneous braking during ACC. Therefore, there is a need for a HALT function, which is a function that temporarily stops the driving support control before the driving support control causes such an operation and notifies the user of the temporary stop. However, if the driving support control is temporarily stopped, the driver's driving operation burden cannot be reduced, so it is desirable to accurately determine the situation in which the driving support control should be temporarily stopped.

In Japanese Unexamined Patent Application Publication No. 2004-100000, in order to accurately determine the situation in which driving support control should be temporarily stopped, the edge or luminance difference is obtained from first and second captured image data during wiper operation, and the difference is specified. It is described that the stop determination process is not performed when the value is equal to or greater than the value.

JP 2015-088047 A

In Patent Document 1, the stop determination process uses the number of edges, the number of distance data, and luminance in the current frame. It is not possible to determine whether the environment is such that brightness and edges are difficult to appear, or whether brightness and edges are not appearing due to image degradation due to heavy rain or the like. In that case, in environments such as snow-covered roads where brightness and edges are inherently difficult to display, if raindrops or snow adheres to the surface and causes slight deterioration in the image, it will be erroneously determined as HALT and the function (driving support control) will be stopped. Resulting in.

Therefore, in the present invention, when the image deterioration is severe, such as heavy rain, it is judged as HALT and the function is stopped. intended to provide

In order to solve the above problems, one of the typical external world recognition devices of the present invention includes a camera installed in the vehicle interior and an image feature amount calculation unit that obtains an image feature amount from the image acquired by the camera. a recognition unit that recognizes an object outside the vehicle from the image acquired by the camera; a memory that stores a first image feature amount in the first image acquired at a timing determined in a predetermined cycle; Determining whether to suspend the function of the recognition unit based on the first image feature amount or a comparison result between the first image feature amount and a second image feature amount in the current image and a function stop determination unit.

According to the present invention, by making determinations using time-series data, it is possible to determine whether the environment is such that image features such as parallax are difficult to appear in the first place, or whether image features such as parallax are not appearing due to image degradation due to heavy rain or the like. Therefore, when the image deterioration is severe, such as heavy rain, it is determined as HALT and the function is stopped, and when the image is lightly deteriorated, the function is not stopped without HALT determination. Therefore, when the image deterioration is severe, such as heavy rain, the function (driving support control) is temporarily suspended. ) is not stopped, and driving assistance can be continued to reduce the burden of the driver's driving operation.

Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiment.

1 is a configuration diagram of an in-vehicle stereo camera according to an embodiment of the present invention; FIG. Explanatory diagrams of (a) a left camera image, (b) a right camera image, and (c) a parallax image in a normal state. Explanatory drawing of (a) left camera image, (b) right camera image, and (c) parallax image when raindrops adhere. Explanatory drawing about the wiper position before wiper actuation. Explanatory drawing about the wiper position immediately after wiper wiping. Explanatory drawing of the camera image before wiper operation. Explanatory drawing of the camera image immediately after wiper wiping. The flowchart of the function stop determination part of a vehicle-mounted stereo camera. FIG. 10 is an image of the behavior of the stop determination of this system during heavy rain, and is an explanatory diagram of time-series changes in the number of parallaxes. FIG. 10 is an image of the behavior of the stop determination of this system during heavy rain, and is an explanatory diagram of the deterioration determination result in each frame. FIG. 4 is an image of the behavior of the stop determination of this system during heavy rain, and is an explanatory diagram of the HALT occurrence counter. FIG. 10 is a behavior image of stop determination of the system during light rain, and is an explanatory diagram of time-series changes in the number of parallaxes. FIG. 4 is an image of the behavior of the stop determination of this system during light rain, and is an explanatory diagram of the deterioration determination result in each frame. FIG. 4 is an image of how the system stops when it rains lightly, and is an explanatory diagram of a HALT occurrence counter.

An embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 is a representative diagram of this embodiment, and is a configuration diagram of an in-vehicle stereo camera in this embodiment. The in-vehicle stereo camera 1 is mounted on the vehicle and recognizes objects outside the vehicle from images acquired (captured) by (multiple) cameras installed in the vehicle interior, and realizes driving support functions such as ACC and AEB. It has a function as an external world recognition device. The in-vehicle stereo camera 1

stores cameras

2A and 2B (left camera 2A and right camera 2B) as a pair of left and right imaging units, an image feature amount calculation unit 3, a recognition unit 4, and a first image feature amount. It is composed of a memory 5 and a function stop determination unit 6 . The in-vehicle stereo camera 1 includes hardware such as a camera, CPU, memory, and electronic circuit, and software that cooperates with the hardware to assist driving of the vehicle.

The

cameras

2A and 2B in FIG. 1 take images. The

cameras

2A and 2B are fixed to the vehicle so as to be spaced apart in the vehicle width direction, and can image the front through the windshield of the vehicle so as to image the same area.

The recognition unit 4 in FIG. 1 detects and recognizes objects such as vehicles and pedestrians in the surrounding environment (in front here) from the images acquired by the

cameras

2A and 2B, and performs driving support functions such as ACC and AEB. come true.

The image feature amount calculation unit 3 receives an image captured by the left camera 2A (hereinafter also referred to as a left camera image) and an image captured by the right camera 2B (hereinafter also referred to as a right camera image), and calculates the An image feature amount is calculated by analyzing a pixel value (luminance value). The image feature quantity can be calculated using the pixel values themselves, or by defining a local area from multiple pixels and using the average luminance, weighted average luminance, representative luminance, etc. in the local area. You may In this example, the image feature amount calculator 3 calculates the number of parallaxes and the number of edges as the image feature amount.

The number of edges in an image can be calculated from (the image of) the left camera 2A or the right camera 2B in the current frame as a known technique.

For parallax, corresponding points in the images of the left and

right cameras

2A and 2B are detected by pattern matching, and the coordinate shift between the detected corresponding points is calculated. By using this parallax, the distance to the corresponding point in the real space can be calculated according to the principle of triangulation. FIG. 2 shows an image of parallax calculation in the image feature quantity calculation unit 3. As shown in FIG. In FIG. 2, (c) a parallax image is calculated by pattern matching of (a) the left camera image and (b) the right camera image. Parallax is calculated in areas where edges appear, such as people, white lines, and preceding vehicles, but parallax cannot be calculated in areas where edges are difficult to appear, such as the sky, and pattern matching is difficult. The number of effective parallaxes calculated at the portion where the edge appears in the entire parallax image is called the number of parallaxes.

FIG. 3 shows an example in which raindrops adhere to the example in FIG. When raindrops adhere to the left camera (specifically, the windshield within the field of view of the left camera) as in the left camera image in FIG. , (b) pattern matching with the right camera image becomes difficult, and (c) the overall number of parallaxes decreases as in the parallax image.

The memory 5 stores the first image feature quantity in the first image acquired by the image feature quantity calculator 3 . The first image for calculating the first image feature amount is acquired at a timing defined in a predetermined cycle defined by wiper wiping. FIGS. 4A, 4B, 5A, and 5B show wiper positions and camera images immediately after wiper wiping. 4A and 4B, 2A and 2B are left and right cameras, 10 is a windshield, and 11A and 11B are left and right wipers.

Fig. 4A is an image before the wipers are activated. In rainy weather, raindrops adhere to the windshield in front of the camera, and the image obtained from the camera is as shown in Fig. 5A. On the other hand, FIG. 5B is an image immediately after wiping with the wiper. Immediately after the wipers are wiped is defined as the case where the

wipers

11A and 11B both pass in front of the left and right cameras as shown in FIG. For example, in a general vehicle, the

wipers

11A and 11B move from the position before the wiper operation (normal or retracted position) as shown in FIG. , clockwise). Then, after reaching the farthest position (positions AA and BB in FIG. 4B) from the position before the wiper operation, the windshield is moved in a direction opposite to one direction (counterclockwise in the example shown in FIGS. 4A and 4B). ), and returns to the position before the wiper operation (normal or retracted position) as shown in FIG. 4A. Raindrops adhering to the windshield are wiped off by repeating such a series of operations once or more times. In such a vehicle, the

wipers

11A and 11B each pass in front of the left camera 2A and in front of the right camera 2B twice (twice in one direction and twice in the other direction) in the series of operations. Note that depending on the installation position of the camera and the wiping range (operating range) of the wiper, the

wipers

11A and 11B may each pass in front of the left camera 2A twice (4 times in total for the

wipers

11A and 11B), or the

wipers

11A and 11B may may each pass in front of the right camera 2B twice (four times in total with the

wipers

11A and 11B). Generally, the

wipers

11A and 11B respectively pass in front of the left camera 2A and right camera 2B twice in one direction and in the other direction (instead of once in one direction) so that the Then, the wiper wiping is considered to be completed. Therefore, in the series of operations, immediately after the

wipers

11A and 11B completely pass in front of the left camera 2A and in front of the right camera 2B for the second time (in other words, after passing in one direction, they pass in the other direction). Immediately after) is defined as immediately after wiper wipe.

In addition, in the present embodiment, the timing of the wiper wiping operation described above can be the reference for the timing of function stop determination (image deterioration determination). Therefore, the operation cycle (driving cycle) of the

wipers

11A and 11B is defined from immediately after wiper wiping in a series of operations to immediately after wiper wiping in the next series of operations. In other words, the operation period of the

wipers

11A and 11B can be defined as the period from the position before the wipers are activated to the time when the

wipers

11A and 11B operate (rotate) on the windshield and return to the positions before the wipers are activated. Furthermore, in other words, a predetermined timing in a series of wiper wiping operations is set as a reference timing, and the wiper is operated from a predetermined timing in a series of operations to a predetermined timing in the next series of operations. It can also be defined as the operation period of 11A and 11B.

Whether or not the wiper has just been wiped can be determined from the camera image, wiper motor signal, wiper drive cycle, signal from the raindrop sensor, and so on. The image feature amount calculator 3 overwrites and saves the image feature amount of the image immediately after wiping with the wiper in the memory 5 as the image feature amount of the first image.

The function stop determination unit 6 in FIG. It is determined whether the function (driving support function of the recognition unit 4) is stopped by comparing with the amount.

Next, with reference to FIG. 6, specific operation contents of the function stop determination unit 6 of the above-described in-vehicle stereo camera 1 will be described. FIG. 6 is a flow chart showing the procedure of the function stop determination section 6 of the in-vehicle stereo camera 1 according to this embodiment.

First, in S101, it is determined whether the wipers are in operation, and if the wipers are in operation, the process proceeds to S102. If the wiper is not in operation, the function stop determination is terminated. A wiper switch signal or the like is acquired in order to determine whether the wiper is in operation.

In S102, the first image feature amount immediately after wiping with the wiper stored in the memory 5 is compared with the second image feature amount, which is the feature number of the current frame, to obtain the reduction rate d. When the first image feature amount is f1 and the second image feature amount is f2, the reduction rate d[%] is obtained by the following formula (1).
[Number 1]
d = ( f1 - f2 ) ÷ f1 × 100 (1)

In S103, it is determined whether the rate of decrease d is greater than the reference value dTH. If it is greater than the reference value dTH, the current frame is determined to be deteriorated, and the counter value of the HALT occurrence counter is added in S104. If it is less than the reference value dTH, the current frame is determined to be non-degraded, and the counter value of the HALT occurrence counter is decremented in S105.

In S106, it is determined whether to suspend the camera function (HALT) based on the counter value of the HALT generation counter calculated in S104 and S105. If the counter value is equal to or greater than the reference value, the process proceeds to S107 to temporarily stop (HALT) the camera function. In this state, the driving support function performed by the recognition unit 4 is temporarily stopped.

Next, the specific behavior of the above flow will be explained using FIGS. 7A, B, C and 8A, B, C.

　Figures 7A, B, and C are image diagrams of behavior in a heavy rain scene. FIG. 7A is a graph of time-series changes in the number of parallaxes. In the case of heavy rain, the number of parallaxes reaches a peak immediately after the wiper is wiped off (▴), because the image appears sharp (see FIG. 5B). Thereafter, raindrops gradually adhere to the windshield in front of the camera until the next wiper wipes (see FIG. 5A), so the number of parallaxes decreases. Immediately after the next wiper wipe, the number of parallaxes increases again, and the same behavior is repeated thereafter. FIG. 7B shows the result of the deterioration determination for each frame, representing the result of the determination in S103. In the graph of FIG. 7B , 1 indicates deterioration in S104, and 0 indicates non-deterioration in S105. In the case of heavy rain, the rate of decrease in the number of parallaxes increases, so the behavior is such that deterioration (1) and non-deterioration (0) are repeated periodically. FIG. 7C shows HALT occurrence counters in S104 and S105. In the case of FIG. 7B, deterioration (the rate of decrease in the number of parallaxes periodically becomes greater than the reference value multiple times) and non-degradation (the rate of decrease in the number of parallaxes periodically becomes less than or equal to the reference value multiple times) are periodically performed. , the counter value repeats increment and decrement. In this example, the increment value is increased compared to the decrement value in each frame. Here, the counter value and the reference value are compared (S106), and if the counter value exceeds the reference value, the camera function is temporarily stopped (HALT) (S107). As a result, the driving support function performed by the recognition unit 4 is temporarily stopped before it fails to operate or malfunctions.

　Figs. 8A, B, and C are image diagrams of the behavior of light rain in an environment where parallax (or edge) is difficult to appear, such as snow. FIG. 8A is a graph of time-series changes in the number of parallaxes. In the case of light rain, as in heavy rain, the image appears sharp immediately after the wiper is wiped off (▴) (see FIG. 5B), so the number of parallaxes reaches a peak and then decreases, but the rate of decrease is small. FIG. 8B shows the result of the deterioration determination for each frame, representing the result of the determination in S103. In the case of light rain, the reduction rate of the number of parallaxes is not large, so it is determined as non-degraded (0). FIG. 8C shows HALT occurrence counters in S104 and S105. Since there is no deterioration, the counter value decreases (specifically, the counter value is 0 or more, so it does not change and stays the same). Here, the counter value is compared with the reference value (S106), and the counter value is below the reference value. As a result, the driving assistance function performed by the recognition unit 4 is not stopped, and the driving assistance can be continued to reduce the burden of the driver's driving operation.

In this way, in the function stop determination unit 6, the reduction rate of the number of parallaxes is calculated by comparing with the current frame immediately after the wiper wipe (the wiper completely passes in front of the left and right cameras and the image is clearly displayed) as a reference. By doing so, it is possible to determine whether the environment is such that parallax is difficult to appear in the first place, or whether parallax is not appearing due to image deterioration due to heavy rain or the like (in other words, it is possible to distinguish between environments such as normal rain and snow that are originally difficult to produce parallax).

In the above embodiment, a HALT occurrence counter is used to determine whether a function has stopped. However, another method of capturing periodic changes in the number of parallaxes as shown in FIG. 7A may be used. For example, it is possible to determine the stoppage of a function based on the periodic change (periodicity) of the rising interval and the falling interval as a result of deterioration determination in each frame as shown in FIG. 7B.

Further, in the above-described embodiment, the timing immediately after wiper wiping (the state in which the wiper completely passes in front of the left and right cameras and the image is clearly captured) is compared with the current frame as the timing for determining whether the wiper has been wiped. The function stop was determined by calculating the reduction rate of the image feature amount of the current frame with respect to the image feature amount of the immediately following image. However, the timing for judging whether to stop functioning is not limited to the above embodiment.

For example, a function stop may be determined by comparing with the current frame after a predetermined time (or after a predetermined number of frames) immediately after wiping the wiper, and calculating the reduction rate of the image feature amount. In addition, the current frame is compared with the image immediately before the wiper is wiped (immediately before the wiper completely passes in front of the left and right cameras, the state of the image in which parallax is unlikely to occur) as a reference, and the current frame is compared with the image feature amount of the image immediately before the wiper is wiped. A function stop may be determined by calculating an increase rate of the image feature amount. In addition, the frame immediately after wiping the wiper is compared with a frame a predetermined time (or a predetermined number of frames before) from the current frame (in other words, for example, a frame immediately before wiping the wiper or a frame after a predetermined time after wiping the wiper) to determine the image characteristics. The outage may be determined by calculating the rate of change (rate of decrease or rate of increase) of the amount. The predetermined period of time (or the predetermined number of frames) used in this determination can be defined from the operation period of the wipers (for example, the period from immediately after wiping with a certain wiper to immediately after wiping with the next wiper).

That is, the function stop determination unit 6 of the present embodiment identifies two different points (two times) within the wiper operation cycle, and changes the image feature amount at two different points (two times) within the wiper operation cycle. By calculating the rate (decrease rate or increase rate), it is possible to determine outage.

Furthermore, in the above-described embodiment, the change rate of the first and second image feature values is used to determine whether the function has stopped. , it may be determined that the function has stopped.

As described above, with the conventional technology, it is not possible to determine whether the environment is such that image features such as parallax are difficult to appear, or whether image features such as parallax are not appearing due to image degradation due to heavy rain or the like. In this case, in environments such as snow-covered roads where image features such as parallax are inherently difficult to obtain, if raindrops or snow adheres to the image and mildly degrades the image, it is erroneously determined to be HALT and functions (driving support control). will stop.

If the image is degraded due to heavy rain, etc. (the entire screen is distorted), image features such as parallax are no longer displayed, and the function (driving support control) is temporarily stopped due to HALT, the user will not feel uncomfortable. On the other hand, on snowy roads, etc., image features such as parallax are originally difficult to appear, and raindrops caused slight deterioration in the image (because part of the screen was distorted). When the driving support control) is temporarily stopped, the user feels uncomfortable. In other words, in places where image features such as parallax are difficult to see (snowy roads or dark road surfaces), if there is a slight deterioration in the image that does not require HALT, it will erroneously generate HALT and function (driving support control). Temporarily stopping the operation gives the user a sense of incongruity.

Therefore, it is necessary to determine whether the environment is such that image features such as parallax are difficult to appear in the first place, or whether image features such as parallax are not appearing due to image deterioration due to heavy rain, etc., and the situation in which the function (driving support control) should be temporarily stopped. It is important to make an accurate judgment, to stop the function by judging it as HALT when the image deterioration is severe, such as heavy rain, and not to stop the function without making a HALT judgment in the case of light rain where the image deterioration is mild. be.

The in-vehicle stereo camera (external recognition device) 1 of the present embodiment described above uses the cameras (left camera 2A, right camera 2B) installed in the vehicle interior and the image feature amount ( a recognition unit 4 for recognizing an object outside the vehicle from the image acquired by the camera; and a timing (

wiper

11A, 11B A memory 5 for storing the first image feature amount in the first image acquired at the timing defined by the operation of (), the first image feature amount, or the first image feature amount and the current image and a function stop determination unit 6 for determining whether or not to suspend the function of the recognition unit 4 (HALT determination) based on the result of comparison with the second image feature amount in .

More specifically, the in-vehicle stereo camera (external recognition device) 1 of this embodiment stores the image feature amount immediately after the wiper is wiped, calculates the rate of decrease from the current image feature amount, and determines the rate of decrease to be the reference value. If the rate of decrease is larger than the reference value and periodically becomes larger than the reference value multiple times, the function of the recognition unit 4 is temporarily stopped.

According to the present embodiment, by performing determination using time-series data (for example, two time-series frames based on immediately after wiping wipers), it is possible to determine whether the environment is such that image features such as parallax are difficult to appear in the first place. Since it is possible to judge whether image features such as parallax are not appearing due to image degradation due to heavy rain, etc., if the image degradation is severe like heavy rain, it will be judged as HALT and the function will be stopped. In the case of , the function is not stopped without HALT judgment. Therefore, when the image deterioration is severe, such as heavy rain, the function (driving support control) is temporarily suspended. ) is not stopped, and driving assistance can be continued to reduce the burden of the driver's driving operation.

It should be noted that the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. Also, part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Moreover, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration. Further, each of the above configurations, functions, processing units, processing means, and the like may be realized by hardware, for example, by designing them in an integrated circuit. Moreover, each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, files, etc. that realize each function can be stored in memory, hard disks, SSD (Solid State Drives) and other recording devices, or IC cards, SD cards, DVDs and other recording media.

1... In-vehicle stereo camera (external world recognition device)
2A, 2B... Cameras (imaging units) (2A: left camera, 2B: right camera)
3... Image feature amount calculator 4... Recognition part 5... Memory 6... Function stop determination part 10...

Windshields

11A, 11B... Wipers

Claims

a camera mounted inside the vehicle,
an image feature amount calculation unit that obtains an image feature amount from the image acquired by the camera;
a recognition unit that recognizes an object outside the vehicle from the image acquired by the camera;
a memory for storing a first image feature amount in the first image acquired at a timing determined in a predetermined cycle;
Determining whether to suspend the function of the recognition unit based on the first image feature amount or a comparison result between the first image feature amount and a second image feature amount in the current image a function stop determination unit;
An external world recognition device characterized by having:
In the external world recognition device according to claim 1,
The external world recognition device, wherein the first image is acquired at a timing defined by the operation of the wiper.
In the external world recognition device according to claim 1,
The external world recognition apparatus, wherein the image feature amount calculation unit uses the number of parallaxes calculated from two or more cameras in the current frame as the image feature amount.
In the external world recognition device according to claim 1,
The external world recognition device, wherein the image feature amount calculation unit uses the number of edges calculated from one camera in the current frame as the image feature amount.
In the external world recognition device according to claim 1,
The external world recognition device, wherein the memory for storing the first image feature quantity stores the image feature quantity in a state of being imaged at a timing defined by the operation of the wiper.
In the external world recognition device according to claim 1,
The function stop determination unit determines the second image feature amount for the first image feature amount from the first image feature amount stored in the memory and the second image feature amount in the current image. An external world recognition device characterized by calculating a change rate of a quantity.
In the external world recognition device according to claim 6,
The external world recognition device, wherein the function stop determination unit stops the function of the recognition unit when the rate of change is greater than a reference value, and does not stop the function of the recognition unit when the rate of change is less than or equal to the reference value.
In the external world recognition device according to claim 7,
The external world recognition device, wherein the function stop determination unit stops the function of the recognition unit when the rate of change periodically exceeds a reference value a plurality of times.
In the external world recognition device according to claim 2,
The function stop determination unit determines whether or not to suspend the function of the recognition unit based on the rate of change of the image feature amount in the time series of two points within the operation cycle of the wiper. External recognition device.
In the external world recognition device according to claim 1,
The function stop determination unit determines whether to suspend the function of the recognition unit based on a comparison result between the absolute values of the first image feature amount and the second image feature amount. Characteristic external recognition device.