WO2021235001A1

WO2021235001A1 - Image processing device

Info

Publication number: WO2021235001A1
Application number: PCT/JP2021/002142
Authority: WO
Inventors: 圭汰仲澤; 哲也山田; 宏治土井
Original assignee: 日立Astemo株式会社
Priority date: 2020-05-22
Filing date: 2021-01-22
Publication date: 2021-11-25
Also published as: JPWO2021235001A1; JP7350168B2

Abstract

Provided is an image processing device capable of realizing accuracy improvement in tracking processing.　The image processing device (1) according to the present invention is characterized by having: a tracking processing unit (10) for setting a search area for an object for each of a plurality of frame images captured by a vehicle-mounted camera, and time-sequentially tracking the object; and a search area calculation unit (11) for changing the search area by using information on a pitch angle of a vehicle.

Description

Image processing device

The present invention relates to an in-vehicle control device installed in a vehicle such as a motorcycle, which acquires an outside world situation from a camera, recognizes the outside world situation, and generates a control signal, and particularly relates to an image processing device.

In order to realize a safe and comfortable car society, the introduction of driving support systems is progressing. For example, ACC: Adaptive Cruise Control) follows the vehicle within the set vehicle speed while maintaining the distance between the vehicle and the vehicle in front.

The difference between a two-wheeled vehicle and a four-wheeled vehicle (so-called automobile) is that the amount of subduction in the front and rear of the vehicle body caused by braking, driving, road surface conditions, etc. is large. In particular, when ACC is realized by using a camera for image recognition, in a motorcycle, when the front and rear of the vehicle body sinks, the tracking process of the camera cannot catch up and there is a high possibility that the tracking fails. When a camera for image recognition is used, not limited to the ACC, there is a problem corresponding to the vehicle behavior, particularly the inclination of the pitch angle, which occurs in the vehicle.

Patent Document 1 discloses a technique of changing the cutting range in consideration of the vehicle behavior when the image processing unit generates an image to be used for recognition from the captured image. In Patent Document 1, in a camera developed for other vehicles such as four wheels, when the image processing unit cuts out an image to be used in the recognition unit from the captured image, the pitch angle generated in the vehicle is obtained from the sensor information, and the pitch is obtained. It is described that a cropping method is used in which the influence of the pitch angle is removed at the stage of generating an image to be passed to the recognition unit by setting the cropping range of the captured image in the image processing unit according to the angle.

Japanese Unexamined Patent Publication No. 2014-143547

When the technique of Patent Document 1 is applied to tracking processing as it is, when the cutting position correction direction based on the pitch angle and the moving direction of the detection target predicted by the tracking processing are different, the image in the direction in which the detection target moves is in the image processing stage. If it is cut off with, it may not be possible to capture the detection target within the search range. Further, in the prior art, since the pitch angle of the own vehicle is estimated from the sensor information and the cutting range is changed, there is a possibility that the cutting range may be erroneous when the sensor information outputs an invalid value. Therefore, the cutting method, which is a conventional technique, cannot solve the problem of accuracy of tracking processing when a pitch angle occurs.

An object of the present invention is to provide an image processing apparatus capable of improving the accuracy of tracking processing.

The image processing device of the present invention uses a tracking processing unit that sets a search range for a target object for each of a plurality of frame images captured by an in-vehicle camera and tracks the target object in chronological order, and vehicle pitch angle information. It is characterized by having a search range calculation unit for changing the search range.

According to the present invention, an image processing device capable of improving the accuracy of tracking processing is provided. Further features relating to the present invention will be apparent from the description herein and the accompanying drawings. In addition, problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

The schematic block diagram of the image processing apparatus in Example 1 of this invention. An example of the tracking process supported by the present invention. An example showing a difference from the prior art in Example 1 of the present invention. An example in which the search range is set according to the reliability in the first embodiment of the present invention. The flowchart of image processing in Example 1 of this invention. An example of application to the roll angle in Example 1 of the present invention. The schematic block diagram of the image processing apparatus in Example 2 of this invention. The flowchart of image processing in Example 2 of this invention. The schematic block diagram of the image processing apparatus in Example 3 of this invention. The flowchart of image processing in Example 3 of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
In each of the following examples, the case where the image processing device of the present invention is applied to a motorcycle will be described as an example, but the present technology is not limited to the motorcycle. For example, it can be applied to all vehicles in which pitching can occur, such as a tricycle such as a three-wheeled motorcycle and a four-wheeled vehicle such as a truck.

Conventionally, there is a tracking process that detects an object by recognizing an image captured by a camera and tracks the detection target in chronological order. An example of the tracking process will be described with reference to FIG. First, as shown in the image shown in FIG. 2A, the detection target is detected from the past frame (frame n-1), and an identifier is added to the detection target. In the example shown in FIG. 2A, the identifier A is added to the motorcycle and the identifier B is added to the preceding vehicle.

Subsequently, the positions and moving directions of the detection targets A and B are predicted using the Kalman filter from the tracking processing results in the past frames. Then, the search range of the detection targets A and B in the current frame (frame n) is set from the prediction result. In the example shown in FIG. 2B, the search range is set by predicting that the motorcycle A moves in the direction approaching the own vehicle, that is, moving to the lower left in the image, and the preceding vehicle B is from the own vehicle. The search range is set by predicting that the vehicle will move upward in the direction away from the image, that is, in the image.

Next, search the search range currently set in the frame. Then, by associating the detection result of the object in the current frame with the prediction result, it is determined whether the detection target detected in the current frame and the detection targets A and B in the past frame are similar objects. If it is determined that the detection target of the current frame is an object similar to the detection targets A and B of the past frame, the same identifier is added, and if it is determined that the object is different from the past frame, a new identifier is added.

The above tracking process recognizes the past frame and the current frame in association with each other, but at that time, it is processed on the premise that the camera position is constant. However, when the pitch angle is generated in the own vehicle, the camera itself is tilted and the position of the detection target in the frame moves. If the position of the detection target in the frame changes depending on the pitch angle, there is a problem that the detection target cannot be associated between the past frame and the current frame, and tracking fails. Therefore, in the tracking process, it is necessary to deal with the vehicle behavior that occurs in the vehicle, particularly the inclination of the pitch angle.

Compared to four-wheeled passenger cars, motorcycles and other two-wheeled vehicles have the characteristic that the amount of subduction in the front and rear of the vehicle body caused by braking, driving, road surface conditions, etc. is large. Therefore, for example, when an image processing device for a passenger car is mounted on a motorcycle as it is and recognition processing is performed using an image captured by an image recognition camera, it becomes impossible to capture the detection target within the search range. There is a possibility that tracking may fail due to the influence of the pitch angle generated on the body of the motorcycle in the tracking process.

In each embodiment, the image processing unit does not cut out an image in consideration of the vehicle behavior as in the conventional case, but the recognition unit of the recognition camera receives information on the inclination of the pitch angle of an arbitrary vehicle. Execute the process to be used. In the present invention, the problem (1) how to set the search range of the tracking process when the pitch angle changes, and the problem (2) how to track when the pitch angle information of the vehicle cannot be accurately acquired. The purpose is to set the search range of the process or to solve these problems (1) and (2).

In each embodiment, for the above problem (1), the search range set from the movement direction of the detection target predicted from the previous frame by the tracking process is searched by using the pitch angle information generated in the vehicle. The range is updated, and for the above problem (2), the reliability for determining whether the acquired pitch angle information is accurate is set, and the search range of the tracking process can be changed according to the pitch angle information and the reliability. By doing so, these problems are solved.

For the above problems (1) and (2), the means for using the pitch angle sensor will be described using the first embodiment, and the second embodiment will be used for the means for using the output of the vehicle control unit instead of the pitch angle sensor. The means for using the control driving force calculated inside the image processing device instead of the pitch angle sensor will be described with reference to Example 3.

[Example 1]
In the first embodiment, the pitch angle information is acquired from the pitch angle sensor 2 for the pitch angle of the vehicle that changes from moment to moment, and the change in the position of the detection target on the recognition image when the pitch angle occurs is used as the deviation information 51. It is output, and whether the output of the deviation information 51 is accurate is output as the reliability 52. Then, a method of setting the search range of the detection target on the recognition image is provided from the deviation information 51, the reliability 52, and the prediction result of the detection target calculated by the tracking processing unit 10.

First, using FIG. 3, the problems when the conventional technology is applied to the tracking process and the solution by the present proposal will be described.

The conventional technology estimates the pitch angle generated in the own vehicle from the sensor information, and adopts a cutting method that changes the cutting position in the image processing unit in response to the change in the position of the detection target in the captured image due to the pitch angle. We are trying to deal with it (see FIG. 3 (a)).
However, in the prior art, the focus is on improving the recognition accuracy mainly in consideration of the matching process for detecting an object, and the tracking process is not specified. The tracking process tracks the detected target by predicting the position and moving direction of the detected target, setting the search range, and associating the detected target with each other in a plurality of frames. In the tracking process, it is necessary to set the search range in consideration of the moving direction of the detection target in addition to the pitch angle information generated in the own vehicle.

When the conventional technique is applied to such a tracking process, for example, as shown in FIG. 3 (b), the cutout position in the image is moved upward in order to cope with the change in the pitch angle. Therefore, in normal tracking processing, the search range is set in consideration of the moving direction of the detection target of the identifiers A and B. Therefore, if the movement direction of the detection target of the identifier A is downward, the image processing unit moves. The image in the direction is cut off, and the search range for identifying the identifier A cannot be sufficiently set. Therefore, there is a possibility that the mapping of the identifier A fails and the detection target of the identifier A cannot be identified.

In the prior art, the problem of pitch angle change is to be addressed by correcting the cutting position when the image processing unit generates a recognition image. On the other hand, in this embodiment, the image processing unit does not perform cutting in consideration of the pitch angle information when generating the recognition image. In this embodiment, the recognition unit 6 uses the pitch angle information to variably set the search range of the detection target in the tracking process according to the pitch angle information (see FIG. 3C).

According to this embodiment, since the image processing unit does not perform cutting in consideration of the pitch angle information, the problem of the prior art that the moving direction of the detection target is cut in the tracking processing does not occur. Since the search range is set based on the pitch angle information, the detection target can be captured within the search range even when the pitch angle changes in the vehicle as long as the detection target exists within the range of the captured image. Therefore, even if the pitch angle changes, it is possible to associate the detection target among a plurality of frames.

The difference between the prior art and the present embodiment has been described with reference to FIG. 3. Next, when the pitch angle of the own vehicle changes, the search range is set by using the pitch angle information in the tracking process. The principle of this embodiment will be described in detail with reference to FIG.

The configuration diagram of this embodiment is shown in FIG. The vehicle is provided with a camera 1 and a pitch angle sensor 2, which are connected using a CAN bus 3. The pitch angle sensor 2 is not particularly limited, but for example, an inertial measurement unit IMU (Inertial Measurement Unit) can be used. In the inertial measurement unit IMU, a three-dimensional angle including a pitch angle, an angular velocity, and an acceleration are obtained. These data are transmitted as data such as CAN and Ethernet.

FIG. 1 shows the internal configuration of the camera 1 which is an image processing device. The camera 1 includes an image pickup unit 4, an image processing unit 5, a recognition unit 6, a control generation unit 7, and a communication interface unit 8. The image pickup unit 4 captures an image with an image pickup element such as a CMOS sensor. The image processing unit 5 uses the image captured by the image pickup unit 4 to generate an image to be used in the recognition process. The recognition unit 6 detects vehicles and lanes, and tracks the time series of detection targets. The control generation unit 7 generates a control signal to an actuator such as a vehicle brake or an accelerator based on the recognition result of the recognition unit 6. The communication interface unit 8 communicates the camera with an external ECU such as CAN.

The recognition unit 6 includes a storage unit 9, a tracking processing unit 10, and a change amount calculation unit 20. The storage unit 9 stores past recognition results such as detection of other vehicles such as the preceding vehicle and lanes of the lane, road surface recognition, and tracking processing results. The tracking processing unit 10 sets a search range from the prediction information that predicts the movement of the detection target by using the Kalman filter for the detection target to which the identifier is added, and performs time-series tracking of the detection target. The change amount calculation unit 20 outputs the vertical change amount of the target object in the frame image as the deviation information 51 from the pitch angle information. The change amount calculation unit 20 outputs the change in the position of the detection target on the recognition image when the pitch angle is generated from the pitch angle information 50 acquired from the pitch angle sensor 2 as the deviation information 51, and also outputs the deviation information. Is output as the reliability 52. The pitch angle information 50 includes a pitch angle generated in the own vehicle, a pitch angle change amount per unit time, and a pitch angle change direction.

Regarding the processing method of the tracking processing unit 10, the tracking method of the detection target using the Kalman filter is described as an example, but the tracking processing method is not limited to this example.

The tracking processing unit 10 has a search range calculation unit 11 that changes the search range from the output of the change amount calculation unit 20 and the prediction information of the detection target with respect to the search range set by the prediction information of the detection target. The tracking processing unit 10 updates the search range to the search range changed by the search range calculation unit 11.

The tracking processing unit 10 determines whether or not the detection target exists in the updated search range by performing a full search in the search range. As a result of the search, if there is a detection target to which tracking is performed and an identifier is added in the past, the same identifier is added to track the detection target. When a new detection target is discovered, a new identifier is added. The deviation information 51 may include a change amount in the vertical direction and a change amount in the left-right direction, and may output the change amount of the detection target on the recognition image as a pixel amount.

The reliability 52 is calculated from the amount of change in the pitch angle per unit time, and when the amount of change in the pitch angle per unit time is larger than the threshold value, it is output as low reliability and the amount of change in the pitch angle per unit time is When it is smaller than the threshold value, it is output as high reliability. In this embodiment, it is assumed that the reliability 52 is treated as a numerical value, but a level, an electric signal, or the like may be used as the reliability.

The reliability 52 is not only calculated from the amount of change in the pitch angle per unit time, but it is also possible to set the reliability according to the road surface condition during traveling based on the road surface recognition information stored in the storage unit 9. be. For example, when it is recognized that the road surface condition is uneven or a steep slope, it is output as low reliability, and conversely, when it is determined that the road surface is flat with few irregularities, it is output as high reliability. The reliability 52 is not limited to high reliability and low reliability, and may be used in multiple stages.

The search range calculation unit 11 changes the search range according to the reliability, as shown in FIG. For example, as shown in FIG. 4A, when the reliability 52 is higher than the threshold value, the search range in the tracking process is reduced as compared with the case where the reliability 52 is the threshold value. This makes it possible to reduce unnecessary CPU load. On the other hand, as shown in FIG. 4B, when the reliability 52 is lower than the threshold value, the search range in the tracking process is expanded as compared with the case where the reliability 52 is the threshold value. This makes it possible to take a large margin and respond to momentary changes in the pitch angle.

Based on the above configuration, FIG. 5 shows a flowchart for setting the search range of the recognition image in the tracking process using the pitch angle information acquired from the pitch angle sensor 2.

First, the detection target is detected from the recognition image, and an identifier is added to the detection target (S1). Then, the past tracking processing result is acquired from the storage unit 9 (S2). The movement direction of the detection target is predicted from the past tracking information, and the search range within the frame is set (S3).

The pitch angle information 50 of the pitch angle sensor 2 is acquired through the communication interface 8 (S4). Next, the change amount calculation unit 20 calculates the deviation information 51 of the detection target in the frame and its reliability 52 from the pitch angle information 50 (S5). Subsequently, it is determined whether the reliability 52 is high or low (S6). When the reliability 52 is larger than the threshold value, it is determined to be high, and when it is smaller than the threshold value, it is determined to be low.
Here, when it is determined that the reliability is low, the vertical direction of the search range set in the frame is expanded beyond the preset reference search range (S7). On the other hand, when it is determined that the reliability is high, the vertical direction of the search range is reduced from the preset reference search range (S8).

After that, the position of the search range is updated according to the deviation information 51 (S9). The tracking processing unit 10 searches within the updated search range, and more specifically, searches the entire search range to see if the detection target to which the identifier is added according to the updated search range exists in the search range. Then, when the detection target exists within the updated search range, the same identifier is added and confirmed (S10). Then, the result of the tracking process performed by the above flow is stored in the storage unit (S11).

The camera 1 of the present embodiment uses a tracking processing unit 10 that sets a search range of a target object for each of a plurality of frame images captured by the camera 1 and tracks the target object in chronological order, and pitch angle information of the vehicle. It also has a search range calculation unit 11 that changes the search range of the tracking processing unit 10.

More specifically, the camera 1 is generated by an image pickup unit 4 that captures an image, an image processing unit 5 that generates an image, a plurality of pitch angles output from the pitch angle sensor 2, and an image processing unit 5. It is provided with a recognition unit 6 that performs image recognition from an image. Then, the recognition unit 6 includes a tracking processing unit 10 that performs tracking processing for tracking an object in the frame in time series, a storage unit 9 that stores past recognition results, and deviation information 51 of the object in the frame from the pitch angle. It has a change amount calculation unit 20 for calculating.

The tracking processing unit 10 includes a search range calculation unit 11 that sets a search range in the tracking processing from the input from the change amount calculation unit 20 and the past recognition result. Then, the change amount calculation unit 20 calculates whether the deviation information 51 is accurate in addition to the deviation information 51, and outputs it as the reliability 52. The search range calculation unit 11 sets the search range in the recognized image by using the deviation information 51, the reliability 52, and the past tracking processing result.

According to the camera 1 of the present embodiment, the search range is set from the movement direction of the detection target predicted from the previous frame by the tracking process, and the pitch angle generated in the motorcycle is used for the set search range. , Change the search range of the tracking process. Then, a reliability 52 for determining whether the acquired pitch angle information is accurate is set, and the search range of the tracking process is made variable by the reliability 52.

The search range calculation unit 11 makes the search range variable according to the reliability 52. For example, if it is determined that the reliability 52 is low, the search range is expanded, and if it is determined that the reliability 52 is high, the search range is reduced.

The change amount calculation unit 20 calculates the deviation information (vertical change amount) 51 of the object in the recognition image from the pitch angle information 50 acquired from the pitch angle sensor 2 as the pixel amount, and calculates the calculation accuracy of the search range calculation unit 11. Improve.

Therefore, even a vehicle such as a motorcycle whose pitch angle changes significantly compared to an automobile can catch the detection target within the search range. Therefore, it is possible to improve the accuracy of the tracking process when the pitch angle occurs.

In the first embodiment, the pitch angle, which is the vehicle behavior that occurs in the front-rear direction of the vehicle, has been focused on, but even when the roll angle occurs in the left-right direction of the vehicle as shown in FIG. The search range can be expanded or contracted as shown in FIG. 6B by the same method based on the information 51 and the reliability 52, and the present invention is applied even when the pitch angle and the roll angle occur at the same time. It is possible.

[Example 2]
In the second embodiment, instead of acquiring the pitch angle information 50 from the pitch angle sensor 2, a method of acquiring the pitch angle information 50 from the output of the control device that controls the braking force and the driving force of the vehicle will be described. It is possible to calculate the pitch angle generated in the vehicle in real time from the output of the control device that controls the control driving force of the vehicle, and the present proposal can be applied to the vehicle that does not have the pitch angle sensor. The method is added to the first embodiment.

The configuration diagram of this embodiment is shown in FIG. The differences from Example 1 are as follows. Instead of the pitch angle sensor 2, a driving force control device 40 and a braking force control device 41 are used. Then, the recognition unit 6 acquires the control driving force control amount 53 of the driving force control device 40 and the braking force control device 41 via the communication interface 8. In this embodiment, the pitch angle information calculation unit 30 for calculating the pitch angle information 50 generated in the vehicle is added to the first embodiment.

A flowchart for calculating the pitch angle information 50 from the control driving force control amount 53 will be described with reference to FIG.

The detection target is detected from the recognition image, and an identifier is added to the detection target (S1). The past tracking processing result is acquired from the storage unit 9 (S2). Then, the moving direction of the detection target is predicted from the acquired past tracking information, and the search range within the frame is set (S3).

The control driving force control amount 53 is acquired from the driving force control device 40 and the braking force control device 41, and the pitch angle information calculation unit 30 calculates the pitch angle information 50 (S20). The calculation method of the pitch angle information 50 is performed by using a known method.

Next, the change amount calculation unit 20 calculates the deviation information 51 of the detection target in the frame and its reliability 52 from the pitch angle information 50 (S5). Subsequently, it is determined whether the reliability 52 is high or low (S6), and if it is determined that the reliability 52 is low, the search range is expanded in the vertical direction (S7). If it is determined that the reliability 52 is high, the vertical direction of the search range is reduced (S8). After that, the position of the search range is updated according to the deviation information 51 (S9).

The tracking processing unit 10 searches within the updated search range, and searches the entire search range to see if the detection target to which the identifier is added exists within the search range. Then, when the detection target exists, the same identifier is added and confirmed (S10). Then, the result of the tracking process performed by the above flow is stored in the storage unit 9 (S11). As described above, the process of S20 is added instead of the process of S4 of the first embodiment.

The image processing device of the second embodiment has a communication interface 8 for communicating with the control device of the vehicle, and the recognition unit 6 has a driving force control amount 53 from the driving force control device 40 and the braking force control device 41. Has a pitch angle information calculation unit 30 for calculating the pitch angle information 50 generated in the own vehicle by using the above. Then, the change amount calculation unit 20 calculates the deviation information 51 of the detection target in the frame by using the pitch angle information 50 output from the pitch angle information calculation unit 30.

As described above, in the second embodiment, the pitch angle information calculation unit 30 that calculates the pitch angle information 50 from the control driving force control amount 53 is added to the configuration of the first embodiment, so that the pitch angle sensor 2 is not provided. It is possible to set the search range for the vehicle in consideration of the vehicle behavior, and it is possible to improve the tracking accuracy.

[Example 3]
In the third embodiment, instead of acquiring the pitch angle information 50 from the pitch angle sensor 2 of the first embodiment, it is output from the control generation unit 7 that generates a control signal to an actuator such as a brake or an accelerator based on the recognition result. It is characterized in that the pitch angle information 50 is acquired from the target control driving force control amount 54. The third embodiment is configured to be completed within the camera 1, and does not require the pitch angle sensor of the first embodiment or the external control driving force control amount 53 as in the second embodiment. Therefore, it can be expected to minimize the work of conforming to each vehicle type. The method is added to the first embodiment.

The block diagram of Example 3 is shown in FIG. The differences from Example 1 are as follows.
Instead of the pitch angle sensor 2 of the first embodiment, a pitch angle information calculation unit 60 that calculates the pitch angle information 50 by inputting the target control driving force control amount 54 output from the control generation unit 7 is added.

Further, since the target driving force control amount 54 is a control amount calculated inside the camera, the change in the control amount due to the override when the driver operates the brake or the accelerator is not included. When an override occurs by the driver, a difference occurs between the pitch angle information 50 calculated by the target driving force control amount 54 and the pitch angle actually generated in the vehicle.

In this embodiment, it is necessary to devise a method for setting the search range when the driver's operation is detected. Specifically, a driver operation detection unit 70 has been added that acquires driver operation information 55 indicating whether or not the driver has operated the brake or accelerator via the communication interface 8 and outputs override information 56. ..

A flowchart for calculating the pitch angle information 50 from the target driving force control amount 54 will be described with reference to FIG.

First, the detection target is detected from the recognition image, and an identifier is added to the detection target (S1). The past tracking processing result is acquired from the storage unit 9 (S2). The movement direction of the detection target is predicted from the past tracking information, and the search range within the frame is set (S3).

The pitch angle information calculation unit 60 calculates the pitch angle information 50 from the target driving force control amount 54 calculated by the control generation unit 7 in the camera from the recognition result of the camera (S30). Next, the change amount calculation unit 20 calculates the deviation information 51 of the detection target in the frame and its reliability 52 from the pitch angle information 50 (S5).

The driver operation information 55 is acquired via the communication interface unit 8 (S31). With the driver operation information 55 as an input, it is determined whether or not the override is detected by the driver operation detection unit 70 (S32). When the driver's operation detection unit 70 detects the driver's override, the search range in the tracking process is expanded to the maximum (S33).

If the override is not detected, it is determined whether the reliability 52 is high or low (S6). Then, when it is determined that the reliability is low, the search range is expanded in the vertical direction (S7). If it is determined that the reliability is high, the vertical direction of the search range is reduced (S8). After that, the position of the search range is updated according to the deviation information 51 (S9).

The tracking processing unit 10 searches within the updated search range, and searches the entire search range to see if the detection target to which the identifier is added exists within the search range. Then, when the detection target exists, the same identifier is added and confirmed (S10). Then, the result of the tracking process performed by the above flow is stored in the storage unit 9 (S11).

As described above, in this embodiment, the processing of S30 is added instead of S4 in Example 1, and the processing of S31, S32, and S33 is further added.

In the third embodiment, the method of calculating the pitch angle information 50 from the target driving force control amount 54 is different, but a method of setting the search range of the tracking process using the deviation information 51 of the detection target and its reliability 52. Is the same as that of the first embodiment.

However, when the driver operation detection unit 70 detects the override, the pitch angle information 50 calculated from the target driving force control amount 54 has a difference from the pitch angle generated in the vehicle, so that the search range is accurate. Will disappear. Therefore, when an override is detected, it is determined that the reliability is the lowest, and the search range is expanded to the maximum.

According to the third embodiment, the camera 1 has a control generation unit 7 that calculates a control amount calculated internally for controlling the vehicle from the recognition result of the recognition unit 6, and the pitch angle information calculation unit 60 has a pitch angle information calculation unit 60. The pitch angle information 50 is calculated from the output of the control generation unit 7, and the deviation information 51 of the object in the frame is calculated from the pitch angle information 50 output from the pitch angle information calculation unit 60 by the change amount calculation unit 20.

The recognition unit 6 has a driver operation detection unit 70 that detects the driver's operation, and when an override is detected, the change amount calculation unit 20 outputs that the reliability is low.

The technology introduced in all the examples can be applied to both stereo cameras and monocular cameras. This proposal is also applicable to LiDAR, which outputs point cloud information. Further, although the pitch angle information is obtained by three methods in all the embodiments, it is not particularly limited as long as it is a means for calculating the pitch angle information in real time.

According to the cameras 1 of the first to third embodiments, the pitch angle can be set in the vehicle equipped with the recognition camera by adding a device corresponding to the pitch angle for the two-wheeled vehicle to the recognition camera developed for the four-wheeled vehicle. Even if it occurs, by setting the search range using the pitch angle information during the tracking process, it is possible to track the detection target in time series as long as the detection target exists in the captured image.

In addition, by adding a new reliability, the search range is expanded when sensor information such as the pitch angle sensor outputs an invalid value, and by increasing the margin, the detection target is out of the search range for tracking. It is possible to solve the problem that fails. If the reliability is high, the search range can be reduced and unnecessary calculation load can be eliminated. In the past, when the calculation load was high, processing such as thinning out the search range was performed, but this invention eliminates the need for processing, which contributes to improvement in recognition accuracy. In addition, effects such as suppressing the temperature rise of the hardware can be expected.

As a result, in a two-wheeled vehicle, stable tracking processing can be performed even when a pitch angle occurs in the vehicle.

According to the second embodiment, the search range can be set in consideration of the vehicle behavior even for a vehicle not provided with the pitch angle sensor, and according to the third embodiment, the present invention can be realized inside the camera. It can be expected to minimize the work of adapting to each vehicle type.
Further, in the first to third embodiments, the technique has been described focusing on the pitch angle generated in the vehicle, but the technique can be applied not only to the pitch angle but also to the case where the pitch angle and the roll angle are generated at the same time.

Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various designs are designed without departing from the spirit of the present invention described in the claims. You can make changes. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

1 ... Camera (image processing device)
2 ... Pitch angle sensor (IMU)
3 ... CAN bus 4 ... Imaging unit 5 ... Image processing unit 6 ... Recognition unit 7 ... Control generation unit 8 ... Communication interface unit 9 ... Storage unit 10 ... Tracking Processing unit 11 ... Search range calculation unit 20 ... Change amount calculation unit 30 ... Pitch angle information calculation unit 40 ... Driving force control device 41 ... Braking force control device 50 ... Pitch angle information 51 ... Misalignment information 52 ... Reliability 53 ... Control driving force control amount 54 ... Target control driving force control amount 55 ... Driver operation information 56 ... Override information 60 ... Pitch Angle information calculation unit 70 ... Driver operation detection unit

Claims

A tracking processing unit that sets a search range for a target object for each of a plurality of frame images captured by an in-vehicle camera and tracks the target object in chronological order.
An image processing device comprising a search range calculation unit that changes the search range using vehicle pitch angle information.
It has a change amount calculation unit that outputs the vertical change amount of the target object in the frame image from the pitch angle information.
The image processing apparatus according to claim 1, wherein the search range calculation unit updates the search range according to the amount of change in the vertical direction.
The change amount calculation unit calculates the reliability indicating whether the output of the vertical change amount is accurate.
The search range calculation unit reduces the search range when it is determined that the reliability is higher than the threshold value, and expands the search range when it is determined that the reliability is lower than the threshold value. The image processing apparatus according to claim 1.
The image processing apparatus according to claim 3, wherein the reliability is calculated based on the amount of change in the pitch angle per unit time.
The image processing device has a storage unit for storing past recognition results.
The image processing apparatus according to claim 3, wherein the reliability is set based on the road surface recognition information stored in the storage unit.
The image processing apparatus according to claim 1, further comprising a pitch angle information calculation unit that calculates pitch angle information from a target driving force calculated by the image processing apparatus.
The image processing device has a driver operation detection unit that detects an accelerator operation or a brake operation by the driver.
The image processing device according to claim 3, wherein the change amount calculation unit determines that the reliability is low when the driver's operation is detected by the driver operation detection unit.