WO2020162498A1

WO2020162498A1 - Information processing device

Info

Publication number: WO2020162498A1
Application number: PCT/JP2020/004366
Authority: WO
Inventors: 育子古村
Original assignee: 株式会社デンソー
Priority date: 2019-02-06
Filing date: 2020-02-05
Publication date: 2020-08-13
Also published as: JP2020127164A; JP7263807B2

Abstract

An information processing device (23) which is mounted on a vehicle and used is provided with: an execution unit (44) configured to execute calibration of cameras (11a, 11b, 11c, 11d) which are mounted on the vehicle and which photograph the periphery of the vehicle, on the basis of a photographed image (61) of the cameras; and a determination unit (45) configured to determine whether a prescribed condition is satisfied. The execution unit is configured to be capable of executing the calibration when the determination unit determines that the prescribed condition is satisfied.

Description

Information processing equipment

Cross-reference of related applications

This international application claims priority based on Japanese Patent Application No. 2019-19622 filed with the Japan Patent Office on February 6, 2019, and is the same as Japanese Patent Application No. 2019-19622. The entire contents of this International Application are incorporated by reference.

The present disclosure relates to an information processing device mounted on a vehicle and used.

When synthesizing a plurality of captured images acquired by a plurality of cameras mounted on a vehicle, the quality of the synthesized image can be improved by correcting the camera posture (hereinafter also referred to as calibration). it can. Patent Document 1 proposes a technique for performing calibration during traveling without using a specific marker for calibration.

JP, 2014-101075, A

However, as a result of a detailed study by the inventor, the calibration method of Patent Document 1 which does not use a specific marker has a problem that the calibration accuracy may be lowered depending on the situation where the vehicle is placed. Was done. Further, as a result of a detailed study by the inventor, it has been found that there is a case that an increase in processing load due to execution of calibration may affect other controls.

In one aspect of the present disclosure, it is preferable to perform calibration at an appropriate timing.

One aspect of the present disclosure is an information processing device mounted on a vehicle and used, and includes an execution unit and a determination unit. The execution unit is configured to execute calibration of a camera mounted on the vehicle to capture an image of the surroundings of the vehicle based on a captured image of the camera. The judgment unit is configured to judge whether a predetermined condition is satisfied. The execution unit is configured to be able to execute the calibration when the determination unit determines that the predetermined condition is satisfied.

With such a configuration, the execution of calibration can be limited to certain situations. Therefore, for example, by not performing the calibration in a situation where the calibration accuracy is low and the probability is high, it is possible to suppress a decrease in the calibration accuracy. Further, for example, by not performing the calibration in a situation where there is a high probability that the increase in the processing load due to the calibration will affect other controls, it is possible to suppress adverse effects on other controls.

FIG. 3 is a block diagram showing a configuration of an image processing unit corresponding to the information processing device of the first embodiment. It is a functional block diagram of an image processing unit. It is a figure explaining the outline of calibration. It is a flow chart of execution judgment processing of a 1st embodiment. It is a figure explaining the relationship between some control based on image processing, and traveling speed. It is a flow chart of execution judgment processing of a 2nd embodiment. It is a flow chart of execution judgment processing of a 3rd embodiment. It is a figure explaining the relationship between memory usage and processing speed fall. It is a flow chart of execution judgment processing of other embodiments.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[1. First Embodiment]
[1-1. Constitution]
A display system 1 shown in FIG. 1 is a system mounted on a vehicle and used, and includes a front camera (hereinafter, F camera) 11a, a rear camera (hereinafter, R camera) 11b, a left side camera (hereinafter, LS camera). 11c, four cameras of the light side camera (henceforth RS camera) 11d, the display 13, and ECU15 are provided. Hereinafter, when referring to all the above-mentioned cameras, they may be simply referred to as the camera 11.

The camera 11 is an imaging device mounted on the vehicle. For the camera 11, for example, a known CCD image sensor or CMOS image sensor can be used. The camera 11 photographs the surroundings of the vehicle at a predetermined time interval (1/15s as an example), and outputs the photographed image to the ECU 15. The F camera 11a photographs the front of the vehicle. The R camera 11b photographs the rear of the vehicle. The LS camera 11c photographs the left side of the vehicle. The RS camera 11d photographs the right side of the vehicle.

The display 13 is a display device such as a liquid crystal display that displays an image. The display 13 displays the combined image generated by the ECU 15 according to the signal input from the ECU 15.

The ECU 15 includes a video signal input unit 21, a communication interface (I/F) 22, an image processing unit 23, a video signal output unit 24, and a power supply unit 25.

The video signal input unit 21 inputs a video signal indicating a captured image captured by the camera 11 from the camera 11, and outputs it to the image processing unit 23.

The communication I/F 22 acquires a signal output to the vehicle-mounted communication bus 17 from one or more control devices or sensors (not shown) mounted on the vehicle and outputs the signal to the image processing unit 23. For example, the communication I/F 22 acquires information about the vehicle such as the traveling speed of the vehicle (hereinafter, vehicle speed), the steering angle of the tire, and the shift range.

The image processing unit 23 includes a microcomputer having a CPU 31 and a semiconductor memory (hereinafter, memory 32) such as a RAM 32a, a ROM 32b, and a flash memory 32c. Various functions of the image processing unit 23 are realized by the CPU 31 executing a program stored in the non-transitional physical recording medium. In this example, the memory 32 corresponds to a non-transitional substantive recording medium storing a program. Further, by executing this program, the method corresponding to the program is executed. The image processing unit 23 may include one microcomputer or a plurality of microcomputers. The image processing unit 23 corresponds to an information processing device.

The image processing unit 23 has an image generation unit 41, an image recognition unit 42, a drawing unit 43, and an execution unit 44, as shown in FIG. 2, as a configuration of functions realized by the CPU 31 executing a program. And a determination unit 45.

The image generation unit 41 generates one or more composite images based on the captured images captured by two or more cameras of the four cameras 11. The composite image is an image in which a plurality of captured images are combined, and the viewpoint of at least one captured image may be changed. For example, it may be an image of a viewpoint different from the captured image actually captured by the camera 11, or an image including a wider range than the captured image of one camera.

In order to generate a composite image, it is necessary to specify the viewpoint of each captured image. In order to specify the viewpoint, parameters of the posture of the camera including the position and angle of each camera 11 installed are used. The above-mentioned parameters of each camera 11 are stored in the storage area formed by the memory 32, and are read and used by the image generation unit 41 when necessary.

The image recognition unit 42 recognizes a predetermined target object from the composite image generated by the image generation unit 41. The predetermined object here may include, for example, a lane marking such as a white line, a moving body such as a person, an automobile, or a bicycle, or a fixed object such as a signal or a sign. The method of recognizing the object from the image is not particularly limited, and various known methods can be used. For example, the target object may be recognized by pattern matching or calculation processing using a learning model. The detected object may be used for various controls.

The drawing unit 43 corrects the brightness and color of the composite image, and when output to the display 13, generates an image that is easily seen by passengers of the vehicle.

The execution unit 44 executes the calibration of the camera 11 based on the image captured by the camera 11 while the vehicle is traveling.

Calibration is to correct the camera posture parameters such as the position and angle of the camera 11 described above. The specific content of the calibration may be, for example, estimating the installation position of the camera 11 with respect to the vehicle and replacing it with the parameter stored in the storage area of the memory 32. It is also possible to recalculate a new parameter using the calculated parameter. In the present embodiment, the calibration is executed without using the marker dedicated to the calibration. The outline of the calibration method will be briefly described with reference to FIG.

In the photographed image 61, a first target image 62 is shown as an image showing the target 73 on the road photographed by the camera 72 fixed to the vehicle 71 at a certain timing t1. When the vehicle travels and the position of the vehicle changes, the position of the target 73 on the captured image changes. An image in which the target 73 captured at the timing t2 after the timing t1 is schematically superimposed on the captured image 61 is referred to as a second target image 63. Estimating the parameters of the camera attitude based on the changes in the positions of the first target image 62 and the second target image 63 on the image, and the movement amount of the vehicle 71 during the period from the timing t1 to the timing t2. You can

Note that the specific calibration method is not particularly limited, and for example, the method disclosed in Japanese Patent Application Publication No. 2014-101075 may be used.

The judgment unit 45 judges whether or not a predetermined condition is satisfied. The predetermined condition is a condition suitable for executing the calibration. The execution unit 44 is configured to be able to execute the calibration when the determination unit 45 determines that the predetermined condition is satisfied.

In the present embodiment, the predetermined condition corresponds to (i) not being a bad environment and (ii) recognizing a white line.

A bad environment is an environment in which it is highly probable that the accuracy will decrease when calibration is performed due to the poor quality of captured images. Examples of the bad environment include that the environment around the vehicle is dark, that the road on which the vehicle is traveling is uneven, and that the camera 1 is dirty. Examples of the uneven road include a road having a large number of irregularities on the road surface and a road having a large number of small obstacles such as pebbles.

　The specific method for detecting a bad environment is not particularly limited. The dark environment around the vehicle can be determined based on, for example, an image captured by the camera 11, or based on the output of an illuminance sensor (not shown) mounted on the vehicle. In addition, the fact that the road on which the vehicle is traveling is uneven may be determined, for example, by detecting the presence of stones or dents by image processing of the captured image, or by a vibration sensor or acceleration not shown mounted on the vehicle. The determination can be made based on the output of the sensor. Further, the fact that the camera 11 is dirty can be determined, for example, from the change with time of the captured image acquired by the camera 11.

The white line can be recognized by the image recognition unit 42. Since the white line is suitable as a target for executing calibration, detection of the white line may be a requirement for executing good calibration.

Return to Figure 1 for explanation. The video signal output unit 24 outputs the combined image generated by the image processing unit 23 to the display 13. The power supply unit 25 supplies electric power to the image processing unit 23 as well as each of the elements configuring the ECU 15.

[1-2. processing]
Next, the execution determination process executed by the CPU 31 of the image processing unit 23 will be described with reference to the flowchart of FIG.

First, in S1, the CPU 31 determines whether or not the vehicle is traveling. For example, the CPU 31 determines that the vehicle is traveling at a predetermined speed (for example, 5 km/h) or more based on the vehicle speed information acquired from a vehicle speed sensor (not shown).

When the CPU 31 determines in S1 that the vehicle is traveling, the CPU 31 proceeds to S2 and acquires a captured image around the vehicle from the camera 11. On the other hand, when the CPU 31 determines in S1 that the vehicle is not traveling, the process of FIG. 4 ends.

In subsequent S3 and S4, the CPU 31 determines whether or not a predetermined condition for permitting execution of calibration is satisfied.

Specifically, in S3, the CPU 31 determines whether the environment of the vehicle is a bad environment. Here, if any one of the requirements that the environment around the vehicle is dark, that the road on which the vehicle is traveling is uneven, and that the camera is dirty, is satisfied in a bad environment. It is determined that there is. When the CPU 31 determines in S3 that the environment is bad, the CPU 31 returns to S1. On the other hand, when the CPU 31 determines in S3 that the environment is not bad, the CPU 31 proceeds to S4.

Next, in S4, the CPU 31 determines whether or not a white line is detected. The detection of the white line itself is realized as a function of the image recognition unit 42. When determining that the white line is detected in S4, the CPU 31 proceeds to S5. On the other hand, when the CPU 31 determines in S4 that the white line is not detected, the CPU 31 returns to S1.

In this way, the CPU 31 functions as the determination unit 45 described above, and if the predetermined condition is not satisfied, the process returns to S1, and if the predetermined condition is satisfied, the process proceeds to S5.

Next, in S5, the CPU 31 executes calibration. Here, the CPU 31 functions as the execution unit 44 described above, and executes the calibration described above.

Next, in S6, the CPU 31 updates the camera parameters stored in the memory 32 to the parameters estimated by the calibration in S5. After S6, the process returns to S1.

[1-3. effect]
According to the first embodiment described in detail above, the following effects are achieved.

(1a) The image processing unit 23 of the display system 1 can execute the calibration when there is a high probability that the calibration can be suitably executed. Therefore, by not performing the calibration in a situation where the calibration accuracy is low and the probability is high, it is possible to suppress deterioration of the calibration accuracy.

(1b) The image processing unit 23 determines whether or not a predetermined condition for performing calibration is satisfied based on at least one of the image captured by the camera 11 and the output of the sensor mounted on the vehicle. You can

(1c) The determination unit 45 satisfies at least one of the requirements that the environment around the vehicle is dark, that the road on which the vehicle is traveling is uneven, and that the camera 11 is dirty. If it does, it is determined that the conditions for executing the calibration are not satisfied. If the above requirements are satisfied, the accuracy of the calibration tends to be low, but if the above requirements are not satisfied, the calibration is not permitted, so that it is possible to suppress the calibration with low accuracy.

(1d) The requirement for the determination unit 45 to determine that the predetermined condition is satisfied includes at least that the captured image acquired by the camera 11 includes a white line. Therefore, the white line can be used as a calibration target, and the calibration accuracy can be improved.

[1-4. Modification of First Embodiment]
In the first embodiment, the configuration in which the white line is used as the calibration target is illustrated, but the target may be something other than the white line. For example, lane markings other than the white line, road markings, curbs, and various other objects on the road can be used as targets. When an object other than the white line is used as the target, it may be determined whether or not the target object is detected instead of the white line in the process of S4 of FIG.

[2. Second Embodiment]
[2-1. Differences from the first embodiment]
The second embodiment has the same basic configuration as that of the first embodiment, and therefore the differences will be described below. The same reference numerals as those in the first embodiment indicate the same configurations, and refer to the preceding description.

In the above-described first embodiment, as a requirement for determining whether or not to perform the calibration, it is illustrated that the environment is a bad environment and that the white line is recognized. On the other hand, the second embodiment is different from the first embodiment in that the traveling speed of the vehicle is included as a requirement for determining whether or not a predetermined condition is satisfied. Further, in the second embodiment, as shown in FIG. 5, various controls using the result of image processing of the captured image are executed according to the vehicle speed. In the present embodiment, a plurality of controls are executed in the low speed range of less than 30 km/h, but those controls are not executed in the medium speed range of 30 km/h to 50 km/h, and in the high speed range of 50 km/h or more. Perform lane detection.

[2-2. processing]
The execution determination process executed by the CPU 23 of the second embodiment in place of the execution determination process of FIG. 4 of the first embodiment will be described with reference to the flowchart of FIG. Since the processing of S11, S12, S14, and S15 in FIG. 6 is the same as the processing of S1, S2, S5, and S6 in FIG. 4, the description is partly simplified.

In the execution determination process of FIG. 6, the CPU 31 proceeds to S13 after acquiring the captured image 61 in S12.

In S13, the CPU 31 determines whether the vehicle speed is in the medium speed range or higher. The vehicle speed can be detected, for example, based on the output signal of a vehicle speed sensor (not shown). Since the image processing unit 23 executes a plurality of controls in the low speed range, the processing load on the image processing unit 23 is high. If calibration is further performed here, there is a high possibility that a delay will occur in other control. Therefore, the CPU 31 does not permit execution of the calibration while traveling in the low speed range. On the other hand, reference numeral 31 permits the execution of calibration because the processing load is small during traveling in the medium speed range and the high speed range. The fact that the vehicle travels in the medium speed range and the high speed range corresponds to the vehicle being in a predetermined state.

When the CPU 31 determines in S13 that the vehicle speed is not higher than the medium speed range, the CPU 31 returns to S11. On the other hand, when the CPU 31 determines in S13 that the vehicle speed is in the medium speed range or higher, the CPU 31 proceeds to S14 and executes the calibration.

[2-3. effect]
According to the second embodiment described in detail above, the following effects are obtained.

(2a) The image processing unit 23 of the display system 1 can execute the calibration when the processing load due to the calibration is unlikely to affect other controls. Therefore, it is possible to prevent the execution of the calibration from adversely affecting other controls.

[2-4. Modification of Second Embodiment]
The second embodiment exemplifies a configuration in which execution of calibration is permitted only when the vehicle speed is in the medium speed range or higher. However, the speed range in which the execution of calibration is permitted is not limited to the above example. For example, the configuration may be such that the execution of calibration is permitted only in a certain speed range. As an example, the execution of calibration may be permitted only in any one of the low speed range, the medium speed range, and the high speed range. Further, the threshold value of the speed for determining whether or not the calibration can be executed is not limited to the speed example disclosed in the second embodiment, and may be various values.

[3. Third Embodiment]
[3-1. Differences from the first embodiment]
The basic configuration of the third embodiment is similar to that of the first and second embodiments, and therefore the differences will be described below. The same reference numerals as those in the first embodiment indicate the same configurations, and refer to the preceding description.

In the third embodiment, the requirement for determining whether or not a predetermined condition is satisfied is that the processing load of the image processing unit 23 is small, specifically, the memory usage amount of the image processing unit 23 is included. This is different from the first embodiment. In the third embodiment, “memory usage” refers to the usage of the RAM 32a. The image processing unit 23 includes software that measures the amount of memory used. Further, in the third embodiment, similar to the second embodiment, various controls using the result of image processing of a captured image are executed.

[3-2. processing]
The execution determination process executed by the CPU 23 of the third embodiment in place of the execution determination process of FIG. 4 of the first embodiment will be described with reference to the flowchart of FIG. 7. Note that the processing of S21, S22, S24, and S25 in FIG. 7 is similar to the processing of S1, S2, S5, and S6 in FIG. 4, so the description is partially simplified.

In the execution determination process of FIG. 7, the CPU 31 acquires the captured image 61 in S22, and then moves to S23.

In S23, the CPU 31 determines whether or not the memory usage amount before the calibration is performed is equal to or larger than the reference value. The reference value here is, for example, as shown in FIG. 8, a value of the memory usage amount before the calibration is performed, which is estimated to cause a decrease in the processing speed. Pattern A in FIG. 8 is an example of the memory usage amount in a range that does not affect the processing speed. Therefore, in this case, execution of calibration is permitted. On the other hand, pattern B in FIG. 8 is an example of the memory usage amount in which the decrease in processing speed is estimated. In this case, execution of calibration is not permitted. The memory usage amount being less than the predetermined reference value corresponds to satisfying the predetermined reference.

When the CPU 31 determines in S23 that the memory usage amount is equal to or greater than the reference value, the CPU 31 returns to S21. On the other hand, when the CPU 31 determines in S23 that the memory usage amount is less than the reference value, the CPU 31 proceeds to S24 and executes calibration.

[3-3. effect]
According to the third embodiment described in detail above, the following effects are obtained.

(3a) The image processing unit 23 of the display system 1 can execute the calibration when the memory usage amount is less than the reference value. Therefore, it is possible to prevent the execution of the calibration from adversely affecting other controls.

[3-4. Modification of Third Embodiment]
In the third embodiment, the reference value is an example of a memory usage amount estimated to cause a decrease in the processing speed of the image processing unit 23. However, the reference value is not limited to the above configuration. For example, in consideration of the increase in the memory usage amount by performing the calibration, the reference value is a threshold value of the memory usage amount at which the processing speed is significantly reduced when the increase amount is added to the reference value. It may be a value of the memory usage amount that will be exceeded.

Further, when a predetermined criterion other than the memory usage amount is satisfied, it may be detected that the processing load of the image processing unit 23 is small. For example, it may be determined that the processing load of the image processing unit 23 is small when a specific control having a large processing load is not executed or when the number of executed controls is less than or equal to a predetermined number.

[4. Other Embodiments]
Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments, and various modifications can be implemented.

(4a) In the first embodiment, the predetermined condition is set so that the calibration can be executed at the timing at which the accuracy of the calibration is not deteriorated. In the second and third embodiments, other control by the calibration is performed. Predetermined conditions were set so that the influence on the above can be suppressed and the calibration can be executed. However, the predetermined condition may be set so that both effects described above can be obtained. For example, like the execution determination process shown in the flowchart of FIG. 9, it may be configured to perform all the determinations of S3, S4, S13, and S23 described above. With such a configuration, it is possible to simultaneously realize a reduction in calibration accuracy and an influence on other controls.

(4b) In the above-described second embodiment, the configuration in which it is determined that the predetermined condition for permitting the execution of the calibration is determined based on the traveling speed of the vehicle has been exemplified, but the configuration is acquired based on the output of the vehicle-mounted sensor. Whether or not the predetermined condition is satisfied may be determined based on the state of the vehicle other than the determined vehicle speed. For example, the requirements for determining that a predetermined condition is satisfied, the vehicle is a state in which a light such as a headlight is not lit, a state in which a moving body is not detected in at least a certain range outside the vehicle, May be included. The fact that the traveling speed of the vehicle is within a predetermined range, that the vehicle travels without turning on the light, and that there is no moving object within the predetermined range correspond to examples where the vehicle is in the predetermined state. ..

(4c) The image processing unit 23 and the method thereof according to the present disclosure are provided by configuring a processor and a memory programmed to execute one or a plurality of functions embodied by a computer program. It may be realized by a dedicated computer. Alternatively, the image processing unit 23 and the method thereof described in the present disclosure may be realized by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the image processing unit 23 and its method described in the present disclosure include a processor and a memory programmed to execute one or a plurality of functions, and a processor configured by one or more hardware logic circuits. It may be realized by one or more dedicated computers configured by combination. Further, the computer program may be stored in a computer-readable non-transition tangible recording medium as an instruction executed by a computer. The method for realizing the function of each unit included in the image processing unit 23 does not necessarily need to include software, and all the functions may be realized by using one or a plurality of hardware.

(4d) A plurality of functions of one constituent element in the above-described embodiment may be realized by a plurality of constituent elements, or one function of one constituent element may be realized by a plurality of constituent elements. .. Further, a plurality of functions of a plurality of constituent elements may be realized by one constituent element, or one function realized by a plurality of constituent elements may be realized by one constituent element. Moreover, you may omit a part of structure of the said embodiment. Further, at least a part of the configuration of the above-described embodiment may be added or replaced with respect to the configuration of the other above-described embodiment.

(4e) In addition to the image processing unit 23 described above, a system having the image processing unit 23 as a constituent element, a program for causing a computer to function as the image processing unit 23, and a non-transitional type such as a semiconductor memory storing the program The present disclosure can also be realized in various forms such as an actual recording medium and a calibration execution method.

Claims

An information processing device (23) mounted on a vehicle and used,
An execution unit (44) configured to execute the calibration of the cameras (11a, 11b, 11c, 11d) mounted on the vehicle and capturing the surroundings of the vehicle based on the captured image (61) of the camera. )When,
A determination unit (45) configured to determine whether or not a predetermined condition is satisfied,
The information processing apparatus, wherein the execution unit is configured to be able to execute the calibration when the determination unit determines that the predetermined condition is satisfied.
The information processing apparatus according to claim 1, wherein
The predetermined condition is an information processing device, which is a condition determined based on at least one of a captured image acquired by the camera and an output of a sensor mounted on the vehicle.
The information processing apparatus according to claim 2, wherein
At least one of the requirement that the environment surrounding the vehicle is dark, the road on which the vehicle is traveling is uneven, and the camera is dirty is satisfied by the determination unit. At times, the information processing apparatus is configured to determine that the predetermined condition is not satisfied.
The information processing apparatus according to claim 2 or 3, wherein
The requirement for the determination unit to determine that the predetermined condition is satisfied includes at least the target (62, 63) that can be used as the calibration reference in the captured image acquired by the camera. An information processing device including:
The information processing apparatus according to any one of claims 2 to 4, wherein:
The information processing apparatus, wherein the requirement for the determination unit to determine that the predetermined condition is satisfied includes that at least the state of the vehicle acquired based on the output of the sensor is a predetermined state.
The information processing apparatus according to any one of claims 1 to 5, wherein
The information processing apparatus, wherein the requirement for the determining unit to determine that the predetermined condition is satisfied includes at least satisfying a predetermined criterion indicating that the processing load of the information processing apparatus is small.