JP2020127164A - Information processing device - Google Patents

Abstract

To provide a technique capable of performing calibration at suitable timing.SOLUTION: An execution unit 44 provided in an image processing unit 23 that is an information processing device is configured to execute calibration of a camera 11 loaded on a vehicle to photograph the surrounding of the vehicle, on the basis of an image 61 captured by the camera 11. A determination unit 45 provided in the image processing unit 23 is configured to determine whether a predetermined condition is satisfied or not. An execution unit 44 is configured to be able to execute calibration when the determination unit 45 determines that the predetermined condition is satisfied.SELECTED DRAWING: Figure 2

Description

The present disclosure relates to an information processing device mounted on a vehicle and used.

When combining a plurality of captured images acquired by a plurality of cameras mounted on a vehicle, it is possible to improve the quality of the combined image by correcting the camera posture (hereinafter, also referred to as calibration). it can. Patent Document 1 proposes a technique for performing calibration during traveling without using a specific marker for calibration.

JP, 2014-101075, A

However, as a result of a detailed study by the inventor, the calibration method of Patent Document 1 that does not use a specific marker has a problem that the accuracy of calibration may be lowered depending on the situation where the vehicle is placed. Was done. Further, as a result of a detailed study by the inventor, it was found that there is a case that an increase in the processing load due to the execution of calibration may affect other controls.

One aspect of the present disclosure provides a technique capable of performing calibration at appropriate timing.

One aspect of the present disclosure is an information processing device that is used by being mounted on a vehicle, and includes an execution unit and a determination unit. The execution unit is configured to execute calibration of a camera mounted on the vehicle to capture an image of the surroundings of the vehicle based on a captured image of the camera. The judgment unit is configured to judge whether a predetermined condition is satisfied. The execution unit is configured to be able to execute the calibration when the determination unit determines that the predetermined condition is satisfied.

With such a configuration, the execution of calibration can be limited to a certain situation. Therefore, for example, by not performing the calibration in a situation in which the calibration accuracy is low and the probability is high, it is possible to suppress a decrease in the calibration accuracy. Further, for example, by not performing calibration in a situation in which it is highly likely that other controls will be affected by an increase in the processing load due to calibration, it is possible to suppress adverse effects on other controls.

FIG. 3 is a block diagram showing a configuration of an image processing unit corresponding to the information processing apparatus of the first embodiment. It is a functional block diagram of an image processing unit. It is a figure explaining the outline of calibration. It is a flow chart of execution judgment processing of a 1st embodiment. It is a figure explaining the relationship of several control based on image processing, and traveling speed. It is a flow chart of execution judgment processing of a 2nd embodiment. It is a flow chart of execution judgment processing of a 3rd embodiment. It is a figure explaining the relationship between memory usage and processing speed fall. It is a flow chart of execution judgment processing of other embodiments.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.
[1. First Embodiment]
[1-1. Constitution]
A display system 1 shown in FIG. 1 is a system mounted on a vehicle and used, and includes a front camera (hereinafter, F camera) 11a, a rear camera (hereinafter, R camera) 11b, a left side camera (hereinafter, LS camera). 11c, four cameras of the light side camera (henceforth RS camera) 11d, the display 13, and ECU15 are provided. Hereinafter, when referring to all the above-mentioned cameras, it may be simply referred to as the camera 11.

The camera 11 is an image pickup device mounted on a vehicle, and can use, for example, a known CCD image sensor or CMOS image sensor. The camera 11 photographs the surroundings of the vehicle at a predetermined time interval (1/15 s as an example), and outputs the photographed image to the ECU 15. The F camera 11a, the R camera 11b, the LS camera 11c, and the RS camera 11d are arranged so as to photograph the front, rear, left, and right sides of the vehicle, respectively.

The display 13 is a display device such as a liquid crystal display that displays an image, and displays a composite image generated by the ECU 15 according to a signal input from the ECU 15.
The ECU 15 includes a video signal input unit 21, a communication interface (I/F) 22, an image processing unit 23, a video signal output unit 24, and a power supply unit 25.

The video signal input unit 21 inputs a video signal indicating a captured image captured by the camera 11 from the camera 11 and outputs it to the image processing unit 23.
The communication I/F 22 acquires a signal output to the vehicle-mounted communication bus 17 from one or more control devices, sensors, or the like (not shown) mounted on the vehicle and outputs the signal to the image processing unit 23. For example, information about the vehicle such as the traveling speed of the vehicle (hereinafter referred to as the vehicle speed), the steering angle of the tire, and the shift range is acquired.

The image processing unit 23 includes a microcomputer including a CPU 31 and a semiconductor memory (hereinafter, memory 32) such as a RAM 32a, a ROM 32b, and a flash memory 32c. Various functions of the image processing unit 23 are realized by the CPU 31 executing a program stored in the non-transitional physical recording medium. In this example, the memory 32 corresponds to a non-transitional substantive recording medium storing a program. Further, by executing this program, the method corresponding to the program is executed. The image processing unit 23 may include one microcomputer or a plurality of microcomputers. The image processing unit 23 corresponds to an information processing device.

The image processing unit 23 has an image generating unit 41, an image recognizing unit 42, a drawing unit 43, and an executing unit 44, as shown in FIG. 2, as a configuration of functions realized by the CPU 31 executing a program. And a determination unit 45.

The image generation unit 41 generates one or more composite images based on the captured images captured by two or more cameras of the four cameras 11. The composite image is an image in which a plurality of captured images are combined, and the viewpoint of at least one captured image may be changed. For example, it may be an image of a viewpoint different from the captured image actually captured by the camera 11, or an image including a wider range than the captured image of one camera.

In order to generate a composite image, it is necessary to specify the viewpoint of each captured image. In order to specify the viewpoint, parameters of the posture of the camera including the position and angle of each camera 11 installed are used. The above-mentioned parameters of each camera 11 are stored in the storage area formed by the memory 32, and are read and used by the image generation unit 41 when necessary.

The image recognition unit 42 recognizes a predetermined target object from the composite image generated by the image generation unit 41. The predetermined object here may include, for example, lane markings such as white lines, moving objects such as people, automobiles, and bicycles, fixed objects such as signals and signs, and the like. The method of recognizing the object from the image is not particularly limited, and various known methods can be used. For example, the target object may be recognized by pattern matching or calculation processing using a learning model. The detected object may be used for various controls.

The drawing unit 43 corrects the brightness and color of the combined image and generates an image that is easily seen by the occupants of the vehicle when output to the display 13.
The executing unit 44 executes the calibration of the camera 11 based on the captured image of the camera 11 while the vehicle is traveling.

The calibration is to correct the parameters of the camera posture such as the position and angle of the camera 11 described above. Specifically, the installation position of the camera 11 with respect to the vehicle is estimated and replaced with the parameter stored in the storage area of the memory 32, or the conventional parameter and the newly estimated parameter are used as new parameters. To recalculate. In the present embodiment, the calibration is executed without using the marker dedicated to the calibration. The outline of the calibration method will be briefly described with reference to FIG.

In the captured image 61, the first target image 62 is shown as an image showing the target 73 on the road captured by the camera 72 fixed to the vehicle 71 at a certain timing t1. When the vehicle travels and the position of the vehicle changes, the position of the target 73 on the captured image changes. An image in which the target 73 captured at the timing t2 after the timing t1 is schematically superimposed on the captured image 61 is referred to as a second target image 63. Estimating the parameters of the camera attitude based on changes in the positions of the first target image 62 and the second target image 63 on the image, and the movement amount of the vehicle 71 during the period from the timing t1 to the timing t2. You can
Note that the specific calibration method is not particularly limited, and for example, the method disclosed in JP2014-101075A may be used.

The judgment unit 45 judges whether or not a predetermined condition is satisfied. The predetermined condition is a condition suitable for executing the calibration. The execution unit 44 is configured to be able to execute the calibration when the determination unit 45 determines that the predetermined condition is satisfied.

In the present embodiment, the predetermined condition corresponds to (i) not being a bad environment and (ii) recognizing a white line.
The bad environment is an environment in which it is considered highly likely that the accuracy will be low when the calibration is executed due to the low quality of the captured image. For example, the environment around the vehicle is dark, the road on which the vehicle is traveling is uneven, and the camera 1 is dirty. Examples of uneven roads include roads having many irregularities on the road surface and roads having many small obstacles such as pebbles.

The specific method for detecting the bad environment is not particularly limited. The dark environment around the vehicle can be determined based on, for example, an image captured by the camera 11, or based on the output of an illuminance sensor (not shown) mounted on the vehicle. In addition, the fact that the road on which the vehicle is traveling is uneven may be determined, for example, by detecting the presence of stones or dents by performing image processing on the captured image, or by a vibration sensor or acceleration not shown mounted on the vehicle. The determination can be made based on the output of the sensor. Further, the fact that the camera 11 is dirty can be determined, for example, from the change with time of the captured image acquired by the camera 11.

The white line can be recognized by the image recognition unit 42. Since the white line is suitable as a target for executing calibration, detection of the white line may be a requirement for executing good calibration.

The description returns to FIG. The video signal output unit 24 outputs the composite image generated by the image processing unit 23 to the display 13. The power supply unit 25 supplies electric power to the image processing unit 23 and each of the elements configuring the ECU 15.

[1-2. processing]
Next, the execution determination process executed by the CPU 31 of the image processing unit 23 will be described with reference to the flowchart of FIG.

First, in S1, the CPU 31 determines whether or not the vehicle is traveling. For example, the CPU 31 determines that the vehicle is traveling at a predetermined speed (for example, 5 km/h) or more based on the vehicle speed information acquired from a vehicle speed sensor (not shown).

When the CPU 31 determines in S1 that the vehicle is traveling, the CPU 31 proceeds to S2 and acquires a captured image of the surroundings of the vehicle from the camera 11. On the other hand, when the CPU 31 determines in S1 that the vehicle is not traveling, the process of FIG. 4 ends.

In subsequent S3 and S4, the CPU 31 determines whether or not a predetermined condition for permitting execution of calibration is satisfied.
Specifically, in S3, the CPU 31 determines whether the environment of the vehicle is a bad environment. Here, if the environment around the vehicle is dark, the road on which the vehicle is running is uneven, and the camera is dirty, then the environment is bad. It is determined that there is. When the CPU 31 determines in S3 that the environment is bad, the CPU 31 returns to S1. On the other hand, when the CPU 31 determines in S3 that the environment is not bad, the CPU 31 proceeds to S4.

In subsequent S4, the CPU 31 determines whether or not a white line is detected. The detection of the white line itself is realized as a function of the image recognition unit 42. When determining that the white line is detected in S4, the CPU 31 proceeds to S5. On the other hand, when the CPU 31 determines in S4 that the white line is not detected, the CPU 31 returns to S1.

In this way, the CPU 31 functions as the above-mentioned determination unit 45, and if the predetermined condition is not satisfied, the process returns to S1, and if the predetermined condition is satisfied, the process proceeds to S5.
Next, in S5, the CPU 31 executes calibration. Here, the CPU 31 functions as the execution unit 44 described above, and executes the calibration described above.

Next, in S6, the CPU 31 updates the camera parameters stored in the memory 32 to the parameters estimated by the calibration in S5. After S6, the process returns to S1.

[1-3. effect]
According to the first embodiment described in detail above, the following effects are obtained.
(1a) The image processing unit 23 of the display system 1 can execute the calibration when there is a high probability that the calibration can be suitably executed. Therefore, by not performing the calibration in a situation where the calibration accuracy is low and the probability is high, it is possible to suppress deterioration of the calibration accuracy.

(1b) The image processing unit 23 determines whether or not a predetermined condition for performing calibration is satisfied based on at least one of an image captured by the camera 11 and an output of a sensor mounted on the vehicle. You can

(1c) The determination unit 45 satisfies at least one of the requirements that the environment around the vehicle is dark, that the road on which the vehicle is traveling is uneven, and that the camera 11 is dirty. If it does, it is determined that the conditions for executing the calibration are not satisfied. If the above requirements are met, the calibration accuracy tends to be low, but if the above requirements are not met, the calibration is not permitted, so that it is possible to prevent the calibration with low accuracy from being executed.

(1d) The determination unit 45 determines at least that the captured image acquired by the camera 11 includes a white line as a requirement for determining that a predetermined condition is satisfied. Therefore, the white line can be used as the calibration target, and the calibration accuracy can be improved.

[1-4. Modification of First Embodiment]
In the first embodiment, the configuration in which the white line is used as the calibration target has been exemplified, but the target may be something other than the white line. For example, lane markings other than white lines, road markings, curbs, and various other objects on the road can be used as targets. When an object other than the white line is used as the target, in the process of S4 of FIG. 4, instead of the white line, it may be determined whether or not the target object is detected.

[2. Second Embodiment]
[2-1. Differences from the first embodiment]
The second embodiment has the same basic configuration as that of the first embodiment, and therefore the differences will be described below. The same reference numerals as those in the first embodiment indicate the same configurations, and refer to the preceding description.

In the above-described first embodiment, as a requirement for determining whether or not to perform the calibration, it is illustrated that the environment is a bad environment and that the white line is recognized. On the other hand, the second embodiment is different from the first embodiment in that the traveling speed of the vehicle is a requirement for determining whether or not a predetermined condition is satisfied. Further, in the second embodiment, as shown in FIG. 5, various controls using the result of image processing of the captured image are executed according to the vehicle speed. In the present embodiment, a plurality of controls are executed in a low speed range of less than 30 km/h, but those controls are not executed in a medium speed range of 30 km/h to 50 km/h, and in a high speed range of 50 km/h or more. Perform lane detection.

[2-2. processing]
The execution determination process executed by the CPU 23 of the second embodiment in place of the execution determination process (FIG. 4) of the first embodiment will be described with reference to the flowchart of FIG. Note that the processing of S11, S12, S14, and S15 in FIG. 6 is the same as the processing of S1, S2, S5, and S6 in the drawing, so the description is partially simplified.

In the execution determination process of FIG. 6, after the captured image 61 is acquired in S12, the CPU 31 proceeds to S13.
In S13, the CPU 31 determines whether the vehicle speed is in the medium speed range or higher. The vehicle speed can be detected, for example, based on the output signal of a vehicle speed sensor (not shown). Since the image processing unit 23 executes a plurality of controls in the low speed range, the processing load of the image processing unit 23 is high. If calibration is further performed here, there is a high possibility that a delay will occur in other control. Therefore, the CPU 31 does not permit execution of the calibration while traveling in the low speed range. On the other hand, 31 indicates that the processing load is small during traveling in the medium speed range and the high speed range, so that the execution of calibration is permitted.

When the CPU 31 determines in S13 that the vehicle speed is not in the medium speed range or higher, the CPU 31 returns to S11. On the other hand, when the CPU 31 determines in S13 that the vehicle speed is in the medium speed range or higher, the CPU 31 proceeds to S14 and executes calibration.

[2-3. effect]
According to the second embodiment described in detail above, the following effects are obtained.
(2a) The image processing unit 23 of the display system 1 can execute the calibration when the processing load due to the calibration is unlikely to affect other controls. Therefore, it is possible to prevent the execution of calibration from adversely affecting other controls.

[2-4. Modification of Second Embodiment]
The second embodiment exemplifies the configuration in which the execution of the calibration is permitted only when the vehicle speed is in the medium speed range or higher. However, the speed range in which the execution of calibration is permitted is not limited to the above example. For example, the configuration may be such that the execution of calibration is permitted only in a certain speed range. As an example, the calibration may be permitted to be executed only in any one of the low speed range, the medium speed range, and the high speed range. Further, the speed threshold value for determining whether or not to execute the calibration is not limited to the speed example disclosed in the second embodiment, and may be various values.

[3. Third Embodiment]
[3-1. Differences from the first embodiment]
The basic configuration of the third embodiment is similar to that of the first and second embodiments, and therefore the differences will be described below. The same reference numerals as those in the first embodiment indicate the same configurations, and refer to the preceding description.

In the third embodiment, the processing load of the image processing unit 23 is small, and specifically, the memory usage of the image processing unit 23 is a requirement for determining whether or not a predetermined condition is satisfied. Different from the embodiment. In the third embodiment, the “memory usage” refers to the usage of the RAM 32a. The image processing unit 23 includes software that measures the amount of memory used. Further, in the third embodiment, as in the second embodiment, various controls using the result of image processing of a captured image are executed.

[3-2. processing]
The execution determination process executed by the CPU 23 of the third embodiment in place of the execution determination process (FIG. 4) of the first embodiment will be described with reference to the flowchart of FIG. 7. Note that the processing of S21, S22, S24, and S25 in FIG. 7 is the same as the processing of S1, S2, S5, and S6 in the drawing, so the description is partially simplified.

In the execution determination process of FIG. 7, the CPU 31 acquires the captured image 61 in S22, and then executes S2.
Move to 3.
In S23, the CPU 31 determines whether or not the memory usage amount before the calibration is performed is equal to or larger than the reference value. The reference value here is, for example, as shown in FIG. 8, a value of the memory usage amount before the calibration is performed, which is estimated to cause a decrease in the processing speed. Pattern A in FIG. 8 is an example of the memory usage amount in a range that does not affect the processing speed. Therefore, in this case, the execution of calibration is permitted. On the other hand, pattern B in FIG. 8 is an example of the memory usage amount in which the decrease in processing speed is estimated. In this case, execution of calibration is not permitted.

When the CPU 31 determines in S23 that the memory usage amount is equal to or larger than the reference value, the process returns to S21. On the other hand, when the CPU 31 determines in S23 that the memory usage is not greater than or equal to the reference value, the CPU 31 proceeds to S24 and executes calibration.

[3-3. effect]
According to the third embodiment described in detail above, the following effects are obtained.
(3a) The image processing unit 23 of the display system 1 can execute the calibration when the memory usage amount is less than the reference value. Therefore, it is possible to prevent the execution of calibration from adversely affecting other controls.

[3-4. Modification of Third Embodiment]
In the third embodiment, the reference value is an example of a memory usage amount estimated to cause a decrease in the processing speed of the image processing unit 23. However, the reference value is not limited to the above configuration. For example, considering that the memory usage amount increases by performing calibration, the reference value is a threshold value of the memory usage amount at which the processing speed is significantly reduced when the increase amount is added to the reference value. It may be a value of the memory usage amount that will be exceeded.

Further, it may be detected that the processing load of the image processing unit 23 is small based on a criterion other than the memory usage amount. For example, it may be determined that the processing load of the image processing unit 23 is small when a specific control having a large processing load is not executed or when the number of executed controls is less than or equal to a predetermined number.

[4. Other Embodiments]
Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments, and various modifications can be implemented.

(4a) In the first embodiment, a predetermined condition is set so that the calibration can be executed at a timing at which the accuracy of the calibration does not deteriorate, and in the second and third embodiments, other control by calibration is performed. Predetermined conditions were set so that the influence on the above can be suppressed and the calibration can be executed. However, the predetermined condition may be set so that both effects described above can be obtained. For example, like the execution determination process shown in the flowchart of FIG. 9, it may be configured to perform all the determinations of S3, S4, S13, and S23 described above. With such a configuration, it is possible to simultaneously realize a reduction in calibration accuracy and an influence on other controls.

(4b) In the above-described second embodiment, the configuration in which it is determined that the predetermined condition for permitting the execution of the calibration is determined based on the traveling speed of the vehicle has been exemplified. Based on this, it may be determined whether or not a predetermined condition is satisfied. For example, it is a condition for satisfying a predetermined condition that a state in which a light such as a headlight is not lit, a state in which a moving body is not detected in at least a certain range outside the vehicle, or the like is detected Good. The traveling speed of the vehicle being within a predetermined range, traveling without turning on the light, and the absence of a moving body within the predetermined range correspond to examples of the vehicle being in the predetermined state. ..

(4c) The image processing unit 23 and the method thereof according to the present disclosure are provided by configuring a processor and a memory programmed to execute one or a plurality of functions embodied by a computer program. It may be realized by a dedicated computer. Alternatively, the image processing unit 23 and the method thereof described in the present disclosure may be realized by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the image processing unit 23 and its method described in the present disclosure include a processor and a memory programmed to execute one or a plurality of functions, and a processor configured by one or more hardware logic circuits. It may be realized by one or more dedicated computers configured by combination. Further, the computer program may be stored in a computer-readable non-transition tangible recording medium as an instruction executed by a computer. The method for realizing the function of each unit included in the image processing unit 23 does not necessarily need to include software, and all the functions may be realized by using one or a plurality of hardware.

(4d) A plurality of functions of one constituent element in the above embodiment may be realized by a plurality of constituent elements, or one function of one constituent element may be realized by a plurality of constituent elements. .. Further, a plurality of functions of a plurality of constituent elements may be realized by one constituent element, or one function realized by a plurality of constituent elements may be realized by one constituent element. Moreover, you may omit a part of structure of the said embodiment. Further, at least a part of the configuration of the above-described embodiment may be added or replaced with respect to the configuration of the other above-described embodiment.

(4e) In addition to the image processing unit 23 described above, a system having the image processing unit 23 as a constituent element, a program for causing a computer to function as the image processing unit 23, and a non-transitional type such as a semiconductor memory storing the program The present disclosure can also be realized in various forms such as an actual recording medium and a calibration execution method.

DESCRIPTION OF SYMBOLS 1... Display system, 11, 11a, 11b, 11c, 11d... Camera, 23... Image processing part, 44... Execution part, 45... Judgment part, 61... Photographed image

Claims (6)

An information processing device (23) mounted on a vehicle for use,
An execution unit (44) configured to execute the calibration of the cameras (11a, 11b, 11c, 11d) mounted on the vehicle and capturing the surroundings of the vehicle based on the captured image (61) of the camera. )When,
A determination unit (45) configured to determine whether or not a predetermined condition is satisfied,
The information processing apparatus, wherein the execution unit is configured to be able to execute the calibration when the determination unit determines that the predetermined condition is satisfied. The information processing apparatus according to claim 1, wherein
The predetermined condition is an information processing device, which is a condition that is determined based on at least one of a captured image acquired by the camera and an output of a sensor mounted on the vehicle. The information processing apparatus according to claim 2, wherein
At least one of the requirement that the environment surrounding the vehicle is dark, the road on which the vehicle is traveling is uneven, and the camera is dirty is satisfied by the determination unit. Sometimes, the information processing apparatus determines that the predetermined condition is not satisfied. The information processing apparatus according to claim 2 or 3, wherein
The determination unit determines that at least the target (62, 63) that can be used as a reference for the calibration is included in the captured image acquired by the camera, to satisfy the predetermined condition. Information processing equipment as a requirement. The information processing apparatus according to any one of claims 2 to 4, wherein:
The information processing apparatus, wherein the determination unit determines that the state of the vehicle acquired based on at least the output of the sensor is a predetermined state as a requirement for determining that the predetermined condition is satisfied. The information processing apparatus according to any one of claims 1 to 5, wherein:
The information processing apparatus, wherein the determination unit sets at least a processing load of the information processing apparatus as a requirement for determining that the predetermined condition is satisfied.