JP7263807B2 - Information processing equipment - Google Patents

Description

The present disclosure relates to an information processing device that is mounted on a vehicle and used.

When synthesizing a plurality of captured images acquired by a plurality of cameras mounted on a vehicle, the quality of the synthesized image can be improved by correcting the camera posture (hereinafter also referred to as calibration). can. Patent Literature 1 proposes a technique of performing calibration while driving without using a specific marker for calibration.

JP 2014-101075 A

However, as a result of detailed studies by the inventors, it was found that the calibration method of Patent Document 1, which does not use specific markers, may result in low calibration accuracy depending on the situation in which the vehicle is placed. was done. Further, as a result of detailed studies by the inventor, a problem was found that an increase in the processing load accompanying the execution of calibration may affect other controls.

One aspect of the present disclosure provides a technique capable of performing calibration at appropriate timing.

One aspect of the present disclosure is an information processing device mounted on a vehicle and used, and includes an execution unit and a determination unit. The executing unit is configured to execute calibration of a camera mounted on a vehicle that photographs the surroundings of the vehicle, based on an image captured by the camera. The determination unit is configured to determine whether or not a predetermined condition is satisfied. The execution unit is configured to be able to execute calibration when the determination unit determines that the predetermined condition is satisfied.

According to such a configuration, execution of calibration can be limited to certain situations. Therefore, for example, by not performing calibration in a situation where there is a high probability that calibration accuracy is low, it is possible to suppress deterioration in calibration accuracy. Further, for example, by not performing calibration in a situation where there is a high probability that an increase in the processing load due to calibration will affect other controls, it is possible to suppress adverse effects on other controls.

2 is a block diagram showing the configuration of an image processing unit corresponding to the information processing apparatus of the first embodiment; FIG. 4 is a functional block diagram of an image processing unit; FIG. It is a figure explaining the outline|summary of calibration. 4 is a flowchart of execution determination processing according to the first embodiment; It is a figure explaining the relationship between several control based on image processing, and running speed. 10 is a flowchart of execution determination processing according to the second embodiment; 10 is a flowchart of execution determination processing according to the third embodiment; It is a figure explaining the relationship between the amount of memory used, and a processing speed fall. 10 is a flowchart of execution determination processing according to another embodiment;

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.
[1. First Embodiment]
[1-1. composition]
A display system 1 shown in FIG. 1 is a system mounted on a vehicle and includes a front camera (hereinafter referred to as F camera) 11a, a rear camera (hereinafter referred to as R camera) 11b, and a left side camera (hereinafter referred to as LS camera). 11c and four right-side cameras (hereinafter referred to as RS cameras) 11d, a display 13, and an ECU 15 are provided. Hereinafter, when referring to all of the cameras described above, they may simply be referred to as camera 11 .

The camera 11 is an imaging device mounted on a vehicle, and can use, for example, a known CCD image sensor or CMOS image sensor. The camera 11 photographs the surroundings of the vehicle at predetermined time intervals (eg, 1/15 s) and outputs the photographed images to the ECU 15 . The F camera 11a, R camera 11b, LS camera 11c, and RS camera 11d are arranged so as to photograph the front, rear, left, and right sides of the vehicle, respectively.

The display 13 is a display device such as a liquid crystal display that displays images, and displays a synthesized image generated by the ECU 15 according to a signal input from the ECU 15 .
The ECU 15 includes a video signal input section 21 , a communication interface (I/F) 22 , an image processing section 23 , a video signal output section 24 and a power supply section 25 .

The video signal input unit 21 receives from the camera 11 a video signal representing an image captured by the camera 11 and outputs the video signal to the image processing unit 23 .
The communication I/F 22 acquires signals output to the in-vehicle communication bus 17 from one or more control devices, sensors, and the like (not shown) mounted in the vehicle, and outputs the signals to the image processing unit 23 . For example, vehicle-related information such as the running speed of the vehicle (hereinafter referred to as vehicle speed), the steering angle of the tires, and the shift range is acquired.

The image processing unit 23 includes a microcomputer having a CPU 31 and semiconductor memories such as a RAM 32a, a ROM 32b, and a flash memory 32c (hereinafter referred to as memory 32). Various functions of the image processing unit 23 are realized by the CPU 31 executing a program stored in a non-transitional substantive recording medium. In this example, the memory 32 corresponds to a non-transitional substantive recording medium storing programs. Also, by executing this program, a method corresponding to the program is executed. Note that the image processing unit 23 may include one microcomputer, or may include a plurality of microcomputers. This image processing unit 23 corresponds to an information processing device.

The image processing unit 23 includes an image generation unit 41, an image recognition unit 42, a drawing unit 43, and an execution unit 44, as shown in FIG. , and a determination unit 45 .

The image generation unit 41 generates one or more composite images based on images captured by two or more of the four cameras 11 . A composite image is an image obtained by combining a plurality of captured images, and the viewpoint of at least one captured image may be changed. For example, it may be an image from a viewpoint different from the captured image actually captured by the camera 11, or an image including a wider range than the captured image of one camera.

In order to generate a composite image, it is necessary to specify the viewpoint of each captured image. In order to specify the viewpoint, camera orientation parameters including the position and angle at which each camera 11 is installed are used. The above-described parameters of each camera 11 are stored in a storage area formed by the memory 32, and are read out and used by the image generating section 41 when necessary.

The image recognition section 42 recognizes a predetermined target object from the composite image generated by the image generation section 41 . The predetermined object here can include, for example, lane markings such as white lines, mobile objects such as people, automobiles, and bicycles, and fixed objects such as traffic lights and signs. A method for recognizing an object from an image is not particularly limited, and various known methods can be used. For example, the target object may be recognized by pattern matching or calculation processing using a learning model. The detected object may be used for various controls.

The drawing unit 43 corrects the brightness and color of the synthesized image, and generates an image that is easy for the passengers of the vehicle to see when output to the display 13 .
The execution unit 44 performs calibration of the camera 11 based on the captured image of the camera 11 while the vehicle is running.

Calibration is to correct parameters of the camera orientation such as the position and angle of the camera 11 described above. Specifically, the installation position of the camera 11 with respect to the vehicle is estimated and replaced with the parameters stored in the storage area of the memory 32, or the conventional parameters and newly estimated parameters are used to create new parameters. recalculate. In this embodiment, calibration is performed without using a marker dedicated to calibration. An overview of the calibration method will be briefly described with reference to FIG.

The captured image 61 shows a first target image 62 as an image showing a target 73 on the road captured by a camera 72 fixed to the vehicle 71 at a certain timing t1. When the vehicle travels and the position of the vehicle changes, the position of the target 73 on the captured image changes. A second target image 63 is an image in which the target 73 captured at timing t2 after timing t1 is schematically superimposed on the captured image 61 . Estimating camera attitude parameters based on changes in the positions of the first target image 62 and the second target image 63 on the images, and the amount of movement of the vehicle 71 during the period from timing t1 to timing t2. can be done.
Note that a specific calibration method is not particularly limited, and for example, the method disclosed in Japanese Patent Application Laid-Open No. 2014-101075 may be used.

The determination unit 45 determines whether or not a predetermined condition is satisfied. Predetermined conditions are conditions suitable for performing calibration. The execution unit 44 is configured to be able to execute calibration when the determination unit 45 determines that a predetermined condition is satisfied.

In this embodiment, the predetermined conditions correspond to (i) the environment is not bad and (ii) the white lines are recognized.
A bad environment is an environment in which there is a high probability that the accuracy of calibration will be lowered due to the low quality of the captured image. For example, the environment around the vehicle is dark, the road on which the vehicle is traveling is bumpy, and the camera 1 is dirty. Examples of bumpy roads include roads with many bumps on the road surface and roads with many small obstacles such as pebbles.

A specific technique for detecting a bad environment is not particularly limited. Whether the environment around the vehicle is dark can be determined, for example, based on an image captured by the camera 11 or based on the output of an illuminance sensor (not shown) mounted on the vehicle. In addition, whether the road on which the vehicle is traveling is uneven can be determined by, for example, detecting the presence of stones or dents by image processing the captured image, or by detecting the presence of a vibration sensor (not shown) mounted on the vehicle or by detecting an acceleration sensor. It can be determined based on the output of the sensor. Further, whether the camera 11 is dirty can be determined, for example, from changes over time in the captured image acquired by the camera 11 .

The white line can be recognized by the image recognition section 42 . Since white lines are suitable targets for performing calibration, detection of white lines may be a requirement for performing good calibration.

Returning to FIG. The video signal output section 24 outputs the composite image generated by the image processing section 23 to the display 13 . The power supply unit 25 supplies electric power to each element constituting the ECU 15 in addition to the image processing unit 23 .

[1-2. process]
Next, execution determination processing executed by the CPU 31 of the image processing unit 23 will be described with reference to the flowchart of FIG.

First, in S1, the CPU 31 determines whether or not the vehicle is running. For example, based on vehicle speed information obtained from a vehicle speed sensor (not shown), the CPU 31 determines that the vehicle is running when the vehicle speed is equal to or higher than a predetermined speed (eg, 5 km/h).

When the CPU 31 determines in S<b>1 that the vehicle is running, the CPU 31 proceeds to S<b>2 and acquires a captured image of the surroundings of the vehicle from the camera 11 . On the other hand, when the CPU 31 determines in S1 that the vehicle is not running, the process of FIG. 4 ends.

In subsequent S3 and S4, the CPU 31 determines whether a predetermined condition for permitting execution of calibration is satisfied.
Specifically, in S3, the CPU 31 determines whether the environment of the vehicle is bad. Here, if any one of the following conditions is satisfied: the environment around the vehicle is dark, the road on which the vehicle is traveling is bumpy, and the camera is dirty. It is determined that there is When the CPU 31 determines in S3 that the environment is bad, the process returns to S1. On the other hand, when the CPU 31 determines in S3 that the environment is not adverse, the process proceeds to S4.

In subsequent S4, the CPU 31 determines whether or not a white line is detected. The white line detection itself is implemented as a function of the image recognition section 42 . When the CPU 31 determines that the white line is detected in S4, the process proceeds to S5. On the other hand, when the CPU 31 determines in S4 that no white line has been detected, the process returns to S1.

In this way, the CPU 31 functions as the determination unit 45 described above, returns to S1 when the predetermined condition is not satisfied, and proceeds to S5 when the predetermined condition is satisfied.
Next, in S5, the CPU 31 executes calibration. Here, the CPU 31 functions as the execution unit 44 described above and executes the calibration described above.

Next, in S6, the CPU 31 updates the camera parameters stored in the memory 32 to the parameters estimated by the calibration in S5. After S6, the process returns to S1.

[1-3. effect]
According to 1st Embodiment detailed above, there exist the following effects.
(1a) The image processing unit 23 of the display system 1 can perform calibration when there is a high probability that calibration can be preferably performed. Therefore, it is possible to prevent the accuracy of calibration from deteriorating by not performing calibration in a situation where the accuracy of calibration is likely to be low.

(1b) The image processing unit 23 determines whether or not a predetermined condition for calibration is satisfied based on at least one of the image captured by the camera 11 and the output of a sensor mounted on the vehicle. can be done.

(1c) The determination unit 45 determines that at least one of the conditions that the environment around the vehicle is dark, that the road on which the vehicle is traveling is uneven, and that the camera 11 is dirty is satisfied. If it does not meet the conditions for executing the calibration. When the above requirements are met, the accuracy of calibration tends to be low, but when the above requirements are not met, calibration is not permitted, so execution of low-accuracy calibration can be suppressed.

(1d) The determining unit 45 sets at least inclusion of a white line in the photographed image acquired by the camera 11 as a requirement for determining that a predetermined condition is satisfied. Therefore, the white line can be used as a calibration target, and the accuracy of calibration can be improved.

[1-4. Modification of First Embodiment]
In the first embodiment, the white line is used as the calibration target, but the target may be something other than the white line. For example, various objects on the road such as lane markings other than white lines, road markings, and curbs can be used as targets. If an object other than the white line is used as the target, in the process of S4 in FIG. 4, instead of the white line, it may be determined whether or not the object is detected.

[2. Second Embodiment]
[2-1. Differences from First Embodiment]
Since the basic configuration of the second embodiment is the same as that of the first embodiment, differences will be described below. Note that the same reference numerals as in the first embodiment indicate the same configurations, and refer to the preceding description.

In the above-described first embodiment, the requirements for determining whether or not to perform calibration are a bad environment and recognition of white lines. On the other hand, the second embodiment differs from the first embodiment in that the traveling speed of the vehicle is used as a requirement for determining whether or not a predetermined condition is satisfied. Further, in the second embodiment, as shown in FIG. 5, various controls using the result of image processing of the captured image are executed according to the vehicle speed. In this embodiment, a plurality of controls are executed in the low speed range of less than 30 km/h. Perform lane detection.

[2-2. process]
Execution determination processing executed by the CPU 23 of the second embodiment in place of the execution determination processing (FIG. 4) of the first embodiment will be described with reference to the flowchart of FIG. Note that the processes of S11, S12, S14, and S15 in FIG. 6 are the same as the processes of S1, S2, S5, and S6 in the figure, so the description is partially simplified.

In the execution determination process of FIG. 6, the CPU 31 acquires the photographed image 61 in S12, and then proceeds to S13.
In S13, the CPU 31 determines whether or not the vehicle speed is in the middle speed range or higher. Vehicle speed can be detected, for example, based on an output signal from a vehicle speed sensor (not shown). Since the image processing unit 23 executes a plurality of controls in the low speed range, the processing load of the image processing unit 23 is high. If calibration is further executed here, the possibility of causing delays in other controls increases. Therefore, the CPU 31 does not permit execution of calibration while the vehicle is running in the low speed range. On the other hand, 31 permits execution of calibration during running in the medium speed range and high speed range because the processing load is small.

When the CPU 31 determines in S13 that the vehicle speed is not in the middle speed range or higher, the process returns to S11. On the other hand, when the CPU 31 determines in S13 that the vehicle speed is equal to or higher than the medium speed range, the CPU 31 proceeds to S14 and executes calibration.

[2-3. effect]
According to the second embodiment described in detail above, the following effects are obtained.
(2a) The image processing unit 23 of the display system 1 can perform calibration when the processing load due to calibration is unlikely to affect other controls. Therefore, it is possible to prevent the execution of calibration from adversely affecting other controls.

[2-4. Modification of Second Embodiment]
In the second embodiment, the configuration was exemplified in which execution of calibration is permitted only when the vehicle speed is in the middle speed range or higher. However, the speed range in which execution of calibration is permitted is not limited to the above example. For example, the configuration may be such that execution of calibration is permitted only within a certain speed range. As an example, execution of calibration may be permitted only in one of the low speed range, medium speed range, and high speed range. Further, the speed threshold for determining whether calibration can be executed is not limited to the speed example disclosed in the second embodiment, and may be various values.

[3. Third Embodiment]
[3-1. Differences from First Embodiment]
Since the basic configuration of the third embodiment is the same as that of the first and second embodiments, differences will be described below. Note that the same reference numerals as in the first embodiment indicate the same configurations, and refer to the preceding description.

In the third embodiment, the processing load on the image processing unit 23 is small, and more specifically, the amount of memory used by the image processing unit 23 is used as a requirement for determining whether or not a predetermined condition is satisfied. It differs from the embodiment. In the third embodiment, "memory usage" refers to usage of the RAM 32a. The image processing unit 23 has software for measuring memory usage. Further, in the third embodiment, as in the second embodiment, various controls using the result of image processing of the captured image are executed.

[3-2. process]
Execution determination processing executed by the CPU 23 of the third embodiment in place of the execution determination processing (FIG. 4) of the first embodiment will be described with reference to the flowchart of FIG. Note that the processes of S21, S22, S24, and S25 in FIG. 7 are the same as the processes of S1, S2, S5, and S6 in the figure, so the description is partially simplified.

In the execution determination process of FIG. 7, the CPU 31 obtains the photographed image 61 in S22, and then
Go to 3.
In S23, the CPU 31 determines whether or not the memory usage before calibration is greater than or equal to the reference value. Here, the reference value is the value of the memory usage before calibration, at which it is estimated that the processing speed is degraded, as shown in FIG. 8, for example. Pattern A in FIG. 8 is an example of memory usage within a range that does not affect the processing speed. Therefore, in this case, execution of calibration is permitted. On the other hand, pattern B in FIG. 8 is an example of memory usage for which a decrease in processing speed is estimated. In this case, execution of calibration is not permitted.

When the CPU 31 determines in S23 that the memory usage is equal to or greater than the reference value, the process returns to S21. On the other hand, when the CPU 31 determines in S23 that the memory usage is not equal to or greater than the reference value, the CPU 31 proceeds to S24 and executes calibration.

[3-3. effect]
According to the third embodiment described in detail above, the following effects are obtained.
(3a) The image processing unit 23 of the display system 1 can perform calibration when the memory usage is less than the reference value. Therefore, it is possible to prevent the execution of calibration from adversely affecting other controls.

[3-4. Modification of Third Embodiment]
In the third embodiment, the reference value is the amount of memory usage estimated to cause the processing speed of the image processing unit 23 to decrease. However, the reference value is not limited to the above configuration. For example, considering that the amount of memory usage increases due to the execution of calibration, the reference value is the memory usage threshold at which the decrease in processing speed becomes noticeable when the increase is added to the reference value. It may be the value of the memory usage that will be exceeded.

Further, it may be detected that the processing load of the image processing unit 23 is small based on a criterion other than the memory usage amount. For example, it may be determined that the processing load of the image processing unit 23 is small when a specific control with a large processing load is not being executed, or when the number of controls being executed is less than or equal to a predetermined number.

[4. Other embodiments]
Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments, and various modifications can be made.

(4a) In the first embodiment, the predetermined conditions are set so that the calibration can be performed at a timing that does not reduce the accuracy of the calibration. Predetermined conditions were set so that calibration could be performed while suppressing the influence on However, a predetermined condition may be set so that both effects described above can be obtained. For example, like the execution determination process shown in the flowchart of FIG. 9, it may be configured to perform all of the above-described S3, S4, S13, and S23 determinations. With such a configuration, it is possible to simultaneously achieve suppression of deterioration in calibration accuracy and suppression of influence on other controls.

(4b) In the above-described second embodiment, the configuration is exemplified in which it is determined that the predetermined condition for permitting execution of calibration is satisfied based on the running speed of the vehicle. Based on this, it may be determined whether or not a predetermined condition is satisfied. For example, a condition in which lights such as headlights are not turned on, a condition in which a moving object is not detected within at least a certain range outside the vehicle, etc. is detected as a requirement for a predetermined condition. good too. Examples of the predetermined state of the vehicle are that the vehicle travel speed is within a predetermined range, that the vehicle is traveling without turning on the lights, and that there is no moving object within the predetermined range. .

(4c) The image processor 23 and techniques described in this disclosure were provided by configuring a processor and memory programmed to perform one or more functions embodied by a computer program. It may also be implemented by a dedicated computer. Alternatively, the image processor 23 and techniques described in this disclosure may be implemented by a dedicated computer provided by configuring the processor with one or more dedicated hardware logic circuits. Alternatively, the image processing unit 23 and techniques described in this disclosure are a combination of a processor and memory programmed to perform one or more functions and a processor configured by one or more hardware logic circuits. It may also be implemented by one or more dedicated computers configured in combination. Computer programs may also be stored as computer-executable instructions on a computer-readable non-transitional tangible storage medium. The method for realizing the function of each part included in the image processing part 23 does not necessarily include software, and all the functions may be realized using one or a plurality of pieces of hardware.

(4d) A plurality of functions possessed by one component in the above embodiment may be realized by a plurality of components, or a function possessed by one component may be realized by a plurality of components. . Also, a plurality of functions possessed by a plurality of components may be realized by a single component, or a function realized by a plurality of components may be realized by a single component. Also, part of the configuration of the above embodiment may be omitted. Moreover, at least part of the configuration of the above embodiment may be added or replaced with respect to the configuration of the other above embodiment.

(4e) In addition to the image processing unit 23 described above, a non-transitional system such as a system having the image processing unit 23 as a component, a program for causing a computer to function as the image processing unit 23, a semiconductor memory storing this program, etc. The present disclosure can also be implemented in various forms such as a physical recording medium and a calibration execution method.

DESCRIPTION OF SYMBOLS 1... Display system 11, 11a, 11b, 11c, 11d... Camera 23... Image processing part 44... Execution part 45... Judgment part 61... Photographed image

Claims (4)

An information processing device (23) mounted on a vehicle and used,
an execution unit (44) configured to perform calibration of cameras (11a, 11b, 11c, 11d) mounted on the vehicle and capturing the surroundings of the vehicle, based on an image (61) captured by the camera; )and,
a determination unit (45) configured to determine whether a predetermined condition is satisfied;
The execution unit is configured to be able to execute the calibration when the determination unit determines that the predetermined condition is satisfied,
The determination unit is configured to determine that at least the target (62, 63) that can be used as a reference for the calibration is included in the captured image acquired by the camera, and that the predetermined condition is satisfied. as a requirement,
The target is at least one of lane markings and road markings,
Further, the execution unit performs the calibration by estimating parameters of the camera posture based on a change in position of the target in the captured image during a specific period and a movement amount of the vehicle during the specific period. information processing device. The information processing device according to claim 1 ,
Based on at least one of the captured image acquired by the camera and the output of a sensor mounted on the vehicle, the determination unit determines that the environment around the vehicle is dark and that the vehicle is running. An information processing device that determines that the predetermined condition is not satisfied when at least one of the conditions that the road is bumpy and that the camera is dirty is satisfied. The information processing device according to claim 1 or claim 2 ,
The determination unit uses, as a requirement for determining that the predetermined condition is satisfied, that the state of the vehicle acquired based on at least the output of a sensor mounted on the vehicle is a predetermined state. processing equipment. The information processing device according to any one of claims 1 to 3 ,
The information processing device, wherein the determination unit determines that at least the processing load of the information processing device is small as a requirement for determining that the predetermined condition is satisfied.

Images