JP2013055410A - Vehicle peripheral image display control device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a vehicle peripheral image display control device for combining real and history images to show a vehicle peripheral image on display, which reduces displacement between the real and the history images compared with conventional one in the case where photographed image acquisition timing and vehicle movement amount information acquisition timing are different.SOLUTION: When vehicle movement amount information is acquired at a point of time t1, a vehicle peripheral image display control device 5 estimates, on the basis of the acquired vehicle movement amount information, a vehicle movement amounts at points of time t11 to t13 at which a photographed image is scheduled to be acquired after the point of time t1. When a photographed image is acquired at each of the points of time t11 to t13, the image based on the photographed images acquired this time is made a real image. And, on the basis of the estimated movement amount at the points of time t11 to t13 estimated at the point of time t1, a history image is created from past photographed images so that the vehicle at the points of time t11 to t13 and the actual positional relation of the periphery matches. This history image is combined with the real image for display.

Description

  The present invention relates to a vehicle peripheral image display control device.

  2. Description of the Related Art Conventionally, a vehicle periphery image display control device that repeatedly acquires captured images from a camera that captures the periphery of the vehicle, creates a vehicle periphery image using the acquired captured image, and displays the vehicle periphery image on a display is known. In such a vehicle periphery image display control device, a real image based on the latest photographed image and a history image based on the past photographed image are combined to form a vehicle periphery image, so that the latest photographed image has a blind spot. It is also known that the peripheral portion of the vehicle that has been displayed is continuously displayed on the display (for example, see Patent Document 1).

JP 2003-9141 A

  As described above, when the actual image based on the latest photographed image and the history image based on the past photographed image are combined and displayed, for the history image, the vehicle is moved according to the movement of the vehicle based on the past photographed image. It is necessary to make it so as to match the actual positional relationship with. That is, in order to create a history image, it is necessary to detect the movement of the vehicle. As a method of detecting the movement of the vehicle, a method using vehicle movement amount information including an amount related to the movement of the vehicle (for example, a vehicle speed pulse, a steering angle) is conceivable.

  Here, the inventor has focused on the point that the acquisition timing of the captured image does not necessarily match the acquisition timing of the vehicle movement amount information. Specifically, it is conceivable that captured images are repeatedly acquired at a first time interval, and vehicle movement amount information is repeatedly acquired at a second time interval that is longer than the first time interval.

  In such a case, as shown in FIG. 7, the actual image is updated at the first time interval (30 ms interval in the example of FIG. 7), whereas the history image is updated at the second time interval (example of FIG. 7). In 100 ms).

  In this case, the vehicle peripheral image displayed on the display is an actual image as shown in FIG. 8A when a history image is not yet created as shown in FIG. Only 51 and the vehicle 50 are displayed. Then, at the time point tb when the actual image is updated twice after the history image is updated, the history image remains behind the actual image, so as shown in FIG. Deviation occurs at the boundary between 61 and the history image 62. This shift is alleviated or eliminated as shown in FIG. 8C at the time tc when the actual image is updated immediately after the history image is taken. If the actual image is updated without being updated, a shift occurs again at the boundary between the actual image 61 and the history image 62 as shown in FIG.

  In this way, if there is frequent deviation at the boundary between the actual image and the history image, there is a high possibility that the user will feel uncomfortable. In view of this point, the present invention provides a vehicle peripheral image display control device that combines a real image and a history image to display a vehicle peripheral image on a display, and the captured image acquisition timing and the vehicle movement amount information acquisition timing are different. It is an object of the present invention to reduce the deviation between the actual image and the history image as compared with the conventional case.

  In order to achieve the above object, the invention according to claim 1 is characterized in that an image input interface (2) for repeatedly acquiring captured images of the periphery of a vehicle captured by a camera at a first time interval, and the first time interval. A vehicle signal input interface (3) for repeatedly acquiring vehicle movement amount information including an amount related to movement of the vehicle in the latest cycle (hereinafter referred to as movement amount) in a cycle of a second time interval longer than Each time the image input interface (2) acquires a captured image, the actual image based on the captured image acquired this time and the history image based on the captured image acquired before this time are combined to obtain a vehicle peripheral image. And a calculation unit (5) that displays the generated vehicle periphery image on a display, and the calculation unit (5) includes a first signal input interface (3). When the vehicle movement amount information is acquired at the movement amount information acquisition time (t1), the image input interface (2) takes the captured image after the first movement amount information acquisition time based on the acquired vehicle movement amount information. Vehicle movement amount estimation means (210-260) for estimating the expected movement amount of the vehicle at the first captured image acquisition time point (t11, t12, t13) to be acquired, and the first captured image acquisition time point ( Based on the fact that the image input interface (2) acquired the captured image at t11, t12, t13), the image based on the captured image acquired this time is used as a real image, and is estimated by the vehicle movement amount estimating means Based on the estimated movement amount of the vehicle at the time of acquisition of the first captured image, the first captured image is acquired using a captured image acquired before this time. A history image that matches the actual positional relationship between the vehicle and the surroundings at the point (t11, t12, t13) is created, and the current vehicle peripheral image is obtained by synthesizing the history actual image and the actual image. A vehicle surrounding image display control device comprising display control means (110-140) for creating and displaying the created vehicle surrounding image for this time on the display.

  Thus, based on having acquired the vehicle movement amount information, the vehicle movement amount at the subsequent first captured image acquisition time is estimated, and when the first captured image acquisition time point comes, the estimated vehicle movement amount By using this to create a history image, the positional deviation at the boundary between the actual image and the history image becomes very small.

  According to a second aspect of the present invention, in the vehicle periphery image display control device according to the first aspect, the display control means (110-140) is configured to acquire the first captured image acquisition time (t11, t12, t13). ) On the basis of the fact that the image input interface (2) has acquired the captured image, the estimated movement amount of the vehicle at the first captured image acquisition time point estimated by the vehicle movement amount estimation means is used, The vehicle periphery image output to the display at the previous captured image acquisition time point (t03, t11, t12) that goes back by the first time interval from the first captured image acquisition time point (t11, t12, t13) The history image is corrected to match the actual positional relationship between the vehicle and the surroundings at the time of shooting (t11, t12, t13) as the history image. The By combining with the real image, creates a current vehicle surroundings image, the current vehicle surroundings images generated, characterized in that to be displayed on the display.

  According to a third aspect of the present invention, in the vehicle periphery image display control device according to the second aspect, the vehicle movement amount estimating means (210-260) is configured such that the vehicle signal input interface (3) is a first one. When the vehicle movement amount information is acquired at the movement amount information acquisition time point (t1), the second time interval elapses from the first movement amount information acquisition time point based on the acquired vehicle movement amount information. When the estimated movement amount of the vehicle in the period is estimated, and the vehicle signal input interface (3) acquires the vehicle movement amount information at the second movement amount information acquisition time point, at the second movement amount information acquisition time point A shift amount between the movement amount included in the acquired vehicle movement amount information and the movement amount at the second movement amount information acquisition time estimated at the time of acquisition of the first movement amount information is calculated. Based on the vehicle movement amount information acquired at the time when the second movement amount information is acquired, the image input interface (2) is scheduled to acquire a captured image after the second movement amount information acquisition point. The expected moving distance that the vehicle moves in a period that goes back by the first time interval from the captured image acquisition time point (t21, t22, 23) and the vehicle at the second captured image acquisition time point (t21, t22, t23). Estimating the expected movement amount, and correcting the predicted movement amount at the time point (t21) closest to the second movement amount information acquisition time point among the second captured image acquisition time points based on the deviation amount. Features.

  By doing so, it is assumed that the expected movement amount at the second movement amount information acquisition time point estimated at the first movement amount information acquisition time point is deviated from the actual movement amount at the second movement amount information acquisition time point. In addition, since the correction for mitigating the deviation is applied to the estimated movement amount estimated for the next second captured image acquisition time point, the actual image and the history image are displayed in the vehicle peripheral image displayed at the second captured image acquisition time point. Deviation is further reduced.

  In addition, the code | symbol in the bracket | parenthesis in the said and the claim shows the correspondence of the term described in the claim, and the concrete thing etc. which illustrate the said term described in embodiment mentioned later. .

It is a block diagram of the vehicle periphery image display control apparatus which concerns on embodiment of this invention. It is a figure which illustrates the change of the real image 11 and the log | history image 12 accompanying a vehicle reverse. It is a figure which illustrates the timing of acquisition of vehicle movement amount information, the update of a real image, and the update of a history image. It is a flowchart of the process which a calculating part performs. It is a flowchart of the process which a calculating part performs. It is a figure which shows the correction | amendment using the difference D of the estimated moving amount Wr and the actual moving amount Wr. It is a figure showing the update timing of a real image and the update timing of a history image in an assumption example. It is a figure which shows the relationship between the real image 61 and the log | history image 62 in an assumption example.

  Hereinafter, an embodiment of the present invention will be described. In FIG. 1, the structure of the vehicle periphery image display control apparatus 1 which concerns on this embodiment is shown. The vehicle peripheral image display control device 1 is mounted on a vehicle, and repeatedly acquires captured images from a camera that is mounted on the vehicle and captures the rear of the vehicle, and also relates to the amount of movement of the vehicle from the in-vehicle LAN such as CAN (hereinafter, Vehicle movement amount information including the movement amount) is repeatedly obtained, a vehicle peripheral image is created based on the captured image and the vehicle movement amount information, and the created vehicle peripheral image is displayed on a display in the vehicle interior. Yes.

  And the vehicle periphery image display control apparatus 1 of this embodiment combines a real image based on the latest captured image and a history image created based on vehicle movement amount information using a past captured image. By displaying it on the display as a peripheral image, the peripheral portion where the vehicle has moved and became a blind spot of the camera is also displayed on the display.

  Specifically, at the start of display of the vehicle periphery image (at the start of reverse of the vehicle), as shown in FIG. 2A, the latest photographed image is converted to a bird's eye view (to an image of a viewpoint looking down from directly above the vehicle). ) And the vehicle image 10 representing the host vehicle are synthesized, and a vehicle peripheral image in which the other part (the blind spot part of the camera) is monochromatic (for example, black) is generated and displayed on the display. Thereafter, when the vehicle continues to move backward, as shown in FIG. 2B, in addition to the real image 11 obtained by bird's-eye conversion of the latest captured image at that time, and the vehicle image 10 described above, the past that is not the latest at that time. A history image 12 is created using the captured images, and the history image is superimposed on the blind spot portion of the camera in the vehicle periphery image. The history image 12 is synthesized with the actual image 11 so as to match the actual positional relationship with the current vehicle based on the past photographed image. Therefore, as shown in FIG. 2C, the range of the history image 12 is expanded in the vehicle peripheral image as the vehicle moves backward.

  As shown in FIG. 1, the vehicle periphery image display control apparatus 1 includes an image input I / F 2, a vehicle signal input I / F 3, an image memory 4, a calculation unit 5, an image output I / F 6, and the like.

  The image input I / F 2 is an interface that repeatedly acquires captured images around the vehicle captured by the camera at a first time interval (specifically, an interval of 30 milliseconds).

  The vehicle signal input I / F 3 is an amount related to the movement of the vehicle in the latest period (hereinafter referred to as an amount of movement) at a period of a second time interval (specifically, 100 milliseconds) longer than the first time interval. It is an interface that repeatedly acquires vehicle movement amount information including The vehicle signal input I / F 3 acquires this vehicle movement amount information from the in-vehicle LAN as described above.

  Here, the movement amount included in the vehicle movement amount information will be described. The movement amount in the vehicle movement amount information acquired by the vehicle signal input I / F 3 at each time point T is a period that goes back from the time point T by the second time interval (that is, the latest cycle for acquiring the vehicle movement amount information). And the latest steering angle of the vehicle detected during the period.

  The moving distance of the vehicle in the period is represented by the number of vehicle speed pulse signals output from one wheel speed sensor during the period. Further, the steering angle of the vehicle can be detected by a steering angle signal output by the steering angle sensor.

  Such vehicle movement amount information is obtained by the ECU (not shown) mounted on the vehicle sequentially acquiring the vehicle speed pulse signal from the wheel speed sensor and the steering angle signal from the steering angle sensor. The vehicle movement amount information includes the number of vehicle speed pulse signals acquired within the second time interval and the rudder angle specified based on the latest rudder angle signal every time the vehicle time interval elapses. Output to. The vehicle signal input I / F 3 sequentially acquires vehicle movement amount information repeatedly output from the ECU to the in-vehicle LAN.

  The image memory 4 is a storage medium that accumulates a predetermined amount (for example, the latest 10 seconds) of captured images repeatedly acquired by the image input I / F 2.

  Each time the image input I / F2 acquires a captured image, the calculation unit 5 obtains an actual image based on the captured image acquired by the current image input I / F2 and the image input I / F2 acquired before this time. This is a circuit for synthesizing a history image based on a photographed image to create a vehicle surrounding image and outputting the created vehicle surrounding image to the image output I / F 6. As the calculation unit 5 that realizes such a function, a known microcontroller including a CPU, a RAM, a ROM, an I / O, and the like can be used. The image output I / F 6 sequentially outputs the vehicle periphery image received from the calculation unit 5 to the display. As a result, a passenger such as a driver in the vehicle cabin can see the vehicle periphery image displayed on the display.

  Here, the functional configuration of the calculation unit 5 will be described. As shown in FIG. 1, the calculation unit 5 functions as a movement distance estimation unit 51, a steering angle estimation unit 52, a vehicle movement amount calculation unit 53, an image composition calculation unit 54, and an output image generation unit 55.

  Each time the vehicle signal input I / F 3 acquires the vehicle movement amount information, the movement distance estimation unit 51 includes the movement distance (or the vehicle movement amount acquired before the previous time in addition to this time) included in the vehicle movement amount information acquired this time. Based on the movement distance included in the information), the image input I / F 2 acquires a captured image during a period from when the next vehicle movement amount information is acquired (when the second time interval elapses from now). The estimated amount of movement of the vehicle at the time point to be performed (more specifically, the amount of movement in a period going back from the time point by the first time interval) is estimated.

  Whenever the vehicle signal input I / F 3 acquires the vehicle movement amount information, the steering angle estimation unit 52 includes the steering angle included in the vehicle movement amount information acquired this time (or the vehicle movement amount acquired before the previous time in addition to this time). Based on the rudder angle included in the information), the estimated rudder angle of the vehicle at the time when the image input I / F 2 obtains the captured image in the period until the next time when the vehicle movement amount information is obtained is estimated. .

  When the image input I / F 2 acquires a captured image, the vehicle movement amount calculation unit 53 is based on the current predicted movement amount and the predicted steering angle that have already been estimated by the movement distance estimation unit 51 and the steering angle estimation unit 52. Then, it is calculated how the vehicle has moved from the time when the previous image input I / F 2 acquired the captured image to the present time.

  When the image input I / F 2 acquires a captured image and records it in the image memory 4, the image composition calculation unit 54 sets the latest captured image recorded in the image memory 4 as a real image. Further, the image composition calculation unit 54, based on the calculation result of the vehicle movement content calculated by the vehicle movement amount calculation unit 53, the actual positional relationship of the vehicle periphery with the current vehicle, and the vehicle and history in the vehicle peripheral image. A history image is created so that the positional relationship of the images matches, and the actual image and the vehicle image are combined to obtain a vehicle peripheral image. The output image generation unit 55 sequentially outputs the vehicle periphery image to the image output I / F 6.

  The operation of the vehicle periphery image display control device 1 configured as described above will be described with reference to FIGS. FIG. 3 is a diagram illustrating timing of acquisition of vehicle movement amount information, actual image update, and history image update in the vehicle surrounding image display control device 1. In this example, the update timing of the actual image and the history image is actually delayed by the processing time with respect to the captured image acquisition timing by the image input I / F 2, but this delay is the captured image acquisition cycle ( Since the timing is sufficiently smaller than the first time interval), it is assumed that both timings are the same.

[Operation at time t0, t01, t02, t03]
In the example of FIG. 3, the vehicle peripheral image display control device 1 starts to operate based on the fact that the shift position of the vehicle becomes the reverse position (R) at time t <b> 0. At time t0, the vehicle signal input I / F 3 first acquires the vehicle movement amount information, and then at time t01, t02, and t03, the image input I / F 2 acquires a captured image and records it in the image memory 4. At each time point t01, t02, t03, the calculation unit 5 uses only the real image among the real image and the history image based on the fact that the vehicle movement amount information has only been acquired once since the start of operation. A surrounding image is created, and the created vehicle surrounding image is displayed on the display. Specifically, the actual image obtained by bird's-eye conversion of the latest photographed image at that time and the own vehicle image that is an image of the own vehicle are combined. As shown in FIG. 2, the composition method arranges the actual image 11 at the rear end of the own vehicle image 10 so that the positional relationship between the own vehicle image 10 and the display content in the actual image 11 can be changed. Match the vehicle and surrounding position. And the vehicle periphery image after a synthesis | combination is displayed on a display by outputting to image output I / F6.

[Operation at time t1]
Next, when the vehicle signal input I / F 3 acquires the vehicle movement amount information for the second time at the time t1 when the second time interval has elapsed from the time t0, the calculation unit 5 Based on the fact that the vehicle movement amount information is acquired twice or more from the start of operation, the processing shown in FIG. 4 and the processing shown in FIG.

  In the process of FIG. 4, it is determined in step 110 whether or not the image input I / F 2 has acquired a captured image. However, since it is determined that it has not been acquired at this time t1, the captured image is acquired next. Until then, the processing of step 110 is repeated.

  On the other hand, in the process of FIG. 5, it is determined in step 210 whether or not the vehicle movement amount information has been acquired from the vehicle signal input I / F 3, but at this time t1, it is determined that it has been acquired, and the process is performed. Proceed to step 220.

  In step 220, a difference calculation process with respect to the previous estimation result is performed. At time t1, the number of times the vehicle signal input I / F 3 has acquired the vehicle movement amount information after the operation of the vehicle peripheral image display control device 1 is started. Based on being less than three times, the process of step 220 is bypassed and the process proceeds to step 230.

  In step 230, the image frame timing is specified. Specifically, within the period A (however, excluding the current time t1) from the current time t1 to the acquisition time t2 of the next vehicle movement amount information at the current time (time t2 after the second time interval from the current time) The time points t11, t12, and t13 (hereinafter referred to as first captured image acquisition time points) at which the input I / F 2 is scheduled to acquire captured images are specified. Specifically, the time t03 at which the image input I / F 2 acquired the last captured image (that is, the last before the current time) is the starting point, and a time that is an integral multiple of a predetermined first time interval from the starting point. All elapsed time points t11, t12, and t13 are extracted within the period A, and these time points t11, t12, and t13 are set as image frame timings (first captured image acquisition time points).

  Subsequently, in step 240, the estimated movement amount of the vehicle at each image frame timing t11, t12, t13 specified in step 230 is estimated. Here, as the predicted movement amount of the vehicle at the image frame timings t11, t12, and t13, the predicted movement distance of the vehicle in a period that goes back from the image frame timing by the first time interval, and the predicted steering of the vehicle at the image frame timing. Adopt a corner.

  Taking the image frame timing t11 as an example, the expected travel distance of the vehicle in a period that goes back by the first time interval from the image frame timing t11, that is, the period from the time t03 to the time t11, and the predicted steering of the vehicle at the time t11. Calculate the corner.

  The predicted movement amount may be estimated based on the vehicle movement amount information acquired at the current time point t1 and the vehicle movement amount information acquired at the acquisition time point t0 of the previous vehicle movement amount information. You may estimate based only on the vehicle movement amount information acquired in the time t1.

  For example, for the predicted travel distance, the travel distance of the vehicle (the travel distance of the vehicle in the period up to the time point t0 of the second time interval) included in the vehicle travel amount information acquired at the previous time point t0 is L0. Yes, if the moving distance of the vehicle (the moving distance of the vehicle in the period up to the time point t1 of the second time interval) included in the vehicle movement amount information acquired at the current time point t1 is L1, It may be estimated that the moving distance of the vehicle during the second time interval is a value (2 × L1−L0) obtained by linear extrapolation using the values L0 and L1 at the time points t0 and t1. You may estimate that it is a value, ie, L1.

  The predicted travel distance of the vehicle from time t03 to time t11 is calculated on the assumption that the vehicle moves at a constant speed with the linear extrapolation value or the current value. That is, the linear extrapolation value or a value obtained by multiplying the current value by (t11−t03) / (t2−t1) is set as an expected movement distance from time t03 to time t11. Similarly, the predicted travel distance of the vehicle from time t11 to time t12 is the value obtained by multiplying the linear extrapolation value or the current value by (t12-t11) / (t2-t1), and from time t12 to time t13. The predicted travel distance of the vehicle is a value obtained by multiplying the linear extrapolation value or the current value by (t13−t12) / (t2−t1).

  As for the predicted steering angle, the steering angle of the vehicle included in the vehicle movement amount information acquired at the previous time point t0 is W0, and the movement distance of the vehicle included in the vehicle movement amount information acquired at the current time point t1 is If it is W1, the rudder angle of the vehicle at the next time point t2 may be estimated to be a value (2 × W1−W0) obtained by linear extrapolation using the values W0 and W1 at the time points t0 and t1. Then, it may be estimated that the current value, that is, W1.

  The predicted steering angle of the vehicle from time t03 to time t11 is calculated on the assumption that the vehicle moves at a constant speed with the linear extrapolation value or the current value. That is, the linear extrapolation value or a value obtained by multiplying the current value by (t11−t03) / (t2−t1) is set as the predicted steering angle from time t03 to time t11. Similarly, the predicted steering angle of the vehicle from time t11 to time t12 is a value obtained by multiplying the linear extrapolation value or the current value by (t12-t11) / (t2-t1), and from time t12 to time t13. The predicted steering angle of the vehicle is a value obtained by multiplying the linear extrapolation value or the current value by (t13-t12) / (t2-t1).

  In step 240, in addition to the image frame timings t11, t12, and t13, an estimated movement amount of the vehicle at the time point t2 after the second time has elapsed from the current time point t1 is also estimated. Specifically, the predicted travel distance of the vehicle in the period from time t1 to time t2 and the predicted steering angle of the vehicle at time t2 are calculated. As these values, the above-described linear extrapolation values or current values may be adopted.

  Subsequently, in step 250, correction using the difference calculated in step 220 is performed. At time t1, the vehicle signal input I / F 3 acquires vehicle movement amount information after the operation of the vehicle peripheral image display control device 1 starts. If the number of times is less than 3, the process of step 250 is bypassed and the process proceeds to step 260.

  In step 260, the predicted movement amount (specifically, the predicted movement distance and the predicted steering angle) at time points t11, t12, t13, and t2 estimated in step 240 is recorded in a storage medium (for example, a RAM (not shown) in the calculation unit 5). To do. Thereafter, the process returns to step 210, and the determination process of step 210 is repeated while determining that the vehicle movement amount information is not acquired until the vehicle signal input I / F 3 acquires the vehicle movement amount information again.

[Operation at time t11]
Thereafter, when time elapses and time t11 is reached, the image input I / F 2 acquires a captured image and records it in the image memory 4. Then, in the process of FIG. 4, the calculation unit 5 first determines in step 110 that a captured image has been acquired, proceeds to step 120, and reads the predicted movement amount (expected movement distance and predicted steering angle) of the vehicle at the current time t11. . The expected movement amount of the vehicle at time t11 is already recorded in step 260 of FIG. 5 at time t1.

  Subsequently, in step 130, based on the predicted movement amount at the time t11 that has been read, a real image and a history image at the current time t11 are created, and a vehicle peripheral image is created by combining the created real image and the history image.

  Here, a method of creating a real image and a history image at the current time t11 will be described. The actual image at the current time t11 is a top view image obtained by performing bird's-eye conversion on the captured image acquired this time (that is, the latest captured image recorded in the image memory 4).

  The history image at the current time t11 is obtained by performing bird's-eye conversion, rotation conversion, or the like on a captured image acquired before this time (that is, a captured image that is not the latest among the captured images recorded in the image memory 4). It is a top view image as a result of processing.

  Specifically, the captured image (captured image at time t03) acquired by the image input I / F 2 one time before the current time t11 is bird's-eye converted, and the image after the bird's-eye conversion is converted from time t03 to time t11. The vehicle is rotated in the same manner as the position around the vehicle changes as the vehicle moves.

  For this purpose, it is necessary to calculate the movement contents of the vehicle from the time point t03 to the current time point t11. However, since the vehicle movement amount information was finally acquired at the time point t1, the actual position of the current vehicle must be specified. I can't. Accordingly, the actual movement content of the vehicle is estimated using the predicted movement amount (expected movement distance, predicted steering angle) at time t11 read in step 120.

  Specifically, from time t03 to time t11, it is estimated that the vehicle has moved backward by the read predicted movement distance according to a well-known Ackermann model while maintaining the read predicted steering angle. Thereby, the turning center of the vehicle from the time point t03 to the time point t11 and the turning angle of the vehicle as seen from the turning center can be estimated. For the image obtained by bird's-eye conversion of the photographed image at time t03, rotation conversion is performed so that the rotation center is the same as the rotation angle and the rotation angle is the same as the rotation angle. And a history image.

  Then, the history image created in this way and the above-described actual image are synthesized. At the time of synthesis, the history image is deleted for the portion where the history image and the actual image overlap, Prioritize. Further, the vehicle image is synthesized with the image synthesized in this way. At this time, by making the positional relationship between the vehicle image and the actual image as shown in FIG. 2A, the positional relationship between the vehicle image and the history image is automatically set to the actual vehicle and the actual image at time t11. It almost matches the positional relationship with the surroundings of the vehicle. The image after combining the own vehicle image becomes the vehicle peripheral image to be displayed on the display.

  Subsequently, in step 140, the vehicle periphery image is output to the image output I / F 6 to be displayed on the display. At this time, as shown in FIG. 2B, a vehicle peripheral image in which the own vehicle image 10, the actual image 11, and the history image 12 are combined is displayed on the display. At this time, the positional deviation at the boundary between the real image 11 and the history image 12 becomes very small. This is because the history image 12 is not created to reflect the positional relationship between the vehicle and the surroundings at the time t1 when the vehicle movement amount information was last acquired, but the latest photographed image that is the basis of the actual image 11 is displayed. This is because it is created so as to reflect the positional relationship between the vehicle and the surrounding area at the time t11 obtained. After step 140, the process returns to step 110, and the determination process of step 110 is repeated while determining that the captured image is not acquired until the image input I / F 2 acquires the vehicle movement amount information again.

[Operation at time t12]
Thereafter, when time elapses and time t12 is reached, the image input I / F 2 acquires a captured image and records it in the image memory 4. Then, in the process of FIG. 4, the calculation unit 5 first determines in step 110 that the captured image has been acquired, proceeds to step 120, and reads the predicted movement amount (expected movement distance and predicted steering angle) of the vehicle at the current time t <b> 12. . The expected movement amount of the vehicle at time t12 is already recorded in step 260 of FIG. 5 at time t1.

  Subsequently, at step 130, based on the read estimated movement amount at time t12, a real image and a history image at the current time t12 are created, and a vehicle peripheral image is created by combining the created real image and the history image. The method for creating the actual image at the current time t12 is the same as the method at the time t11.

  Further, regarding the creation of the history image at the current time t12, a method different from the time t11 that is the first time is based on the fact that the history image is created after the second operation after the operation of the vehicle peripheral image display control device 1. Create a history image with. That is, rotation conversion or the like is performed on the vehicle peripheral image created at the time t11 (corresponding to the previous captured image acquisition time that is back by the first time interval from the first captured time) when the previous captured image is acquired. By performing the correction, the top view image corrected to match the actual positional relationship between the vehicle and the vehicle at time t12 (corresponding to the first shooting time) is set as the current vehicle peripheral image.

  For this purpose, the movement contents of the vehicle from the time point t11 to the current time point t12 are calculated. The method is the same as the method at the time point t11, and the predicted movement amount (expected movement distance, predicted steering angle) at the time point t12 read in step 120. Is used to estimate the actual movement of the vehicle.

  Specifically, from time t11 to time t12, it is estimated that the vehicle has moved back by the read predicted moving distance according to a well-known Ackermann model while maintaining the read predicted steering angle. As a result, the turning center of the vehicle from the time point t11 to the time point t12 and the turning angle of the vehicle estimated from the turning center can be estimated. For the vehicle peripheral image at time t11, the rotation conversion is performed so that the turning center is the rotation center and the rotation angle is the same size as the turning angle, and the rotation angle is reversed. To do. Note that the vehicle peripheral image created at time t11 is created using a captured image acquired before time t12 of this time.

  Then, the history image created in this way and the above-described actual image are synthesized. At the time of synthesis, the history image is deleted for the portion where the history image and the actual image overlap, Prioritize. Further, the vehicle image is synthesized with the image synthesized in this way. At this time, by making the positional relationship between the vehicle image and the actual image as shown in FIG. 2B, the positional relationship between the vehicle image 10 and the history image 12 is automatically set to the vehicle at time t12. It almost matches the actual positional relationship with the surroundings of the vehicle. The image after combining the own vehicle image becomes the vehicle peripheral image to be displayed on the display.

  Subsequently, in step 140, the vehicle periphery image is output to the image output I / F 6 to be displayed on the display. At this time, as shown in FIG. 2C, a vehicle peripheral image in which the own vehicle image 10, the actual image 11, and the history image 12 are combined is displayed on the display. At this time, the positional deviation at the boundary between the real image 11 and the history image 12 becomes very small as already described.

[Operation at time t13]
Thereafter, when time elapses and time t13 is reached, the image input I / F 2 acquires a captured image and records it in the image memory 4. Then, in the process of FIG. 4, the calculation unit 5 first determines in step 110 that the captured image has been acquired, proceeds to step 120, and reads the predicted movement amount (expected movement distance and predicted steering angle) of the vehicle at the current time t <b> 13. . Subsequently, in step 130, based on the read estimated movement amount at time t13, a real image and a history image at the current time t13 are created, and a vehicle peripheral image is created by combining the created real image and the history image. The method for creating and synthesizing the actual image and the history image at the current time t12 is the same as the method at the time t12. Subsequently, in step 140, the generated vehicle periphery image is output to the image output I / F 6 to be displayed on the display. At this time, the positional deviation at the boundary between the real image and the history image becomes very small as described above.

[Operation at time t2]
Thereafter, when time elapses and time t2 is reached, it is determined in step 210 that the vehicle movement amount information has been acquired from the vehicle signal input I / F 3 in the process of FIG. 5, and the process proceeds to step 220.

  In step 220, at the time t2, the previous estimation is performed based on the fact that the number of times the vehicle signal input I / F 3 has acquired the vehicle movement amount information after the start of the operation of the vehicle periphery image display control device 1 is three times or more. A difference calculation process with the result is performed. In this difference extraction process, the movement distance Lr (the movement distance of the vehicle in the period from the time point t1 to the time point t2) included in the vehicle movement amount information acquired at the time point t2 (corresponding to the second movement amount information acquisition time point) and Between the steering angle Wr (the latest steering angle detected at time t2) and the predicted movement distance Le and the predicted steering angle We at time t2 estimated at time t1 (corresponding to the first movement amount information acquisition time). A deviation amount (specifically, a difference) is calculated. The predicted travel distance and the predicted steering angle at time t2 are estimated and recorded at step 240 in FIG. 5 at time t1.

  More specifically, the travel distance difference Lr−Le obtained by subtracting the predicted travel distance Le from the travel distance Lr, and the steering angle difference obtained by subtracting the predicted steering angle We from the steering angle Wr. Wr-We is calculated. These differences are indices indicating how much the travel distance and the steering angle estimated at the time point t1 deviate from the actual travel distance and the steering angle. If the expected travel distance (expected travel distance and predicted rudder angle) is significantly different from the actual travel distance (travel distance and rudder angle), there will be a large image shift at the boundary between the actual image and the history image. There is a high possibility that the user will feel uncomfortable.

  In step 230, the image frame timing is specified. Specifically, in the period B (excluding the current time t2) from the current time t2 to the acquisition time t3 of the next vehicle movement amount information at the current time (time t3 after the second time interval from the current time) The time points t21, t22, t23, and t24 (hereinafter referred to as first captured image acquisition time points) at which the input I / F 2 is scheduled to acquire a captured image are specified. Specifically, the time t13 at which the image input I / F 2 acquired the previous captured image (that is, the last before the current time) is the starting point, and a time that is an integral multiple of a predetermined first time interval from that starting point. All of the elapsed time points t21, t22, t23, and t24 are extracted within the period B, and these time points t21, t22, t23, and t24 are set as image frame timings (first captured image acquisition time points).

  Subsequently, in step 240, the predicted movement amount (predicted movement distance and predicted steering angle) of the vehicle at each image frame timing t21, t22, t23, t24 specified in step 230 is calculated in the same manner as in step 240 at time t1, presume.

  Taking the image frame timing t21 as an example, the expected travel distance of the vehicle in a period that goes back from the image frame timing t21 by the first time interval, that is, the period from the time t13 to the time t21, and the predicted steering of the vehicle at the time t21. Calculate the corner. The calculation method may adopt the current value at the time t2 as the predicted movement distance and the predicted steering angle as in step 240 at the time t1, or linear compensation using the movement distance and the steering angle at the time t1 and the time t2. The outside value may be used as the predicted moving distance and the predicted steering angle.

  In step 240, in addition to the image frame timings t21, t22, and t23, an estimated movement amount of the vehicle at the time point t3 after the second time has elapsed from the current time point t2 is also estimated. Specifically, the predicted travel distance of the vehicle in the period from time t2 to time t3 and the predicted steering angle of the vehicle at time t3 are calculated. As these values, the above-described linear extrapolation values or current values may be adopted. This expected movement amount at time t2 is used in step 220 at a later time t3.

  Subsequently, in step 250, based on the fact that the number of times the vehicle signal input I / F 3 has acquired the vehicle movement amount information after the start of the operation of the vehicle periphery image display control device 1 is 3 or more times at the time t2. Correction using the difference calculated in 220 is performed.

  The target of this correction is the predicted movement amount (expected moving distance and predicted steering angle) at the time t21 that is the image frame timing closest to the current time after the current time t2. This correction eliminates the previous deviation at the next display timing t21 so that the deviation amount between the predicted movement amount and the actual movement amount at the time point t2 does not adversely affect the vehicle surrounding image display after the time point t2. It is a correction to do.

  Specifically, correction is performed by adding the movement distance difference Lr−Le calculated in step 220 to the predicted movement distance at time t21 calculated in step 240. Similarly, as shown in FIG. 6, the steering angle difference D = Wr−We calculated in step 220 is added to the predicted steering angle W21 at time t21 calculated in step 240.

  In step 250, the predicted travel distance (expected travel distance and predicted steering angle) at time t3, which is the next vehicle travel distance information acquisition timing, is also corrected. Specifically, correction is performed by adding the moving distance difference Lr−Le calculated in step 220 to the predicted moving distance at time t3 calculated in step 240. Similarly, correction is performed by adding the steering angle difference Wr-We calculated in step 220 to the predicted steering angle calculated at step 240 at the time point t3.

  Subsequently, in step 260, the predicted movement amount (specifically, the predicted movement distance and the predicted steering angle) at the time points t21, t22, t23, t24, t2 estimated in step 240 is stored in a storage medium (for example, not shown in the calculation unit 5). RAM). Thereafter, the process returns to step 210, and the determination process of step 210 is repeated while determining that the vehicle movement amount information is not acquired until the vehicle signal input I / F 3 acquires the vehicle movement amount information again.

[Operation at time t21]
Thereafter, when time elapses and time t21 is reached, the image input I / F 2 acquires a captured image and records it in the image memory 4. Then, in the process of FIG. 4, the calculation unit 5 first determines in step 110 that the captured image has been acquired, proceeds to step 120, and reads the predicted movement amount (expected movement distance and predicted steering angle) of the vehicle at the current time t <b> 21. . The predicted movement amount read here is the predicted movement amount corrected in step 250 of FIG. 5 at time t2.

  Subsequently, in step 130, based on the read estimated movement amount at time t21, a real image and a history image at the current time t21 are created, and a vehicle peripheral image is created by combining the created real image and the history image. The actual image and history image creation method and composition method at the current time t21 are the same as the methods at the time points t12 and t13. Subsequently, in step 140, the generated vehicle periphery image is output to the image output I / F 6 to be displayed on the display. At this time, the positional deviation at the boundary between the real image and the history image becomes very small as described above.

  Further, since the read predicted movement amount at time t21 is corrected so as to cancel out the difference between the predicted movement amount at time t2 and the actual movement amount, the error of the predicted movement amount before time t2 is later than time t2. There is no dragging.

[Operation after time t22]
The operation of the vehicle surrounding image display control device 1 after time t22 is the same as that at time t21 at times t22, t23, t24, t31, t32, t33, t41, t42, and t43. Further, time t3, t4, and t5 are the same as time t2.

  As described above, the calculation unit 5 of the vehicle periphery image display control device 1 according to the present embodiment acquires vehicle movement amount information at the first movement amount information acquisition time points t1, t2, t3, t4, and t5 (steps). 210), based on the acquired vehicle movement amount information, the first from the first captured image acquisition time point (for example, time point t11) where the captured image is scheduled to be acquired after the first movement amount information acquisition time point (for example, time point t1). The estimated travel distance that the vehicle moves in a period that goes back by this time interval (for example, the period from time t03 to time t11) and the predicted steering angle of the vehicle at the first captured image acquisition time (for example, time t11) are estimated ( Steps 230, 240).

  Further, the computing unit 5 acquires the captured image at the first captured image acquisition time point (t11, t12, t13, t21, t22, t23, t24, t31, t32, t33, t41, t42, t43). (Step 110), an image based on the captured image acquired this time is used as a real image, and the first captured image is based on the estimated travel distance and the estimated steering angle at the time of the first estimated captured image. The vehicle peripheral image output to the display at the previous captured image acquisition time point (for example, time point t03) that goes back by the first time interval from the acquisition time point (for example, time point t11) is the vehicle and surroundings at the first shooting time point (for example, t11). The vehicle surrounding image of this time is created by modifying the history image as a result of the correction to match the actual positional relationship and the actual image (Step) 20,130), and displays the current vehicle periphery image generated on the display (step 140).

  Thus, based on having acquired the vehicle movement amount information, the vehicle movement amount (expected moving distance, predicted rudder angle) at the subsequent first captured image acquisition time point is estimated, and the first captured image acquisition time point is By creating a history image using the estimated amount of vehicle movement when visiting, the positional deviation at the boundary between the actual image and the history image becomes very small.

  In addition, when the vehicle movement amount information is acquired at the first movement amount information acquisition time point (for example, time point t1) (step 210), the calculation unit 5 performs the first movement amount based on the acquired vehicle movement amount information. The estimated travel distance that the vehicle moves during the period from the information acquisition time point until the second time interval elapses, and the second movement amount information acquisition time point when the second time interval elapses from the first movement amount information acquisition time point ( For example, when the predicted steering angle of the vehicle at time t2) is estimated (step 240), and the vehicle movement amount information is acquired at the second movement amount information acquisition time point (step 210), the second movement amount information acquisition time point The travel distance and rudder angle included in the vehicle travel distance information acquired in step 1 and the predicted travel at the second travel distance information acquisition time estimated at the first travel distance information acquisition time (for example, time t1) The distance between the distance and the predicted steering angle is calculated (step 220), and a captured image is acquired after the second movement amount information acquisition time based on the vehicle movement amount information acquired at the second movement amount information acquisition time. The estimated moving distance and the second shooting in which the vehicle moves in a period (for example, a period from time t13 to time t21) that goes back by the first time interval from the second captured image acquisition time point (for example, time point t21) to be performed. The estimated steering angle of the vehicle at the image acquisition time is estimated (steps 230 and 240), and the predicted movement at the time closest to the second movement amount information acquisition time (for example, time t21) among these second captured image acquisition time points. The distance and the predicted rudder angle are corrected based on the amount of deviation (step 250).

  By doing in this way, the predicted travel distance (expected travel distance and predicted steering angle) at the second travel distance information acquisition time point estimated at the first travel distance information acquisition time point is the second travel distance information acquisition time point. Even if there is a deviation from the actual movement amount, since the correction for mitigating the deviation is applied to the estimated movement amount estimated for the next second captured image acquisition time point, the vehicle displayed at the second captured image acquisition time point In the peripheral image, the deviation between the actual image and the history image is further reduced.

  In the above embodiment, the calculation unit 5 functions as an example of a display control unit by executing steps 110 to 140 in FIG. 4, and as an example of a vehicle movement amount estimation unit by executing steps 210 to 260. Function. Further, the functions of the movement distance estimation unit 51 and the steering angle estimation unit 52 in FIG. 1 are realized by the processing of steps 230 and 240 in FIG. 5, and the functions of the vehicle movement amount calculation unit 53 and the image composition calculation unit 54 are illustrated in FIG. 4, and the function of the output image generation unit 55 is realized by the process of step 140 in FIG. 4.

  As mentioned above, although embodiment of this invention was described, the scope of the present invention is not limited only to the said embodiment, The various form which can implement | achieve the function of each invention specific matter of this invention is included. It is.

1 Vehicle Peripheral Image Display Control Device 2 Image Input I / F
3 Vehicle signal input I / F
4 Image memory 5 Calculation unit 6 Image output I / F
51 Travel Distance Estimator 52 Steering Angle Estimator 53 Vehicle Movement Amount Calculator 54 Image Composition Calculator 55 Output Image Generator

Claims (3)

An image input interface (2) for repeatedly acquiring captured images around the vehicle captured by the camera at a first time interval;
A vehicle signal input interface for repeatedly acquiring vehicle movement amount information including an amount related to movement of the vehicle in the latest cycle (hereinafter referred to as movement amount) in a cycle of a second time interval longer than the first time interval. (3) and
Each time the image input interface (2) acquires a captured image, the actual image based on the captured image acquired this time and the history image based on the captured image acquired before this time are combined to obtain a vehicle peripheral image. And a calculation unit (5) for displaying the created vehicle periphery image on a display,
The calculation unit (5)
When the vehicle signal input interface (3) acquires the vehicle movement amount information at the first movement amount information acquisition time point (t1), the first movement amount information acquisition time point based on the acquired vehicle movement amount information. Vehicle movement amount estimation means (210-260) for estimating the expected movement amount of the vehicle at a first captured image acquisition time point (t11, t12, t13) at which the image input interface (2) intends to acquire a captured image later. When,
Based on the fact that the image input interface (2) has acquired a captured image at the first captured image acquisition time point (t11, t12, t13), an image based on the currently acquired captured image is set as a real image, and Based on the estimated movement amount of the vehicle at the first captured image acquisition time estimated by the vehicle movement amount estimating means, the first captured time point (t11, A history image that matches the actual positional relationship between the vehicle and the surroundings at t12, t13) is created, and the current vehicle surrounding image is created by synthesizing the history actual image and the actual image. Display control means (110-140) for displaying the current vehicle periphery image on the display, the vehicle periphery image display control device.   The display control means (110-140) estimates the vehicle movement amount based on the fact that the image input interface (2) has acquired a captured image at the first captured image acquisition time point (t11, t12, t13). Previously, going back from the first captured image acquisition time point (t11, t12, t13) by the first time interval using the estimated movement amount of the vehicle at the first captured image acquisition time point estimated by the means. The vehicle periphery image output to the display at the captured image acquisition time (t03, t11, t12) is corrected so as to match the actual positional relationship between the vehicle and the surrounding at the first captured time (t11, t12, t13). Then, as the history image, by synthesizing the history image with the actual image, a current vehicle peripheral image is created, and the created current vehicle A vehicle periphery image display control device according to the peripheral image to claim 1, characterized in that to be displayed on the display. The vehicle movement amount estimation means (210-260)
When the vehicle signal input interface (3) acquires the vehicle movement amount information at the first movement amount information acquisition time point (t1), the first movement amount information acquisition time point based on the acquired vehicle movement amount information. To estimate the expected movement amount of the vehicle in a period until the second time interval elapses,
When the vehicle signal input interface (3) acquires the vehicle movement amount information at the time of the second movement amount information acquisition, the movement amount included in the vehicle movement amount information acquired at the time of the second movement amount information acquisition; Calculating a deviation amount from the movement amount at the second movement amount information acquisition time point estimated at the first movement amount information acquisition time point;
Based on the vehicle movement amount information acquired at the time when the second movement amount information is acquired, the second shooting that the image input interface (2) intends to acquire a shot image after the second movement amount information acquisition time point. The expected moving distance that the vehicle moves in a period that goes back by the first time interval from the image acquisition time point (t21, t22, 23) and the prediction of the vehicle at the second captured image acquisition time point (t21, t22, t23) Estimate the amount of movement,
The expected movement amount at a time point (t21) closest to the second movement amount information acquisition time point among the second captured image acquisition time points is corrected based on the deviation amount. The vehicle periphery image display control apparatus described.

Images