JP2017175268A - Image processing system, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system that is able to hinder a gap between a composite image and an actual situation.SOLUTION: An image processing system 1 comprises: an image acquisition unit 7 that acquires an image of the surroundings of a vehicle; a conversion unit 9 that converts an image into a bird-eye view image; a signal acquisition unit 11 that acquires a vehicle signal from an on-vehicle network; a position determination unit 13 that determines a positional relation between a plurality of bird-eye view images; a composing unit 15 that forms a composite image; an estimation unit 17 that estimates a vehicle speed at the point in time further ahead by a signal delay time; and a stop determination unit 19 that determines whether the vehicle is stationary or not. If it is determined that the vehicle is stationary, the position determination unit determines a positional relation acquired when the vehicle is stationary.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  Conventionally, the following image processing apparatuses are known. The image processing apparatus acquires a plurality of images around the vehicle with a time difference using a camera mounted on the vehicle. The image processing device converts each of the plurality of images into a bird's eye view image. The image processing apparatus acquires a vehicle signal representing the behavior of the vehicle from the in-vehicle network. The image processing apparatus determines the positional relationship between the plurality of bird's-eye view images based on the vehicle signal. The image processing apparatus creates a composite image by arranging a plurality of bird's-eye view images in the positional relationship determined as described above. Such an image processing apparatus is disclosed in Patent Document 1.

Japanese Patent No. 4156214

  The vehicle signal acquired from the in-vehicle network is delayed. If the composite image is generated by the above method using the delayed vehicle signal when the behavior of the vehicle is not constant, a divergence occurs between the composite image and the actual situation.

  The present invention has been made in view of these problems, and an object thereof is to provide an image processing apparatus, an image processing method, and a program that can suppress a deviation between a composite image and an actual situation.

  An image processing apparatus (1) of the present disclosure includes an image acquisition unit (7) that acquires a plurality of images around a vehicle with a time difference using a camera (21) mounted on the vehicle, and a plurality of images. A conversion unit (9) that converts each to a bird's eye view image to create a plurality of bird's eye view images, and a signal that acquires a vehicle signal representing the behavior of the vehicle from the in-vehicle network (25) when the image acquisition unit acquires an image. An acquisition unit (11), a position determination unit (13) that determines the positional relationship of a plurality of bird's-eye view images using the vehicle signals acquired by the signal acquisition unit, and at least a part of the plurality of bird's-eye view images is a position determination unit And a synthesis unit (15) for creating a synthesized image arranged in the positional relationship determined by (1).

  The image processing apparatus according to the present disclosure further includes a prediction unit (17) for predicting the vehicle speed of the vehicle at a time point ahead by the signal delay time of the in-vehicle network using the time-dependent change of the vehicle signal acquired by the signal acquisition unit. When the vehicle speed predicted by the unit is equal to or lower than a preset threshold, a stop determination unit (19) that determines whether or not the vehicle is stopped by comparing a plurality of images acquired at different times using a camera. And). When the stop determination unit determines that the vehicle is stopped, the position determination unit determines the positional relationship when the vehicle is stopped.

According to the image processing device of the present disclosure, even when the behavior of the vehicle is not constant, the divergence between the synthesized image and the actual situation can be suppressed.
The image processing method of the present disclosure uses a camera mounted on a vehicle to acquire a plurality of images around the vehicle with a time difference (S1), and converts each of the plurality of images into a bird's eye view image to generate a plurality of images. When a bird's-eye view image is created (S4) and an image is acquired using a camera, a vehicle signal representing the behavior of the vehicle is acquired from the in-vehicle network (S2), and the positional relationship of a plurality of bird's-eye view images using the vehicle signal And a composite image in which at least some of the plurality of bird's-eye view images are arranged in the positional relationship is created (S9).

  The image processing method of the present disclosure further predicts the vehicle speed of the vehicle at a time point ahead by the signal delay time of the in-vehicle network using the time-dependent change of the vehicle signal (S5), and the predicted vehicle speed is less than or equal to a preset threshold In this case, it is determined whether or not the vehicle is stopped by comparing a plurality of images acquired at different times using a camera (S13, S14), and when it is determined that the vehicle is stopped. Then, the positional relationship when the vehicle is stopped is determined (S15).

According to the image processing method of the present disclosure, even when the behavior of the vehicle is not constant, the divergence between the synthesized image and the actual situation can be suppressed.
In addition, the code | symbol in the parenthesis described in this column and a claim shows the correspondence with the specific means as described in embodiment mentioned later as one aspect, Comprising: The technical scope of this invention is shown. It is not limited.

1 is a block diagram illustrating a configuration of an image processing device 1. FIG. 2 is a block diagram illustrating a functional configuration of the image processing apparatus 1. FIG. FIG. 3 is an explanatory diagram showing the arrangement of a camera 21 and a display 23 in a host vehicle 27. 3 is a flowchart illustrating processing executed by the image processing apparatus 1. It is explanatory drawing showing the method of estimating the vehicle speed of the own vehicle in the time ahead (DELTA) T from the present time T. FIG. It is explanatory drawing showing the method of producing the synthetic image G (i) at the present time. It is explanatory drawing showing the method of producing the synthesized image G (i) at the present time, when the movement amount ΔX is 0. 11 is an explanatory diagram illustrating a display example on the display 23. FIG. It is explanatory drawing showing the method of contrasting image P (i) and image P (i-1). It is explanatory drawing showing the own vehicle 27, the pole 35, and the stop line 37. FIG. It is a composite image after ΔT from the stop point, and is an explanatory diagram showing a composite image when a movement amount ΔX is always created based on a vehicle signal. It is explanatory drawing showing synthetic image G (i) which the image processing apparatus 1 produced after (DELTA) T from the stop time. It is explanatory drawing showing the method of contrasting image P (i) and image P (i-1) based on the position of the target 41. FIG.

Embodiments of the present invention will be described with reference to the drawings.
<First Embodiment>
1. Configuration of Image Processing Device 1 The configuration of the image processing device 1 will be described with reference to FIGS. The image processing device 1 is an in-vehicle device mounted on a vehicle. Hereinafter, the vehicle on which the image processing apparatus 1 is mounted is referred to as the own vehicle. The image processing apparatus 1 is configured around a known microcomputer having a CPU 3 and a semiconductor memory (hereinafter referred to as a memory 5) such as a RAM, a ROM, and a flash memory. Various functions of the image processing apparatus 1 are realized by the CPU 3 executing a program stored in a non-transitional physical recording medium. In this example, the memory 5 corresponds to a non-transitional tangible recording medium that stores a program. Further, by executing this program, a method corresponding to the program is executed. The number of microcomputers constituting the image processing apparatus 1 may be one or more.

  As shown in FIG. 2, the image processing apparatus 1 has a function configuration realized by the CPU 3 executing a program. As illustrated in FIG. 2, the image acquisition unit 7, the conversion unit 9, the signal acquisition unit 11, and the position determination unit 13. A synthesis unit 15, a prediction unit 17, a stop determination unit 19, and a display unit 20. The method for realizing these elements constituting the image processing apparatus 1 is not limited to software, and some or all of the elements may be realized using hardware that combines a logic circuit, an analog circuit, and the like. .

  In addition to the image processing apparatus 1, the host vehicle includes a camera 21, a display 23, and an in-vehicle network 25. As shown in FIG. 3, the camera 21 is mounted on the rear portion of the host vehicle 27. The camera 21 captures a landscape behind the vehicle 27 and creates an image. The rear of the host vehicle corresponds to the periphery of the host vehicle.

  When the host vehicle 27 is viewed from above, the optical axis 29 of the camera 21 is parallel to the longitudinal axis of the host vehicle 27. The optical axis 29 has a depression angle. When the host vehicle 27 is used as a reference, the optical axis 29 is always constant. Therefore, if the host vehicle 27 is not inclined and the road is flat, the range 31 included in the image captured by the camera 21 is always at a fixed position with respect to the host vehicle 27. The range 31 includes a road surface.

  As shown in FIG. 3, the display 23 is provided in the passenger compartment of the host vehicle 27. The driver of the host vehicle 27 can visually recognize the display 23. The display 23 is controlled by the image processing apparatus 1 and displays an image.

  The in-vehicle network 25 is connected to the image processing apparatus 1. The image processing apparatus 1 can acquire a vehicle signal representing the behavior of the host vehicle from the in-vehicle network 25. Specifically, the vehicle signal is a signal representing the vehicle speed of the host vehicle. An example of the in-vehicle network 25 is CAN (registered trademark).

  The vehicle signal transmitted by the in-vehicle network 25 is delayed by the signal delay time ΔT. That is, the vehicle signal that the image processing apparatus 1 acquires from the in-vehicle network 25 at time T is a vehicle signal that represents the behavior of the host vehicle at time (T−ΔT). The signal delay time ΔT is a positive value unique to the in-vehicle network 25 and is a known value.

2. Image Processing Performed by Image Processing Device 1 Image processing that is repeatedly executed by the image processing device 1 at a fixed period I will be described with reference to FIGS. The unit of period I is time. In the following, executing the process shown in FIG. 4 once may be referred to as one cycle.

In step 1 of FIG. 4, the image acquisition unit 7 acquires one image using the camera 21. The image processing apparatus 1 repeats the cycle with a constant period I. Therefore, the image acquisition unit 7 acquires a plurality of images with a time difference corresponding to the period I. For example, the image acquisition unit 7 acquires a first image in a cycle executed at time t ₀ , acquires a second image in a cycle executed at time (t ₀ + I), and acquires time (t ₀ + 2I). The third image is acquired in the cycle executed at), and the fourth image is acquired in the cycle executed at time (t ₀ + 3I).

  In step 2, the signal acquisition unit 11 acquires a vehicle signal from the in-vehicle network 25. Since each cycle includes step 2 together with step 1, the signal acquisition unit 11 acquires a vehicle signal when the image acquisition unit 7 acquires an image.

In step 3, the signal acquisition unit 11 stores the vehicle signal acquired in step 2 in the memory 5 in association with the time of acquisition.
In step 4, the conversion unit 9 converts the image acquired in step 1 in the current cycle into a bird's eye view image. A well-known method can be used as a method for converting into a bird's eye view image. As an example of a method for converting to a bird's eye view image, for example, a method described in JP-A-10-211849 can be used.

  Note that the image processing apparatus 1 repeatedly executes the process of step 4 for each cycle. The conversion unit 9 converts the image acquired in step 1 in the same cycle into a bird's eye view image each time the process of step 4 is performed. Therefore, the conversion unit 9 converts each of the plurality of images acquired in Step 1 into a bird's eye view image. The conversion unit 9 stores the bird's eye view image in the memory 5.

  In step 5, it is determined whether or not the timing for starting creation of the composite image has already arrived. The timing at which the creation of the composite image is started is the timing at which the host vehicle has moved by a preset distance since the bird's eye view image was first stored in step 4. If it is already time to start creating a composite image, the process proceeds to step 6; otherwise, the process ends.

In step 6, the prediction unit 17 predicts the vehicle speed of the host vehicle at a time ahead by ΔT from the current time T as follows.
The prediction unit 17 first reads out the vehicle signal stored in Step 3 in the current cycle and in Step 3 in the previous cycle. As described above, the vehicle signal represents the vehicle speed of the host vehicle. Next, as shown in FIG. 5, the prediction unit 17 plots the read vehicle signal on a graph in which the vertical axis is the vehicle speed and the horizontal axis is the time when the vehicle signal is acquired.

  Next, the prediction unit 17 calculates an approximate curve 33 representing the relationship between the time when the vehicle signal is acquired and the vehicle speed, using the time-dependent change in the vehicle speed appearing in this graph. The approximate curve 33 may be a temporary function related to time, a quadratic function, or a cubic function. Next, the prediction unit 17 uses the approximate curve 33 to predict the vehicle speed of the host vehicle at a time point that is ahead of the current time T by ΔT.

  In step 7, the stop determination unit 19 determines whether or not the vehicle speed predicted in step 6 is equal to or less than a preset threshold value. If the predicted vehicle speed exceeds the threshold value, the process proceeds to step 8, and if the predicted vehicle speed is equal to or less than the threshold value, the process proceeds to step 12.

  In step 8, the movement amount ΔX of the host vehicle in the period from the previous cycle to the current cycle is calculated. The movement amount ΔX is a value obtained by multiplying the vehicle speed represented by the vehicle signal acquired in step 2 in the current cycle by the period I. For example, when the vehicle speed is 60 km / h and the period I is 33 msec, ΔX is 0.55 m.

  In step 9, the synthesis unit 15 creates a synthesized image as follows. If the composition unit 15 has already started creating a composite image in the previous cycle, the composition unit 15 reads the content of the composite image stored in the memory 5 in step 10 in the previous cycle and the position of the composite image. The position of the composite image means a position when displayed on the display 23. Further, when the creation of a composite image is started in this cycle, the composite image is not read out.

  FIG. 6 shows an example of the read composite image. In FIG. 6, a composite image G (i-1) is a composite image created and stored in the previous cycle. The composite image G (i-1) is, for example, a combination of bird's-eye view images TV (i-1), TV (i-2), TV (i-3), and TV (i-4). TV (i-1) means the bird's eye view image created in step 4 in the cycle before (n + 1) times, and TV (i-2) means step 4 in the cycle before (n + 2) times. TV (i-3) means the bird's eye view image created in step 4 in the previous cycle (n + 3), and TV (i-4) means (n + 4) Means a bird's eye view image created in step 4 in the previous cycle. n is a natural number of 1 or more, and is a constant number.

  The composite image G (i-1) is created by the following process. First, in step 9 in the cycle four times before, a composite image G (i-4) composed of the bird's eye view image TV (i-4) was created. Next, in step 9 in the cycle three times before, the bird's eye view image TV (i-3) is added to the composite image G (i-4) to create a composite image G (i-3). Next, in step 9 in the previous two cycles, the bird's eye view image TV (i-2) was added to the composite image G (i-3) to create a composite image G (i-2). Next, in step 9 in the previous cycle, the bird's eye view image TV (i-1) is added to the composite image G (i-2), and the composite image G (i-1) is created.

  Next, the synthesis unit 15 moves the position of the synthesized image G (i−1) by kΔX in the direction opposite to the traveling direction D of the host vehicle. kΔX is a value obtained by multiplying the constant k and the movement amount ΔX. The constant k is the pixel pitch. The unit of pixel pitch is m / pix.

  Next, the synthesis unit 15 includes the bird's-eye view image TV (i) created in the cycle one time after the latest bird's-eye view image TV (i-1) included in the synthesized image G (i-1). In addition to the composite image G (i-1), a composite image G (i) in this cycle is created. For example, if TV (i−1) is the bird's eye view image created in step 4 in the cycle (n + 1) times before, TV (i) is created in step 4 in the cycle n times ago. It is a bird's-eye view image.

  The position of the bird's eye view image TV (i) to be newly added is always constant. Therefore, the positional relationship between the bird's eye view image TV (i) and the bird's eye view image TV (i-1), TV (i-2), TV (i-3), and TV (i-4) is determined by kΔX. The That is, kΔX corresponds to the positional relationship between a plurality of bird's-eye view images.

  The position of the bird's eye view image TV (i) is a position shifted to the direction D side by kΔX from the bird's eye view image TV (i−1). When the bird's-eye view image TV (i) and the composite image G (i-1) have an overlapping region, the content of the composite image G (i) in the overlapping region is the content of the bird's-eye view image TV (i). is there.

  There is a certain upper limit for the length in the vertical direction of the composite image G (i). Even if the bird's-eye view image TV (i) is added until the length in the vertical direction of the composite image G (i) reaches the upper limit value, the old bird's-eye view image is not deleted.

  On the other hand, as a result of adding the bird's eye view image TV (i), when the length of the composite image G (i) in the vertical direction exceeds the upper limit, the oldest bird's eye view image included in the composite image G (i) Is deleted. In the example shown in FIG. 6, the bird's eye view image TV (i-4) is deleted.

  In step 14 to be described later, when the movement amount ΔX becomes 0, the position of the composite image G (i−1) does not move as shown in FIG. The position of the bird's eye view image TV (i-1) in the composite image G (i-1) is the same as the position of the bird's eye view image TV (i) to be added. Therefore, the bird's eye view image TV (i-1) and the bird's eye view image TV (i) overlap with each other. The composite image G (i) is a composite image obtained by replacing the portion of the bird's eye view image TV (i-1) in the composite image G (i-1) with the bird's eye view image TV (i).

In step 10, the composition unit 15 stores the content and position of the composite image G (i) created in step 9 in the memory 5.
In step 11, the display unit 20 displays an image including the composite image G (i) created in step 9 on the display 23. An example of the display is shown in FIG. The composite image G (i) is displayed on the left middle of the display 23. In FIG. 8, an area displayed as “image in the traveling direction” is an area for displaying a real-time image of the camera 21. The area displayed as “actual image” is an area for displaying the bird's eye view image created in step 4 in the current cycle. The area displayed as “blank” is an area where nothing is displayed.

  The upper left area of the display 23 is an area for displaying a bird's eye view image obtained by converting an image acquired using the front camera. FIG. 8 is an image displayed when the host vehicle moves backward. The front camera is not used when the host vehicle is moving backward. Therefore, in FIG. 8, the upper left area on the display 23 is “blank”. When the host vehicle moves forward, a bird's eye view image obtained by converting an image acquired using the front camera is displayed in the upper left area of the display 23.

  If an affirmative determination is made in step 7, the process proceeds to step 12. In step 12, the stop determination unit 19 compares the image P (i) acquired in step 1 in the current cycle with the image P (i-1) acquired in step 1 in the previous cycle. . The image P (i) and the image P (i-1) correspond to a plurality of images acquired at different times using the camera 21.

  The comparison is performed as follows. As illustrated in FIG. 9, pixels having the same coordinates in the image P (i) and the image P (i−1) are compared, and differences in luminance, color, and the like are calculated. This process is performed for all the pixels in the image P (i) and the image P (i-1), and the difference of the entire image is calculated.

  In step 13, the stop determination unit 19 determines whether or not the host vehicle is stopped based on the comparison result in step 12. That is, if the difference of the entire image calculated in step 12 is equal to or less than a preset threshold value, it is determined that the host vehicle is stopped. Otherwise, the host vehicle is not stopped. to decide. If it is determined that the host vehicle is stopped, the process proceeds to step 14, and if it is determined that the host vehicle is not stopped, the process proceeds to step 8.

In step 14, the movement amount ΔX is set to zero. A movement amount ΔX of 0 corresponds to the host vehicle being stopped. Thereafter, the process proceeds to step 9.
3. Effects produced by the image processing apparatus 1 (1A) The image processing apparatus 1 predicts the vehicle speed of the host vehicle at a time point ahead by the signal delay time ΔT. Then, when the predicted vehicle speed is equal to or lower than a preset threshold, the image processing apparatus 1 compares the plurality of images acquired at different times using the camera 21 to determine whether the host vehicle is stopped. Judging. Furthermore, when the image processing apparatus 1 determines that the host vehicle is stopped, the image processing apparatus 1 sets the movement amount ΔX to zero.

Thereby, even when the behavior of the host vehicle changes, it is possible to suppress the difference between the synthesized image and the actual situation. This effect will be described with reference to the examples shown in FIGS.
As shown in FIG. 10, the host vehicle 27 moves backward between the left and right poles 35 and stops before the stop line 37. In other words, the behavior of the host vehicle 27 changes from a backward traveling state to a stopped state. The image processing apparatus 1 repeatedly creates a composite image and displays it on the display 23 when the host vehicle 27 moves backward and stops.

  It is assumed that the processing in steps 6, 7, 12, 13, and 14 is not performed and the movement amount ΔX is always calculated in step 8. In this case, a composite image at a time after ΔT from the time when the host vehicle 27 stops before the stop line 37 (hereinafter referred to as the stop time) is as shown in FIG. In this composite image, the position of the pole 35 </ b> A closest to the stop line 37 among the poles 35 is displayed above the actual position 39. That is, the synthesized image and the actual situation are different.

  This is due to the following reason. The image processing apparatus 1 acquires a vehicle signal at a time point that is traced back by the signal delay time ΔT. Therefore, the vehicle signal in the period from the stop time point to the time point after ΔT is a vehicle signal representing the vehicle speed when the host vehicle 27 is still moving backward. That is, the image processing apparatus 1 acquires a vehicle signal that represents the vehicle speed when the host vehicle 27 is moving backward, although the host vehicle 27 is actually stopped during the period from the stop point to the point after ΔT. Therefore, the movement amount ΔX calculated at a time after ΔT from the stop time becomes a value larger than the actual value.

  Then, when the composite image G (i) is created at a time after ΔT from the stop time, the composite image G (i−1) moves upwardly more than actual as shown in FIG. As a result, as shown in FIG. 11, the pole 35A appearing in the composite image G (i-1) moves upward, and is displayed above the actual position 39 in the composite image G (i). End up. In addition, the own vehicle 27 in FIG. 11 and FIG. 12 mentioned later is displayed by computer graphics.

  On the other hand, the image processing apparatus 1 can set the movement amount ΔX to 0 by making an affirmative determination in Step 7 after the stop point and making an affirmative determination in Step 13. Then, when creating the composite image G (i) at the time after ΔT from the stop time, the image processing apparatus 1 does not move the composite image G (i−1) as shown in FIG. As a result, as shown in FIG. 12, the pole 35A appearing in the composite image G (i-1) does not move upward and is correctly displayed at the actual position 39 in the composite image G (i). That is, the difference between the composite image and the actual situation is suppressed.

(1B) The vehicle signal acquired by the image processing device 1 represents the vehicle speed of the host vehicle. Therefore, the movement amount ΔX can be easily calculated.
(1C) The image processing apparatus 1 determines whether or not the host vehicle is stopped by comparing pixels having the same coordinates in a plurality of images. This makes it possible to easily and accurately determine whether the host vehicle is stopped.
Second Embodiment
1. Differences from the First Embodiment Since the basic configuration of the second embodiment is the same as that of the first embodiment, description of the common configuration will be omitted, and differences will be mainly described. Note that the same reference numerals as those in the first embodiment indicate the same configuration, and the preceding description is referred to.

In step 12, the stop determination unit 19 compares the image P (i) and the image P (i-1) by the following method.
As illustrated in FIG. 13, the stop determination unit 19 recognizes the same target 41 in each of the image P (i) and the image P (i−1). A known image recognition technique can be used for the recognition of the target 41. The target 41 is preferably one that does not move relative to the earth. Examples of such a target include a white line, a building, and a structure. Whether or not the target recognized in the image P (i) and the target recognized in the image P (i) are the same can be determined by the similarity of the shape, size, color, etc. of both.

Next, the stop determination unit 19 compares the position of the target 41 recognized in the image P (i) and the position of the target 41 recognized in the image P (i-1), and calculates their distance. .
In step 13, the stop determination unit 19 determines whether or not the host vehicle is stopped based on the comparison result in step 12. That is, if the distance calculated in step 12 is equal to or less than a preset threshold value, it is determined that the host vehicle is stopped, and otherwise, it is determined that the host vehicle is not stopped. If it is determined that the host vehicle is stopped, the process proceeds to step 14, and if it is determined that the host vehicle is not stopped, the process proceeds to step 8.

2. Effects Produced by Image Processing Device 1 According to the second embodiment described in detail above, the following effects are obtained in addition to the effects (1A) and (1B) of the first embodiment described above.

(2A) The image processing apparatus 1 recognizes the same target 41 in each of the image P (i) and the image P (i-1). And the image processing apparatus 1 judges whether the own vehicle has stopped by comparing the position of the target 41 in the image P (i) and the image P (i-1). This makes it possible to easily and accurately determine whether the host vehicle is stopped.
<Other embodiments>
As mentioned above, although the form for implementing this invention was demonstrated, this invention is not limited to the above-mentioned embodiment, It can implement in various deformation | transformation.

  (1) The camera 21 may be attached to the front end of the host vehicle 27. In that case, the image processing apparatus 1 can acquire an image of the front of the host vehicle using the camera 21. The front of the host vehicle corresponds to the periphery of the host vehicle. The image processing apparatus 1 can create a composite image using an image taken in front of the host vehicle.

  (2) The vehicle signal may be a signal representing the position of the host vehicle or the amount of movement of the host vehicle. For example, the vehicle signal may be one or more selected from the value of the pulse counter, the traveling direction of the host vehicle, and the steering angle of the host vehicle.

(3) The image to be compared in step 12 may be a bird's eye view image.
(4) A plurality of functions of one constituent element in the above embodiment may be realized by a plurality of constituent elements, or a single function of one constituent element may be realized by a plurality of constituent elements. . Further, a plurality of functions possessed by a plurality of constituent elements may be realized by one constituent element, or one function realized by a plurality of constituent elements may be realized by one constituent element. Moreover, you may abbreviate | omit a part of structure of the said embodiment. In addition, at least a part of the configuration of the above embodiment may be added to or replaced with the configuration of the other embodiment. In addition, all the aspects included in the technical idea specified only by the wording described in the claim are embodiment of this invention.

  (5) In addition to the image processing apparatus described above, a system including the image processing apparatus as a constituent element, a program for causing a computer to function as the image processing apparatus, and a non-transitory actual recording such as a semiconductor memory storing the program The present invention can also be realized in various forms such as a medium and an image processing method.

DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus, 3 ... CPU, 5 ... Memory, 7 ... Image acquisition unit, 9 ... Conversion unit, 11 ... Signal acquisition unit, 13 ... Position determination unit, 15 ... Composition unit, 17 ... Prediction unit, 19 ... Stop Judgment unit, 20 ... display unit, 21 ... camera, 23 ... display, 25 ... vehicle-mounted network, 27 ... own vehicle, 29 ... optical axis, 31 ... range, 33 ... approximate curve, 35 ... pole, 37 ... stop line, 41 ... target

Claims (9)

An image processing device (1),
An image acquisition unit (7) for acquiring a plurality of images around the vehicle with a time difference using a camera (21) mounted on the vehicle (27);
A conversion unit (9) for converting each of the plurality of images into a bird's eye view image to create a plurality of bird's eye view images;
A signal acquisition unit (11) for acquiring a vehicle signal representing the behavior of the vehicle from the in-vehicle network (25) when the image acquisition unit acquires the image;
A position determination unit (13) for determining a positional relationship between the plurality of bird's-eye view images using the vehicle signal acquired by the signal acquisition unit;
A synthesis unit (15) for creating a composite image in which at least some of the plurality of bird's-eye view images are arranged in the positional relationship determined by the position determination unit;
A prediction unit (17) for predicting a vehicle speed of the vehicle at a time point ahead by a signal delay time of the in-vehicle network using a time-dependent change of the vehicle signal acquired by the signal acquisition unit;
When the vehicle speed predicted by the prediction unit is equal to or lower than a preset threshold value, it is determined whether or not the vehicle is stopped by comparing a plurality of images acquired at different times using the camera. A stop determination unit (19);
With
When the stop determination unit determines that the vehicle is stopped, the position determination unit determines the positional relationship when the vehicle is stopped. The image processing apparatus according to claim 1,
The image processing apparatus, wherein the vehicle signal is a signal representing one or more selected from a vehicle speed, a pulse counter, a traveling direction, and a steering angle of the vehicle. The image processing apparatus according to claim 1, wherein:
The stop determination unit is an image processing apparatus that determines whether or not the vehicle is stopped by comparing pixels having the same coordinates in the plurality of images. The image processing apparatus according to claim 1, wherein:
The stop determination unit determines whether or not the vehicle is stopped by recognizing the same target in each of the plurality of images and comparing the position of the target in the plurality of images. Image processing device. An image processing method comprising:
Using a camera mounted on the vehicle, a plurality of images around the vehicle are acquired with a time difference (S1),
Each of the plurality of images is converted into a bird's eye view image to create a plurality of bird's eye view images (S4),
When the image is acquired using the camera, a vehicle signal representing the behavior of the vehicle is acquired from the in-vehicle network (S2),
Using the vehicle signal, determine the positional relationship of the plurality of bird's-eye view images (S8, S15),
Creating a composite image in which at least some of the plurality of bird's-eye view images are arranged in the positional relationship (S9);
Using the time-dependent change of the vehicle signal, the vehicle speed of the vehicle at a time point ahead by the signal delay time of the in-vehicle network is predicted (S5),
When the predicted vehicle speed is less than or equal to a preset threshold value, it is determined whether or not the vehicle is stopped by comparing a plurality of images acquired at different times using the camera (S13, S14). ),
If it is determined that the vehicle is stopped, the positional relationship when the vehicle is stopped is determined (S15). The image processing method according to claim 5,
The image processing method, wherein the vehicle signal is a signal representing one or more selected from a vehicle speed, a pulse counter, a traveling direction, and a steering angle of the vehicle. The image processing method according to claim 5, wherein:
The image processing method determines whether or not the vehicle is stopped by comparing pixels having the same coordinates in the plurality of images (S13). The image processing method according to claim 5, wherein:
In each of the plurality of images, the same target is recognized, and the position of the target in the plurality of images is compared to determine whether or not the vehicle is stopped (S13). .   The program which makes a computer function as each unit in the image processing apparatus of any one of Claims 1-4.

Images