JP2008042759A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus in which a vehicle can detect movements, without having to use car speed sensors. <P>SOLUTION: Four cameras 5-8 are mounted on a vehicle, to perform imaging in longitudinal and lateral directions. Motion vectors of objects photographed in images are acquired, respectively, from two images acquired by performing imaging using the cameras 5-8 at different times, and it is decided for each of the cameras 5-8 whether the vehicle is moving. Only when it is decided that the vehicle is moving, for all the cameras 5-8, will the movement of the vehicle finally be detected. Furthermore, when the movement of the vehicle is not detected and a moving object exists in the vicinity the vehicle, the image acquired by imaging the object is displayed on a liquid crystal display 20. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus capable of capturing an outside world with a camera mounted on a vehicle and detecting whether or not the vehicle is moving based on the obtained image.

  Conventionally, a camera is mounted on the front side, rear side, or left and right sides of a vehicle to capture the outside world of the vehicle, and various images are provided to the driver by displaying an image acquired by the imaging on a display provided in the vehicle. The system to do is put into practical use. For example, by installing a camera that captures a blind spot from the driver's seat of the vehicle, a portion that cannot be directly seen by the driver can be displayed on the display, so that an accident can be prevented.

  In Patent Document 1, there is an optical flow type rear information detection device that captures an image of the back of the host vehicle with a predetermined period by an imaging unit, calculates an optical flow from a plurality of obtained images, and detects an object approaching the host vehicle. Proposed. In this apparatus, the average value of the magnitude of the optical flow corresponding to the object approaching the host vehicle is calculated, and the risk of approaching the object is determined based on the average value. Can be given to the driver.

The optical flow is a representation of the movement of an object in an image obtained by imaging with a camera as a vector value, and compares two images, a current image and a previous image captured at different times, This can be obtained by searching for the corresponding position in the current image of the object copied in the previous image. By calculating a motion vector based on an image captured and acquired by a camera mounted on a vehicle, it can be determined whether or not a moving object such as a person or another vehicle exists around the vehicle. It is possible to prevent accidents by notifying the driver of the presence of moving objects. In particular, when starting a vehicle stopped at a parking lot or the like, it is highly effective in preventing accidents to notify the presence of an object moving around the vehicle.
JP 11-259634 A

  However, when determining the presence of a moving object by calculating an optical flow based on an image obtained by an image captured by a camera mounted on the vehicle, the object captured in the image has moved or the vehicle There is a problem that it is difficult to judge whether it has moved or not. The same applies to the optical flow type rear information detecting device described in Patent Document 1. Since the vehicle is equipped with a vehicle speed sensor that detects the vehicle speed, it can be determined that the vehicle has moved according to the output of the vehicle speed sensor, but the vehicle is moving at a low speed of about 3 km / h or less. Sometimes, the vehicle speed sensor cannot detect the vehicle speed in many cases, and there is a possibility that an erroneous determination occurs. Since there is a high possibility that the vehicle will move at a low speed in a parking lot, etc., and there is a high possibility that there will be people around the vehicle, without using a vehicle speed sensor to reliably prevent accidents in this situation Therefore, it is required to accurately determine whether or not the vehicle is moving.

  The present invention has been made in view of such circumstances, and an object of the present invention is to capture different directions around the vehicle with a plurality of imaging means and to copy the images obtained by each imaging means. When each change in the image of the target object is acquired, whether the vehicle has moved based on the acquired change is determined for each imaging means, and when it is determined that the vehicle has moved for all the imaging means An object of the present invention is to provide an image processing apparatus capable of accurately detecting movement of a vehicle from information in a plurality of directions by adopting a configuration for detecting movement of the vehicle.

  Another object of the present invention is to obtain a motion vector of an object captured in an image from a plurality of images captured at different time points by an imaging means, and whether or not the motion vector is a predetermined pattern. Therefore, it is possible to accurately determine whether or not the vehicle has moved from an image obtained by capturing an image in one direction, and more accurately determine whether or not the vehicle has moved. It is another object of the present invention to provide an image processing apparatus that can detect the above.

  Another object of the present invention is that the imaging means for imaging the front or rear of the vehicle moves when the acquired motion vector diffuses radially from the predetermined position or converges to the predetermined position. An image processing apparatus that can accurately determine by using an imaging unit that images the front or rear of the vehicle and that can more accurately detect forward or backward movement of the vehicle. is there.

  Another object of the present invention is that the imaging means for imaging the side of the vehicle is configured to determine that the vehicle has moved when the acquired motion vector is directed forward or backward of the vehicle. Accordingly, it is an object of the present invention to provide an image processing apparatus that can accurately determine using an imaging unit that images the side of the vehicle and can more accurately detect forward or backward movement of the vehicle.

  Another object of the present invention is to capture the periphery of the vehicle by an imaging means having a lens whose degree of curvature of the image formed at the end portion is larger than the central portion of the field of view, and correct the curved portion of the image. Thus, an object of the present invention is to provide an image processing device that can capture the periphery of a vehicle over a wide range and can detect movement of the vehicle more accurately using an imaging means. .

  Another object of the present invention is that a plurality of imaging means capture and acquire images based on a change in an image of an object captured in an image when movement of the vehicle is not detected. An object of the present invention is to provide an image processing apparatus capable of displaying an object moving around a vehicle and notifying a driver by selecting one or more images from a plurality of images and displaying them. .

  An image processing apparatus according to a first aspect of the present invention is mounted on a vehicle, picks up an image of the outside world of the vehicle, acquires an image, and moves the vehicle based on the image acquired by the image pickup unit. In the image processing apparatus including the detecting means for detecting, the imaging means is a plurality, the plurality of imaging means have different imaging directions, and based on a plurality of images obtained by capturing the images at different time points, respectively. An acquisition unit that acquires, for each of the imaging units, a change in the image of the object captured in the image, and whether or not the vehicle has moved for each of the imaging units based on the change acquired by the acquisition unit And determining means for detecting the movement of the vehicle when the determining means determines that the vehicle has moved for all the imaging means. To do.

  In the present invention, a plurality of imaging means are mounted on a vehicle to perform imaging in different directions. A change in the image of the object captured in the image is acquired from images acquired by the respective image pickup means at different times, and whether or not the vehicle has moved is determined for each image pickup means based on the change. To do. When it is determined that the vehicle has moved for all the imaging means, the movement of the vehicle is detected. If it is determined that the vehicle has been moved only by any one of the imaging units, and the remaining imaging units have determined that the vehicle has not moved, the movement of the vehicle is not detected. Since it is determined in a plurality of directions whether or not the vehicle has moved, even if an erroneous determination occurs in any direction, the movement of the vehicle is not erroneously detected in the end.

  In the image processing apparatus according to the second aspect of the invention, the acquisition unit acquires a change in the image of the object captured in the image as a motion vector, and the determination unit includes the acquisition unit. Whether or not the vehicle has moved is determined for each of the imaging means according to whether or not the acquired motion vector is a predetermined pattern.

  In the present invention, the motion vector is acquired by searching for the movement destination of the object captured in the image from a plurality of images captured at different times by the imaging means. Whether or not the vehicle has moved is determined according to whether or not the acquired motion vector is a predetermined pattern. If the imaging direction by the imaging means is determined, for example, the motion vector pattern when the vehicle moves forward or backward can be examined and stored in advance, so the acquired pattern is compared with the stored pattern. Thus, it can be determined whether or not the vehicle has moved.

  In the image processing apparatus according to the third aspect of the present invention, the plurality of imaging means include imaging means for imaging the front or rear of the vehicle, and the determination means is configured to acquire the image from the image acquired by the imaging means. When the motion vector acquired by the acquisition means diffuses radially from a predetermined position or converges to the predetermined position, it is determined that the vehicle has moved.

  In the present invention, the front or rear of the vehicle is imaged by the imaging means. When the above-mentioned motion vector is acquired from the image acquired by the imaging means, the motion vector diffuses radially around a predetermined position, so-called vanishing point, or converges to the vanishing point. It is determined that the vehicle has moved. For example, in an imaging unit that images the front of a vehicle, it is known that a motion vector that can be acquired from an image diffuses radially around a vanishing point when the vehicle moves forward. Therefore, it is possible to determine whether or not the vehicle has moved based on whether or not the acquired motion vector indicates this pattern.

  In the image processing apparatus according to a fourth aspect of the present invention, the plurality of imaging means includes imaging means for imaging the side of the vehicle, and the determination means acquires the image from the image acquired by the imaging means. When the motion vector acquired by the means is directed forward or backward of the vehicle, it is determined that the vehicle has moved.

  In the present invention, the side of the vehicle is imaged by the imaging means. When a motion vector is acquired from an image captured and acquired by the imaging unit, it is determined that the vehicle has moved when the motion vector is directed forward or backward. For example, in the imaging means for imaging the right side of the vehicle, when the vehicle moves forward, the motion vectors are substantially parallel from the front to the rear of the vehicle, that is, from the left to the right of the captured image toward the vanishing point at infinity. It is known to be. Therefore, it can be determined whether or not the vehicle has moved based on whether or not the acquired motion vector indicates this pattern.

  In the image processing apparatus according to the fifth aspect of the present invention, the imaging unit includes a lens having a greater degree of curvature of the image formed at the end than the center of the field of view, and the image captured by the imaging unit is curved. The image processing apparatus includes a correction unit that corrects a portion, and the acquisition unit acquires a motion vector from an image corrected by the correction unit.

  In the present invention, the periphery of the vehicle is imaged by an imaging means having a lens whose degree of curvature of the image formed at the end portion is larger than the central portion of the field of view, for example, a camera having a fish-eye lens or a wide-angle lens. Thereby, the periphery of a vehicle can be imaged over a wider range. Further, the curved portion of the captured image is corrected, and a motion vector is acquired from the corrected image. When a motion vector is acquired from an image in which the curved portion is not corrected, it is difficult to determine whether the pattern and the motion vector match as described above. To make a decision.

  According to a sixth aspect of the present invention, there is provided an image processing apparatus, comprising: a display unit configured to display an image captured and acquired by the imaging unit; and a detection unit configured to detect the movement of the vehicle for each imaging unit. And a selection unit configured to select an image to be displayed on the display unit from a plurality of images captured and acquired by the plurality of imaging units based on the acquired change.

  In the present invention, display means such as a liquid crystal display for displaying an image acquired by the image pickup means is provided. When the movement of the vehicle is not detected, one or a plurality of images are selected and displayed from a plurality of images captured and acquired by a plurality of imaging means. At this time, the image is selected by acquiring a change in the object captured in the image based on the images acquired and acquired by the respective imaging means at different times, and selecting an image having a change or an image having a large change. Thereby, the moving object around the vehicle can be displayed on the display means.

  In the case of the first invention, a plurality of image pickup means are mounted on the vehicle to pick up images in different directions, and the changes in the image of the object captured in the images obtained by the respective image pickup means are acquired and acquired. Whether or not the vehicle has moved based on the change is determined for each imaging unit, and when it is determined that the vehicle has moved for all the imaging units, either Even if a misjudgment occurs in the direction, the movement of the vehicle is not erroneously detected in the end, so that the movement of the vehicle can be detected accurately, and the vehicle can be detected without using a vehicle speed sensor. Since the movement can be detected, the movement can be detected even when the vehicle is moving at a low speed, and the reliability of the image processing apparatus in detecting the movement of the vehicle can be improved.

  In the case of the second invention, a motion vector is obtained by searching for a movement destination of an object captured in the image from a plurality of images captured at different time points by the imaging means, and the motion vector is a predetermined value. By determining whether or not the vehicle has moved according to whether or not it is a pattern, it is possible to more accurately determine the movement of the vehicle for each imaging unit, so that the movement of the vehicle is more accurately performed. Therefore, the reliability of the image processing apparatus in detecting the movement of the vehicle can be further improved.

  In the case of the third invention, the front or rear of the vehicle is imaged by the imaging means, and it is determined that the vehicle has moved when the acquired motion vector diffuses radially from the predetermined position or converges to the predetermined position. By adopting the configuration, it is possible to accurately detect that the vehicle has moved forward or backward from the image captured and acquired by the imaging unit, so that the reliability of the movement detection of the vehicle of the image processing apparatus can be further improved.

  According to the fourth aspect of the invention, the imaging means is configured to take an image of the side of the vehicle and determine that the vehicle has moved when the acquired motion vector is directed forward or backward of the vehicle. Since it is possible to accurately detect that the vehicle has moved forward or backward from an image acquired by imaging, it is possible to further improve the reliability of the image processing apparatus in detecting the movement of the vehicle.

  In the case of the fifth invention, the periphery of the vehicle is imaged by an imaging means having a lens whose degree of curvature of the image formed at the end portion is larger than the center portion of the field of view, and the motion vector is corrected by correcting the curved portion of the image. Since the vehicle periphery can be imaged over a wider range and the vehicle movement determination based on the motion vector can be performed more accurately, the movement detection of the vehicle of the image processing apparatus can be performed. The reliability in the can be further improved.

  Further, in the case of the sixth invention, when the movement of the vehicle is not detected, a plurality of images acquired and acquired by a plurality of imaging means based on the change in the image of the object captured in the image. By selecting and displaying one or more images from the images, objects moving around the vehicle can be displayed on a liquid crystal display or the like to give a warning to the driver. Safety can be improved and accidents can be prevented.

  Hereinafter, the present invention will be specifically described with reference to the drawings showing embodiments thereof. FIG. 1 is a schematic plan view showing the configuration of a vehicle equipped with an image processing apparatus according to the present invention. In the figure, reference numeral 1 denotes a vehicle, and the vehicle 1 includes four cameras 5 to 8 for capturing an image of the periphery. In FIG. 1, the imaging range of the cameras 5 to 8 is illustrated as a region surrounded by a broken line.

  A camera 5 that captures the front is mounted on the front of the vehicle 1, and a camera 6 that captures the rear is mounted on the rear. The left and right door mirrors of the vehicle 1 are equipped with cameras 7 and 8 that image the left and right sides of the vehicle 1. The cameras 5 to 8 are cameras having an angle of view of 180 ° or more, and by mounting the four cameras 5 to 8 on the four sides of the vehicle 1, the horizontal periphery of the vehicle 1 extends over 360 °. An image can be taken. By using these cameras 5 to 8, for example, the captured image is displayed on a display disposed in the vicinity of the driver's seat of the vehicle 1, and the driver indirectly visually observes a blind spot around the vehicle 1. So that you can drive.

  FIG. 2 is a block diagram showing the configuration of the image processing apparatus according to the present invention. An image processing apparatus mounted on the vehicle 1 includes an imaging unit 9 having cameras 5 to 8 and an ECU (Electronic Control Unit) 10 that mainly performs processing for converting an image captured by the cameras 5 to 8 into a display image. And a liquid crystal display 20 provided in the vicinity of the driver's seat of the vehicle 1 for displaying an image given from the ECU 10. The cameras 5 to 8 and the ECU 10 of the imaging unit 9 are connected to the ECU 10 and the liquid crystal display 20 by a connection cable or a network cable, respectively.

  The cameras 5 to 8 of the imaging unit 9 each have an imaging element in which image sensors by CCD or CMOS are arranged in a matrix, and one or a plurality of lenses for condensing light on the imaging element. It is an apparatus that performs imaging at a frequency of about 30 times per second by detecting light collected by a lens by an image sensor of an imaging device. As the lenses of the cameras 5 to 8, lenses having a wide angle of view such as wide-angle lenses or fish-eye lenses are used. The cameras 5 to 8 capture images around the vehicle 1 to acquire images, and give the acquired images to the ECU 10 respectively.

  The ECU 10 is provided with an image processing unit 11 that receives images from the cameras 5 to 8 of the imaging unit 9 and performs various image processing on the given images. In addition, the image processing unit 11 is provided with an image memory 12 for temporarily storing various images such as images given from the cameras 5 to 8 and images after image processing, and the cameras 5 to 8. A correction table memory 13 in which a correction table for correcting an image is stored is connected. The image memory 12 is constituted by a rewritable memory element such as SRAM or DRAM, and the correction table memory 13 is constituted by a non-volatile memory element such as mask ROM or EEPROM. In addition, a liquid crystal display 20 is connected to the image processing unit 11, and image processing is performed on images given from the cameras 5 to 8 so that the images can be displayed on the liquid crystal display 20.

  Since the cameras 5 to 8 of the imaging unit 9 perform imaging using a lens such as a wide-angle lens or a fisheye lens, the degree of curvature of the captured image gradually increases from the center toward the end. Therefore, the image processing unit 11 corrects the curved portion of the image acquired by the cameras 5 to 8 and displays the corrected image on the liquid crystal display 20. The curved portion is corrected using a correction table stored in advance in the correction table memory 13. The image processing unit 11 corrects the curved portion of the image by moving each pixel of the image given from the cameras 5 to 8 to a position determined by the correction table. The movement destination of the pixel is stored.

  The ECU 10 is mounted on the vehicle 1 based on detection results obtained from sensors (not shown) such as a vehicle speed sensor, an acceleration sensor, or a steering angle sensor mounted on the vehicle 1, and a processing result of the image processing unit 11. CPU 15 for controlling the electronic device and the image processing unit 11 is provided, and ROM 14 and RAM 16 are connected to CPU 15. The ROM 14 stores various control programs executed by the CPU 15 in advance, and is configured by a nonvolatile memory element such as a mask ROM, an EEPROM, or a flash memory. The RAM 16 stores temporary data generated when the CPU 15 performs control processing, arithmetic processing, or the like, and includes a rewritable memory element such as SRAM, DRAM, or flash memory. The CPU 15 reads out a control program from the ROM 14 and executes it to perform electrical control of the vehicle 1 such as ABS (Antirock Bracket System) or TCS (Traction Control System).

  Further, the image processing apparatus according to the present invention images the periphery of the vehicle 1 with the cameras 5 to 8 when the vehicle 1 is stopped, and a moving object such as a person or another vehicle exists around the vehicle 1. In this case, the moving object is displayed on the liquid crystal display 20 to perform a process of giving a warning to the driver. First, the image processing apparatus determines whether the vehicle 1 is moving or stopped based on the information on the vehicle speed detected by the vehicle speed sensor. If the vehicle 1 is at a low speed of about 3 km / h or less, the vehicle speed In addition, the movement of the vehicle 1 is detected based on the images acquired by the cameras 5 to 8 capturing the periphery.

  The movement detection of the vehicle 1 is obtained from the four cameras 5 to 8 by determining whether or not the vehicle 1 has moved for each of the cameras 5 to 8 based on images acquired by the cameras 5 to 8. One judgment result is comprehensively performed. 3 to 5 are schematic diagrams for explaining a method of determining the movement of the vehicle 1 from an image captured and acquired by one camera. As an example, the case of the camera 5 that images the front of the vehicle 1 is illustrated. It is shown.

  The movement determination of the vehicle 1 is performed based on two images captured and acquired by the camera 5 at different times. FIG. 3A is an image acquired by the camera 5 captured at time t, and FIG. 3B is an image acquired by the camera 5 captured after a predetermined time Δt from time t, that is, at time t + Δt. is there. Since the camera 5 performs imaging using a lens such as a wide-angle lens or a fish-eye lens, an image obtained by imaging is an image with a greatly curved end. In such an image, it is difficult to determine the moving distance, moving direction, and the like of the object captured in the image. Therefore, in the image processing apparatus according to the present invention, the image processing unit 11 performs correction processing to remove image distortion. It is like that.

  As described above, the image correction processing can be performed by reading a correction table stored in advance in the correction table memory 13 and moving each pixel of the image to a predetermined position. FIG. 4A is an image obtained by correcting the image acquired by the camera 5 at time t, and FIG. 4B is an image obtained by correcting the image acquired by the camera 5 at time t + Δt.

  Next, the image processing apparatus searches the movement destination and the movement amount of the object in the image at the time t + Δt of the object copied in the image at the time t on the basis of the two corrected images. It is obtained as a motion vector. The motion vector may be acquired for all the pixels in the image, but some target pixels may be extracted from the image at time t, and the motion vector may be acquired only for the extracted target pixel. A plurality of motion vectors can be acquired from the two images at time t and time t + Δt, and this motion vector is a so-called optical flow. FIG. 5 is an image in which a plurality of motion vectors are superimposed on the image at time t + Δt.

  Note that the method for searching for the destination of the object captured in the image is a known technique, and the details thereof will be omitted. However, a pixel of interest is extracted from the image at time t, and a predetermined centered on this pixel of interest is extracted. This is done by extracting an image area of a size and examining the identity of the image area for each image area of the same size in the image at time t + Δt. The identity of the image area can be checked, for example, by calculating the sum of the differences between the luminance values of the pixels included in the image area.

  After acquiring the motion vector based on the image at time t and the image at time t + Δt, the image processing apparatus determines whether the vehicle 1 has moved by examining the pattern of the motion vector. When the vehicle 1 moves forward, all the objects captured in the image approach the vehicle 1 with the camera 5 that captures the front, and as shown in FIG. 5, the motion vector of the entire image is one point (vanishing point) of the image. ), And the pattern diffuses radially. On the other hand, for example, when a person moves in front of the vehicle 1 (illustration is omitted), a motion vector exists in the moving direction of the person only in the portion where the person is shown in the image, and other parts. Has no motion vector. Therefore, in the image processing apparatus according to the present invention, it is determined that the vehicle 1 has moved (moved forward) when the acquired motion vector diffuses radially around the vanishing point.

  Although illustration is omitted, when the vehicle 1 moves backward, the motion vector converges toward the vanishing point, which can be determined. The same applies to the camera 6 that captures the rear of the vehicle 1. When the vehicle 1 moves forward, the motion vector converges toward the vanishing point, and when the vehicle 1 moves backward, the motion vector changes to the vanishing point. This can be determined because it diffuses radially in the center.

  6 and 7 are schematic diagrams for explaining a method of determining the movement of the vehicle 1 from an image captured and acquired by one camera, and in the case of the camera 8 that captures the right side of the vehicle 1 as an example. Is shown. However, FIG. 6 shows an image after correcting the distortion of the edge of the image captured and acquired by the camera 8, and the illustration of the image before the correction is omitted.

  The image processing apparatus according to the present invention also obtains and corrects an image acquired at time t (see FIG. 6A) and a time t + Δt for the camera 8 that captures the right side as well as the camera 5 that captures the front. Based on the acquired and corrected image (see FIG. 6B), a motion vector is acquired (see FIG. 7), and it is determined whether or not the vehicle 1 has moved. When the vehicle 1 moves forward, as shown in FIG. 7, the motion vector of the entire image converges to the traveling direction of the vehicle 1, that is, a substantially parallel pattern from the left to the right of the image (the vanishing point at the infinity point). Pattern). Further, when the vehicle 1 moves backward, the motion vector of the entire image becomes a substantially parallel pattern in the backward direction of the vehicle 1, that is, from the right to the left of the image. Therefore, the image processing apparatus can determine whether or not the vehicle 1 has moved with respect to the image acquired by the camera 8 based on whether or not the motion vector has such a substantially parallel pattern. .

  Although not shown, the same applies to the camera 7 that captures the left side of the vehicle 1. When the vehicle 1 moves forward, the motion vector becomes a substantially parallel pattern from the right to the left of the image. Can be determined since the motion vector has a substantially parallel pattern from the left to the right of the image.

  Further, even when the vehicle 1 is turning, if the time interval Δt between the two images is a minute time, the turning movement of the vehicle 1 can be approximated by forward or backward movement. It is possible to determine the movement of the vehicle 1 by the same method. For the cameras 5 and 6 that image the front or rear of the vehicle, the vanishing point moves to the left and right according to the steering angle of the steering, and the motion vector diffuses or converges radially from the moved vanishing point. It is possible to make a determination according to

  As described above, the image processing apparatus according to the present invention determines whether or not the vehicle 1 has moved in each of the front, rear, left, and right directions of the vehicle 1 based on the images acquired by the four cameras 5 to 8. It is like that. If the determination is made only from the image captured and acquired by one camera, for example, when the determination is made only from the image captured and acquired by the camera 5 that images the front of the vehicle 1, the other is directed toward the camera 5. Since the motion vector when the vehicle approaches is diffused radially, it may be erroneously determined that the vehicle 1 has moved. Therefore, in the image processing apparatus according to the present invention, the results of the determinations made for each of the four cameras 5 to 8 are totaled, and only when it is determined that the vehicle 1 has moved by all the cameras 5 to 8 Thus, the movement of the vehicle 1 is detected. Therefore, even if an erroneous determination occurs in any of the four cameras 5 to 8, the movement of the vehicle 1 can be detected more accurately because the final detection result is not affected. It is.

  Further, the image processing apparatus according to the present invention determines that the vehicle 1 is stopped when the movement of the vehicle 1 is not detected by the above-described method, and the cameras 5 to 8 are stopped while the vehicle 1 is stopped. When a moving object such as a person or another vehicle is imaged, this image is displayed on the liquid crystal display 20 to give a warning to the driver. Whether or not a moving object has been imaged by the cameras 5 to 8 can be determined by acquiring a motion vector in the same manner as described above.

  FIG. 8 is a flowchart showing the procedure of the moving object display processing around the vehicle performed by the image processing apparatus according to the present invention. The CPU 15 reads and executes a control program stored in advance in the ROM 14 to control the image processing unit 11. It is a process performed by doing. First, the image processing apparatus captures the periphery of the vehicle 1 with the cameras 5 to 8 (step S1), and performs image correction to correct the distortion at the edge of the acquired image as shown in FIGS. (Step S2), the corrected image is stored in the image memory 12 (Step S3).

  Next, the vehicle speed of the vehicle 1 is acquired from a vehicle speed sensor (not shown) (step S4), and it is checked whether or not the vehicle 1 is moving (step S5). When the vehicle 1 is moving (S5: YES), the process returns to step S1, and imaging by the cameras 5 to 8 is repeated until the vehicle 1 stops. When the vehicle 1 is not moving, that is, when the vehicle is stopped (S5: NO), a vehicle movement detection process for detecting the movement of the vehicle 1 based on images acquired by the cameras 5 to 8 is performed (step S6). .

  9 and 10 are flowcharts showing the procedure of the vehicle movement detection process performed by the image processing apparatus according to the present invention, which is the process performed in step S6 of the flowchart shown in FIG. In the vehicle movement detection process, first, one of the four cameras 5 to 8 is selected as a processing target camera (step S21). The latest image (image at time t + Δt) captured by the selected camera is read from the image memory 12 (step S22), and the image before the predetermined time Δt (image at time t) is read from the image memory 12 (step S23). Based on the read two images, a motion vector is acquired by the above-described method (step S24).

  Next, it is checked whether or not the processing target camera selected in step S21 is a side camera of the camera 7 that captures the left side or the camera 8 that captures the right side (step S25). If the camera is not a side camera, that is, the camera 5 that captures the front or the camera 6 that captures the rear (S25: NO), is the pattern in which the motion vector acquired in step S24 radiates from the vanishing point? If the pattern diffuses radially from the vanishing point (S26: YES), it is determined that the vehicle 1 is moving from the image acquired by capturing with this camera (step S29). .

  If the motion vector is not a pattern that diffuses radially from the vanishing point (S26: NO), it is further checked whether the motion vector is a pattern that converges to the vanishing point (step S27). When the motion vector is a pattern that converges to the vanishing point (S27: YES), it is determined that the vehicle 1 is moving from the image acquired by capturing with this camera (step S29). If the motion vector is not a pattern that converges to the vanishing point (S27: NO), it is determined that the vehicle 1 is stopped from the image captured and acquired by this camera (step S30).

  When the camera selected as the processing target is a side camera (S25: YES), the motion vector acquired in step S24 is parallel to the traveling direction or backward direction of the vehicle 1, that is, the left or right direction of the image. It is checked whether the pattern is correct (step S28). When the motion vector is a parallel pattern (S28: YES), it is determined that the vehicle 1 is moving from the image acquired by capturing with this camera (step S29). When the motion vector is not a parallel pattern (S28: NO), it is determined that the vehicle 1 is stopped from the image acquired by capturing with this camera (step S30).

  After determining whether the vehicle 1 has moved or stopped in step S29 or S30, it is checked whether the determination has been completed for all four cameras 5 to 8 (step S31). If all the cameras 5 to 8 have not been determined (S31: NO), the process returns to step S21, an unselected camera is selected as the camera to be processed, and the above determination is performed for all the cameras 5 to 8. . When the determination is completed for all the cameras 5 to 8 (S31: YES), it is checked whether or not the movement determination of step S29 has been made for all the cameras 5 to 8 (step S32). When the movement determination is made by all the cameras 5 to 8 (S32: YES), the movement of the vehicle 1 is detected (step S33), the vehicle movement detection process is terminated, and the process returns to the flowchart shown in FIG. When the movement determination is not made in all the cameras 5 to 8, that is, when the stop determination in step S30 is made by at least one of the cameras (S32: NO), without detecting the movement of the vehicle 1, It is detected that an object around the vehicle 1 is moving (step S34), the vehicle movement detection process is terminated, and the process returns to the flowchart shown in FIG.

  After the vehicle movement detection process is completed, it is checked from the detection result whether the vehicle 1 is moving (step S7). When the vehicle 1 is moving (S7: YES), the process returns to step S1, and imaging by the cameras 5 to 8 is repeated until the vehicle 1 stops. When the vehicle 1 is not moving, that is, when the vehicle is stopped (S7: NO), whether or not there is a moving object such as a person or other vehicle around the vehicle 1 from images acquired by the cameras 5 to 8 (Step S8). Whether there is a moving object around the vehicle 1 can be examined based on the motion vector acquired in the vehicle movement detection process in step S6. When there is a moving object around the vehicle 1 (S8: YES), a camera that picks up the moving object is selected from the four cameras 5 to 8 (step S9), and an image acquired by the selected camera is acquired. The image is displayed on the liquid crystal display 20 (step S10). At this time, when there are a plurality of moving objects around the vehicle 1 and there are a plurality of cameras that image the moving objects, the images acquired by the plurality of cameras may be divided and displayed on the liquid crystal display 20. One camera may be selected and displayed based on a predetermined priority order. Further, when there is no moving object around the vehicle 1 (S8: NO), or after displaying an image in step S10, it is checked whether or not an instruction to end the moving object display process is given by the driver ( In step S11), if the end instruction is not given (S11: NO), the process returns to step S1, the above-described processing is repeated, and if the end instruction is given (S11: YES), the image processing apparatus. Ends the moving object display process.

  In the image processing apparatus having the above configuration, motion vectors are acquired from images captured and acquired by the four cameras 5 to 8, and whether or not the vehicle 1 has moved based on the motion vectors is determined. It is possible to detect the movement of the vehicle 1 by making a configuration that detects the movement of the vehicle 1 only when the determination of the movement of the vehicle 1 is made by each of the cameras 5 to 8 and the movement of the vehicle 1 is determined. This can be performed accurately, and the moving object can be reliably displayed on the liquid crystal display 20 when the vehicle 1 is stopped. For the camera 5 that captures the front of the vehicle 1 and the camera 6 that captures the rear, the vehicle 1 moves when the acquired motion vector is a pattern that diffuses radially from the vanishing point or a pattern that converges to the vanishing point. By determining that the vehicle 1 is moving, the movement of the vehicle 1 can be reliably determined when the vehicle 1 moves forward or backward. For the camera 7 that captures the left side of the vehicle 1 and the camera 8 that captures the right side of the vehicle 1, the acquired motion vector is a parallel pattern toward the traveling direction or backward direction of the vehicle 1, that is, to the right or left of the image. Therefore, when the vehicle 1 moves forward or backward, the movement of the vehicle 1 can be reliably determined.

  In the present embodiment, four cameras 5 to 8 are mounted on the vehicle 1, but the present invention is not limited to this, and it is only necessary to mount at least two cameras. Moreover, although each camera 5-8 acquired the motion vector from the two images imaged and acquired at the time t and the time t + (DELTA) t, it was set as the structure which determines, It does not restrict to this but changes to two images It is good also as a structure which determines only whether there exists, and detects that the vehicle 1 is moving, when there exists a change in two images about all the cameras 5-8. Further, the cameras 5 to 8 are cameras having an angle of view of 180 ° or more. However, the present invention is not limited to this, and a camera having an angle of view within 180 ° may be used.

1 is a schematic plan view showing a configuration of a vehicle equipped with an image processing apparatus according to the present invention. It is a block diagram which shows the structure of the image processing apparatus which concerns on this invention. It is a schematic diagram for demonstrating the method to determine the movement of a vehicle from the image which the one camera imaged and acquired. It is a schematic diagram for demonstrating the method to determine the movement of a vehicle from the image which the one camera imaged and acquired. It is a schematic diagram for demonstrating the method to determine the movement of a vehicle from the image which the one camera imaged and acquired. It is a schematic diagram for demonstrating the method to determine the movement of a vehicle from the image which the one camera imaged and acquired. It is a schematic diagram for demonstrating the method to determine the movement of a vehicle from the image which the one camera imaged and acquired. It is a flowchart which shows the procedure of the moving object display process of the vehicle periphery which the image processing apparatus which concerns on this invention performs. It is a flowchart which shows the procedure of the vehicle movement detection process which the image processing apparatus which concerns on this invention performs. It is a flowchart which shows the procedure of the vehicle movement detection process which the image processing apparatus which concerns on this invention performs.

Explanation of symbols

1 vehicle 5, 6, 7, 8 camera (imaging means)
9 Imaging unit 10 ECU (detection means, acquisition means, determination means, correction means, selection means)
11 Image Processing Unit 12 Image Memory 13 Correction Table Memory 14 ROM
15 CPU
16 RAM
20 Liquid crystal display (display means)

Claims (6)

An image processing apparatus that is mounted on a vehicle and includes an imaging unit that captures an image of the outside world of the vehicle and acquires an image; and a detection unit that detects movement of the vehicle based on an image captured and acquired by the imaging unit In
The imaging means is a plurality, and the plurality of imaging means have different imaging directions,
An acquisition unit that acquires, for each imaging unit, a change in an image of an object captured in an image, based on a plurality of images acquired and acquired at different points in time by the imaging unit;
Determination means for determining whether or not the vehicle has moved for each of the imaging means based on the change acquired by the acquisition means;
The image processing apparatus according to claim 1, wherein the detection unit is configured to detect movement of the vehicle when the determination unit determines that the vehicle has moved for all of the imaging units. The acquisition means is configured to acquire a change in an image of an object captured in an image as a motion vector,
2. The determination unit according to claim 1, wherein the determination unit determines, for each of the imaging units, whether or not the vehicle has moved according to whether or not the motion vector acquired by the acquisition unit is a predetermined pattern. The image processing apparatus described. The plurality of imaging means includes imaging means for imaging the front or rear of the vehicle,
The determination means is that the vehicle has moved when the motion vector acquired by the acquisition means from the image acquired by the imaging means diffuses radially from a predetermined position or converges to the predetermined position. The image processing apparatus according to claim 2, wherein the determination is made. The plurality of imaging means includes imaging means for imaging the side of the vehicle,
The determination unit is configured to determine that the vehicle has moved when a motion vector acquired by the acquisition unit from an image captured and acquired by the imaging unit is directed forward or backward of the vehicle. Item 3. The image processing apparatus according to Item 2. The imaging means has a lens with a large degree of curvature of the image formed at the end portion than at the center portion of the visual field
A correction unit that corrects a curved portion of an image captured by the imaging unit;
The image processing apparatus according to claim 2, wherein the acquisition unit acquires a motion vector from an image corrected by the correction unit. Display means for displaying an image captured and acquired by the imaging means;
When the detection unit does not detect the movement of the vehicle, the display unit displays a plurality of images captured and acquired by the plurality of imaging units based on the change acquired for each imaging unit. The image processing apparatus according to claim 1, further comprising a selection unit that selects an image to be performed.

Images