JP2005318568A - Image compensation device and image compensation method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To compensate blurring at a photographed image caused by run of a vehicle. <P>SOLUTION: The image compensation device of the present invention comprises: a camera for photographing at the front of a vehicle continuously; a move vector calculation means for computing a move vector within an image for each object, based on the position of each object in each image continuously photographed with the camera; and a blurring compensation means for compensating blurring in the photographed image from the camera, by using the computed move vector from the move vector calculation means. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image correction apparatus and an image correction method for imaging the front of a vehicle. In particular, the present invention relates to an image correction apparatus and an image correction method for correcting a shake generated in an image due to traveling of a vehicle.

  2. Description of the Related Art Conventionally, a camera that corrects shake by following an object by moving an optical system is known. By mounting such a camera on a vehicle, it is possible to acquire an image in which blurring due to vibration or the like generated in the vehicle is corrected. In addition, the literature which disclosed the technique about such a vehicle-mounted camera was not found.

  Here, when the front is continuously imaged from the traveling vehicle, the object in the image appears to move in the image so as to flow from the center to the periphery. For this reason, when the vehicle travels at high speed, each object in the captured image is shaken. However, a camera with the above-described camera shake correction function can correct camera shake when the camera moves up and down or left and right with respect to the object, but if the distance between each object and the camera changes, Since blur occurs in a different direction every time, the blur cannot be corrected.

  Therefore, an object of the present invention is to provide an image correction apparatus and an image correction method that can solve the above-described problems. This object is achieved by a combination of features described in the independent claims. The dependent claims define further advantageous specific examples of the present invention.

  That is, according to the first aspect of the present invention, based on the camera that continuously images the front of the vehicle and the position of the object in each image continuously captured by the camera, the movement vector in the image of each object There is provided an image correction apparatus comprising: a movement vector calculation unit that calculates the image blur; and a blur correction unit that corrects a blur of an image captured by the camera using the movement vector calculated by the movement vector calculation unit. As a result, the image correction apparatus of the present invention can obtain a blur-free image. In particular, when an image is captured by a camera mounted on the vehicle while the vehicle is traveling at a high speed, the captured image is greatly shaken. However, the image correction apparatus of the present invention can reduce such blurring. In addition, since the image correction apparatus of the present invention can correct blur, it can take an image with a longer exposure time than a camera without a function for correcting blur. Thereby, even when the surroundings are dark, a bright image can be acquired.

  The movement vector calculation means obtains the vanishing point of the optical flow of the image captured by the camera, and obtains the direction of the movement vector on the assumption that the movement vector of each object is directed radially from the vanishing point. Good. Further, the movement vector calculation means may determine the length of the movement vector in the radial direction based on the position of the object in each image continuously captured by the camera. Thereby, the direction and length of the movement vector of each object can be easily calculated. In addition, for example, if roadside objects, road paint, similar vehicles, etc. in which the same pattern such as a safety zone display is repeated at equal intervals are included in the image as objects, the feature points of other objects will be mistaken in the process of calculating the movement vector. May be extracted. In such a case, an error is included in the calculated movement vector, but the image correction apparatus of the present invention can easily correct such an error in the direction and length of the movement vector.

  The image correction apparatus includes a distance measuring unit that measures a distance from the vehicle to each object, a speed measuring unit that measures the speed of the vehicle, a speed measured by the speed measuring unit, and each object measured by the distance measuring unit. The movement vector calculation means further includes an optical flow calculation means for calculating an optical flow in calculation of each object based on the distance to the first image based on the optical flow calculated by the optical flow calculation means. The position of the object in the second image may be obtained by calculating the position of the included object in the second image and searching for the same object in a region centered on the position. Thus, the position of the object can be quickly obtained by using the optical flow. Further, since the same object is searched for in an area centered on the position determined by the optical flow, the movement vector of the object can be obtained quickly and accurately.

  The image correction apparatus further includes a turning speed calculating means for calculating a turning speed of the vehicle based on the speed measured by the speed measuring means for measuring the speed of the vehicle and the rudder angle measured by the rudder angle measuring means. The calculation means may further calculate a calculation optical flow of each object based on the turning speed calculated by the turning speed calculation means. Thereby, the position of the object can be obtained quickly and accurately.

  The blur correction unit may correct the blur by adding an image obtained by differentiating the image inside each object in the direction of the movement vector to the original image. As a result, it is possible to easily reduce the blurring generated in each object.

  The image correction apparatus may further include a display device that displays an image in which the shake correction unit corrects the shake. Thereby, a clear image can be displayed to the driver of the vehicle.

  The image correction apparatus further includes an exposure time control unit that controls the exposure time of the camera and an acceleration measurement unit that measures the acceleration of the vehicle, and the exposure time control unit is configured to measure the acceleration of the vehicle measured by the acceleration measurement unit. If it is larger, the exposure time of the camera may be controlled to be shorter.

  The acceleration measuring means further measures a change in the turning speed of the vehicle, and the exposure time control means controls the exposure time of the camera to be shorter when the change in the turning speed of the vehicle measured by the acceleration measuring means is larger. May be.

  The acceleration measuring means measures at least one of the accelerator depression amount, the brake depression amount, and the steering wheel turning amount of the vehicle, and the exposure time control means includes an accelerator depression amount, a brake depression amount, and a steering wheel amount. When at least one change amount of the turning-back amount is larger, the exposure time of the camera may be controlled to be shorter.

  The above blur correction means corrects the blur in the region near the right side and the region near the left side of each image continuously captured by the camera by correcting the blur in the region near the upper side and the region near the lower side of the image. Alternatively, it may be performed with higher accuracy.

  The blur correction unit does not need to correct the blur in the region near the upper side and the region near the lower side of each image continuously captured by the camera.

  The movement vector calculation means obtains the vanishing point of the optical flow of the image captured by the camera, and the blur correction means corrects the blur in the area near the vanishing point of the optical flow, and the area near the vanishing point. It may be performed with lower accuracy than other areas.

  The blur correction unit may perform blur correction in a substantially central region of each image continuously captured by the camera with a lower accuracy than regions other than the substantially central region.

  According to the second aspect of the present invention, based on the step of continuously capturing the front of the vehicle with the camera and the position of the object in each image continuously captured by the camera, the movement vector in the image of each object A movement vector calculation step for calculating the image, a blur correction step for correcting the blur of the image captured by the camera using the movement vector calculated in the movement vector calculation step, and a steering angle measurement step for measuring the steering angle of the vehicle An image correction method comprising:

  The movement vector calculation step described above obtains the vanishing point of the optical flow of the image captured by the camera and corrects the direction of the movement vector assuming that the movement vector of each object is radially directed from the vanishing point. Also good.

  In the movement vector calculation step, the length of the movement vector may be corrected based on the corrected direction of the object in each image continuously captured by the camera.

  The image correction method includes a distance measuring step for measuring a distance from the vehicle to each object, a speed measuring step for measuring the speed of the vehicle, a speed measured by the speed measuring step, and each object measured by the distance measuring step. And an optical flow calculation step of calculating an optical flow in calculation of each object based on the distance to the first image based on the optical flow calculated by the optical flow calculation step. The position of the object in the second image may be obtained by calculating the position of the object included in the second image and searching for the same object in a region centered on the position.

  The image correction method further includes a turning speed calculation step for calculating a turning speed of the vehicle based on the speed measured by the speed measurement step for measuring the speed of the vehicle and the rudder angle measured by the rudder angle measurement step. The calculating step may further calculate a calculation optical flow of each object based on the turning speed calculated by the turning speed calculating step.

  In the blur correction step, blur may be corrected by adding an image obtained by differentiating an image inside each object in the direction of the movement vector to the original image.

  The image correction method may further include a display step for displaying an image in which the shake correction step corrects the shake.

  The blur correction step described above is to correct the blur in the area near the right side and the area near the left side of each image continuously captured by the camera by correcting the blur in the area near the upper side and the area near the lower side of the image. Alternatively, it may be performed with higher accuracy.

  In the blur correction step described above, it is not necessary to perform blur correction in the region near the upper side and the region near the lower side of each image continuously captured by the camera.

  The movement vector calculation step obtains the vanishing point of the optical flow of the image captured by the camera, and the blur correction step corrects the blur in the region near the vanishing point of the optical flow, and the region near the vanishing point. It may be performed with lower accuracy than other areas.

  In the blur correction step, the blur correction in a substantially central region of each image continuously captured by the camera may be performed with a lower accuracy than the region other than the central region.

  The above summary of the invention does not enumerate all the necessary features of the present invention, and sub-combinations of these feature groups can also be the invention.

  Hereinafter, the present invention will be described through embodiments of the invention. However, the following embodiments do not limit the invention according to the scope of claims, and all combinations of features described in the embodiments are included. It is not necessarily essential for the solution of the invention.

  FIG. 1 shows an example of an image correction apparatus 100 mounted on a vehicle 10 according to this embodiment. The image correction apparatus 100 includes a camera inside, and continuously images the front of the vehicle 10. FIG. 2 shows an image 12 that is an example of an image acquired by the image correction apparatus 100 according to the present embodiment. When the camera captures a series of frames while the vehicle 10 is traveling, the object 602 included in the approximate center of the image is a peripheral portion of the image 12 in the next frame when the vehicle 10 is traveling straight at a constant speed. Appears as an object 604 moved to. Here, when the vehicle 10 is traveling at high speed, the imaged object 604 may be shaken. Accordingly, an object of the image correction apparatus 100 of the present invention is to provide a driver with an image obtained by correcting object shake.

  When the positions of the same object in a plurality of frames are connected, a movement vector corresponding to the relative speed of the object with respect to the vehicle 10 is obtained. When the object is stationary, the movement vector generated by the traveling of the vehicle 10 is small near the center of the image and large around. Moreover, the movement vector of each object becomes larger as the distance from the vehicle 10 to the object is shorter. In this specification, the movement of each object in the image determined by the traveling state of the vehicle 10 and the distance of the object is called an optical flow, and the actual movement of the object in the image is called a movement vector. The optical flow of each object occurs radially from the vanishing point 600 when the traveling direction of the vehicle 10 matches the optical axis direction of the camera. Another object of the present invention is to correct blur using this optical flow.

  FIG. 3 shows an example of a functional configuration of the image correction apparatus 100 according to the present embodiment. The image correction apparatus 100 of this example includes a speed measurement unit 40, a camera 41, a steering angle measurement unit 42, a movement vector calculation unit 43, a shake correction unit 44, a turning speed calculation unit 46, and a display device 51.

  The camera 41 is disposed in advance in the vehicle 10 so that the optical axis coincides with the traveling direction of the vehicle 10 on which the image correction apparatus 100 is mounted, and continuously images the front of the vehicle 10.

  The movement vector calculation unit 43 calculates a movement vector in the image of each object based on the position of the object in each image continuously captured by the camera 41. The movement vector calculation means 43 of this example calculates a movement vector by obtaining the difference in the position of the same feature point between different frames in each object. Thereby, the optical flow of the image captured by the camera 41 is calculated. Here, the movement vector calculation means 43 calculates the vanishing point of the image from the optical flow of the image. The movement vector calculation means 43 of this example generates a straight line obtained by extending the movement vector of each object, and calculates a point at which the sum of the squares of the distances to each straight line is minimum as a vanishing point.

  Here, for example, when roadside objects, road paint, similar vehicles, etc. in which the same pattern such as a safety zone display is repeated at equal intervals are included in the image as objects, the feature points of other objects in the process of calculating the movement vector May be extracted. In such a case, the calculated movement vector includes an error. Therefore, the image correction apparatus 100 of the present invention corrects the direction and length of the movement vector.

  FIG. 4 is a diagram illustrating an example of an operation for correcting a movement vector according to the present embodiment. The movement vector calculation means 43 of the present invention corrects the direction of the movement vector on the assumption that the movement vector of each object is directed radially from the vanishing point. In this case, the direction of the corrected movement vector is a direction from the vanishing point to the position of each object. In the example of this figure, when correcting the movement vector 606, the movement vector calculation means 43 sets the direction from the vanishing point 600a to the object position 605 as the direction of the corrected movement vector.

  Moreover, the movement vector calculation means 43 corrects the length of the movement vector based on the corrected direction. The movement vector calculation means 43 of this example corrects the length of the movement vector by calculating the component of the movement vector before correction in the corrected direction. In the example of this figure, the length of the vector 608 corresponds to the corrected length. Thereby, the movement vector calculation means 43 can calculate easily the direction and length of the movement vector of each object.

  The blur correction unit 44 corrects the blur of the image captured by the camera 41 using the movement vector calculated by the movement vector calculation unit 43. The detailed operation of the shake correction unit will be described later. The display device 51 displays an image obtained by correcting the shake by the shake correction unit 44 to the driver. Thereby, a clear image can be displayed to the driver of the vehicle.

  As described above, the image correction apparatus 100 according to the present invention can generate a blur-free image. In particular, large blurring occurs in an image captured when the vehicle 10 is operated at high speed. However, the image correction apparatus 100 of the present invention can reduce such blurring. In addition, since the image correction apparatus 100 of the present invention can correct blur, it can take an image with a longer exposure time compared to a camera that does not have the function of correcting blur. Thereby, even when the surroundings are dark, a brighter image can be displayed to the driver.

  The camera 41 of the present invention may be a 3D camera including two cameras inside. In this case, the camera 41 calculates the distance from the vehicle 10 to the object using the shift of the position of the feature point of the object imaged by the two cameras included therein. In this case, the camera 41 is an example of a distance measuring unit, and measures the distance to the object with reference to the vehicle 10 using the positional deviation of the feature points of the object imaged by each camera.

  The speed measuring means 40 measures the speed of the vehicle 10, and the steering angle measuring means 42 measures the steering angle of the vehicle 10. The turning speed calculating means 46 calculates the turning speed of the vehicle based on the speed measured by the speed measuring means 40 and the rudder angle measured by the rudder angle measuring means 42. The turning speed calculating means 46 of this example calculates the turning speed of the vehicle 10 by multiplying the speed measured by the speed measuring means 40 by the magnitude of the change in the steering angle measured by the steering angle measuring means 42.

  The movement vector calculation means 43 has an optical flow calculation means 48. Here, the position of the vanishing point in the image is determined in advance from the traveling direction of the vehicle 10 and the direction of the optical axis of the camera 41. In this case, when the vehicle 10 travels, an optical flow in a direction from the vanishing point toward the object occurs in each object. The length of the optical flow is determined from the speed of the vehicle 10, the distance from the vehicle 10 to the object, and the direction of the object with respect to the vehicle 10. Therefore, the optical flow calculation means 48 of this example calculates the optical flow in calculation of each object based on the speed measured by the speed measurement means 40, the distance to each object, and the direction of the object, that is, the position of the object in the image. Is calculated. In this case, the calculated optical flow is an optical flow in an image generated in each object when the vehicle 10 goes straight.

  Further, when the vehicle 10 turns in a minute time, an optical flow in the left and right direction is generated in the object in the image. The direction of this optical flow is opposite to the direction in which the vehicle turns, and its magnitude is proportional to the turning speed of the vehicle 10. Therefore, the turning speed calculating means 46 calculates the turning speed from the speed calculated by the speed measuring means 40 and the rudder angle measured by the rudder angle measuring means 42, and the optical flow calculating means 48 uses the calculated turning speed. Calculate the optical flow calculated by turning. In this example, the optical flow of each object that occurs when the vehicle 10 turns is added to the optical flow of each object that occurs when the vehicle 10 goes straight, thereby calculating each object when the vehicle 10 moves while turning. Calculate the optical flow above.

  FIG. 5 is a diagram for explaining another example of the operation for correcting the movement vector according to the present embodiment. The movement vector calculation unit 43 calculates the position of the object included in the first image in the second image based on the optical flow in calculation of each object calculated by the optical flow calculation unit 48. Here, the first image and the second image are images captured continuously. In this example, the second image is an image captured after the first image is captured. The optical flow calculation means 48 obtains the position of the object in the second image by searching for the same object in a region centered on the calculated position using, for example, pattern matching processing.

  In the example of this figure, the movement vector calculation means 43 calculates a position 612 in the second image by calculating a calculation optical flow 610 starting from the vanishing point 600b, and an area centered on the position 612. In 614, the position 616 of the object in the second image is determined. The movement vector calculation unit 43 calculates an object movement vector 618 from the object position 611 in the first image and the calculated object position 616 in the second image. Thus, the position of the object can be quickly obtained by using the optical flow. Further, since the same object is searched for in the area 614 centered on the position determined by the optical flow, the movement vector of the object can be obtained quickly and accurately.

  Then, the shake correction unit 44 corrects the shake using the direction and length of the movement vector 618 calculated by the movement vector calculation unit 43, and the display device 51 displays the image corrected by the shake correction unit 44 to the driver. indicate.

  As described above, the image correction apparatus 100 according to the present invention can generate a blur-free image. In particular, the size of the object shake caused by the traveling of the vehicle 10 varies depending on the distance from the vehicle 10 to the object. However, in the present invention, since the shake is corrected after calculating the size of the shake according to the distance of each object, the shake of each object can be corrected appropriately.

  FIG. 6 is a flowchart illustrating an example of the operation of the image correction apparatus 100 according to the present embodiment. The camera 41 is arranged in advance in the vehicle 10 so that the optical axis coincides with the traveling direction of the vehicle 10 on which the image correction apparatus 100 is mounted, and continuously images the front of the vehicle 10 (S100).

  The movement vector calculation means 43 calculates the movement vector in the image of each object based on the position of the object in each image continuously captured by the camera 41 (S105). Then, the movement vector calculation unit 43 calculates the vanishing point of the image from the optical flow of the image (S110).

  Next, the movement vector calculation means 43 corrects the direction and length of each movement vector of each object (S115). Specifically, the direction of the movement vector is corrected assuming that the movement vector of each object is directed radially from the vanishing point. Moreover, the movement vector calculation means 43 calculates | requires the length of the radial direction of a movement vector based on the position of the object in each image imaged continuously with the camera. In this case, the radiation direction is a direction from the vanishing point toward the starting point of the movement vector.

  Then, the shake correction unit 44 corrects the shake of the image captured by the camera 41 using the movement vector calculated by the movement vector calculation unit 43 (S120). An image with the correction corrected is displayed to the driver (S125). As a result, the image correction apparatus 100 of the present invention can display an image without blurring for the driver of the vehicle 10.

  FIG. 7 is a flowchart showing an example of detailed operation of step S120. The shake that occurs in the image captured by the camera 41 is caused by a change in the relative position of the object and the vehicle 10 within the exposure time when the image is captured. Therefore, the direction of the generated blur is equal to the direction of the movement vector of the object, and the width of the generated blur is equal to the distance that the object has moved in the image within the exposure time.

  Here, the blur correction unit 44 receives the direction and length of the movement vector calculated by the movement vector calculation unit 43. Then, the shake correction unit 44 determines an area for correcting shake based on the direction and length of the received movement vector (S200). In this case, the shake correction unit 44 determines the direction indicated by the movement vector as the direction for correcting the shake with respect to the movement vector of each feature point. The blur correction unit 44 calculates the blur width based on the received movement vector and the exposure time of the image. The blur correction unit 44 of this example calculates the blur width by multiplying the value obtained by dividing the exposure time by the timing interval at which the movement vector is calculated, by the movement vector. As a result, the direction and width in which blurring is to be corrected for each feature point is determined.

  Then, the blur correction unit 44 differentiates the image inside each object in the direction of the movement vector in an area determined by the direction and width in which the blur is to be corrected with respect to the feature point (S205), and the obtained image. Is added to the original image (S210) to correct the blur. The shake correction unit 44 corrects the shake generated in each object by performing the above-described processing on all the objects. As a result, the image correction apparatus 100 according to the present invention can easily reduce blurring that occurs in each object. Then, the flowchart proceeds to step S125.

  FIG. 8 is a flowchart illustrating an example of another operation of the image correction apparatus 100 according to the present embodiment. In this operation example, the image correction apparatus 100 performs step S150 described below instead of the operations from step S105 to step S115 described in FIG.

  The camera 41 may be a 3D camera including two cameras inside, and the camera 41 uses the displacement of the position of the feature point of the object imaged by the two cameras included therein to move from the vehicle 10 to the object. Is measured (S300).

  The speed measuring means 40 measures the speed of the vehicle 10, and the steering angle measuring means 42 measures the steering angle of the vehicle 10 (S305). Further, the turning speed calculating means 46 calculates the turning speed of the vehicle 10 based on the speed measured by the speed measuring means 40 and the rudder angle measured by the rudder angle measuring means 42 (S310). The turning speed calculating means 46 of this example calculates the turning speed of the vehicle 10 by multiplying the speed measured by the speed measuring means 40 by the magnitude of the change in the steering angle measured by the steering angle measuring means 42.

  The optical flow calculation means 48 calculates the optical flow in calculation of each object based on the speed measured by the speed measurement means 40, the distance to each object measured by the camera 41, and the position of the object in the horizontal direction in the image. Is calculated (S315). Further, the optical flow calculation means 48 calculates the optical flow generated by the turning based on the turning speed calculated by the turning speed calculation means 46 and the direction in which the steering angle change has occurred, and adds it to the previously calculated optical flow. Thus, the optical flow in calculation of each object is calculated.

  Then, based on the optical flow calculated by the optical flow calculation unit 48, the movement vector calculation unit 43 calculates the position of the object in the next image to be captured (S320), and the movement vector calculation unit 43 is an area centered on the calculated position. By searching for the same object using, for example, pattern matching processing, the position of the object in the next image to be captured is obtained (S325). Then, the movement vector calculation unit 43 calculates the movement vector of the object from the calculated position of the object (S330). Then, the flowchart proceeds to step S120.

  FIG. 9 is a block diagram illustrating an example of a functional configuration of the image correction apparatus 105 according to the modification of the present embodiment. In addition to the functions of the image correction apparatus 100 described with reference to FIGS. 1 to 8, the image correction apparatus 105 according to the modification of the present embodiment controls the exposure time when an image is captured, or changes the captured image to the captured image. On the other hand, an object of the present invention is to perform shake correction more accurately and more efficiently by providing a function of correcting shake with different accuracy for each region.

  The image correction apparatus 105 includes a speed measurement unit 40, a steering angle measurement unit 42, a turning speed calculation unit 46, an acceleration measurement unit 47, an exposure time control unit 45, a camera 41, a movement vector calculation unit 43, a shake correction unit 44, and a display. A device 51 is provided. Of the elements included in the image correction apparatus 105 shown in this figure, elements denoted by the same reference numerals as the elements included in the image correction apparatus 100 shown in FIG. 3 correspond to those described with reference to FIGS. Since it has substantially the same function and configuration as the element, its description is omitted except for the differences.

  The acceleration measuring unit 47 measures the acceleration of the vehicle 10. Specifically, the acceleration measuring unit 47 may receive the speed of the vehicle 10 measured by the speed measuring unit 40 and measure the acceleration of the vehicle 10 based on the time change of the received speed of the vehicle 10. Then, the acceleration measuring unit 47 outputs the measured acceleration of the vehicle 10 to the exposure time control unit 45. The exposure time control unit 45 controls the exposure time of the camera 41 based on the acceleration of the vehicle 10 received from the acceleration measurement unit 47. The blur correction unit 44 corrects the blur of the image captured by the camera 41 in the same manner as the blur correction unit 44 included in the image correction apparatus 100 described with reference to FIGS. However, the shake correction unit 44 according to the modification of the present embodiment divides the captured image into a plurality of areas and corrects the shake with different accuracy in at least some of the plurality of areas. To do.

  FIG. 10 shows a first example of exposure time control by the exposure time control means 45 according to a modification of the present embodiment. In this example, the exposure time control means 45 controls the exposure time of the camera 41 to be shorter when the acceleration of the vehicle 10 measured by the acceleration measurement means 47 is larger. For example, when the acceleration α of the vehicle 10 is 0 or more and less than A, the exposure time control means 45 may cause the camera 41 to capture an image with an exposure time that is predetermined as a standard exposure time by a user or the like. . Further, when the acceleration α of the vehicle 10 is greater than or equal to A and less than B, the exposure time control means 45 may cause the camera 41 to capture an image with an exposure time obtained by reducing the standard exposure time by a reduction rate of 90%. . Further, when the acceleration α of the vehicle 10 is greater than or equal to B and less than C, the exposure time control means 45 may cause the camera 41 to capture an image with an exposure time obtained by shortening the standard exposure time by a reduction rate of 81%. .

  According to the image correction apparatus 105 according to the modification of the present embodiment, the shake correction unit 44 can correct the shake in the captured image. However, when the amount of blur in the image is very large, it may be difficult for the blur correction unit 44 to correct the blur with sufficiently high accuracy. However, as described above, according to the image correction apparatus 105 according to the modification of the present embodiment, when the acceleration of the vehicle 10 is larger, the exposure time is shortened, so that the image at the time of image capture The blur can be reduced. Therefore, the display device 51 can display a high-quality image without blurring.

  Further, the exposure time control means 45 may control the exposure time of the camera 41 based on a change in the turning speed of the vehicle 10. In this case, the acceleration measuring means 47 may receive the turning speed of the vehicle 10 from the turning speed calculation means 46 and measure the change in the turning speed of the vehicle 10 based on the time change of the received turning speed. The exposure time control unit 45 may control the exposure time of the camera 41 to be shorter when the change in the turning speed of the vehicle 10 received from the acceleration measurement unit 47 is larger.

  As described above, even when the change in the turning speed of the vehicle 10 is larger, it is possible to reduce the blurring of the image when the image is captured by shortening the exposure time. Therefore, the display device 51 can display a high-quality image without blurring.

  FIG. 11 is a flowchart illustrating an example of a processing flow in the image correction apparatus 105 according to the modification of the present embodiment. Of the steps shown in this figure, steps denoted by the same reference numerals as those described in FIGS. 6 to 8 perform substantially the same processing as the corresponding steps described in FIGS. Except for the differences, the description is omitted.

  First, the acceleration measuring means 47 measures the acceleration of the vehicle 10 (S200). Subsequently, the exposure time control means 45 controls the exposure time of the camera 41 based on the measured acceleration (S205). Subsequently, the image correction apparatus 105 performs blurring in the captured image by performing substantially the same processing as the steps denoted by the same reference numerals shown in FIGS. 6 to 8 in each step from S100 to S125. Is corrected and the corrected image is displayed.

  FIG. 12 shows another example of exposure time control by the exposure time control means 45 according to a modification of the present embodiment. FIG. 12A shows a second example in the exposure time control by the exposure time control means 45. FIG. 12B shows a third example of exposure time control by the exposure time control means 45. FIG. 12C shows a fourth example in the exposure time control by the exposure time control means 45.

  The acceleration measuring means 47 measures the acceleration of the vehicle 10 based on the speed of the vehicle 10 measured by the speed measuring means 40 as described in FIG. 9 and FIG. A change in the turning speed of the vehicle 10 may be measured based on the calculated turning speed of the vehicle 10. However, instead of this, the acceleration measuring means 47 may measure at least one of the accelerator depression amount, the brake depression amount, and the steering wheel turning amount of the vehicle 10. In this case, the exposure time control unit 45 determines that the camera 41 has a larger change amount of at least one of the accelerator stepping amount, the brake stepping amount, and the steering wheel turning amount measured by the acceleration measuring unit 47. The exposure time may be controlled to be shorter.

  For example, as shown in FIG. 12A, when the change amount V1 of the accelerator depression amount is 0 or more and less than A1, the exposure time control means 45 is predetermined as a standard exposure time by the user or the like. The camera 41 may capture an image with the exposure time. When the change amount V1 of the accelerator depression amount is not less than A1 and less than B1, the exposure time control means 45 takes an image on the camera 41 with the exposure time obtained by shortening the standard exposure time by 90%. You may let me. Further, when the change amount V1 of the accelerator depression amount is B1 or more and less than C1, the exposure time control means 45 takes an image on the camera 41 with an exposure time obtained by shortening the standard exposure time by a reduction rate of 80%. You may let me.

  For example, as shown in FIG. 12 (b), when the change amount V2 of the brake depression amount is not less than 0 and less than A2, the exposure time control means 45 displays an image on the camera 41 with the standard exposure time. You may make it image. When the amount of change V2 in the brake depression amount is not less than A2 and less than B2, the exposure time control means 45 captures an image on the camera 41 with the exposure time obtained by shortening the standard exposure time by 90%. You may let me. Further, when the change amount V2 of the brake depression amount is B2 or more and less than C2, the exposure time control means 45 captures an image on the camera 41 with an exposure time obtained by shortening the standard exposure time by a reduction rate of 80%. You may let me.

  Further, for example, as shown in FIG. 12C, when the change amount V3 of the turning amount of the handle is 0 or more and less than A3, the exposure time control means 45 displays an image on the camera 41 with the standard exposure time. You may make it image. When the change amount V3 of the turning amount of the handle is greater than or equal to A3 and less than B3, the exposure time control means 45 captures an image on the camera 41 with an exposure time obtained by shortening the standard exposure time by 90%. You may let me. When the change amount V3 of the steering wheel turning amount is B3 or more and less than C3, the exposure time control means 45 captures an image on the camera 41 with the exposure time obtained by shortening the standard exposure time by a reduction rate of 80%. You may let me.

  By controlling the exposure time by measuring changes in the acceleration and turning speed of the vehicle based on the amount of accelerator depression, brake depression, and steering turnback, as described above, the exposure time is controlled. Compared to the case where the exposure time is controlled based on the measurement result of the speed by the measuring unit 40 and the calculation result of the turning speed by the turning speed calculating unit 46, a change in the vehicle acceleration and the turning speed is detected with a simpler configuration. be able to.

  FIG. 13 shows an image 700 that is a first example of an image captured by the camera 41 according to a modification of the present embodiment. The blur correction unit 44 divides the captured image into a plurality of areas and corrects at least a part of the plurality of areas when correcting blur of each image continuously captured by the camera 41. In this case, the shake may be corrected with different accuracy. For example, as shown in the figure, the blur correction unit 44 divides the captured image 700 into four diagonal lines, and corrects the blur in the region 710 near the right side and the region 720 near the left side of the image. The correction may be performed with higher accuracy than the blur correction in the region 730 near the upper side and the region 740 near the lower side.

  Here, the accuracy in shake correction may be, for example, the number of feature points when the shake correction unit 44 corrects shake for feature points included in each image. The accuracy in blur correction is, for example, that the blur correction unit 44 differentiates an image in the vicinity of the feature point in each image in the direction of the movement vector, and adds the obtained image to the original image. It may be the rank of the derivative when correcting the shake at.

  Further, the movement vector calculation unit 43 may calculate the movement vector with different accuracy in at least some of the plurality of areas in the captured image in the same manner as the blur correction unit 44. Here, the accuracy in the calculation of the movement vector may be, for example, the detection accuracy of the correlation region when a known cross-correlation method is used for detecting corresponding objects in a plurality of images.

  When an image is taken from the front of a vehicle, the sky is usually shown at the top of the image, and the road surface is often shown at the bottom of the image. However, image blurring in the sky or on the road surface often does not matter to users such as the driver of the vehicle 10. Therefore, the image quality perceived by the user can be suppressed by correcting the blur for areas where the blur is not noticeable to the user, such as the upper and lower parts of the image, with lower accuracy than the left and right areas of the image. However, the time required for the shake correction process can be shortened.

  Further, the blur correction unit 44 may not perform blur correction in at least some of the plurality of regions in each captured image. For example, the shake correction unit 44 does not have to perform shake correction in the region 730 near the upper side and the region 740 near the lower side of the captured image 700. As a result, the image correction apparatus 105 can further reduce the time required for blur correction processing.

  FIG. 14 shows an image 800 that is a second example of an image captured by the camera 41 according to a modification of the present embodiment. In this example, the movement vector calculation unit 43 uses the optical flow calculation unit 48 to obtain the optical flow vanishing point 810 of the image 800 as described with reference to FIGS. Then, the blur correction unit 44 performs blur correction on the image 800 in a region near the vanishing point 810 of the optical flow with a lower accuracy than a region other than the region near the vanishing point 810. Specifically, the shake correction unit 44 performs shake correction in the region 820 near the vanishing point 810 with lower accuracy than the region 830 existing outside the region 820. Further, the shake correction unit 44 may correct the shake in the region 830 with a lower accuracy than the region 840 having a larger distance from the vanishing point 810 than the region 830.

  The blurring in the image captured by the camera 41 provided in the vehicle becomes smaller in a region closer to the vanishing point. That is, the blurring in the region closer to the vanishing point is often not noticed by the user. Therefore, by performing blur correction in a region closer to the vanishing point with lower accuracy, it is possible to reduce the time required for blur correction while suppressing deterioration in image quality felt by the user.

  Note that the blur correction unit 44 does not have to perform blur correction in the region near the vanishing point, instead of lowering the blur correction in the region near the vanishing point as compared with other regions. Thereby, the blur correction unit 44 can further reduce the time required for blur correction.

  FIG. 15 shows an image 900 that is a third example of an image captured by the camera 41 according to a modification of the present embodiment. In this example, the blur correction unit 44 performs blur correction in a substantially central region of the image 900 with a lower accuracy than regions other than the substantially central region. Specifically, the blur correction unit 44 performs blur correction in the region 920 near the image center 910 with a lower accuracy than the region 930 existing outside the region 920. In addition, the blur correction unit 44 may perform blur correction in the region 930 with a lower accuracy than the region 940 having a larger distance from the image center 910 than the region 930.

  When the front of the vehicle is imaged by the camera 41, the camera 41 is often provided so that the center point of the captured image is positioned near the vanishing point of the optical flow. Therefore, by performing blur correction in the central area of the captured image with lower accuracy than other areas, it is possible to reduce the time required for blur correction while suppressing the deterioration in image quality felt by the user. . In this case, since it is not necessary to perform the process of calculating the vanishing point of the optical flow, the time required for blur correction can be further shortened.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

1 is a diagram illustrating an image correction apparatus 100 mounted on a vehicle 10 according to an embodiment of the present invention. It is a figure which shows the image 12 which is an example of the image which the image correction apparatus 100 which concerns on embodiment of this invention acquires. It is a figure which shows an example of a function structure of the image correction apparatus 100 which concerns on embodiment of this invention. It is a figure explaining an example of the operation | movement which corrects the movement vector which concerns on embodiment of this invention. It is a figure explaining the other example of the operation | movement which corrects the movement vector which concerns on embodiment of this invention. It is a flowchart which shows an example of operation | movement of the image correction apparatus 100 which concerns on embodiment of this invention. It is a flowchart which shows an example of the detailed operation | movement of step S120. It is a flowchart which shows an example of other operation | movement of the image correction apparatus 100 which concerns on embodiment of this invention. It is a block diagram which shows an example of a function structure of the image correction apparatus 105 which concerns on the modification of embodiment of this invention. It is a figure which shows the 1st example of control of the exposure time by the exposure time control means 45 which concerns on the modification of embodiment of this invention. It is a flowchart which shows an example of the flow of a process in the image correction apparatus 105 which concerns on the modification of embodiment of this invention. It is a figure which shows the other example of control of the exposure time by the exposure time control means 45 which concerns on the modification of embodiment of this invention. It is a figure which shows the image 700 which is a 1st example of the image imaged with the camera 41 which concerns on the modification of embodiment of this invention. It is a figure which shows the image 800 which is the 2nd example of the image imaged with the camera 41 which concerns on the modification of embodiment of this invention. It is a figure which shows the image 900 which is the 3rd example of the image imaged with the camera 41 which concerns on the modification of embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Vehicle, 12 ... Image, 40 ... Speed measurement means, 41 ... Camera, 42 ... Steering angle measurement means, 43 ... Movement vector calculation means, 44 ... Shake correction Means 45 ... Exposure time control means 46 ... Turning speed calculation means 47 ... Acceleration measurement means 48 ... Optical flow calculation means 51 ... Display device 100 ... Image correction Device, 105... Image correction device

Claims (25)

A camera that continuously images the front of the vehicle;
A movement vector calculation means for calculating a movement vector in the image of each object based on the position of the object in each image continuously captured by the camera;
Using the movement vector calculated by the movement vector calculation means, a shake correction means for correcting blur of the image captured by the camera;
An image correction apparatus comprising:   The movement vector calculation means obtains a vanishing point of the optical flow of the image captured by the camera, and assumes that the movement vector of each object is radially directed from the vanishing point. The image correction apparatus according to claim 1, wherein the correction is performed.   The image correction apparatus according to claim 2, wherein the movement vector calculation unit corrects the length of the movement vector based on a corrected direction of an object in each image continuously captured by the camera. Distance measuring means for measuring a distance from the vehicle to each object;
Speed measuring means for measuring the speed of the vehicle;
Optical flow calculation means for calculating an optical flow in calculation of each object based on the speed measured by the speed measurement means and the distance to each object measured by the distance measurement means, The movement vector calculation means Calculates the position of the object included in the first image in the second image based on the optical flow calculated by the optical flow calculation means, and searches for the same object in a region centered on the position. The image correction apparatus according to claim 1, wherein the position of the object in the second image is obtained. Rudder angle measuring means for measuring the rudder angle of the vehicle;
A turning speed calculating means for calculating a turning speed of the vehicle based on the speed measured by the speed measuring means and the rudder angle measured by the rudder angle measuring means; and the optical flow calculating means further comprises the turning speed. The image correction apparatus according to claim 4, wherein an optical flow in calculation of each object is calculated based on the turning speed calculated by the calculation unit.   The image correction apparatus according to claim 1, wherein the shake correction unit corrects shake by adding an image obtained by differentiating an image inside each object in a direction of the movement vector to an original image.   The image correction apparatus according to claim 6, further comprising a display device that displays an image in which the shake correction unit corrects the shake. Exposure time control means for controlling the exposure time of the camera;
An acceleration measuring means for measuring the acceleration of the vehicle,
The image correction apparatus according to claim 1, wherein the exposure time control unit controls the exposure time of the camera to be shorter when the acceleration of the vehicle measured by the acceleration measurement unit is larger. The acceleration measuring means further measures a change in the turning speed of the vehicle;
The image correction apparatus according to claim 8, wherein the exposure time control unit controls the exposure time of the camera to be shorter when the change in the turning speed of the vehicle measured by the acceleration measurement unit is larger. The acceleration measuring means measures at least one of an accelerator depression amount, a brake depression amount, and a steering wheel turning amount of the vehicle;
The exposure time control unit controls the exposure time of the camera to be shorter when at least one change amount of the accelerator depression amount, the brake depression amount, and the steering wheel return amount is larger. The image correction apparatus described in 1.   The blur correction unit corrects blurring in a region near the right side and a region near the left side of each image continuously captured by the camera, and corrects a blur in a region near the top side and a region near the bottom side of the image. The image correction apparatus according to claim 1, wherein the image correction apparatus is performed with higher accuracy.   The image correction apparatus according to claim 11, wherein the shake correction unit does not perform shake correction in a region near the upper side and a region near the lower side of each image continuously captured by the camera. The movement vector calculation means obtains a vanishing point of the optical flow of the image captured by the camera,
The image correction apparatus according to claim 1, wherein the blur correction unit performs blur correction in a region near the vanishing point of the optical flow with a lower accuracy than a region other than a region near the vanishing point.   The image according to claim 1, wherein the blur correction unit corrects blur in a substantially central region of each of the images continuously captured by the camera with a lower accuracy than regions other than the substantially central region. Correction device. Continuously capturing the front of the vehicle with a camera;
A movement vector calculating step for calculating a movement vector in the image of each object based on the position of the object in each image continuously captured by the camera;
Using the movement vector calculated in the movement vector calculation step, a blur correction step of correcting blur of an image captured by the camera;
An image correction method comprising: a rudder angle measuring step for measuring a rudder angle of the vehicle.   The movement vector calculation step obtains the vanishing point of the optical flow of the image captured by the camera, and assumes the movement vector of each object is directed radially from the vanishing point. The image correction method according to claim 15 to be corrected.   The image correction method according to claim 16, wherein the movement vector calculation step corrects the length of the movement vector based on a corrected direction of an object in each image continuously captured by the camera. A distance measuring step for measuring a distance from the vehicle to each object;
A speed measuring step for measuring the speed of the vehicle;
An optical flow calculation step of calculating an optical flow in calculation of each object based on the speed measured in the velocity measurement step and the distance to each object measured in the distance measurement step; Calculates the position in the second image of the object included in the first image based on the optical flow calculated in the optical flow calculation step, and searches for the same object in a region centered on the position. The image correction method according to claim 15, wherein the position of the object in the second image is obtained. Rudder angle measuring step for measuring the rudder angle of the vehicle;
A turning speed calculating step of calculating a turning speed of the vehicle based on the speed measured by the speed measuring step and the rudder angle measured by the rudder angle measuring step;
The image correction method according to claim 18, wherein the optical flow calculation step further calculates a calculation optical flow of each object based on the turning speed calculated by the turning speed calculation step.   The image correction method according to claim 15, wherein the blur correction step corrects blur by adding an image obtained by differentiating an image inside each object in a direction of the movement vector to an original image.   The image correction method according to claim 20, further comprising a display step of displaying an image in which the blur correction step corrects the blur.   The blur correction step includes correcting blurs in a region near the right side and a region near the left side of each image continuously captured by the camera, and correcting a blur in a region near the top side and a region near the bottom side of the image. The image correction method according to claim 15, wherein the image correction method is performed with higher accuracy. The image correction method according to claim 22, wherein the shake correction step does not perform shake correction in a region near the upper side and a region near the lower side of each image continuously captured by the camera. The movement vector calculation step obtains a vanishing point of the optical flow of the image captured by the camera,
The image correction method according to claim 15, wherein the blur correction step performs blur correction in a region near the vanishing point of the optical flow with a lower accuracy than a region other than a region near the vanishing point.   The image correction according to claim 15, wherein the blur correction step performs blur correction in a substantially central region of each of the images continuously captured by the camera with lower accuracy than a region other than the central region. Method.

Images