JP2007240422A - Depression angle calculator, depression angle calculation method, depression angle calculation program, and image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To calculate a depression angle of a monocular camera, even when parallel lines such as road white lines are not imaged within an image picked up by the monocular camera. <P>SOLUTION: A vehicle controller shifts vertically and horizontally a current image and an image just before, when the parallel lines are not recognized within the image, detects a shift position (for example, (x, y)) where the both images have the highest correlation, and calculates a vertical shift amount (for example, y) of an object (for example, tree) imaged in the current image and the image just before, based on the detected shift position. The vehicle controller calculates the depression angle of the monocular camera, based on the calculated vertical shift amount. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a depression angle calculation device, a depression angle calculation method, a depression angle calculation program, and an image processing apparatus that calculate a depression angle of the camera from a plurality of images captured by an in-vehicle camera.

  Conventionally, there is a conventional technique for calculating a distance to a front object in an image captured by an in-vehicle camera (for example, a monocular camera or a stereo camera) (for example, see Patent Document 1). The calculation of such a distance requires the depression angle of the camera. As a method of calculating this depression angle, for example, when parallel lines such as road white lines are projected in an image captured by a monocular camera, A general method is to calculate an image vanishing point from the intersection of the parallel lines and calculate a depression angle based on the image vanishing point.

  Specifically, the angle of depression of the camera indicates the angle at which the optical axis of the camera is attached to a horizontal road surface. Then, as a principle of calculating the depression angle variation from the image vanishing point, for example, when the vehicle is on a horizontal road, assuming that the position of the image vanishing point on the image is always constant, the vehicle is on the road. When passing through a certain unevenness, the position of the image vanishing point is shifted according to the inclination of the unevenness, and the depression angle variation amount can be grasped from the displacement amount as illustrated in FIG.

  As shown in the figure, when calculating the distance to the object based on the position of the object seen on the image (for example, L), the depression angle variation obtained from the image vanishing point shift amount is used. By calculating the absolute depression angle between the current optical axis and the horizontal road and correcting the distance to the object with the absolute depression angle, an accurate distance is calculated.

Japanese Patent Laid-Open No. 9-48298

  By the way, as described below, the above-described conventional technique has a problem that the depression angle of the camera may not be calculated. That is, for example, when a parallel line such as a road white line is not displayed in an image captured by an in-vehicle monocular camera (that is, when a road where no road white line or the like is present is captured by a monocular camera) There is a problem that the image vanishing point cannot be calculated, and as a result, the depression angle of the monocular camera cannot be calculated.

  Therefore, the present invention has been made to solve the above-described problems of the prior art, and even if a parallel line such as a road white line is not projected in an image captured by an in-vehicle camera, the camera An object of the present invention is to provide a distance calculation device, a distance calculation method, and a distance calculation program capable of calculating the depression angle of the distance.

  In order to solve the above-described problems and achieve the object, the invention according to claim 1 is a depression angle calculation device that calculates a depression angle of a camera from a plurality of images captured by an in-vehicle camera. Based on the current image currently input and the previous image input before the current image, the vertical shift amount of the specific object projected in the current image and the previous image is determined. A shift amount calculating means for calculating, and a depression angle calculating means for calculating the depression angle of the camera based on the shift amount of the object calculated by the shift amount calculating means.

  Further, in the invention according to claim 2, in the above invention, the current image and the previous image are overlapped to shift both images in the vertical direction and the horizontal direction, and the correlation between the two images is highest. Shift position detection means for detecting a shift position is further provided, wherein the shift amount calculation means calculates the shift amount of the object based on the shift position detected by the shift position detection means.

  The invention according to claim 3 is the above invention, further comprising information acquisition means for acquiring information on a moving amount of the vehicle in the horizontal direction, wherein the shift position detection means is acquired by the information acquisition means. The shift position is corrected and detected in the horizontal direction based on information on the amount of movement of the vehicle in the horizontal direction.

  According to a fourth aspect of the present invention, in the above invention, the shift position detection means weights the center of the current image and the previous image low and weights the outside high, and the most correlation between the two images. The shift position at which the value becomes high is detected.

  The invention according to claim 5 further comprises image vanishing point obtaining means for obtaining an image vanishing point from parallel lines in an image captured by the camera in the above invention, wherein the depression angle calculating means is the image angle calculating means. When the image vanishing point is acquired by the vanishing point acquisition unit, the depression angle of the camera is calculated based on the image vanishing point instead of the shift amount of the object calculated by the shift amount calculation unit. Features.

  According to a sixth aspect of the present invention, in the above invention, in the depression angle calculation method for calculating the depression angle of the camera from the plurality of images captured by the in-vehicle camera, the plurality of images are actually input. Based on the current image and the previous image input before the current image, a shift amount calculating step for calculating a vertical shift amount of the specific object displayed in the current image and in the previous image And a depression angle calculating step of calculating a depression angle of the camera based on the shift amount of the object calculated by the shift amount calculation step.

  According to a seventh aspect of the present invention, in the above invention, in the depression angle calculation program for causing a computer to execute a method of calculating the depression angle of the camera from a plurality of images captured by an in-vehicle camera, Based on the current image currently input and the previous image input before the current image, the vertical shift amount of the specific object projected in the current image and the previous image is determined. The computer is caused to execute a shift amount calculation procedure to calculate and a depression angle calculation procedure to calculate the depression angle of the camera based on the shift amount of the object calculated by the shift amount calculation procedure.

  The invention according to claim 8 is an image processing apparatus that calculates a depression angle of the camera from a plurality of images captured by an in-vehicle camera and performs image processing using the depression angle. Based on the current image currently input and the previous image input before the current image, the vertical shift amount of the specific object displayed in the current image and the previous image is calculated. Shift angle calculation means for calculating the depression angle of the camera based on the shift amount of the object calculated by the shift amount calculation means, and the depression angle of the camera calculated by the depression angle calculation means. And a distance correction means for correcting the distance to the candidate object imaged by the camera.

  According to the first, sixth, and seventh aspects of the present invention, among a plurality of images captured by an in-vehicle camera (for example, a monocular camera or a stereo camera), a current image and a previous image input before the current image; The shift amount obtained by calculating the shift amount of a specific object (for example, a structure such as a utility pole or road sign or a pedestrian in the image) projected in the current image and the previous image based on The angle of depression of the in-vehicle camera can be calculated even when parallel lines such as road white lines are not shown in the image captured by the in-vehicle camera. It becomes.

  According to the second aspect of the present invention, the current image and the previous image are shifted in a superimposed state, and the shift position where the correlation between the current image and the previous image is highest (for example, the current image and the previous image are changed). The position where the correlation of the pixel values of the overlapped portion becomes the largest) is detected, and the shift amount of the object projected in the current image and the previous image is calculated based on this shift position, so that it is reflected in the image. Therefore, the shift amount of the object can be easily calculated without requiring recognition of the detected object, and the depression angle of the in-vehicle camera can be easily calculated.

  According to the invention of claim 3, the shift position having the highest correlation between the current image and the previous image is detected, and the shift position is detected as information on the amount of horizontal movement of the vehicle (for example, information based on the steering angle and yaw rate). ), The shift position where the correlation between the two images is highest is corrected and detected in the horizontal direction, so that the shift position of both images can be detected with high accuracy, even if the vehicle steers left and right. It becomes possible to accurately calculate the depression angle of the on-vehicle camera.

  According to the invention of claim 4, since the center of the current image and the previous image is weighted low and the outside is weighted high, the shift position having the highest correlation between the two images is detected. As a result, the weight of the center of the image that is likely to show moving objects is low, and the weight outside the image that shows stationary objects such as buildings, utility poles, and road signs is high. The shift position with the highest correlation can be accurately detected, and the depression angle of the in-vehicle camera can be accurately calculated.

  According to the invention of claim 5, when the image vanishing point can be acquired from an image captured by an in-vehicle camera (for example, a monocular camera or a stereo camera), the depression angle of the camera is set based on the image vanishing point. Rather than calculating the depression angle based on the shift amount of a specific object (for example, a structure such as a power pole or a road sign or a pedestrian in the image) projected in the current image and the previous image, It is possible to prioritize the calculation of the depression angle based on the image vanishing point, and it is possible to calculate the depression angle of the monocular camera with higher accuracy than always calculating the depression angle of the in-vehicle camera based on the shift amount of the object. It becomes.

  According to the eighth aspect of the present invention, the present image and the previous image are displayed based on the current image currently input and the previous image input before the current image among the plurality of images. The vertical shift amount of a specific target object (for example, a structure such as a power pole or road sign in the image or a pedestrian) is calculated, and the camera depression angle calculated based on the shift amount of the target object is calculated. To correct the distance to the candidate object (for example, a pedestrian), so even if there is a change in the depression angle of the camera due to pitching due to unevenness on the road, etc., based on the shift amount of the object on the image It is possible to correct and obtain the actual distance to the candidate object on the image using the depression angle of the camera to which the calculated depression angle variation amount is added.

  Embodiments of a depression angle calculation device, depression angle calculation method, and depression angle calculation program according to the present invention will be described below in detail with reference to the accompanying drawings. In the following, an embodiment in which an image processing device that realizes the depression angle calculation method according to the present invention is applied to a vehicle control device will be described.

  In the following first embodiment, the outline and features of the vehicle control device according to the first embodiment, the configuration of the vehicle to which the vehicle control device is applied, and the flow of processing of the vehicle control device will be described in order, and finally the effects of the first embodiment will be described. Will be explained.

[Outline and Features of Vehicle Control Device (Example 1)]
First, the outline and features of the vehicle control apparatus according to the first embodiment will be described with reference to FIG. FIG. 1 is a diagram for explaining the outline and features of the vehicle control device according to the first embodiment.

  The vehicle control apparatus according to the first embodiment is generally configured to calculate the depression angle of a camera from a plurality of images captured by a vehicle-mounted camera (for example, a monocular camera or a stereo camera). The main feature is that the depression angle of the camera is calculated even when parallel lines such as road white lines are not projected in the image.

  The outline and features will be described in detail. The vehicle control device recognizes, for example, whether there is a parallel line such as a road white line in an image captured by a monocular camera ((1) in FIG. 1). reference). As a result, when a parallel line is recognized in the image, the vehicle control device acquires an image vanishing point based on the parallel line in the image (see (2) in FIG. 1). And based on the acquired image vanishing point, this vehicle control apparatus calculates the depression angle of a monocular camera (refer (3) of FIG. 1).

  On the other hand, when a parallel line cannot be recognized in the image, the vehicle control device calculates a shift amount of a specific object (for example, a structure such as a tree, a road sign, a building, or a pedestrian). (See (4) in FIG. 1). Specifically, the vehicle control device shifts, for example, the current image currently input and the immediately preceding image input immediately before the current image in the vertical direction and the horizontal direction, and both images have the highest correlation. A position (for example, (x, y)) is detected, and based on the detected shift position, a vertical shift amount (for example, y) of an object (for example, a tree) displayed in the current image and the immediately preceding image is detected. ) Is calculated. Then, the vehicle control device calculates the depression angle of the monocular camera based on the calculated vertical shift amount (see (3) in FIG. 1).

  More specifically, if each frame of the image is captured at an interval of 10 msec, for example, the capture interval is extremely short compared to the travel amount of the vehicle, so that the amount of movement of the object in the image can be ignored. On the other hand, when a vehicle passes on a road with unevenness that causes pitching, depending on the degree of the unevenness, the influence on the image due to the change in the depression angle of the camera becomes very large compared to the movement of the vehicle (see FIG. 2). ).

  In other words, when a vehicle passes on an uneven road where pitching occurs, the camera moves with the vehicle in the vertical direction at the moment when the vehicle touches the unevenness on the road, and the image is displayed even during one frame. Since the object inside changes greatly, the depression angle variation of the camera is calculated using the shift amount of the object in the image. In order to avoid noise due to vehicle movement or object movement, it is desirable to perform filtering and determine that there is a depression angle variation amount due to pitching if there is a certain vertical movement amount or more.

  In addition, when the vehicle control device determines that there is a depression angle variation amount, the vehicle control device obtains the absolute depression angle between the current optical axis and the horizontal road from the depression angle variation amount, and determines the position of the object (for example, a pedestrian's foot). The actual distance may be calculated by correcting the distance on the image calculated by (position) with an absolute depression angle.

  For this reason, the vehicle control apparatus according to the first embodiment, like the main feature described above, even when parallel lines such as road white lines are not projected in the image captured by the in-vehicle camera, It becomes possible to calculate the depression angle of the in-vehicle camera. Even if there is a change in the depression angle of the camera due to pitching due to unevenness on the road, the depression angle of the camera is added using the depression angle of the camera added with the depression angle variation calculated based on the shift amount of the object on the image. The actual distance to the candidate object can be corrected and obtained.

[Configuration of Vehicle (Example 1)]
Next, the configuration of the vehicle to which the vehicle control device according to the first embodiment is applied will be described with reference to FIG. FIG. 3 is a block diagram illustrating a configuration of the vehicle to which the vehicle control device according to the first embodiment is applied. As shown in the figure, this vehicle has the following configuration. That is, the vehicle includes an imaging device (monocular camera) 10, a navigation unit 11, a preprocessing unit 12, an in-vehicle notification unit 13, and an image processing device 20.

  Among them, the navigation unit 11 is a means for setting and guiding a travel route from the specified vehicle position and map data by communicating with a GPS (Global Positioning System) artificial satellite. In addition, the navigation unit 11 supplies various information useful for vehicle driving operations such as vehicle position information, road shape, road width, and inclination to the driver via an in-vehicle notification unit 13 described later.

  The preprocessing unit 12 is a processing unit that performs preprocessing on an image transmitted from the imaging apparatus 10 (for example, a monocular camera), and includes a filter unit 12a and a contour extraction unit 12b. Among these, the filter unit 12a is a means for performing preprocessing (for example, sharpness, contrast adjustment, and saturation adjustment) for enhancing the outline of an object projected in an image. The contour extracting unit 12b is means for extracting the contour of the object in the image based on the filtering performed by the filter unit 12a.

  The in-vehicle notification unit 13 is a means for notifying information from the navigation unit 11 and a notification control unit 23 described later, and is configured by a monitor, a speaker, and the like. For example, an instruction from the notification control unit 23 is received, and an image indicating the presence of a pedestrian is output to a monitor to alert the driver, or a message or alarm sound is sent from a speaker to warn the driver. Or do.

  The storage unit 21 of the image processing apparatus 20 is a storage unit (storage unit) that stores data and programs necessary for various types of processing performed by the control unit 22. Particularly, the image storage unit 21a is closely related to the present invention. And a depression angle storage unit 21b.

  Among these, the image storage unit 21a is a unit that stores image data of an image captured by the imaging device 10 (for example, a monocular camera). Specifically, image data is received and stored from an image acquisition unit 22a described later.

  The depression angle storage unit 21b is means for storing the depression angle of the imaging device 10 (for example, a monocular camera). Specifically, in addition to storing the depression angle of the imaging device 10 set in advance before traveling the vehicle, the latest depression angle of the imaging device 10 (monocular camera) calculated by the depression angle calculation unit 22e described later is used as the depression angle calculation unit. Accept from 22e and store.

  The control unit 22 of the image processing apparatus 20 is a processing unit that has a predetermined control program, a program that defines various processing procedures, and an internal memory for storing required data, and executes various processes using these. In particular, those closely related to the present invention include an image acquisition unit 22a, an image vanishing point acquisition unit 22b, an image shift position detection unit 22c, a shift amount calculation unit 22d, a depression angle calculation unit 22e, and a distance calculation. Part 22f.

  The image vanishing point acquisition unit 22b corresponds to the “image vanishing point acquisition unit” recited in the claims, and the image shift position detection unit 22c similarly corresponds to the “shift position detection unit” and calculates the shift amount. The unit 22d corresponds to the “shift amount calculation unit”, and the depression angle calculation unit 22e corresponds to the “depression angle calculation unit”.

  Among these, the image acquisition unit 22a is a processing unit that receives and acquires an image input from the imaging device 10 (monocular camera), and outputs the received image to the image storage unit 21a as image data.

  When the image acquisition unit 22a receives an image input from the imaging device 10, the image vanishing point acquisition unit 22b recognizes whether there is a parallel line such as a road white line in the image. And a processing unit that acquires an image vanishing point based on the parallel lines. Specifically, when the image vanishing point acquisition unit 22b performs Hough transform (Hough transform) on the image data read from the image storage unit 21a and recognizes (extracts) parallel lines, for example, an optical flow is performed. An image vanishing point is acquired from the intersection of parallel lines calculated by a method or the like. Then, the image vanishing point acquisition unit 22b outputs the acquired image vanishing point to the depression angle calculation unit 22e. On the other hand, when the parallel line cannot be recognized (extracted), the image vanishing point acquisition unit 22b outputs a processing start signal to the image shift position detection unit 22c.

  The image shift position detection unit 22c is a processing unit that detects a shift position having the highest correlation between the current image currently input from the imaging apparatus 10 (monocular camera) and the immediately preceding image input immediately before the current image. is there. Specifically, when receiving the processing start signal from the image vanishing point acquisition unit 22b, the image shift position detection unit 22c reads the current image data and the immediately preceding image data from the image storage unit 21a, and displays the current image and the immediately preceding image. Shift position with the highest correlation between both images by shifting in the horizontal direction and the vertical direction (for example, a position (x, y) where the correlation value calculated from the pixel value of the overlapped portion of the current image and the previous image is the largest) ) Is detected. Then, the image shift position detection unit 22c outputs the detected shift position (for example, (x, y)) to the shift amount calculation unit 22d.

  The image shift position detection unit 22c performs object recognition (for example, object recognition by pattern matching) on the immediately preceding image, and uses the recognized object as a target for detecting the shift position of the image (for example, , Pedestrians, etc.) in the image, and determine whether the target object is substantially the same in the current image. Assuming that the correlation of the target object is high, the shift position of the image may be detected by detecting the shift position of the object.

  Note that the amount of change in the object in the image due to the movement of the vehicle can be ignored during one frame from the previous image to the current image. It can be an object for detection. If the stationary object is an object for detecting the shift position of the image, noise due to the movement of the object itself can be reduced, and the shift position can be detected with high accuracy.

  In the first embodiment, the current image that is an image of the current frame and the immediately preceding image that is an image of the frame immediately before the current frame are compared on the basis of the image frame capture interval. Is not limited to this. For example, an average image of the immediately preceding image, which is an image of a frame immediately before the immediately preceding image, and the immediately preceding image is generated, and the current image is compared with the generated average image. You may do it.

  The shift amount calculation unit 22d is a processing unit that calculates the shift amount of the object displayed in the current image and the immediately preceding image. Specifically, the shift amount calculation unit 22d determines that both images are based on the shift position (for example, (x, y)) having the highest correlation between the current image and the previous image received from the image shift position detection unit 22c. A vertical shift amount (for example, y) of the object (for example, a tree) being projected is calculated. That is, the shift amount calculated by the shift amount calculation unit 22d becomes the depression angle variation amount. Then, the shift amount calculation unit 22d outputs the calculated shift amount to the depression angle calculation unit 22e.

  The depression angle calculation unit 22e is a processing unit that calculates the depression angle of the imaging apparatus 10 (for example, a monocular camera) based on the image vanishing point or the image shift amount. Specifically, when the depression angle calculation unit 22e detects a deviation in the position of the image vanishing point by the image vanishing point acquisition unit 22b, the depression angle fluctuation amount grasped from the deviation amount is set in advance before the vehicle travels. The current depression angle (absolute depression angle) is calculated by adding to the depression angle.

  On the other hand, when the image vanishing point is not acquired by the image vanishing point acquisition unit 22b, the depression angle calculation unit 22e calculates the depression angle variation amount based on the vertical shift amount of the object received from the shift amount calculation unit 22d. The current depression angle (absolute depression angle) is calculated by calculating and adding to a depression angle that is set in advance before the vehicle travels. Then, the depression angle calculation unit 22e outputs the calculated depression angle to the distance calculation unit 22f.

  The distance calculation unit 22f is a processing unit that calculates the distance to the object displayed in the image. Specifically, the distance calculation unit 22f calculates the distance on the image to the object based on the ground contact position (for example, the pedestrian's foot position) of the object displayed in the image. The distance calculation unit 22f calculates the actual distance by correcting the distance to the object calculated based on the position of the object on the image with the absolute depression angle received from the depression angle calculation unit 22e. Then, the distance calculation unit 22f outputs the calculated distance information to the object to the notification control unit 23 and the vehicle control unit 24.

  The notification control unit 23 is a processing unit that performs notification control based on data (distance data to a candidate object) received from the distance calculation unit 22f. Specifically, in the first embodiment, the distance calculation unit 22d receives a distance to an object (for example, a pedestrian) in the image, and the notification control unit 23 determines whether the in-car notification unit is in accordance with the distance to the object. For example, a warning instruction by an image or sound indicating the presence of a pedestrian is output.

  The vehicle control unit 24 is a processing unit that performs vehicle travel control based on data (distance data to a candidate object) received from the distance calculation unit 22f. Specifically, in the first embodiment, the distance from the distance measurement unit 22d to the object (for example, a pedestrian) in the image is received, and the vehicle control unit 24 determines, for example, a brake according to the distance to the object. A speed reduction command by control, a collision avoidance command by steering wheel control, and a pre-crash command by vehicle braking control are output to each processing unit.

[Processing of vehicle control apparatus (Example 1)]
Then, the flow of the process of the vehicle control apparatus which concerns on Example 1 is demonstrated using FIG. FIG. 4 is a flowchart illustrating a process flow of the vehicle control device according to the first embodiment. In addition, the process demonstrated below is repeatedly performed during operation | movement of a vehicle control apparatus.

  As shown in the figure, when the power source (for example, ignition) of the vehicle control device is turned on (Yes at step 401) and an input of an image (one frame) captured by the imaging device 10 (for example, a monocular camera) is received ( In step S402, the image vanishing point acquisition unit 22b reads the image data from the image storage unit 21a and performs parallel line recognition processing (step S403). As a result, when a parallel line is recognized (Yes at Step S404), the image vanishing point acquisition unit 22b performs an image vanishing point acquisition process (Step S405). Subsequently, the image vanishing point acquisition unit 22b outputs the image vanishing point acquired by the image vanishing point acquisition process to the depression angle calculation unit 22e. The depression angle calculation unit 22e that has acquired the image vanishing point from the image vanishing point acquisition unit 22b calculates the depression angle of the imaging device 10 (step S406).

  Specifically, when the image vanishing point is acquired by the image vanishing point acquiring unit 22b, the depression angle calculating unit 22e preliminarily calculates the depression angle fluctuation amount obtained from the displacement amount of the image vanishing point before the vehicle travels. The current depression angle (absolute depression angle) is calculated by adding to the set depression angle.

  Here, returning to the description of step S404, when the parallel line cannot be recognized (No in step S404), the image shift position detection unit 22c detects the shift position of the image (step S407). Specifically, the image shift position detection unit 22c reads the current image data and the previous image data from the image storage unit 21a, shifts the current image and the previous image in the horizontal direction and the vertical direction, and sets the most of both images. A highly correlated shift position (for example, a position (x, y) where the correlation value calculated from the pixel value of the overlapped portion of the current image and the immediately preceding image is the largest) is detected, and the detected shift position (for example, ( x, y)) is output to the shift amount calculation unit 22d.

  Receiving the shift position from the image shift position detector 22c, the shift amount calculator 22d calculates the shift amount of the image (step S408). Specifically, the shift amount calculation unit 22d determines that both images are based on the shift position (for example, (x, y)) having the highest correlation between the current image and the previous image received from the image shift position detection unit 22c. A vertical shift amount (for example, y) of the object (for example, a tree) being projected is calculated, and the calculated shift amount is output to the depression angle calculation unit 22e.

  Receiving the shift amount from the shift amount calculation unit 22d, the depression angle calculation unit 22e calculates the depression angle of the imaging device 10 based on the vertical shift amount of the target object instead of the above-described image vanishing point (step S406). . Specifically, the depression angle fluctuation amount is calculated based on the vertical shift amount of the object received from the shift amount calculation unit 22d, and is added to the depression angle that is set in advance before the vehicle travels to thereby obtain the current depression angle ( (Absolute depression angle) is calculated.

  Subsequently, when a pedestrian is recognized in the image by the control unit 22 of the image recognition device 20 (Yes in step S409), the distance calculation unit 22f, for example, based on the pedestrian's foot position, the pedestrian The distance on the image is calculated (step S410). Next, the distance calculation unit 22f calculates the actual distance by correcting the distance on the pedestrian image by the absolute depression angle received by the depression angle calculation unit 22e (step S411). Then, the control unit 22 of the image recognition device 20 outputs pedestrian information to the notification control unit 23 and the vehicle control unit 24, and the distance calculation unit 22f sends the distance information to the notification control unit 23 and the vehicle control unit 24. The data is output (step S412). Although not shown in the figure, the notification control unit 23 and the vehicle control unit 24 that have received the output of the distance from the distance calculation unit 22f respectively perform notification control and vehicle travel control according to the received pedestrian information and distance information. I do.

  Here, returning to the description of step S409, when the pedestrian is not recognized in the image by the control unit 22 of the image recognition device 20 (No in step S409), the vehicle control device performs the processing on the image (frame). The process ends.

[Effect of Example 1]
As described above, according to the first embodiment, the vehicle control device includes a current image and a right before the current image among a plurality of images captured by a vehicle-mounted camera (for example, a monocular camera or a stereo camera). Based on the input immediately preceding image, calculate the shift amount of the object (for example, a structure such as a power pole or a road sign or a pedestrian in the image) projected in the current image and in the immediately preceding image, Since the depression angle of the monocular camera is calculated based on the acquired shift amount, the depression angle of the in-vehicle camera is calculated even when parallel lines such as road white lines are not shown in the image captured by the in-vehicle camera. It becomes possible to do.

  Further, according to the first embodiment, the vehicle control device shifts the current image and the immediately preceding image in a superimposed state, and shifts the position having the highest correlation between the current image and the immediately preceding image (for example, the current image and the previous image). The position where the correlation value calculated from the pixel value of the overlapped portion with the immediately preceding image is the largest) is detected, and the shift amount of the object displayed in the current image and the immediately preceding image is determined based on this shift position. Since it is calculated, the shift amount of the object can be calculated easily without the need to recognize whether the object projected in the image is a stationary object, and the angle of depression of the in-vehicle camera can be easily calculated It is possible to calculate.

  Further, according to the first embodiment, when the vehicle control apparatus can acquire the image vanishing point from the image captured by the vehicle-mounted camera, the vehicle control device calculates the depression angle of the camera based on the image vanishing point. Rather than calculating the depression angle based on the shift amount of the object (for example, a structure such as a power pole or road sign in the image or a pedestrian) displayed in the image and the immediately preceding image, the depression angle based on the image vanishing point is calculated. The calculation can be prioritized, and the depression angle of the monocular camera can be calculated with higher accuracy compared to the case where the depression angle of the vehicle-mounted camera is always calculated based on the shift amount of the object.

  Further, according to the first embodiment, the current image and the previous image are displayed on the basis of the current image currently input and the previous image input before the current image among the plurality of images. Calculate the vertical shift amount of a specific object (for example, a structure such as a utility pole or road sign or a pedestrian in the image), and use the depression angle of the camera calculated based on the shift amount of the object Since the distance to the candidate object (for example, a pedestrian) is corrected, even if there is a change in the depression angle of the camera due to pitching due to unevenness on the road, it was calculated based on the shift amount of the object on the image It is possible to correct and obtain the actual distance to the candidate object on the image using the depression angle of the camera to which the depression angle variation is added.

  Although the embodiments of the present invention have been described so far, the present invention may be implemented in various different forms other than the embodiments described above. Therefore, another embodiment included in the present invention will be described below.

(1) Shift position detection considering the amount of horizontal movement of the vehicle In the first embodiment, the image shift position detection unit acquires information on the amount of horizontal movement of the vehicle (for example, steering angle and yaw rate). Then, the shift position of the image may be detected by correcting it in the horizontal direction. For example, a target object (for example, a tree or the like) is stored by performing object recognition in the immediately preceding image, and correction is performed to restore the object by the amount of movement in the horizontal direction of the vehicle as illustrated in FIG. After detecting, the shift position is detected.

  For this reason, the vehicle control device detects the shift position having the highest correlation between the current image and the immediately preceding image, and determines the shift position based on information on the amount of movement of the vehicle in the horizontal direction (for example, the steering angle and yaw rate). Information), the shift position having the highest correlation between the two images is corrected and detected in the horizontal direction, so that the shift position of both images can be accurately detected, and the vehicle is steered left and right. In addition, it becomes possible to accurately calculate the depression angle of the on-vehicle camera.

(2) The shift position is detected by increasing the weight outside the image and decreasing the weight at the center. In the first embodiment, the image shift position detector is located outside the current image and the previous image. The shift position of the image may be detected by increasing the weight and decreasing the center weight. For example, as illustrated in FIG. 6, an image having a high possibility that a stationary object such as a tree is likely to appear in the image is high and a moving object such as a person is likely to be reflected in the image. The center weight is lowered to detect the shift position of the image.

  For this reason, the vehicle control device detects the shift position where the correlation between the two images is the highest after weighting the center of the current image and the immediately preceding image low and increasing the outside. The weight of the center of an image that is likely to show moving objects such as a person is low, and the weight outside the image that shows static objects such as buildings, utility poles, and road signs is high, resulting in a gap between both images. Thus, it is possible to accurately detect the shift position having the highest correlation, and to accurately calculate the depression angle of the monocular camera.

  Alternatively, the shift amount of the object may be detected first on the outer side and the inner side of the image, respectively, and the final shift amount may be calculated by the equation shown in FIG.

(3) Calculation of depression angle only by shift amount In the first embodiment, when an image vanishing point cannot be obtained from an image, the depression angle of the vehicle-mounted camera is calculated based on the shift amounts of the current image and the immediately preceding image. Although the case has been described, the present invention is not limited to this, and the depression angle of the in-vehicle camera may be always calculated based on the shift amount of the current image and the immediately preceding image. For example, the first depression angle of the monocular camera based on the attachment angle when attaching the monocular camera to the vehicle is stored in the depression angle storage unit, and the change amount of the depression angle calculated based on the shift amount of the current image and the immediately preceding image is The depression angle of the monocular camera is calculated by adding to the initial depression angle of the monocular camera stored in the depression angle storage unit.

  For this reason, this vehicle control device is provided with an in-vehicle camera even when parallel lines such as a road white line are not projected in an image captured by the in-vehicle camera and an image vanishing point cannot be obtained. It becomes possible to calculate the depression angle.

(4) Device configuration, etc. In addition, among the processes described in the above embodiments, all or a part of the processes described as being automatically performed can be performed manually or manually. All or part of the processing described as being performed can be automatically performed by a known method. In addition, the processing procedure, control procedure, specific name, and information including various data and parameters shown in the above-described document and drawings can be arbitrarily changed unless otherwise specified.

  Further, each component of the image processing apparatus 20 shown in FIG. 2 is functionally conceptual and does not necessarily need to be physically configured as illustrated. That is, the specific form of dispersion / integration of the image processing apparatus 20 is not limited to the one shown in the figure. For example, the image shift position detection unit 22c and the shift amount calculation unit 22d may be integrated in whole or in part. Depending on the load and usage status of the device, it can be configured to be functionally or physically distributed and integrated in arbitrary units. Further, all or a part of each processing function performed in the image processing apparatus 20 is realized by a CPU and a program analyzed and executed by the CPU, or is realized as hardware by wired logic. obtain.

  Also, as described in the first embodiment, the depression angle calculated by the depression angle calculation unit is received, and the distance calculation unit calculates an actual distance to a candidate object (for example, a pedestrian) in the image, and the actual distance. The vehicle control unit may perform vehicle travel control based on the above, and the notification control unit may configure a vehicle control device that performs notification control.

  The depression angle calculating method (see FIG. 3) described in the first embodiment can be realized by executing a program prepared in advance on a computer such as a personal computer or a workstation. This program can be distributed via a network such as the Internet. The program can also be executed by being recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, an MO, and a DVD, and being read from the recording medium by the computer.

  As described above, the depression angle calculation device, depression angle calculation method, and depression angle calculation program according to the present invention are useful when calculating the depression angle of the monocular camera from a plurality of images captured by the in-vehicle monocular camera, It is suitable for calculating the depression angle of a monocular camera even when a parallel line such as a road white line is not projected in the image captured by the monocular camera.

BRIEF DESCRIPTION OF THE DRAWINGS It is a figure for demonstrating the outline | summary and the characteristic of the vehicle control apparatus which concern on Example 1. FIG. BRIEF DESCRIPTION OF THE DRAWINGS It is a figure for demonstrating the outline | summary and the characteristic of the vehicle control apparatus which concern on Example 1. FIG. 1 is a block diagram illustrating a configuration of a vehicle to which a vehicle control device according to a first embodiment is applied. 3 is a flowchart illustrating a process flow of the vehicle control device according to the first embodiment. FIG. 10 is a diagram for explaining a case where the shift position of an image is corrected based on information on a horizontal movement amount of a vehicle according to a second embodiment. FIG. 10 is a diagram for explaining a case where a shift position is detected by increasing the weight on the outside of the image according to the second embodiment and decreasing the weight at the center. FIG. 10 is a diagram illustrating mathematical formulas according to the second embodiment. It is a figure for demonstrating a prior art.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image pick-up device 11 Navi part 12 Pre-processing part 13 In-car notification part 20 Image processing apparatus 21 Storage part 21a Image storage part 21b Angle storage part 22 Control part 22a Image acquisition part 22b Image vanishing point acquisition part 22c Image shift position detection part 22d Shift Quantity calculation unit 22e Depression angle calculation unit 22f Distance calculation unit 23 Notification control unit 24 Vehicle control unit

Claims (8)

In a depression angle calculating device that calculates the depression angle of the camera from a plurality of images captured by an in-vehicle camera,
Of the plurality of images, based on the current image currently input and the previous image input before the current image, the current object and the specific object displayed in the previous image Shift amount calculating means for calculating the shift amount in the vertical direction;
A depression angle calculating means for calculating a depression angle of the camera based on the shift amount of the object calculated by the shift amount calculation means;
A depression angle calculating device comprising: Further comprising a shift position detecting means for superimposing the current image and the previous image, shifting both images in the vertical direction and the horizontal direction, and detecting a shift position having the highest correlation between the two images,
2. The depression angle calculating device according to claim 1, wherein the shift amount calculation unit calculates a shift amount of the object based on a shift position detected by the shift position detection unit. It further comprises information acquisition means for acquiring information on the amount of movement of the vehicle in the horizontal direction,
The shift position detection unit detects the shift position by correcting the shift position in the horizontal direction based on information on a horizontal movement amount of the vehicle acquired by the information acquisition unit. The depression angle calculation device described.   The shift position detection means detects the shift position having the highest correlation between the two images after weighting the center of the current image and the previous image low and increasing the outside. Item 4. The depression angle calculating device according to Item 2 or 3. An image vanishing point acquiring means for acquiring an image vanishing point from parallel lines in the image captured by the camera;
When the image vanishing point is acquired by the image vanishing point acquiring unit, the depression angle calculating unit is not based on the shift amount of the object calculated by the shift amount calculating unit, but based on the image vanishing point. The depression angle calculating apparatus according to claim 1, wherein the depression angle of the camera is calculated. In a depression angle calculation method for calculating the depression angle of the camera from a plurality of images captured by an in-vehicle camera,
Of the plurality of images, based on the current image currently input and the previous image input before the current image, the current object and the specific object displayed in the previous image A shift amount calculating step for calculating a shift amount in the vertical direction;
A depression angle calculating step of calculating a depression angle of the camera based on the shift amount of the object calculated by the shift amount calculation step;
The depression angle calculation method characterized by including. In a depression angle calculation program for causing a computer to execute a method of calculating the depression angle of the camera from a plurality of images captured by an in-vehicle camera,
Of the plurality of images, based on the current image currently input and the previous image input before the current image, the current object and the specific object displayed in the previous image A shift amount calculation procedure for calculating the shift amount in the vertical direction;
A depression angle calculation procedure for calculating a depression angle of the camera based on the shift amount of the object calculated by the shift amount calculation procedure;
A depression angle calculation program that causes a computer to execute. In an image processing apparatus that calculates a depression angle of the camera from a plurality of images captured by a vehicle-mounted camera and performs image processing using the depression angle,
Of the plurality of images, based on the current image currently input and the previous image input before the current image, the current object and the specific object displayed in the previous image Shift amount calculating means for calculating the shift amount in the vertical direction;
A depression angle calculating means for calculating a depression angle of the camera based on the shift amount of the object calculated by the shift amount calculation means;
Distance correcting means for correcting the distance to the candidate object imaged by the camera using the depression angle of the camera calculated by the depression angle calculating means;
An image processing apparatus comprising:

Images