JP4555635B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method, and to an optical flow detection method using stereo image processing together.

2. Description of the Related Art A monitoring device that captures a scene in front by a camera mounted on a host vehicle and monitors a traveling state in a monitoring area based on information obtained thereby is widely known. In this type of monitoring device, the driving situation is recognized using a known image processing method such as stereo image processing or optical flow. In addition, not only image information but also the yaw rate (angular velocity) of the vehicle is referred to from the viewpoint of improving recognition accuracy in the outside monitoring apparatus. The angular velocity generated in the vehicle is usually directly detected using a sensor or the like. However, in the case of sensor detection, there is a disadvantage that the recognition accuracy is lowered due to a response delay or the like. Therefore, a method of detecting an angular velocity based on image information is also used. For example, in the method disclosed in Patent Document 1, an area where a distant view is projected is assumed to be infinity (a point far from infinity) for convenience, and the angular velocity is calculated based on the optical flow at infinity.
Japanese Patent Laid-Open No. 11-51650

  However, the scenery projected in the captured image may not necessarily include a distant scenery, and in this case, there is an inconvenience that it becomes difficult to specify the optical flow at infinity.

  The present invention has been made in view of such circumstances, and an object of the present invention is to estimate an optical flow at infinity regardless of the scenery projected in an image such as a distant view or a close view.

In order to solve this problem, the first invention provides an image processing apparatus. This image processing apparatus is mounted on a moving body, and captures a scene around the moving body to output a pair of image data in time series, and a pair of image data to be processed. A stereo image processing unit that calculates disparity based on dimensional matching and outputs distance data in which a disparity group related to image data corresponding to one frame is associated with a position on an image plane defined by the image data; Based on the distance data, a recognition unit that recognizes a plurality of three-dimensional objects and identifies each three-dimensional position of the three-dimensional object, and one image data of the pair of image data has a time-series context. based certain pair of image data into two-dimensional matching the processing target, voting a detection unit for detecting the respective optical flow of the three-dimensional object, a plurality of three-dimensional objects As an elephant, a two-dimensional histogram is generated with the parallax on one axis and the optical flow on the other axis. And an arithmetic unit for estimating an optical flow at infinity.

In the first invention, the calculation unit may approximate the voting pattern with a curve and estimate the optical flow at infinity based on the approximation result.

  Furthermore, in the first invention, it is preferable that the calculation unit calculates an angular velocity of the moving body based on an optical flow at infinity.

According to a second aspect of the present invention, there is provided an image processing method for processing a pair of image data mounted in a moving body and output in a time series from a stereo camera that captures a scene around the moving body. In this image processing method, a parallax is calculated based on two-dimensional matching with a pair of image data as processing targets, a parallax group related to image data corresponding to one frame, and a position on an image plane defined by the image data A first step of outputting distance data associated with each other, a second step of recognizing a plurality of three-dimensional objects based on the distance data, and specifying each three-dimensional position of the three-dimensional object, and a pair of A third step of detecting each optical flow of the three-dimensional object based on the two-dimensional matching in which one of the image data is processed in a pair of image data in a time-series context; as voting target multiple three-dimensional object, comprising the steps of: generating a two-dimensional histogram for the one axis parallax, the other axis and the optical flow, double As a constraint condition that the arrangement of the three-dimensional object is linear, based on the tendency of the voting pattern in the two-dimensional histogram, and a fourth step of estimating the optical flow in the infinity.

  According to the present invention, distance data is calculated based on a pair of captured images output in time series from a stereo camera, and a three-dimensional object is recognized using the distance data. In addition, the optical flow of the three-dimensional object is detected using one of the pair of captured images as a processing target. Then, the optical flow at infinity is estimated on the condition of the recognized arrangement of three-dimensional parts, the parallax of the three-dimensional object, and the optical flow of the three-dimensional object. Therefore, by providing these three conditions, the optical flow at infinity can be estimated even when an image is used so that a distant view is not projected. Thereby, the optical flow at infinity can be estimated regardless of the scenery projected on the image such as a distant view or a close view.

(First embodiment)
FIG. 1 is a block diagram of a monitoring apparatus with an angular velocity calculation function to which the image processing apparatus according to the present embodiment is applied. The monitoring device 1 is mounted on a vehicle (moving body) and monitors the traveling state ahead by image recognition. Further, the monitoring device 1 detects the optical flow OF based on the captured image, and calculates the angular velocity ω of the vehicle that is necessary for recognition of the driving situation.

  A stereo camera that captures a scene including a monitoring area is attached in the vicinity of a room mirror, for example. This stereo camera is composed of a pair of cameras 2 and 3, and each of the cameras 2 and 3 includes an image sensor (for example, a CCD or CMOS sensor). The main camera 2 captures a reference image necessary for performing stereo image processing, and the sub camera 3 captures a comparative image. Each analog image (stereo image) output in time series from the cameras 2 and 3 in a state where they are synchronized with each other is converted into a predetermined luminance gradation (for example, 256th floor) by the A / D converters 4 and 5. Tones).

  A pair of digitized image data is subjected to brightness correction, image geometric conversion, and the like in the image correction unit 6. Usually, there is an error in the mounting position of the pair of cameras 2 and 3 although there is a difference in degree. Therefore, a shift caused by the error occurs in the left and right images. In order to correct this deviation, geometrical transformation such as image rotation or translation is performed using affine transformation or the like.

  Through such image processing, reference image data is obtained from the main camera 2, and comparison image data is obtained from the sub camera 3. These image data are a set of luminance values (0 to 255) of each pixel. Here, the image plane defined by the image data is expressed in an ij coordinate system. For example, the lower left corner of the image is the origin, the horizontal direction is the i coordinate axis, and the vertical direction is the j coordinate axis. Stereo image data corresponding to one frame is output to the subsequent stereo image processing unit 7 and stored in the image data memory 8. As will be described later, the image data memory 8 has a storage capacity capable of storing at least two frames of stereo image data in order to detect the optical flow OF based on a pair of image data in a time-series context. Have.

  The stereo image processing unit 7 calculates distance data D related to the captured image corresponding to one frame based on the reference image data and the comparison image data. Here, the “distance data” is a set of parallax d calculated for each small area on the image plane defined by the image data, and each parallax d is associated with a position (i, j) on the image plane. It has been. Each parallax d is calculated for each pixel block of a predetermined area (for example, 4 × 4 pixels) constituting a part of the reference image.

  FIG. 2 is an explanatory diagram of pixel blocks set in the reference image. For example, when the reference image is composed of 512 × 200 pixels, a disparity group corresponding to the number of pixel blocks PBij (128 × 50) can be calculated from a captured image corresponding to one frame. As is well known, the parallax d is the amount of horizontal displacement with respect to the pixel block PBij that is the calculation unit, and has a large correlation with the distance to the object projected in the pixel block PBij. In other words, the closer the object projected in the pixel block PBij is to the cameras 2 and 3, the larger the parallax d of the pixel block PBij is, and the farther the object is, the smaller the parallax d is (if the object is infinitely far (infinite Far) and parallax d becomes 0).

  When calculating the parallax d related to a certain pixel block PBij (correlation source), a region (correlation destination) having a correlation with the luminance characteristic of the pixel block PBij based on two-dimensional matching with the reference image and the comparison image as processing targets Is specified in the comparative image. As described above, the distance from the cameras 2 and 3 to the object appears as a horizontal shift amount between the reference image and the comparison image. Therefore, when searching for the correlation destination in the comparison image, it is only necessary to search on the same horizontal line (epipolar line) as the j coordinate of the pixel block PBij as the correlation source. The stereo image processing unit 7 shifts the correlation between the correlation source candidate and the correlation destination candidate while shifting one pixel at a time on the epipolar line within a predetermined search range set based on the i coordinate of the correlation source. Sequential evaluation (stereo matching). In principle, the amount of horizontal deviation of the correlation destination (one of the correlation destination candidates) determined to have the highest correlation is defined as the parallax d of the pixel block PBij.

  The correlation between the two pixel blocks PBij can be evaluated by, for example, calculating the city block distance CB. Formula 1 shows the basic form of the city block distance CB. In the equation, p1ij is the luminance value of the ijth pixel of one pixel block, and p2ij is the ijth luminance value of the other pixel block. The city block distance CB is the total sum of the differences (absolute values) of the luminance values p1ij and p2ij corresponding to each other in the entire pixel block, and the smaller the difference, the greater the correlation between both pixel blocks. .

  Basically, of the city block distances CB calculated for each pixel block PBij existing on the epipolar line, the pixel block PBij having the smallest value is determined as the correlation destination. The amount of deviation between the correlation destination specified in this way and the correlation source is the parallax d. Note that the hardware configuration of the stereo image processing unit 7 for calculating the city block distance CB is disclosed in Japanese Patent Laid-Open No. 5-1114099, so refer to it if necessary. The distance data D calculated through such processing, that is, a set of parallax d associated with the position (i, j) on the image is stored in the distance data memory 9.

  The microcomputer 10 includes a CPU, a ROM, a RAM, an input / output interface, and the like, and includes a recognition unit 11, a detection unit 12, and a calculation unit 13 when functionally grasping this. The recognition unit 11 specifies the three-dimensional position of each three-dimensional object On based on the distance data D. The detection unit 12 detects the optical flow OF related to the three-dimensional object On based on two-dimensional matching for a pair of reference image data that are in chronological order. The calculation unit 13 estimates the optical flow OFinf at infinity based on the distance data D, the arrangement of the plurality of three-dimensional objects On in the real space, and the detected optical flow OF. Then, based on the estimated optical flow OFinf, the calculation unit 13 calculates the angular velocity ω of the stereo camera, that is, the vehicle on which the stereo camera is mounted.

  When the angular speed ω of the host vehicle is calculated, the microcomputer 10 calculates, for example, the moving speed of the three-dimensional object On in consideration of the angular speed ω, thereby recognizing the traveling state ahead. If it is determined that the front three-dimensional object On is dangerous for the host vehicle, an alarm device (not shown) such as a monitor or a speaker is activated to alert the driver. Further, control such as downshifting or brake operation is executed by controlling a control device (not shown) as necessary.

  FIG. 3 is a flowchart showing a procedure for calculating the angular velocity ω according to the present embodiment. The routine shown in this flowchart is called at predetermined intervals and executed by the microcomputer 10. Hereinafter, a method for calculating an angular velocity around the vertical axis of the vehicle (hereinafter referred to as “yaw angular velocity”) ωy as the angular velocity ω will be described. First, in step 1, distance data D is read from the distance data memory 9, and reference image data (hereinafter simply referred to as “image data Pn”) captured at the latest imaging timing is read from the image data memory 8. It is.

  In step 2, the road shape ahead of the host vehicle is recognized. This road shape can be recognized, for example, by calculating a road model. The road model is expressed by a straight line expression in the horizontal and vertical directions in the three-dimensional coordinate system that defines the real space, and is calculated by setting the parameters of the straight line expression to values that match the actual road shape. Is done. For example, a white line (or yellow line) defining a lane drawn on the road surface is effective in recognizing the road shape. Therefore, the road model is calculated by recognizing the position of the white line in the real space with reference to the distance data D and the image data Pn and connecting the positions. The coordinate system that defines the real space is set on the basis of the vehicle, with the road surface directly below the center of the stereo camera as the origin, the vehicle width direction is the y axis, the vehicle height direction is the z axis, and the vehicle length direction is the x axis. And

  In step 3, the three-dimensional object On is recognized. In the recognition of the three-dimensional object On, first, three-dimensional data is calculated. This three-dimensional data is data indicating a three-dimensional position (a position in the real space) in the monitoring area, and is uniquely calculated based on the distance data D by referring to a known coordinate conversion formula. . The calculated three-dimensional data is compared with a road model, and data above the road surface is extracted as a three-dimensional object On. For specific methods regarding the recognition of the road shape and the three-dimensional object, JP-A-5-265547, JP-A-6-266828, JP-A-10-283461, JP-A-10-283477, Since it is disclosed in Japanese Laid-Open Patent Publication No. 11-213138, Japanese Patent Laid-Open No. 2001-92970, etc., please refer to it if necessary.

  In step 4, an optical flow OF is detected using a pair of image data having a time-series context. The optical flow OF represents a flow (movement amount) of a pixel between frames in an image captured in time series as a vector. In general, the imaging timing is changed from a previous image to a later image. The flow of the pixel is shown. Therefore, first, reference image data Pn-1 (hereinafter referred to as “previous image data” as appropriate) captured before the image data Pn read in Step 1 is read from the image data memory 8. It is. Then, the optical flow OF is detected using the pair of image data Pn and Pn-1 that are in chronological order as a processing target. As will be described later, in this embodiment, the optical flow OFinf at infinity is estimated based on the parallax d regarding the three-dimensional object On and the optical flow OF regarding the three-dimensional object On. Therefore, from the viewpoint of omitting complicated calculation processing, the detection of the optical flow OF is performed based on the pixel block PBij which is a unit for calculating the parallax d. In other words, this optical flow OF indicates the amount of movement from the previous image data Pn-1 for each pixel block PBij constituting the image data Pn.

  Specifically, for each pixel block PBij, a region (correlation destination) having a correlation with the luminance characteristic of the pixel block PBij (correlation source) is specified in the previous image data Pn-1. When detecting the optical flow OF, unlike the stereo matching process described above, the highest correlation is obtained by offsetting the pixel block PBij as a correlation source one pixel at a time in the horizontal / vertical direction over the entire area of the previous image data Pn-1. (2D matching) is specified. Then, the coordinate position (i, j) of the correlation source and the correlation destination is compared, the positional deviation amount Δj in the vertical direction (j-axis direction) from the previous image data Pn-1 to the image data Pn, and the horizontal direction ( The positional deviation amount Δi in the i-axis direction) is the optical flow OF related to the pixel block PBij. The detected optical flow OF (horizontal component Δi, vertical component Δj) is associated with the position (i, j) on the image plane. Note that the reference image data Pn and Pn-1 that are targets of calculation of the optical flow OF need only have the imaging timings before and after in time series, and the imaging timings do not have to be continuous. Further, the detection area of the optical flow OF does not have to be the entire area of the image plane, and it is sufficient that at least the optical flow OF related to the recognized three-dimensional object On is detected.

  FIG. 4 is an explanatory diagram of a two-dimensional histogram. In step 5, a two-dimensional histogram showing the frequency distribution between the parallax d calculated for each pixel block PBij and the horizontal component Δi of the optical flow OF is generated. In step 4, first, a voting space is prepared in which one axis is parallax d and the other axis is a horizontal component Δi of the optical flow OF. Then, each pixel block PBij is sequentially voted to the corresponding voting space based on the parallax d related to itself and the horizontal component Δi of the optical flow OF. When this voting process is performed for all the pixel blocks PBij, the process proceeds to step 5.

  In Step 6, based on the generated two-dimensional histogram, the optical flow OFinf at infinity (in this embodiment, the horizontal component Δiinf of the optical flow OF) is estimated. Specifically, first, the voting pattern for each pixel block PBij is approximated by a curve (in the present embodiment, a quadratic expression) based on the two-dimensional histogram. As an approximation method to the quadratic expression, Hough transform can be used. The Hough transform is a well-known image processing technique for detecting an object having a straight line or an arbitrary shape from an image. When extracting a curve using the Hough transform, first, the two-dimensional histogram is binarized according to the frequency. Then, by applying Hough transform to the binarized pattern, a quadratic curve corresponding to the voting pattern is extracted. For approximation to the quadratic expression, a least square method or the like can be used in addition to the Hough transform.

FIG. 5 is an explanatory diagram of the extracted quadratic curve. The figure shows four quadratic curves extracted by the Hough transform, and each quadratic curve can be expressed by the following equation. Here, in the equation, Pn, Qn, and Δiinf are coefficients (curve parameters) in a quadratic curve.

  Then, the 0th order term in the extracted quadratic curve, that is, the constant term Δiinf becomes the horizontal component of the optical flow OFinf at infinity. As shown in the figure, when a plurality of quadratic curves are extracted (n = 1 to 4), the horizontal component Δiinf has a characteristic that reflects the tendency of the constant term Δiinf in each quadratic curve. It is calculated as a value (average value, mode value, etc.).

In step 7, the yawing angular velocity ωy is calculated based on the estimated optical flow OFinf (horizontal component Δiinf) (see the following equation).

  Here, Wi is a horizontal width per pixel of the image sensor in the cameras 2 and 3, and f is a focal length of the cameras 2 and 3. Δt is an imaging interval between the imaging timing of the previous image data Pn-1 and the imaging timing of the image data Pn.

  FIG. 6 is an explanatory diagram of an estimation concept of the optical flow OFinf at infinity. In the figure, the three-dimensional object On observed from the cameras 2 and 3 is indicated by black dots. These three-dimensional objects On are made to turn a vehicle (speed in the forward direction (hereinafter referred to as “forward speed”) u, speed in the lateral direction positive in the right direction (hereinafter referred to as “transverse speed”) v, yawing angular speed ωy. ), The three-dimensional object On exists at the position P (t0) at the imaging timing at a certain time t0. When a predetermined time has elapsed from this time t0 (that time t1 = t0 + Δt), the object On moves to the position P (t1) as the vehicle turns. In the figure, the movement amount of the three-dimensional object On from the time t0 to the time t1 is indicated by an arrow, and the movement amount of the three-dimensional object On is represented by an optical flow OF between frames with the imaging timings at the times t0 and t1. Equivalent to.

  When the vehicle is turning, the detected optical flow OF of the three-dimensional object On includes the pixel flow caused by the vehicle speed components (forward speed u, lateral speed v) and the rotational component of the vehicle (yawing angular velocity). pixel flow caused by ωy). By the way, when the host vehicle and the three-dimensional object On are close in position, the main factor of the pixel flow can be regarded as a speed component of the vehicle. On the other hand, when the own vehicle and the three-dimensional object On are separated from each other, the influence of the speed component of the vehicle on the pixel flow can be regarded as almost zero as the distance increases. . Therefore, out of the optical flow OF on the image plane, the optical flow OF in the region where the distant view is projected is compared with the optical flow OF in the region where the distant view is projected, and the flow of pixels due to the velocity component is suppressed, and the rotation component The pixel flow resulting from the above becomes remarkable. In other words, the pixel flow caused by the velocity component does not act on the optical flow OFinf at infinity, and the pixel flow caused by the rotation of the cameras 2 and 3 (or the moving body on which the cameras 2 and 3 are mounted). (Thus, the yawing angular velocity ωy can be calculated). In this case, regarding the three-dimensional object On (or a plurality of three-dimensional object groups) arranged on a certain straight line from the near view to the distant view, the suppression tendency of the pixels of the speed component regularly appears in the optical flow OF. Therefore, by specifying the regularity of such an optical flow OF, even if the scene at infinity is not projected on the image, the optical flow at infinity is bounded by the arrangement of the three-dimensional object On. OFinf can be estimated.

Of course, the three-dimensional object On recognized in the outside monitoring includes a three-dimensional object On such as a preceding vehicle and an obstacle, but there are also three-dimensional objects On arranged in a straight line such as a guardrail and a side wall. In the three-dimensional coordinate system based on the vehicle, the straight line formed by the three-dimensional object On can be expressed by the following equation. In the equation, Gn and Hn are straight line parameters (coefficients of a linear expression), and n straight line groups can be expressed by the subscript n.

When the three-dimensional object On constituting these straight line groups is observed on a two-dimensional plane composed of the horizontal coordinate i of the image and the parallax d, the above n straight lines are expressed by the following equation. In the following formula, L is the camera baseline length in the stereo camera.

The optical flow OF of the three-dimensional object On aligned on the straight line is observed from the cameras 2 and 3 that move at the forward speed u, the right lateral speed v, and the yawing angular speed ωy. In this case, the parallax d of the object On and the horizontal component Δi of the optical flow OF are expressed by the following quadratic expression.

  In the present embodiment, based on such a concept, a two-dimensional histogram is generated on the assumption that the solid objects On aligned in a straight line are observed from the cameras 2 and 3 (constraint conditions). And the curve parameter of Formula 6 (namely, Formula 2) is identified from the vote pattern in this histogram. Thereby, the optical flow OFinf at infinity (d = 0) can be estimated. In other words, based on the parallax d of the three-dimensional object On (one axis of the two-dimensional histogram), the optical flow OF of the three-dimensional object (the other axis of the two-dimensional histogram), and the arrangement of the three-dimensional object, it is infinite. The optical flow OFinf at can be estimated. Thereby, even if it is a case where an image is used so that a distant view is not projected, the optical flow OFinf at infinity can be estimated. Further, by estimating the optical flow OFinf at infinity, the yawing angular velocity ωy of the vehicle, that is, the moving body on which the cameras 2 and 3 are mounted can be calculated.

(Second Embodiment)
One of the features of the first embodiment is that the voting pattern is approximated by a curve, and the optical flow OFinf at infinity is estimated based on this approximation result. On the other hand, one of the features of the second embodiment is that the optical flow OFinf at infinity is estimated using a simpler configuration than the first embodiment. Specifically, in the second embodiment, the weighted average value of the voting pattern is calculated based on the above-described two-dimensional histogram (see the following formula), and the optical flow OFinf at infinity is calculated based on the calculation result. Presumed.

As shown in the first embodiment, in the two-dimensional histogram, the voting pattern of the three-dimensional objects On arranged in a straight line tends to converge to the optical flow OFinf at infinity. Therefore, this equation uses the frequency h (d, Δi) and the weight w (d) of the two-dimensional histogram to increase the weight of the distant view data, thereby suppressing the influence of the distant view data and converting the distant view data into the distant view data. Based on this, the optical flow OFinf at infinity is estimated. This weight w (d) can be set as shown in the following equation, for example. In the following equation, D1 is a first parallax value indicating a boundary between a distant view and a middle foreground, D2 is a second parallax indicating a boundary between the middle and foreground, and the total number of votes in the two-dimensional histogram Is set in advance in consideration of the above.

  As described above, the optical flow OFinf at infinity (parallax d = 0) is estimated based on the weighted average value of the voting pattern (more precisely, the weighted average value of the optical flow OF). According to this method, as in the first embodiment, the optical flow OFinf at infinity can be estimated, and the approximation process to the quadratic expression can be omitted, so that the process is simplified. be able to. Thereby, the processing load by the computer can be reduced.

  In the first and second embodiments, the method for calculating the yaw rate angular velocity ωy as the angular velocity has been described. However, the present invention is not limited to this, and based on the optical flow OFinf at infinity, the angular velocity around the horizontal axis of the vehicle, that is, the angular velocity in the pitching direction (hereinafter referred to as “pitching angular velocity”) can also be calculated. In the above-described embodiment, as a premise for detecting the yawing angular velocity ωy, one axis of the two-dimensional histogram is set to the rotation component (yaw component) pixel flow, that is, the horizontal component Δi of the optical flow OF. However, when calculating the pitching angular velocity, this axis is set to the pixel flow of the rotation component (pitching component), that is, the vertical component Δj of the optical flow OF. In this two-dimensional histogram, the voting pattern is approximated by a quadratic expression, and the vertical component of the optical flow OFinf at infinity is estimated. Therefore, the pitching angular velocity can be uniquely calculated based on the estimated vertical component of the optical flow OF.

  Further, as shown in Formula 6, the coefficients of the respective orders are also identified by approximating the voting pattern by a quadratic expression based on the two-dimensional histogram. This makes it possible to estimate an unknown number of the coefficients, for example, the forward speed u and the lateral speed v.

  In each of the above-described embodiments, the angular velocity of the vehicle is calculated based on the optical flow OFinf at infinity on the premise of a monitoring device that monitors the outside of the vehicle, but the present invention is not limited to this. That is, the case where the cameras 2 and 3 are mounted is not limited to the vehicle, but can be mounted on various moving bodies including a flying body and can be widely applied to the angular velocity calculation. In addition, since the optical flow OFinf at infinity can effectively extract the rotation component of the camera, the image processing apparatus according to the present invention is applied as a camera shake correction function for shooting moving images and still images. be able to. That is, in the present invention, the moving body on which the cameras 2 and 3 are mounted widely includes elements that cause the cameras 2 and 3 to shake.

Block configuration diagram of a vehicle exterior monitoring device with an angular velocity calculation function to which the image processing device according to the present embodiment is applied. Explanatory drawing of the pixel block set to the reference image Flow chart showing the calculation procedure of angular velocity Illustration of two-dimensional histogram Illustration of the extracted quadratic curve Illustration of optical flow OFinf estimation concept at infinity

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Monitoring apparatus 2 Main camera 3 Sub camera 4,5 A / D converter 6 Image correction part 7 Stereo image processing part 8 Image data memory 9 Distance data memory 10 Microcomputer 11 Recognition part 12 Detection part 13 Calculation part OF Optical flow (DELTA) i Horizontal Component Δj Vertical component D Distance data u Forward speed v Lateral speed

Claims (4)

In the image processing apparatus,
A stereo camera that is mounted on a moving body and outputs a pair of image data in a time series by imaging the scenery around the moving body,
The parallax is calculated based on the two-dimensional matching with the pair of image data as processing targets, and the parallax group related to the image data corresponding to one frame is associated with the position on the image plane defined by the image data. A stereo image processing unit for outputting the obtained distance data;
A recognition unit that recognizes a plurality of three-dimensional objects based on the distance data, and identifies each three-dimensional position of the three-dimensional object;
Detection that detects the optical flow of each of the three-dimensional objects based on two-dimensional matching in which one of the pair of image data is a processing target of a pair of image data in a time-series context. And
While generating a two-dimensional histogram with the plurality of three-dimensional objects as voting targets, with one axis being parallax and the other axis being an optical flow, the constraint that the arrangement of the plurality of three-dimensional objects is linear, An image processing apparatus comprising: an arithmetic unit that estimates an optical flow at infinity based on a tendency of a voting pattern in the two-dimensional histogram . The image processing apparatus according to claim 1 , wherein the arithmetic unit approximates the voting pattern with a curve and estimates an optical flow at the infinity based on the approximation result. The arithmetic unit, based on the optical flow in the infinity, the image processing apparatus according to claim 1 or 2, characterized in that to calculate the angular velocity of the moving body. In an image processing method for processing a pair of image data mounted in a moving body and output in a time series from a stereo camera that captures the scenery around the moving body,
The parallax is calculated based on the two-dimensional matching with the pair of image data as processing targets, and the parallax group related to the image data corresponding to one frame is associated with the position on the image plane defined by the image data. A first step of outputting the obtained distance data;
A second step of recognizing a plurality of three-dimensional objects based on the distance data and specifying the three-dimensional position of each of the three-dimensional objects;
A first optical data of the three-dimensional object is detected based on two-dimensional matching in which one of the pair of image data is a processing target of a pair of image data having a time-series context. 3 steps,
The step of generating a two-dimensional histogram in which the plurality of three-dimensional objects are to be voted, one axis is parallax and the other axis is an optical flow, and the arrangement of the plurality of three-dimensional objects is linear. And a fourth step of estimating an optical flow at infinity based on the tendency of the voting pattern in the two-dimensional histogram .

Images