JP5267330B2 - Image processing apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable a user to easily understand the movement of a moving body in an image processing apparatus for making a display device display the pickup image of a driver recorder. <P>SOLUTION: A three-dimensional position information calculation part 11 calculates a three-dimensional position information from a time-series stereo pickup image from a drive recorder, and a moving body extraction part 12a extracts the same moving body, and a face setting part 13 sets a projection face desired by a user when displaying a pickup image, that is, the direction of a line of sight, and a three-dimensional position information integration part 15 integrates time-series images on the set projection face, and a three-dimensional position information calculation part 16 calculates each position of the moving body on the integrated screen, and a three-dimensional position information projection part 14 makes a display device 3 display it. Therefore, it is possible to convert the movement of the moving body analyzed from the time-series three-dimensional pickup image into an image viewed from the line of sight of a driver or the line of sight of the eyewitness of an accident for display. Thus, it is possible for a user to easily understand the movement of the moving body. <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to an image processing apparatus and method for analyzing a motion of a moving object of interest from a time-series captured image by an in-vehicle camera or the like and displaying it on a display device.

  In recent years, various systems for the purpose of improving safety have been studied in the automobile industry. For this purpose, by using a sensor or a camera to acquire information around the vehicle, particularly distance information, the collision risk is determined, which is useful for avoiding accidents. In particular, in a crisis avoidance system using a camera, obstacles around the vehicle are identified and the movement of the obstacles is analyzed based on the captured image of the camera to avoid the obstacles. .

  On the other hand, when an accident occurs without avoiding danger, a system has been developed that extracts various information useful for investigating the cause by analyzing images before and after the accident. For example, Patent Document 1 discloses a system that analyzes images of an accident such as the speed of an accident vehicle by acquiring images before and after the accident with a camera installed at an intersection or the like and analyzing the images. Yes. This system installs cameras in advance at intersections where accidents can occur (where accidents occur frequently), and prepares plan view data of stationary bodies at such intersections such as road surfaces and pedestrian crossings. By projecting images before and after the accident (depicting a vehicle trajectory), the situation of the accident can be analyzed.

  Further, in Patent Document 2, a camera vector value (CV value) indicating a three-dimensional position and posture of a camera mounted on a moving body is obtained from feature points of a monocular moving image, and based on the obtained CV value. The camera position is superimposed on the image.

JP 2004-102426 A JP 2008-5478 A

  However, although the prior art disclosed in Patent Document 1 can analyze the situation of the vehicle in detail, it is necessary to prepare the plan view data, so that it can only analyze the situation of an accident at a predetermined location. There is also a problem that it cannot be handled.

  On the other hand, the conventional technique disclosed in Patent Document 2 has a problem that it is difficult to calculate an accurate CV value for an image in which a moving object region is dominant. Furthermore, although the movement of the camera, that is, the moving body (own vehicle) can be obtained, it is not considered to obtain the positions of other moving bodies in the image.

  An object of the present invention is to provide an image processing apparatus and method for displaying a motion of a moving object analyzed from a time-series three-dimensional captured image in a form that can be easily understood by a user.

The image processing apparatus of the present invention, when the obtained image data which can be collected by the three-dimensional position information of the subject in the imaging field image series as an input, the image processing apparatus to be displayed on the display device processes the input image data A moving object extraction unit that extracts the same moving object and a stationary object from the time-series captured images;
A plane setting unit for setting a projection plane for the display, and the three-dimensional position information of the moving object extracted by the moving object extraction unit on the projection plane set by the plane setting unit based on the stationary object And a three-dimensional position information projection unit for creating a projected display image.

The image processing method of the present invention, when taken three-dimensional position information of the subject in the imaging field image series as input the resulting possible image data, a method for creating a display image by processing the input image data The step of extracting the same moving body and stationary body from the time-series captured image, the step of setting a projection plane for the display, and the three-dimensional position of the extracted moving body on the set projection plane Creating a display image in which information is projected based on the stationary body .

According to the above configuration, as input image data, for example, a time-series captured image by a stereo camera is input, and a corresponding point search process is performed on an image between the left and right of the stereo camera, so that the subject in the captured image is detected. By obtaining 3D position information, or obtaining 3D position information of a subject by inputting distance information from a radar or the like to a time-series captured image of a monocular camera, a time-series captured image is included in the captured image. In the image processing apparatus that acquires the three-dimensional position information of the subject at the same time, processes the input image data and displays it on the display device, a moving object extraction unit, an input unit, and a three-dimensional position information projection unit are provided. . The moving object extraction unit extracts the same moving object and stationary object from the time-series captured images, while the surface setting unit sets a projection plane desired by the user for the display, that is, a line-of-sight direction. Thus, the three-dimensional position information projection unit displays the three-dimensional position information at each point in the time series of the moving object extracted by the moving object extraction unit on the projection plane set by the surface setting unit. For example, a display image projected based on the stationary object is created by superimposing lines and points of a locus or a moving body image of a real image, and displayed on the display device.

  Therefore, the motion of the moving object analyzed from the time-series three-dimensional captured image can be converted into an image viewed from the driver's eye or the eye of the accident witness, for example, and the motion of the moving object can be used. Can be easily understood.

Furthermore, in the image processing apparatus of the present invention, the output of the previous SL moving object extraction unit, and the 3-dimensional information integration section a three-dimensional position to integrate the basis of the stationary body between the frames of the moving object, the three-dimensional information integration 3D position information calculation for calculating the 3D position of the moving object in the remaining frames and outputting the 3D position information to the 3D position information projecting unit based on the 3D position of the moving object in a certain frame as a reference. And a section.

  According to the above configuration, since the position information of the moving body between the frames is integrated based on the stationary body and then the projection is performed, the camera that obtains the time-series captured images is moving, that is, the camera is used as a drive recorder. Even in the case of an in-vehicle camera such as the above, it is possible to create a display image not only from the driver's eyes but also from any eye (projection plane) direction such as the eyes of the witness on the road.

In the image processing apparatus of the present invention, the pre-Symbol 3-dimensional position information projection unit, for each frame, on the projection plane, and projecting the three-dimensional position information of the moving object extracted by the moving object extraction unit and a stationary member The projected image integration unit further includes a projection image integration unit that integrates the projection planes by aligning the stationary body in the projected frame images.

  According to the above configuration, since the projection results for each frame are integrated with reference to a stationary body, the camera that obtains the time-series captured images is moving, that is, the camera is an in-vehicle camera such as a drive recorder In addition, not only the driver's line of sight but also a display image from any line of sight (projection plane) direction such as the line of sight of the witness on the road can be created.

Furthermore, in the image processing apparatus of the present invention, before Symbol plane setting unit are inputted an output of the moving object extraction unit, a virtual projection plane relative to the extraction result in a certain frame is set, the three-dimensional position information The projection unit integrates the three-dimensional position of the moving body in each frame from the output of the moving body extraction unit on the basis of the stationary body, and projects it on the virtual projection plane set by the surface setting unit. To do.

  According to the above configuration, since the position information of the moving body between the frames is integrated with respect to the stationary body and then projected onto the virtual projection plane, the camera that obtains the time-series captured image is moving, that is, Even when the camera is a vehicle-mounted camera such as a drive recorder, it is possible to create a display image not only from the driver's line of sight but also from an arbitrary line of sight (projection plane) such as the line of sight of the witness on the road. it can.

  In the image processing device according to the aspect of the invention, the three-dimensional position information projection unit may create the display image by combining the three-dimensional position information with information regarding the motion of each moving object extracted by the moving object extraction unit. It is characterized by.

  According to the above configuration, the user can more easily understand the movement of the moving object by displaying the information on the movement of each moving object such as an arrow or a numerical value indicating the magnitude of the speed together with the three-dimensional position information. can do.

  Furthermore, in the image processing apparatus according to the present invention, in the three-dimensional position information projection unit, the projection plane and the three-dimensional position information projected thereon are the angles set by the surface setting unit. It is a real image converted into an image seen from the above.

  According to the above configuration, the moving image is created by converting the angle of the photographed image, so that reality can be provided.

  In the image processing apparatus of the present invention, in the three-dimensional position information projection unit, the projection plane is a schematic road surface viewed from an angle set by the surface setting unit, and the three-dimensional position information Consists of landmarks of actual photographs extracted from the input image data.

  According to the above configuration, since a display image is created by pasting a landmark based on a live-action image on a road surface such as a computer graphic, it is possible to provide a reality with ease of recognition.

  Furthermore, in the image processing apparatus of the present invention, the projection plane and the three-dimensional position information projected thereon are composed of a schematic drawing viewed from an angle set by the plane setting unit. To do.

  According to the above configuration, a schematic drawing, for example, the above-described schematic road surface, such as an identification symbol for road traffic such as the above-described schematic traffic division, pedestrian crossing, and signal is synthesized. In addition, since the position of the moving object can be projected and the user's line-of-sight direction can be arbitrarily set and displayed, the user can move the movement of the moving object from an image such as a computer graphic displaying the minimum necessary information. Can be understood more easily.

  As described above, the image processing apparatus and method according to the present invention uses, as input, image data that can be acquired by combining a time-series captured image with the three-dimensional position information of the subject in the captured image. In the image processing apparatus that processes the image and displays it on the display device, the moving object extraction unit extracts the same moving object from the time-series captured images, while the surface setting unit determines that the user desires for the display. The projection plane to be operated, that is, the line-of-sight direction is set, and the three-dimensional position information projection unit according to this sets the time series of the moving object extracted by the moving object extraction unit on the projection plane set by the surface setting unit. A display image is created by projecting the three-dimensional position information at each point by, for example, superimposition of trajectory lines and points, or the moving body image itself, and displayed on the display device.

  Therefore, the motion of the moving object analyzed from the time-series three-dimensional captured image can be converted into an image viewed from the eyes of the driver or the eyewitness of the accident, for example, and the motion of the moving object can be displayed. It can be easily understood by the user.

It is a block diagram which shows the electrical constitution of the accident verification system provided with the image processing apparatus which concerns on the 1st Embodiment of this invention, and uses a fixed point camera. It is a block diagram which shows the electric constitution of the accident verification system provided with the image processing apparatus which concerns on the 1st Embodiment of this invention, and uses a drive recorder. It is a figure for demonstrating the specific operation | work of the moving body corresponding | compatible image part by a user. It is a figure which shows the integrated display example of the time-sequential position of the moving body by one Embodiment of this invention, and shows the locus | trajectory of the bicycle which drive | works on a road surface. FIG. 10 is a schematic diagram for explaining a method of switching the line-of-sight direction in the integrated display according to the embodiment of the present invention. It is a figure for demonstrating the extraction method of a stationary body. It is a figure for demonstrating the said integration process of three-dimensional information. It is a figure for demonstrating matching of the moving body in a time series image. It is a figure which shows the result which integratedly displayed the time-sequential position of the moving body of FIG. 3 from two gaze directions. FIG. 5 is a diagram schematically showing a pictorial drawing when displaying time-series positions of moving objects in an integrated manner. It is the figure which projected together the information regarding a motion on the schematic image shown in FIG. It is a figure for demonstrating the pop-up display of other movement information. It is a figure which shows the other example of the schematic image shown in FIG. It is a block diagram which shows the electric constitution of the accident verification system provided with the image processing apparatus which concerns on the 2nd Embodiment of this invention. It is a figure for demonstrating position alignment of the projection surface by the pictorial drawing by the 2nd Embodiment of this invention. It is a block diagram which shows the electric constitution of the accident verification system provided with the image processing apparatus which concerns on the 3rd Embodiment of this invention. It is a figure for demonstrating the projection method to the virtual projection surface by the 3rd Embodiment of this invention. It is a figure for demonstrating the method of the three-dimensional calculation (distance calculation) with respect to the output image from a stereo camera. It is a figure for demonstrating the corresponding point search method of the reference image with respect to the reference | standard image in calculating | requiring the right-and-left parallax of the stereo camera used for the said three-dimensional calculation. It is a figure for demonstrating the multi-resolution strategy effective for the said corresponding point search. It is a graph which shows the correlation value by a phase only correlation method (POC). It is a figure which shows an example of the corresponding point search in the said phase only correlation method. It is a figure for demonstrating the corresponding point search range in the said phase only correlation method. It is a figure for demonstrating an ICP algorithm.

(Embodiment 1)
FIG. 1 and FIG. 2 are block diagrams showing the electrical configuration of an accident verification system including the image processing apparatuses 1 and 1a according to the first embodiment of the present invention. The accident verification system of FIG. 1 is configured by including the image processing apparatus 1 and the display apparatus 3 installed in a traffic monitoring room or the like on a fixed point camera 2 installed at an accident-prone intersection or the like like an intersection monitoring camera. The The accident verification system in FIG. 2 includes the drive recorder 4 mounted on a vehicle such as a taxi and the like, and includes the image processing device 1a and the display device 3 installed in a business office or the like.

  The fixed point camera 2 creates image data that can be acquired by combining a time-series captured image with the three-dimensional position information of the subject in the captured image. For this reason, as shown in FIG. 1, it is possible to realize a configuration in which images captured by the stereo camera 21 are continuously recorded in the image recording unit 22. In this case, the image processing apparatus 1 includes the image Three-dimensional position information for obtaining three-dimensional position information of a subject in the captured image by inputting a time-series captured image from the recording unit 22 and performing a corresponding point search process on the left and right images of the stereo camera 21 A calculation unit 11 is provided. A plurality of examples of the three-dimensional measurement using the captured image from the stereo camera 21 will be described later. On the other hand, if the fixed-point camera 2 is to record time-series captured images of a monocular camera together with distance information from a radar or the like, a three-dimensional position information calculation unit 11 is provided in the image processing apparatus 1. It does not have to be. In other words, the image processing apparatus 1 may input data that can be acquired by combining time-series captured images with the three-dimensional position information of the subject in the captured images as input image data.

  The drive recorder 4 includes a temporary storage unit 41 and a trigger generation unit 42 in the stereo camera 21 and the image recording unit 22, and stores a captured image of the stereo camera 21 in a temporary storage unit 41 such as a ring buffer. Then, from the temporary storage unit 41 to the image recording unit 22 for a predetermined time before a predetermined time of the trigger timing generated by the trigger generation unit 42 such as an acceleration sensor (a dangerous timing due to the possibility of a collision or the like). This can be realized with a configuration in which captured image data is transferred and recorded. The image recording unit 22 is configured to overwrite old image data when the image data to be recorded exceeds the recording capacity. When the image recording unit 22 has a sufficient recording capacity, the image recording unit 22 continuously records images from the stereo camera 21 and the trigger generation unit 42 generates a trigger, or Marking may be applied to image data from a previous timing for a predetermined time so that it can be easily extracted later.

  It should be noted that the image processing apparatus 1 acquires the time-series captured image as described above together with the three-dimensional position information of the subject in the captured image, and causes the display apparatus 3 to display the processed image. The moving object extracting unit 12 that extracts the same moving object from the time-series captured images, the surface setting unit 13 that sets a projection surface for the display, and the projection plane set by the surface setting unit 13 And a three-dimensional position information projecting unit 14 for creating a display image obtained by projecting the three-dimensional position information on the moving object extracted by the moving object extracting unit 12. It should be noted that the image processing apparatus 1a includes a three-dimensional information integration unit 15 in addition to the three-dimensional position information calculation unit 11, the moving object extraction unit 12, the surface setting unit 13, and the three-dimensional position information projection unit 14. The three-dimensional position information calculation unit 16 is provided.

  The three-dimensional position information calculation unit 11 calculates three-dimensional positions of a plurality of identical moving objects that are temporally different. When the stereo camera 21 is fixed like the fixed point camera 2 in FIG. What is necessary is just to calculate the three-dimensional position of the moving body obtained by each image. On the other hand, when the stereo camera 21 is mounted on the vehicle as in the drive recorder 4 of FIG. 2, the three-dimensional position of the moving object obtained in each image is the three-dimensional position relative to the position of the camera at the time of shooting. Therefore, when the vehicle on which the stereo camera 21 is mounted and the moving object being photographed are moving at the same speed in the same direction, the three-dimensional position of the moving object always has the same value. It is necessary to match the coordinate system of the camera 21 with a certain reference frame.

  The moving object extraction unit 12 extracts the same moving object from the time-series captured images. Here, the moving body refers to an object that is actually moving with respect to the ground, such as a vehicle such as an automobile or a motorcycle, a bicycle, or a pedestrian. In the following description, the stereo camera 21 is mounted on a moving body such as a vehicle and performs imaging, that is, moving object extraction in the case of the drive recorder 4 will be described. In this case, since the mounted vehicle itself moves, even if there is a relatively moving object in the captured image of the stereo camera 21, it is not necessarily a moving object. Therefore, a method for specifying the moving object corresponding image portion in the time-series image generated by the stereo camera 21 will be described below. In specifying the moving object corresponding image portion, the moving object extracting unit 12 uses the three-dimensional position information such as the three-dimensional coordinates, the two-dimensional motion vector, and the three-dimensional motion vector obtained by the three-dimensional position information calculating unit 11. . Note that specifying a moving object-corresponding image portion on an image specifically refers to specifying a portion where a moving object is displayed among objects represented as an image and acquiring the three-dimensional image information. . The moving object corresponding image portion refers to a portion corresponding to the moving object displayed in the image.

  First, there is a method of specifying a moving object corresponding image portion using a vanishing point of movement. Here, the vanishing point of the motion is a point where a straight line obtained by extending the motion vector at each point on the image along the direction intersects. This vanishing point is determined according to the moving direction of the object on the image. That is, when the camera is moving in the same direction, if it is the same object, it has moved in the same direction, so there is a vanishing point for that object. In addition, when the object on the image is a stationary object, the same vanishing point exists for all the objects that are stationary objects ("Examination of moving object recognition method using principal component analysis", Information Processing Society of Japan Report-Computer Vision and Image Media Vol. 1996, No. 31, 1995-CVIM-099, literature number: IPSJ-CVIM 9509008). Note that most of the subject images captured by the stereo camera 21 are considered to be occupied by a stationary object-corresponding image portion corresponding to a stationary object such as a traffic light, a road surface, a pedestrian crossing, or a wall. Here, the stationary object-corresponding image portion refers to a portion displayed in the image and corresponding to the stationary object. Then, assuming that, the vanishing point for the most motion vectors is estimated to be the vanishing point of the stationary object corresponding to the stationary object corresponding image portion. Therefore, it can be estimated that each vanishing point that exists after removing vanishing points for the most motion vectors among vanishing points existing in the image is the vanishing point of the moving object corresponding to the moving object corresponding image portion.

  Therefore, the moving object extraction unit 12 extends the motion vector obtained in the time-series image calculated by the three-dimensional position information calculation unit 11 along the direction, and obtains a vanishing point on the image where they intersect. Of these vanishing points, each vanishing point other than the vanishing points for the most motion vectors is estimated to be a vanishing point corresponding to the moving object corresponding image portion. Further, based on the vanishing point of the estimated moving object corresponding image portion in this way, the moving object corresponding image portion on the image is specified, and its three-dimensional image information is acquired. In this way, the moving object corresponding image portion in each time-series image can be specified. Since the motion vector is calculated by the three-dimensional position information calculation unit 11, it is not necessary to calculate a new motion vector to obtain the vanishing point, and the vanishing point can be easily calculated.

  Next, a method for specifying a moving object corresponding image portion by pattern recognition or template matching will be described. For example, for a moving object that is expected to exist as a subject, such as a vehicle such as a car, a motorcycle, or a bicycle, or a pedestrian, the moving object corresponding image portion in the image may be specified using pattern recognition or template matching. Good. In the pattern recognition, the moving object extraction unit 12 stores data for pattern recognition related to the moving object in advance, and performs pattern recognition in the captured image using the stored data, so that a moving object corresponding image portion is obtained. The three-dimensional position information is acquired. Furthermore, in pattern recognition, for example, a moving object corresponding image portion can be identified more efficiently by learning pattern data using a technique such as SVM (Support Vector Machine) or AdaBoost. . In template matching, the moving object extraction unit 12 stores a template related to the moving object in advance, and searches for a portion having a high correlation value with the template in the same manner as in the corresponding point search described above, thereby obtaining an image. The upper moving object-corresponding image portion is specified, and its three-dimensional position information is acquired.

  As in the above pattern recognition and template matching, as a method for specifying a moving object corresponding image portion using a moving object candidate, there is a method for specifying a vehicle on an image from edge distribution in the image, left-right symmetry, and the like ( For example, see JP-A-7-334800). By this method, the moving object extraction unit 12 may specify a moving object corresponding image portion such as a vehicle on the image and acquire the three-dimensional position information thereof.

  In addition, the three-dimensional motion vector obtained from the stereo time-series image is corrected by the moving speed (vehicle speed) of the stereo camera 21 that has generated the stereo time-series image, so that the stationary object corresponding image portion on the image is corrected. There is also a method for discriminating a moving object-corresponding image portion (see, for example, JP-A-2006-134035). When this method is used, the moving object extraction unit 12 receives speed information of the vehicle on which the stereo camera 21 is mounted, and uses the three-dimensional motion vector calculated by the three-dimensional position information calculation unit 11 to move the moving object on the image. The corresponding image portion can be specified and its three-dimensional position information can be acquired.

  Further, the moving object corresponding image portion may be specified by the user selecting the moving object corresponding image portion from the image while viewing the image generated by the stereo camera 21. FIG. 3 is a diagram for explaining a case where the user specifies a moving object corresponding image portion. When the user instructs the display device 3 and the image recording unit 22 using an input unit (not shown) or the like, the display device 3 displays the photographed image recorded in the image recording unit 22 in FIG. As shown, it is displayed as it is. The user can select a part of the image displayed on the display device 3 by operating the mouse or the like of the input unit, as shown by the frame in FIG. And it is sufficient. The selected location is specified as the moving object corresponding image portion.

  Specifically, the image for which the three-dimensional position information calculation unit 11 calculates the three-dimensional position information is read from the image recording unit 22 and displayed on the display device 3. In addition to the displayed image, the display device 3 displays a cursor or the like that can operate the position of the display device 3 on the screen with a mouse, for example, and by selecting a specific portion on the screen with the cursor, The three-dimensional position information of the selected part is input to the moving object extraction unit 12, and the moving object corresponding image part is specified. For example, as shown in FIG. 3A, from the image displayed on the display device 3 by the user, the moving object corresponding image portions M1 and M2 including automobiles m1 and m2, and the moving object corresponding image portion including pedestrian m3. By selecting M3 with the input unit, the moving object extracting unit 12 specifies moving object corresponding image portions on these images and acquires the three-dimensional position information thereof.

  Such selection of the moving object-corresponding image portion may be performed for each image, but since it is complicated, the moving object extraction unit 12 may automatically perform tracking and selection. For example, FIG. 3C is an image after Δt seconds after FIGS. 3A and 3B. As shown in FIG. M1, M2; M3 are selected. Such automatic tracking of the moving object corresponding image portion includes not only the above-described method based on the corresponding point search but also a method using an operation for calculating a motion vector such as the Lucas-Kanade method described later. The Lucas-Kanade method is a method for obtaining a motion vector between images, but by obtaining a motion vector, association between images is possible, and tracking of a moving object corresponding image portion is also possible.

  Moreover, the moving body extraction part 12 may specify a moving body corresponding | compatible image part by one method among the methods mentioned above, and may specify it using any method selectively. For example, the moving object corresponding image portion is first specified by pattern recognition or template matching, and if the moving object corresponding image portion cannot be specified by these methods, the user specifies the moving object corresponding image portion using the input unit. It is good as well.

  On the other hand, the plane setting unit 13 sets a projection plane that the user desires for the display, that is, a line-of-sight direction. In response to this, the three-dimensional position information projection unit 14 on the projection plane set by the plane setting unit 13, the three-dimensional point at each point in the time series of the moving object extracted by the moving object extraction unit 12. A display image on which the position information is projected is created and displayed on the display device 3. FIG. 4 is a diagram showing an example of the projection result, in which the three-dimensional position information obtained from the captured image of the fixed point camera 2 is projected as it is. FIG. 4 shows the locus of the bicycle 52 traveling on the road surface 51, and the solid line 53 shows the locus of the bicycle 52 projected on the road surface 51. In FIG. 4, in addition to the road surface 51 and the locus lines and points of the CG-combined bicycle 52 by the actual image, the actual image itself of the bicycle 52 is also superimposed and projected. Which of these trajectory lines and points and the moving body image itself is selected and projected may be appropriately selected so as to be easily understood within a range that is not complicated (difficult to see).

  On the other hand, FIG. 5 schematically shows the switching of the line-of-sight direction. 5A is an image viewed from the installation position of the fixed point camera 2 corresponding to FIG. 4, but FIG. 5B is an accident witness indicated by reference numeral 54 in FIG. 5A. It is the image seen from the line-of-sight direction.

  Here, in the image processing apparatus 1 of the fixed point camera 2 shown in FIG. 1, the three-dimensional position information accompanying the motion of the moving object extracted by the moving object extracting unit 12 may be projected in order. However, in the image processing device 1a of the drive recorder 4 shown in FIG. 2, since the host vehicle is also moving, alignment between the frames is necessary. For this reason, in the image processing apparatus 1a, the moving object extraction unit 12a extracts the stationary object as well, and it should be noted that the image processing apparatus 1a receives the moving object from the output of the moving object extraction unit 12a. Between the frames integrated by the 3D information integration unit 15 and the 3D information integration unit 15 that integrates the 3D positions between the frames with reference to the stationary body. As a reference, a three-dimensional position information calculation unit 16 that calculates the three-dimensional position of the moving object in the remaining frames and outputs the calculated three-dimensional position information projection unit 14 is further provided.

  FIG. 6 is a diagram for explaining a method of extracting the stationary body. The moving object extraction unit 12 also includes still body corresponding image portions S1 to S4 in each image based on the three-dimensional coordinates, the two-dimensional motion vector, the three-dimensional motion vector, and the like calculated by the three-dimensional position information calculation unit 11. Is identified. Here, the stationary body is a landmark such as a traffic light, a road surface, a pedestrian crossing, a signboard, a wall, etc., and is fixed to the ground. In FIG. 6, a stationary object corresponding image portion S1 including the vicinity of the boundary between the road and the sidewalk and a wall surface and the like, a stationary object corresponding image portion S2 including a traffic light and a road surface such as a pedestrian crossing, and a stationary object corresponding to the sidewalk, the road surface and the wall surface, etc. The stationary object corresponding image part S4 including the image part S3 and the road surface and the lane formed on the road surface is selected, and the moving object extracting unit 12 also specifies the stationary object corresponding image parts S1 to S4 on these images, and 3 Get dimensional image information.

  Here, since the stereo camera 21 is mounted on the vehicle, the stereo camera 21 itself is also moved, and the stationary object corresponding image portions S1 to S4 are moved on the time-series image. As described above, there are the following methods for identifying the stationary object corresponding image portions S1 to S4 in the stationary object that are not fixed on the image but are not actually moved from the image. The moving body extraction unit 12 specifies the still body corresponding image portions S1 to S4 from the captured image using these methods. Moreover, you may specify as a stationary body other than the moving body corresponding | compatible image parts M1-M3 which this moving body extraction part 12 specified on the captured image.

  First, a method for specifying a stationary object corresponding image portion using a vanishing point of motion will be described. The moving object extraction unit 12 obtains vanishing points in the time-series image calculated by the three-dimensional position information calculation unit 11, and among these vanishing points, the vanishing points for the most motion vectors correspond to the stationary object corresponding image portion. Presumed to be the vanishing point of the stationary object. Furthermore, based on the vanishing point of the stationary object estimated in this way, the stationary object corresponding image part on the image is specified, and the three-dimensional image information is acquired. In this way, it is possible to specify the stationary object corresponding image portion in each time-series image. Since the motion vector is calculated by the three-dimensional position information calculation unit 11, it is not necessary to calculate a new motion vector in order to obtain the vanishing point, and the vanishing point can be easily calculated.

  Further, the moving object extraction unit 12 detects a stationary object that is expected to exist, that is, a landmark such as a traffic light, a sign, a signboard, or the like by pattern recognition or template matching, so that a stationary object corresponding image portion is obtained. You may specify. The pattern data and template used at this time may be stored in advance in the moving object extraction unit 12. In this way, the moving object extraction unit 12 specifies the still object corresponding image portion on the image and acquires the three-dimensional image information. As in the case of moving object extraction, the stationary object corresponding image portion can be identified more efficiently by learning the pattern data. Alternatively, the still object-corresponding image portion may be specified by the user selecting the still object-corresponding image portion from the image while viewing the image generated by the stereo camera 21.

  FIG. 7 is a diagram for explaining the processing of the three-dimensional information integration unit 15. When a captured image as shown in FIG. 7A is obtained at time T and a captured image as shown in FIG. 7B is obtained at time T + Δt, the captured image is stationary as shown in FIG. When the body region is extracted, images as shown in FIGS. 7C and 7D are obtained. In these FIG. 7 (c) and FIG. 7 (d), the moving object region is blacked out. First, as shown in FIGS. 8A and 8B, the two images are associated with each other. For this purpose, a corresponding point search method described later may be used, or the aforementioned Lucas-Kanade method may be used.

  FIGS. 8 (a) and 8 (b) correspond to FIGS. 7 (c) and 7 (d), respectively. When the association between the two images is thus performed, the first integration is performed. In the method, three points that are not collinear are selected from the associated points 61 to 65, and the rotations (surfaces) are made to coincide with each other at the time T and the time T + Δt. ) And a translational component (match any one point, or match the centroid position of three points). In the second method, any number of the associated points 61 to 65 is selected, and the selected several points are used as initial values to rotate and translate components using ICP (Iterative Closest Points). (The ICP algorithm will be described later).

  Next, the three-dimensional information at time T + Δt is converted using the calculated rotation and translation components. When the three-dimensional information at time T + Δt after conversion and the three-dimensional information at time T are overlapped, the stationary body region matches, but the moving body region does not match, and two identical subjects exist. By superimposing three-dimensional positions at different times on the same subject, the locus of the moving object can be easily known as shown in FIG. FIG. 9A shows an integrated image of a schematic diagram of the line of sight by the driver shown in FIGS. 3B and 3C, and FIG. 9B shows an overhead view of FIG. 9A. Images are shown. From FIG. 9, it can be seen that the vehicle m2 on the right side is faster than the vehicle m1 on the left side.

  As described above, the method for integrating two pieces of three-dimensional information at different times has been described. However, more time-series images can be integrated in the same manner, and a photographed image as shown in FIG. 4 can be obtained. . Specifically, in the first method, the three-dimensional information at time T + Δt, time T + 2Δt,... Is aligned using the image at time T as a reference. In the second method, the three-dimensional information is sequentially aligned based on the image at the previous time.

  Also, the points selected for integration may be different points at each time. For example, in the first method, the point selected by the pair of time T and time T + Δt does not always exist at time T and time T + 2Δt. It is preferable to update. If the selected points are close to each other, the three-dimensional match in the local part is calculated. Therefore, the three-dimensional match in the local region can be accurately performed, but can be seen in the entire image. Since the result tends to be unstable, it is preferable to select the three points to be separated as much as possible because a stable result can be obtained. Furthermore, in the first method, although three points are selected, a plurality of sets of three points may be selected, and a solution may be obtained in a least square manner from the set of these three points. By doing so, the solution can be obtained stably. Furthermore, in the case of the drive recorder 4, by using the image at the time of the trigger as a reference, the three-dimensional information at the time of the trigger can be output with high accuracy, and errors are accumulated when integrating the three-dimensional information. However, since the error of the three-dimensional information before and after the occurrence of the trigger is reduced, it is useful for accident analysis and the like.

  By configuring in this way, the three-dimensional motion of the moving object analyzed from the time-series three-dimensional captured image can be obtained by, for example, an overhead image as shown in FIG. 4 or FIG. 5A or a driving as shown in FIG. The motion of the moving object can be displayed after being converted into a two-dimensional image in an arbitrary line-of-sight direction, such as an image from the viewer's eye, an image from the eye of the accident witness as shown in FIG. Can be easily understood by the user.

  Further, in the image processing apparatus 1a, since the position information of the moving body between the respective frames is integrated based on the stationary body and then the projection is performed, the stereo camera 21 that obtains the time-series captured images is moving, that is, Even when the stereo camera 21 is an in-vehicle camera of the drive recorder 4, a display image is generated not only from the driver's line of sight but also from any line of sight (projection plane) such as the line of sight of the witness on the road. be able to.

  Preferably, in the three-dimensional position information projection unit 14, the projection plane and the three-dimensional position information projected thereon may be a schematic drawing viewed from the angle set by the plane setting unit 13. . An example is shown in FIG. FIG. 10A is a photographed image similar to the above-described FIG. 4, and FIG. Reference numeral 54 is a travel locus of the host vehicle at the same timing as the bicycle 52.

  In this way, a schematic drawing, for example, the above-described schematic road surface 51, on which the above-mentioned schematic traffic division 55, pedestrian crossing 56, identification symbols for road traffic such as signals are synthesized. In addition, by projecting the position of the moving object, and further enabling display by arbitrarily setting the user's line-of-sight direction, the user can display the moving object from an image such as a computer graphic displaying the minimum necessary information. Can understand movement more easily. The identification symbol for the road traffic may be extracted using pattern recognition or the like. Moreover, you may synthesize | combine the said live-action image partially with the image of the road surface 51 of such a computer graphic. Specifically, the road surface 51 and the moving object position are created by computer graphics, and landmarks such as traffic lights, signs, signboards, and the like use actual images. Thus, it is possible to give reality to a computer graphic image that is easy to recognize.

  Furthermore, from the moving object extraction unit 12 in the image processing apparatus 1 and the three-dimensional information calculation unit 16 in the image processing apparatus 1a, any of the moving object corresponding image portions extracted by the moving object extraction units 12 and 12a is projected. Based on the time-series images in the frames, the three-dimensional coordinates in which the corresponding image portions in the other frames are integrated are calculated. Hereinafter, the three-dimensional coordinates for obtaining the integrated image are referred to as standardized three-dimensional coordinates. Therefore, the calculation method of the standardized three-dimensional coordinates in the three-dimensional information calculation unit 16 in the image processing apparatus 1a that is also moving will be described in detail below. First, the three-dimensional information integration unit 15 selects arbitrary three points included in the stationary object corresponding image portion in the arbitrary reference image set by the surface setting unit 13. Since the three-dimensional coordinates for each of the three images are calculated, the three-dimensional information integration unit 15 can easily select three points that are not on the same straight line. Similarly, the three-dimensional information integration unit 15 acquires three points corresponding to the three points selected in the reference image on an image of a frame different from the reference image for calculating the normalized three-dimensional coordinates. For these three corresponding points, the data calculated by the moving object extraction unit 12a may be used, or the three-dimensional information integration unit 15 may determine the corresponding points by the Lucas-Kanade method described later or the like.

  In this way, the three-dimensional information integration unit 15 selects three points that are not on the same straight line from the still body corresponding image portion of the image at time T, and obtains corresponding points on the image at time T + Δt. The three-dimensional information integration unit 15 then calculates the three-dimensional coordinate of the three points at time T + Δt necessary to match the surface formed by the three points at time T + Δt with the surface formed by the three points at time T. The rotation component and translation component necessary for coordinate transformation are calculated. That is, the three-dimensional information integration unit 15 matches the normal vector of the surface composed of three points at time T + Δt with the normal vector of the surface composed of three points at time T, and A rotation component and a translation component are calculated to perform coordinate transformation such that any one of the three points at time T + Δt is matched with any one point or the centroid of three points at time T is matched with the centroid of three points at time T + Δt. Then, the three-dimensional position information calculation unit 15 converts the three-dimensional coordinates of the specified moving object-corresponding image portion in the image at time T + Δt with the calculated rotation component and translation component, thereby using the image at time T as a reference. Normalized three-dimensional coordinates can be calculated.

  Here, it is preferable that the three points selected in the integrated image are separated from each other in the three-dimensional coordinates. As a result, the stationary object-corresponding image portions match each other in a wide range in the stationary object-corresponding image portion, and not the local matching. Then, the three-dimensional information integration unit 15 may calculate the rotation component and the translation component in a least square manner using the plurality of sets. Thereby, the three-dimensional information integration unit 15 can obtain a more stable solution (rotation component and translation component), and the conversion accuracy of the three-dimensional coordinates is increased.

  Another method will be described as a method of converting the three-dimensional coordinates of the specified moving object corresponding image portion with the integrated image as a reference. Specifically, this is a method using the ICP algorithm. According to this, the three-dimensional information integration unit 15 uses the three-dimensional coordinates at arbitrary points of the stationary object corresponding image portion extracted by the moving object extraction unit 12a as initial values, and corresponds to these multiple points at other times. Acquires a point on the series image. Then, by using the ICP algorithm, the three-dimensional information integration unit 15 corresponds to a plurality of points in the stationary object corresponding image portion of the reference image captured at time T, and corresponds to the stationary object corresponding to the image at time T + Δt. It is possible to calculate a rotation component and a translation component necessary for coordinate conversion so as to match a plurality of points in the image portion in three-dimensional coordinates. Further, the three-dimensional information integration unit 15 converts the three-dimensional coordinates of the specified moving object corresponding image portion in the image at the time T + Δt by the calculated rotation component and translation component, so that the time T + Δt with the image at the time T as a reference. It is possible to calculate the normalized three-dimensional coordinates of the identified moving object corresponding image portion in the image. In this way, by using the ICP algorithm, the three-dimensional information integration unit 15 can perform robust coordinate transformation that is not easily affected by noise for a plurality of corresponding points.

  Note that the conversion of the three-dimensional coordinates of the moving object corresponding image portion specified in the image at the time T + Δt has been described with reference to the reference image at the time T. However, the three-dimensional information integration unit 15 has specified the moving object corresponding to another time-series image. Similarly, the conversion of the three-dimensional coordinates of the image portion may be performed by calculating the rotation component and the translation component. Note that if the moving body on which the stereo camera 1 is mounted is going straight, the stationary body corresponding image portion corresponding to the stationary body at a distant place in the front exists in a plurality of time-series images, but the moving body turns left. Alternatively, when the vehicle is bent, such as turning right, the stationary object corresponding image portion existing in the subsequent time series image changes. Therefore, depending on each time-series image, there may be no corresponding point of the point selected first in the reference image. Even in such a case, the three-dimensional information integration unit 15 adds the selected point to a new point. Change (update) to the point. Then, it is possible to calculate standardized three-dimensional coordinates by performing coordinate transformation a plurality of times. As described above, the three-dimensional information integration unit 15 can calculate the standardized three-dimensional coordinates using the three-dimensional image information of the stationary body without being limited by the movement of the moving body.

  Since the three-dimensional information integration unit 15 calculates the normalized three-dimensional coordinates for the moving object corresponding image portion specified in this way, the three-dimensional information integration unit 15 obtains the standardized three-dimensional coordinates or determines the fixed point camera 2 in advance. It is possible to calculate information relating to the motion of the moving object corresponding to the moving object corresponding image portion from the output of the moving object extracting unit 12 for which the normalized three-dimensional coordinates are obtained. The information is, for example, the speed, acceleration, speed vector, acceleration vector, etc. of the moving object. The motion vector calculated by the three-dimensional image information integration unit 15 is also one piece of information. Therefore, a calculation method for calculating the speed, acceleration, and vector of the moving object corresponding to the specified moving object corresponding image portion using the normalized three-dimensional coordinates will be described. First, calculation results obtained by using standardized three-dimensional coordinates of three consecutive frames in a moving object corresponding image portion corresponding to the same moving object at a frame interval of t seconds are respectively (x1, y1, z1) and (x2, y2). , Z2) and (x3, y3, z3). Next, from (x1, y1, z1) and (x2, y2, z2), the velocity v1 of the moving body when these are imaged can be expressed by the following equation.

v1 = {(Vx1) ² + (Vy1) ² + (Vz1) ² } ^1/2
However,
(Vx1, Vy1, Vz1)
= ((X2-x1) / t, (y2-y1) / t, (z2-z1) / t)
It is.

  Similarly, from (x2, y2, z2) and (x3, y3, z3), the velocity v2 of the moving body when these are imaged can be expressed by the following equation.

^{v2 = {(Vx2) 2 +} (Vy2) 2 + (Vz2) 2} 1/2
However,
(Vx2, Vy2, Vz2)
= ((X3-x2) / t, (y3-y2) / t, (z3-z2) / t)
It is.

  Therefore, the acceleration a of the moving body obtained from the corresponding points of the three images can be expressed by the following equation.

a = {(Ax) ² + (Ay) ² + (Az) ² } ^1/2
However,
(Ax, Ay, Az)
= ((Vx2-Vx1) / t, (Vy2-Vy1) / t, (Vz2-Vz1) / t)
It is.

Also, the three-dimensional motion vectors (Ux1, Uy1, Uz1) and (Ux2, Uy2, Uz2) are
(Ux1, Uy1, Uz1) = (x2-x1, y2-y1, z2-z1)
(Ux2, Uy2, Uz2) = (x3-x2, y3-y2, z3-z2)
It is.

  Preferably, the three-dimensional position information projection unit 14 creates the display image together with information on the motion of the moving body thus obtained. FIG. 11 is a schematic image shown in FIG. 5A in which information related to the motion is projected together. Specifically, FIG. 11A shows motion information between frames as motion vectors (connecting the same positions of moving objects), and the motion vectors are represented by arrows. FIG. 11B shows motion information between frames as a velocity vector, and the velocity vector is represented by an arrow whose length changes according to the velocity. FIG. 11C shows the speed (km / h) superimposed as it is as motion information between frames. Thus, by displaying together the information regarding the movement of each moving object, the user can more easily understand the movement of the moving object.

  On the other hand, FIG. 12 pops up other motion information (the speed in FIG. 11C) by clicking (or simply overlaying) on the motion vector display shown in FIG. It is displayed. In addition, the moving object positions and movement information of all the temporally different frames as shown in FIG. 11 and FIG. 12 are not superimposed and displayed as shown in FIGS. 13 (a) to 13 (c). The position may be displayed in time series (animation display, etc.).

(Embodiment 2)
FIG. 14 is a block diagram showing an electrical configuration of an accident verification system including the image processing apparatus 1b according to the second embodiment of the present invention. This accident verification system is similar to the accident verification system shown in FIG. 1 and FIG. 2 described above, and corresponding portions are denoted by the same reference numerals, and description thereof is omitted. It should be noted that in the present embodiment, the moving object extraction unit 12a also extracts a stationary object, and then the three-dimensional position information projection unit 14 is first placed on the projection plane set by the plane setting unit 13. For each frame, the three-dimensional position information of the moving body and the stationary body extracted by the moving body extraction unit 12a is projected, and the projected image integration unit 17 aligns the stationary body in each of the projected frame images. Then, the projection planes are integrated.

  FIG. 15 is a diagram for explaining the alignment of the projection plane according to the picture drawing. Even if each projection surface is a curved surface, it is possible to perform alignment in the same manner. However, in order to simplify the description, it will be described as a plane. First, FIG. 15A shows a state in which a moving object is projected on a plane in each of four time-series images. In FIG. 15B, the plurality of images different in time are aligned on the basis of the identification symbol for the road traffic that is a stationary body. Specifically, the vertical alignment in the figure is performed at the pedestrian crossing 56, and the horizontal alignment is performed at the traffic division 55. FIG. 15C shows a state in which the planes after alignment are integrated.

  Thus, by integrating the projection results for each frame on the basis of a stationary body, the stereo camera 21 that obtains the time-series captured images is moving. That is, the stereo camera 21 is an in-vehicle camera such as the drive recorder 4. In some cases, a display image can be created not only from the driver's line of sight but also from an arbitrary line of sight (projection plane) such as the line of sight of the witness on the road.

(Embodiment 3)
FIG. 16 is a block diagram showing an electrical configuration of an accident verification system including the image processing apparatus 1c according to the third embodiment of the present invention. This accident verification system is similar to the accident verification system shown in FIG. 1 and FIG. 2 described above, and corresponding portions are denoted by the same reference numerals, and description thereof is omitted. It should be noted that in the present embodiment, the moving object extraction unit 12a extracts a stationary object as well, while the surface setting unit 13c receives the output of the moving object extraction unit 12a, and extracts the extraction result in a certain frame. A virtual projection plane as a reference is set, and the three-dimensional position information projection unit 14c integrates the three-dimensional position of the moving body in each frame based on the stationary body from the output of the moving body extraction unit 12c. Projecting onto the virtual projection plane set at 13c.

  FIG. 17 is a diagram for explaining such a projection method onto the virtual projection plane. Even in the present embodiment, even if the virtual projection plane is a curved surface, it is possible to perform alignment in the same manner. However, in order to simplify the description, it will be described as a plane. In FIG. 15 described above, a moving object is projected onto the plane set in each frame, and then the plane is aligned, thereby creating a plan view allowing the bird's-eye view of the whole. On the other hand, in the present embodiment, a virtual plane that can be viewed from the whole is prepared, and the three-dimensional position information projection unit 14c has a reference plane in the virtual plane of each image as shown in FIG. As shown in FIG. 17B, the plane set by the remaining frames is connected to the virtual plane set by the frame to be created.

  Here, since the virtual plane in the case of the uphill is as shown in FIG. 17C, the positional accuracy of the projected moving object is lowered. In such a case, for example, it is possible to determine whether the road is a slope by using GPS information from the navigation system. Therefore, it is possible to improve the accuracy by changing the inclination of the virtual plane to be set. . In order to further improve the accuracy, a virtual curved surface may be set as shown in FIG.

  Thus, after integrating the position information of the moving body between the frames based on the stationary body, the stereo camera 21 is moved by projecting onto the virtual projection plane, that is, the stereo camera 21 is moved to the drive recorder 4 or the like. Even in the case of the in-vehicle camera, it is possible to create a display image not only from the driver's eyes but also from any eye (projection plane) direction such as the eyes of the witness on the road.

  FIG. 18 is a diagram for explaining a method of three-dimensional calculation (distance calculation) in the three-dimensional position information calculation unit 11 for the output image of the stereo camera 21 (21-1, 21-2). For simplification of explanation, it is assumed that the aberrations of the stereo cameras 21-1, 21-2 are corrected well and are installed in parallel. Even if the actual hardware is not in such a condition, it can be converted into an equivalent image by image processing. The advantage of using an image parallelized by hardware or image processing is that the search area for corresponding points can be limited to one dimension as will be described later with reference to FIG. 19C and FIG. In the case of a technique that is easy to perform a two-dimensional search, such as the limited correlation method, it is possible to perform correspondence using stereo images that have not been parallelized, and to directly three-dimensionalize the obtained corresponding point results ( Since parallelization is performed by image processing, noise is superimposed on the image, so if the correspondence is performed using the parallelized image, the accuracy is reduced. To minimize the effect of noise.)

As the stereo cameras 21-1 and 21-2, those having at least a focal length (f), the number of pixels of the imaging surfaces (CCD) S1 and S2, and the size (μ) of the pixels are equal to each other. When the subject P is photographed with the optical axes L1 and L2 arranged in parallel to each other with the base line (baseline) length (B) spaced apart from each other, the parallax (the number of displaced pixels) on the imaging surfaces S1 and S2 When Δd (= d1 + d2), the distance (D) to the subject P is
D = f · B / Δd
Can be obtained.

  The three-dimensional position (X, Y, Z) of each part of the subject P is calculated as follows, where x and y are positions on the pixel.

X = x · D / f
Y = y · D / f
Z = D
Here, for example, an in-vehicle stereo camera is required to be easy to install due to downsizing as well as needs to measure the distance to a distant preceding vehicle with high accuracy as described above.
The depth direction resolution ΔZ of the stereo camera is
ΔZ = (D ² / B) · (1 / f) · Δd
Therefore, as a method for improving accuracy, a method of increasing the focal length f and a method of increasing the baseline length B can be considered. However, as described above, the former has a disadvantage that the visual field range is narrow, and the latter is large in size. As a method for improving the accuracy without the above drawbacks, there is subpixelization of correspondence. This is because by performing the associating operation with a resolution equal to or less than the pixel unit, the resolution of the parallax Δd can be reduced and the resolution of the stereo three-dimensional measurement can be made fine.

  FIG. 19 illustrates a method for searching for corresponding points of a reference image (stereo camera 21-2) with respect to a reference image (stereo camera 21-1) in obtaining the parallax Δd in the three-dimensional position information calculation unit 11. FIG. First, as shown in FIG. 19A, a two-dimensional window W1 having a predetermined size with the attention point P as the center or the center of gravity is set on the reference image F1. Similarly, as shown in FIG. 19B, a large number of windows W2 having a predetermined size are set on all possible positions on the reference image F2. Here, as shown in FIG. 19C, when the standard image F1 and the reference image F2 are arranged almost in parallel as described above, the Y coordinate position Py of the point of interest P on the standard image F1. Since the corresponding position of the reference image F2 can be assumed, the window W2 may be set only on this line (basically, the window W2 is set while shifting by one pixel).

  Furthermore, when the standard image F1 and the reference image F2 are arranged almost in parallel and the parallax Δd between the attention point P of the standard image F1 and the corresponding position of the reference image F2 is known to some extent, As shown in FIG. 19D, the window W2 may be set only in the range Δd ′ of the parallax Δd.

  On the other hand, in searching for corresponding points, a window setting based on a multi-resolution strategy may be used as a suitable method for searching for many corresponding points or for searching for corresponding points in a short time from a high-resolution image. Good. FIG. 20 is a diagram for explaining the multi-resolution strategy when it is assumed that the base image F1 and the reference image F2 are arranged almost in parallel. In FIG. 20 (a), a plurality of windows W2 of the reference image F2 are set on the Y coordinate position Py of the point of interest P on the standard image F1, as in the above-described FIGS. 19 (a) and (c). Yes. However, as shown in FIG. 20B, the resolutions of the respective images F1 and F2 are converted to create low-resolution images F1 ′ and F2 ′, and between these low-resolution images F1 ′ and F2 ′. Associate with. Therefore, in the low-resolution images F1 'and F2', the number of pixels to be searched is reduced by the amount of the decrease in the number of pixels. For example, when the resolution is halved, the number of search pixels is halved.

  After obtaining the corresponding position P ′ at the low resolution in this way, as shown in FIG. 20C, the search is performed by returning to the high-resolution images F1 and F2, but from the corresponding position obtained at the low resolution, Since the search range is known, a high-resolution search need only be performed within a very narrow range. By narrowing the search range ΔW in this way, it is possible to search many corresponding points as described above within the same time, or to search from a high-resolution image.

  In the above description, the low-resolution image is created in only one stage, but it may be created in a plurality of stages and the search position may be narrowed down sequentially. For example, when the input image is 1280 × 960 pixels, the first-stage low-resolution image is 640 × 480 pixels, the second-stage low-resolution image is 320 × 240 pixels, and the third-stage low-resolution image is 160 Three types of low resolution images of × 120 pixels are created, and corresponding positions are searched in order from an image of 160 × 120 pixels.

  Furthermore, as another method for searching for corresponding points in the three-dimensional position information calculation unit 11, a correlation method that suppresses an amplitude component, which is known as a robust pattern similarity calculation method, can be used. In such a correlation method, the similarity calculation is performed using only the phase component signal in which the amplitude component is suppressed from the frequency resolution signal of the pattern. Therefore, the correlation method is affected by a difference in photographing conditions of the stereo camera 21 and noise. It is difficult to achieve the robust correlation calculation. Further, unlike the conventional two-dimensional correlation method and feature extraction method using grayscale data, it has a feature that it is resistant to disturbances and can calculate with high accuracy even for images with low brightness and contrast. Known techniques for calculating the frequency-resolved signal having such a pattern include Fourier transform, discrete cosine transform, wavelet transform, Hadamard transform, and the like. As for the discrete cosine (DCT) code-only correlation method, for example, “Fusion of image signal processing and image pattern recognition—DCT code-only correlation and its application” (Kishiya Hitoshi, Faculty of System Design, Tokyo Metropolitan University) You can refer to articles from Shop 2007 (2007.3.8-9)).

  A phase-only correlation method (POC), which is a representative correlation method capable of realizing a robust correlation calculation, uses a Fourier transform for conversion, and performs a correlation calculation only for phase components in which the amplitude component of the Fourier series is suppressed. The details of the phase-only correlation method (POC) will be described below.

First, let f (n ₁ , n ₂ ) and g (n ₁ , n ₂ ) be two images F1 and F2 having an image size N ₁ × N ₂ pixels, and for the convenience of formulation, the index of the discrete space is n ₁ = If -M ₁ ,... M ₁ , n ₂ = −M ₂ ,... M ₂ and the image sizes are N ₁ = 2M ₁ +1 pixels and N ₂ = 2M ₂ +1 pixels, 2 of these images The dimensional Fourier transform (2D DFT) is given by the following equation, respectively.

_{Here, k 1 = -M 1, ···} M 1, k 2 = -M 2, ··· M 2

And Σ _n1n2 is

It is. A _F (k ₁ , k ₂ ) and A _G (k ₁ , k ₂ ) are amplitude components, and e ^{jθF (k1, k2)} and e ^{jθG (k1, k2)} are phase components.

Then, the phase only correlation method (POC) performs a correlation calculation of only the phase component in which the amplitude component of the Fourier series thus obtained is suppressed. First, the combined phase spectrum ＲR (k ₁ , k ₂ ) of the patterns f and g is defined as follows.

Here, the complex conjugate of G (k ₁ , k ₂ ) is shown with an overline. Further, θ (k ₁ , k ₂ ) = θ _F (k ₁ , k ₂ ) −θ _G (k ₁ , k ₂ ).

Correlation calculation can be performed by performing inverse Fourier transform on the combined phase spectrum ^ R (k ₁ , k ₂ ). That is, θ (k ₁ , k ₂ ) = θ _F (k ₁ , k ₂ ) −θ _G (k ₁ , k ₂ ), and the POC function ｒr (n ₁ , n ₂ ) is R (k ₁ , n ₂ ). k ₂ ) is a two-dimensional discrete Fourier inverse transform (2D IDFT) and is defined by the following equation.

Where Σ _n1n2 is

It is.

  The POC value obtained by the above POC function processing is known to have a sharp similarity peak in the coordinates of the amount of movement between images (reference window and reference window) as shown in FIG. Robustness is high. The height of the peak of the POC indicates the pattern similarity. Then, the position information calculation unit 11 estimates the position shift amount (= parallax dsub) by estimating the peak position of the POC. At this time, since the POC is obtained discretely, high resolution corresponding region coordinates can be obtained by interpolating and estimating the peak position with subpixels. The peak position interpolation estimation method can be performed by fitting a function such as a parabola. The positional deviation amount Δd between the candidate areas is an amount obtained by adding the positional deviation amount dsub of the sub-pixel obtained by the POC method to the positional deviation quantity dpixel at the pixel level between the candidate areas.

  Therefore, an example of a specific corresponding point search in the phase-only correlation method is as follows. As described above, it is known that the POC value has a steep correlation peak in the coordinates of the movement amount between images (the reference window W1 and the reference window W2). Therefore, as shown in FIG. When the point on the reference image F2 corresponding to the point P on the image F1 is P ′ and the windows W1 and W2 are set such that the points P and P ′ are respectively located at the center of gravity, the POC value between the windows W1 and W2 is A peak appears at the center of gravity of the windows W1 and W2. Therefore, if the reference window W2 is not set so that the point P ′ is located at the center of gravity, but is set so as to be shifted by one pixel in the horizontal direction, the POC value also peaks at a position shifted by one pixel from the center of gravity. Similarly, as shown in FIG. 22B, when the reference window W2 is set so that the point P ′ is further shifted by two pixels laterally from the center of gravity position, the POC value is also shifted by two pixels from the center of gravity position. Since the peak is set, as described with reference to FIG. 19, the window W2 set on the reference image F2 side does not need to be shifted pixel by pixel, and may be set with a certain sampling interval. Therefore, how much sampling interval should be set depends on the searchable range W3, but generally, it is said to be about half the window size as shown in FIG. Therefore, the sampling interval may be set so that, for example, the half of the window size overlaps. Therefore, assuming that the maximum disparity in the base image F1 and the reference image F2 is 128 pixels, the window size is 31 × 31, and the range that can be searched with POC is ± 8 pixels with respect to the center of gravity position, the disparity of 128 pixels at maximum is assumed. In order to perform the search, the windows need only be shifted by 16 pixels, so that eight windows may be set.

Further, when the multi-resolution strategy is used, the search range can be reduced by reducing the image size as described in FIG. Specifically, in FIG. 22 described above, it is necessary to set eight windows on the reference image F2 for a certain point on the standard image F1, but when the image is reduced to ½, the window to be set is set. May be half, four. If the image is further reduced to 1/2, the number of windows to be set is two. If the image is further reduced to 1/2, the number of windows to be set is one. That is, when the maximum parallax is 128 pixels as described above, the maximum parallax is 8 pixels by reducing the image to (1/2) ⁴ = 1/16, so search is performed in one window. Will be able to. Accordingly, when the corresponding position on the image reduced to 1/16 is first obtained, the operation of setting one window as the initial position on the image reduced to 1/8 and obtaining the corresponding position will be described hereinafter. What is necessary is repeated sequentially.

  Still other methods of searching for corresponding points in the three-dimensional position information calculation unit 11 include an SAD (absolute sum of density differences) method, an SSD (sum of squares of density differences) method, and an NCC (normalized cross correlation) method. Etc. can also be used.

  The motion vector calculation by the above-mentioned Lucas-Kaneda method will be described below. The apparent movement between two images such as a time series image is called a motion vector (optical flow). Assuming that the same point has the same brightness on the two images, the following equation holds.

Here, I is the luminance of the image, x and y are the coordinates on the image, and vx and vy are the motion vectors.

  The following equation is obtained by Taylor expansion of the above equation.

Transforming the above equation,

It becomes. This equation is called an optical flow constraint equation.

  By the way, it is not possible to obtain a motion vector for one point (x, y) on the image using the above equation. Therefore, in the Lucas-Kanade method, a window is set around one point (x, y) on the image, and the constraint equation is weighted and combined under the assumption that the motion vector does not change in the window. , (X, y). Specifically, it can be realized by solving the following equation.

Furthermore, the ICP algorithm described above will be described below. The ICP algorithm minimizes the error between corresponding points by iterative calculation, and the processing flow is as follows. As shown in Figure 24, _{N t} point group of points of the individual T = | a _{t i _{i∈N t},} different _{N s} point group of points of the individual _{S = {s i | s∈N s} } , The distance from the point group T at each point s _i of the point group S is as follows.

If a point corresponding to each point _{s i} and _{m i} ∈T, the corresponding point bunching M of the points _{s i,}
M = C (S, T)
It becomes. However, C is a function for obtaining the nearest point.

  When the corresponding point group M of the point group S is thus obtained, the alignment parameters (rotation matrix R, movement vector t) can be obtained by minimizing the following expression.

By repeating this error until it becomes sufficiently small, alignment can be performed.

1, 1a, 1b, 1c Image processing device 2 Fixed point camera 3 Display device 4 Drive recorder 11 Three-dimensional position information calculation unit 12, 12a Moving object extraction unit 13, 13c Surface setting unit 14, 14c Three-dimensional position information projection unit 15 Three-dimensional Information integration unit 16 Three-dimensional position information calculation unit 17 Projected image integration unit 21; 21-1, 21-2 Stereo camera 22 Image recording unit 41 Temporary storage unit 42 Trigger generation unit 51 Road surface 52 Bicycle 53 Trajectory 54 Line-of-sight direction 55 Traffic Category 56 Crosswalk F1 Standard image F2 Reference image m1, m2 Car m3 Pedestrian M1, M2, M3

Claims (12)

An image processing apparatus for time-yield can be image data collected by the three-dimensional position information of the subject in the imaging field image sequence as input, it displayed on the display device processes the input image data,
A moving object extraction unit that extracts the same moving object and a stationary object from the time-series captured images;
A plane setting unit for setting a projection plane for the display;
A three-dimensional position information projection unit that creates a display image obtained by projecting the three-dimensional position information of the moving object extracted by the moving object extraction unit on the projection plane set by the surface setting unit based on the stationary object; An image processing apparatus comprising: The input image data is a time-series captured image by a stereo camera input from a drive recorder,
The three-dimensional position information calculation part which acquires the three-dimensional position information of the to-be-photographed object in the said captured image by carrying out corresponding point search processing of the image between the right and left of the said stereo camera, The further characterized by the above-mentioned. Image processing device.   The image processing apparatus according to claim 1, wherein the input image data includes a time series captured image of a monocular camera and radar distance information. From the output of the previous SL moving object extraction unit, and the 3-dimensional information integration section that integrates the basis of the stationary body a three-dimensional positions between the frames of the moving object,
Between the frames integrated by the three-dimensional information integration unit, the three-dimensional position of the moving object in the remaining frames is calculated with reference to the three-dimensional position of the moving object in a certain frame, and is output to the three-dimensional position information projection unit. The image processing apparatus according to claim 1, further comprising a three-dimensional position information calculation unit. Before Symbol 3-dimensional position information projection unit, for each frame, on the projection plane, and projecting the three-dimensional position information of the moving object extracted by the moving object extraction unit and the stationary body,
The projection image integration part which integrates each said projection surface by aligning the said stationary body in each projected said frame image is further provided, The any one of Claims 1-3 characterized by the above-mentioned. An image processing apparatus according to 1. Before SL plane setting unit are inputted an output of the moving object extraction unit, a virtual projection plane relative to the extraction result in a certain frame is set,
The three-dimensional position information projection unit integrates the three-dimensional position of the moving body in each frame based on the stationary body from the output of the moving body extraction unit, and projects it on the virtual projection plane set by the surface setting unit The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   The said three-dimensional position information projection part produces the said display image combining the information regarding the motion of each moving body extracted in the said three-dimensional position information by the said moving body extraction part, The said display image is characterized by the above-mentioned. The image processing apparatus according to any one of the above.   In the three-dimensional position information projection unit, the projection plane and the three-dimensional position information projected thereon are real images obtained by converting the input image data into an image viewed from an angle set by the plane setting unit. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   In the three-dimensional position information projection unit, the projection surface is a schematic road surface viewed from an angle set by the surface setting unit, and the three-dimensional position information is a real image extracted from the input image data. The image processing apparatus according to claim 8, comprising:   8. The projection plane and the three-dimensional position information projected thereon comprise a schematic drawing viewed from an angle set by the plane setting unit. An image processing apparatus according to 1.   The image processing apparatus according to claim 10, wherein the pictorial drawing is obtained by combining an identification symbol for the schematic road traffic on the schematic road surface. A method for time-obtained image data which can be collected by the three-dimensional position information of the subject in the imaging field image sequence as input, and creates a display image by processing the input image data,
Extracting the same moving object and stationary object from the time-series captured images;
Setting a projection plane for the display;
And a step of creating a display image obtained by projecting the three-dimensional position information of the extracted moving object on the set projection plane based on the stationary object .