JP4553071B1 - 3D information display device and 3D information display method - Google Patents

Abstract

The three-dimensional information display device captures a subject and generates a plurality of time-series images, a three-dimensional information acquisition unit that acquires three-dimensional information, and a time-series image based on the time-series image and the three-dimensional information. A three-dimensional image information calculation unit that calculates three-dimensional image information, a moving object specifying unit that specifies a moving object-corresponding image part in a time-series image, and a time-series of the moving object-corresponding image part specified using the three-dimensional image information An arithmetic unit that calculates three-dimensional coordinates based on any one of the images, and an image in which the identified moving object corresponding image portion in the time-series image is integrated with the reference image based on the three-dimensional coordinates is displayed. Display device.
[Selection] Figure 1

Description

  The present invention relates to a three-dimensional information display apparatus and a three-dimensional information display method for displaying a moving object corresponding image portion in a time-series image by integrating an arbitrary image as a reference.

  In recent years, various systems for the purpose of improving safety have been studied in the automobile industry. In particular, a crisis avoidance system using an image sensor provided with an imaging device has been developed. Specifically, based on the image of the subject imaged by the imaging device, a system has been developed to identify obstacles around the vehicle, analyze the movement of the obstacles, and avoid obstacles. Has been.

  In addition, when an accident occurs without avoiding danger, a system has been developed that extracts various information useful for investigating the cause of the accident by analyzing images before and after the accident. For example, Patent Document 1 discloses a system that analyzes images of an accident such as the speed of an accident vehicle by acquiring images before and after the accident using an imaging device installed at an intersection or the like, and analyzing these images. Is disclosed. This system prepares plan view data, which is data including only stationary objects such as road surfaces such as intersections and pedestrian crossings, which are accident sites, and projects the images before and after the accident onto the plan view data. Then, analyze the situation of the accident. The technique described in Patent Document 2 obtains a camera vector value (CV value) indicating the three-dimensional position and orientation of a camera mounted on a moving body from the feature points of a monocular moving image, and obtains the obtained CV value. Based on the above, the camera position is superimposed on the image.

  However, since the technique disclosed in Patent Document 1 uses an image from a camera that is a fixedly installed imaging device, it cannot be applied to a camera mounted on a vehicle that is a moving body. Further, since it is necessary to prepare plan view data in advance, there is a problem that it can only deal with an accident situation analysis at a predetermined place.

Further, the technique disclosed in Patent Document 2 has a problem that it is difficult to calculate an accurate CV value for an image in which a moving object region is dominant. Furthermore, although the technique disclosed in Patent Document 2 can determine the moving body on which the camera is mounted and the movement of the camera, it does not consider determining the positions of other moving bodies in the image.
JP 2004-102426 A JP 2008-5478 A

  The present invention has been made in view of the above-described circumstances, and an object thereof is to provide a three-dimensional information display device and a three-dimensional information display device that can display a moving object corresponding image portion in a time-series image in an integrated manner with an arbitrary image as a reference. It is to provide an information display method.

  The three-dimensional information display device of the present invention displays an image in which the moving object corresponding image portion in the time series image is integrated with the reference image using the three-dimensional image information in the time series image. Thereby, the temporal movement of a moving body can be shown.

It is a block diagram which shows the structure of the three-dimensional information display apparatus which concerns on one Embodiment of this invention. It is a figure for demonstrating the corresponding point search in a time series stereo image. FIG. 3A is a diagram for explaining a case where an operator specifies a moving object corresponding image portion, and FIG. 3A is a view showing a state where the moving object corresponding image portion is specified in the image at time T, and FIG. ) Is a diagram showing a state in which the moving object corresponding image portion in the image at time T + Δt is specified. FIG. 4A is a diagram for explaining a case where the operator specifies a stationary object-corresponding image portion, and FIG. 4A is a diagram showing a state where the stationary object-corresponding image portion is identified in the image at time T. (B) is a figure which shows the state which specified the stationary body corresponding | compatible image part in the image in time T + (DELTA) t. It is a figure which shows an example of the time series image used for the integration of the image which concerns on one Embodiment of this invention. FIG. 6A is a diagram illustrating an integrated image in an embodiment of the present invention, FIG. 6A is a diagram illustrating an integrated image based on an arbitrary image, and FIG. 6B is an overhead view of the integrated image. It is a figure which shows the image converted into. FIG. 7A is a diagram illustrating an image on which motion information is superimposed and displayed in an embodiment of the present invention. FIG. 7A is a diagram illustrating an image on which motion vectors are superimposed and FIG. FIG. 7C is a diagram showing an image displayed in a superimposed manner, and FIG. FIG. 8A is a diagram showing an image in which moving object corresponding image portions are displayed in time series in one embodiment of the present invention, FIG. 8A is a diagram showing a first display image, and FIG. FIG. 8C illustrates a second display image, and FIG. 8C illustrates a third display image. It is a figure which shows the image which selectively displays the moving information of the moving body in one Embodiment of this invention. FIG. 10A is a diagram showing an image displayed for moving object corresponding image portions of a plurality of different moving objects in an embodiment of the present invention, and FIG. 10A is a diagram showing an image by the first display method; (B) is a figure which shows the image by the 2nd display method. FIG. 11A is a diagram illustrating a display image example of each moving object corresponding image portion when the integrated display is a bird's eye view according to an embodiment of the present invention, and FIG. 11 (B) is a diagram showing an image in which moving object corresponding image portions are integrated and displayed at a desired time interval, and FIG. 11 (C) is a diagram corresponding to each moving object at equal distance intervals. FIG. 11D is a diagram showing an image in which image portions are integrated and displayed, and FIG. 11D integrally displays moving object corresponding image portions corresponding to a reference moving object at equal distance intervals, and about moving object corresponding image portions corresponding to the remaining moving objects. These are the figures which show the image integrally displayed about the flame | frame corresponding to the moving body used as a reference | standard. It is a flowchart which shows operation | movement of the three-dimensional information display apparatus which concerns on one Embodiment of this invention. It is a figure which shows the image integrated in the traffic accident in one Embodiment of this invention. It is a figure which shows another image integrated in the traffic accident in one Embodiment of this invention. It is a figure which shows another image integrated in the traffic accident in one Embodiment of this invention.

  Hereinafter, an embodiment according to the present invention will be described with reference to the drawings. In addition, the structure which attached | subjected the same code | symbol in each figure shows that it is the same structure, The description is abbreviate | omitted. Note that the three-dimensional information display device 100 may be used as a so-called driving recorder that is mounted on an automobile and used to investigate the cause of an accident such as a rear-end collision. Note that the three-dimensional information display apparatus 100 can be used for other purposes.

  First, the configuration of the three-dimensional information display apparatus 100 according to the embodiment of the present invention will be described. FIG. 1 is a block diagram showing a configuration of a three-dimensional information display apparatus according to an embodiment of the present invention. As shown in FIG. 1, the three-dimensional information display device 100 includes a stereo camera 1 that is an imaging unit and a three-dimensional information acquisition unit, an arithmetic processing unit 3, an input unit 4, a display device 5, and a trigger 13. It is prepared for.

  The stereo camera 1 is mounted on a moving body such as a vehicle and acquires a stereo time-series image. The stereo camera 1 is a camera having an image pickup device such as a CCD (Charge-Coupled Devices), for example. The left and right cameras in the stereo camera 1 capture the subject at the same timing to obtain a pair of left and right images. In addition, it is preferable that the aberrations of the left and right cameras are corrected well, and they are installed parallel to each other. Thus, in the stereo camera, each camera is installed in parallel, so that a parallelized image can be obtained, and three-dimensional image information can be easily obtained from these images. Note that the three-dimensional image information refers to three-dimensional coordinates, two-dimensional and three-dimensional motion vectors, etc., which can be obtained from stereo time-series images, based on the position of the camera. The stereo camera 1 repeats imaging at any time with a constant period. In addition, the stereo image that the stereo camera 1 captures and captures the subject includes stereoscopic information.

  For example, when the imaging unit is a monocular camera, the three-dimensional information acquisition unit is configured to include a device capable of three-dimensional measurement, for example, a measuring instrument using laser or millimeter wave, and the three-dimensional information is acquired by this measuring instrument. May be.

  The input unit 4 is, for example, a keyboard, a mouse, a push button, a switch, and the like. The input unit 4 is operated by an operator and inputs an operation command to the arithmetic processing unit 3. The display device 5 is, for example, a CRT (Cathode Ray Tube) display, an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display, a plasma display, or the like, according to the image signal generated by the image generation unit 11. Display the image.

  The arithmetic processing unit 3 includes various electronic components, integrated circuit components, a CPU (Central Processing Unit), a storage device, and the like. Note that the storage device stores, for example, a ROM (Read Only Memory) that stores a control program of the three-dimensional information display device 100, data such as arithmetic processing and control processing, and an image of a subject captured by the stereo camera 1. It comprises an EEPROM (Electrically Erasable Programmable ROM), a RAM (Random Access Memory), a nonvolatile memory such as a flash memory, and the like for temporary storage. The arithmetic processing unit 3 functionally includes an image recording unit 6, a three-dimensional image information calculation unit 7, a moving object specifying unit 8, a stationary object specifying unit 9, a calculating unit 10, a storage unit 12, and an image signal. And a generation unit 12.

  The image recording unit 6 stores images captured by the stereo camera 1 at regular intervals and transmitted as signals to the arithmetic processing unit 3. Further, usually, the image recording unit 6 does not store an image of a predetermined amount or more, and erases the recorded image from the old image at any time while storing a newly generated image. The amount of image stored in the image recording unit 6 may be a preferable value in consideration of the capacity of the image recording unit 6 or the intended use of the three-dimensional information display device 100.

  The three-dimensional image information calculation unit 7 calculates the three-dimensional image information of the image stored in the image recording unit 6. Specifically, the three-dimensional image information calculation unit 7 obtains a three-dimensional coordinate, a motion vector, and the like based on the imaging position of the point on the image. A known method may be used as a method for obtaining three-dimensional image information (such as the three-dimensional coordinates and motion vectors) based on the time-series stereo image. Specifically, the three-dimensional image information in the stereo image is obtained by searching for a point corresponding to a point on a certain image from the image corresponding to the image (corresponding point search). For example, by performing corresponding point search between a pair of stereo images, three-dimensional coordinates at that time can be obtained. In addition, for example, by searching for corresponding points between images of subjects captured with the same camera and having different imaging times, a motion vector at that point is obtained. When the stereo camera 1 is not a stereo camera but a monocular camera, the three-dimensional information display device 100 is configured to include a three-dimensional measuring device such as a laser or millimeter wave, which is a three-dimensional information acquisition unit. Then, the three-dimensional image information calculation unit 7 obtains three-dimensional image information by associating the measurement value obtained by the three-dimensional information acquisition unit with a time-series image generated by imaging a subject with a monocular camera. For example, three-dimensional information obtained by three-dimensional measurement using a laser or millimeter wave emitted in the same direction as the optical axis of the monocular camera may be associated with the subject image captured by the monocular camera.

  For example, as a corresponding point search method, there is a correlation method in which a point (corresponding point) on a reference image corresponding to an arbitrary point of interest on a reference image is searched for. The reference image is an image corresponding to the standard image. Specifically, in a stereo image, one of a pair of images taken at the same time is a standard image, and the other is a reference image. In the time-series images, among images taken by the same camera, the temporally previous image is the reference image, and the temporally subsequent image is the reference image. A template is set for the attention point on the reference image, a window on the reference image corresponding to the template is searched, and a corresponding point is obtained from the searched window.

  A specific corresponding point search performed by the three-dimensional image information calculation unit 7 will be described below. One of the stereo images generated by the stereo camera 1 is a reference image, a point of interest is set in the reference image, and a template including the point of interest is set on the reference image. Here, the template is a range divided by a certain area in the reference image, and has information (image pattern) such as a luminance value of each pixel in the range. Then, a correlation value (similarity) between the template and a plurality of windows set in the reference image (the other image in the stereo image) corresponding to the reference image is calculated. Based on the correlation value, It is determined whether or not the window corresponds. Note that a window is an area in the range of the same size as the template generated in the reference image, and has information (image pattern) such as a luminance value of each pixel in the range. As described above, the correlation value is obtained from the image pattern of the template and the window. For example, the correlation value between the template and one of the windows is obtained, and if these correlation values are low, if it is determined that they do not correspond, for example, it is generated at a position shifted in one direction of one pixel. Correlation values between the determined window and the template are obtained. In this way, the correlation value is obtained while the windows are sequentially changed, and a window in which the correlation value takes a peak value is searched. A window having an image pattern having a peak correlation value with the image pattern of the template is obtained as a window corresponding to the template.

  Such a corresponding point search method is known as described above, and various methods other than the above method have been proposed. For example, various methods for shortening the time for obtaining a window corresponding to a template have been proposed. Some of these methods will be briefly described. For example, as described above, when the standard image is one of the stereo images, the reference image is the other image, and the cameras that generate the images are arranged in parallel, the standard image and the reference image Are arranged almost in parallel. Then, since the corresponding point on the reference image can be assumed to be at the same height position as the target point on the standard image, only the window at this height position needs to obtain the correlation value with the template. Further, when the standard image and the reference image are arranged almost in parallel and the parallax between the standard image and the reference image is known to some extent, the setting range of the window can be further limited. In this way, if the window setting range is limited, the number of windows for which the correlation value with the template is obtained is suppressed, so that the corresponding window can be searched in a short time.

  Another method is called a search method using a multi-resolution strategy. In this method, the resolution of the base image and the reference image is once reduced, and the correlation value calculation is performed in a state where the number of pixels is reduced, and the coordinates at which the correlation value reaches the peak for the attention point are obtained. Thereafter, the resolution is restored to the original, and the corresponding point search is performed by narrowing the window setting range around the coordinates obtained at the low resolution. In the state where the resolution of the standard image and the reference image is low, the information of the image pattern is reduced, and therefore the correlation value can be obtained in a short time. In addition, there should be coordinates where the correlation value at the original resolution has a peak in the vicinity of the coordinates at which the correlation value at the low resolution has a peak. Therefore, by using this method, since the range in which the window corresponding to the template exists is determined in a short time, the corresponding window can also be searched in a short time. In this method, a plurality of low-resolution images divided into several stages may be created, and the search range may be narrowed down gradually.

  Next, a specific method for calculating the correlation value will be described. Specifically, a correlation value is obtained using a function. For example, SAD (Sum of Absolute Difference) method, SSD (Sum of Squared Difference) method (square residual method), NCC (Normalize cross Correlation) method (normalized cross correlation method) and the like are known. ing. For example, the SAD method is a method using a function for obtaining a sum of absolute values of luminance values of a template and a window, and a correlation value for each template and window is obtained using this function. There is also a correlation value calculation method having robustness compared to the SAD method and the like. Specifically, this method is a method of performing similarity calculation using a signal having only a phase component in which an amplitude component is suppressed from a frequency resolved signal of an image pattern. By using this method, it is possible to realize a robust correlation value calculation that is not easily affected by differences in shooting conditions of the left and right cameras in a stereo image, noise, and the like. Note that the method of calculating the frequency-resolved signal of the image pattern is, for example, fast Fourier transform (FFT), discrete Fourier transform (DFT), discrete cosine transform (DCT), discrete sine transform (DST), wavelet transform, Hadamard transform, etc. It has been known. Here, a phase-only correlation method (hereinafter referred to as POC method) among the correlation value computations having such robustness will be briefly described.

  Also in the POC method, a template is set on the standard image and a window having the same size is set on the reference image. Then, a correlation value (POC value) between the template and each window is calculated, and a window corresponding to the template is obtained from the correlation value. First, the template of the standard image and the window of the reference image are each subjected to two-dimensional discrete Fourier transform, normalized, synthesized, and then subjected to two-dimensional inverse discrete Fourier transform. In this way, a POC value that is a correlation value is obtained. Further, since the POC value is obtained discretely for each pixel in the window, the correlation value for each pixel can be obtained. This is different from the above-described SAD method or the like that obtains a correlation value for each window. Thus, in the POC method, since the correlation value can be obtained for each pixel in the window, it is easy to narrow the window setting range, and there is an effect that the processing for obtaining the corresponding points can be performed at high speed. Further, in the correlation value calculation method having robustness such as the POC method, the correlation value can be obtained for each pixel in the window. Therefore, the correlation value is obtained by shifting the window by one pixel as in the SAD method. Even without it, the corresponding window can be searched.

  In the POC method, when calculating the correlation value with the template, the correlation value may be calculated while shifting the window by a plurality of pixels. Specifically, how much can be shifted depends on the searchable range of corresponding points, but is generally said to be about half the window size. In other words, for example, the shifted window and the window before being shifted may be set so as to overlap in about half of the window size. For example, assuming that the maximum parallax between the base image and the reference image is 128 pixels, the window size is 31 × 31, and the range that can be searched by the POC method is ± 8 pixels with respect to the center of gravity of the window, this parallax is searched. In order to do this, the windows need only be shifted by 16 pixels, so eight windows need only be set. In the POC method, the search method based on the above multi-resolution strategy can be used. In the above example, it is sufficient to set eight windows. However, if the resolution is reduced to 1/16, for example, by using a search method based on a multi-resolution strategy, only one window may be set. This makes it possible to search for corresponding points more easily.

  In addition to the POC method, a method is known in which a correlation value calculation is performed using a signal having only a phase component in which an amplitude component is suppressed from a frequency resolution signal of an image pattern. For example, DCT code limited correlation method (“Fusion of image signal processing and image pattern recognition-DCT code limited correlation and its application”, Hitoshi Kiya, Tokyo Metropolitan University, Faculty of System Design, Dynamic Image Processing Realization Workshop 2007, 2007.3. .. 8-9), and the correlation value calculation may be performed using these.

  When a corresponding point corresponding to the attention point is obtained by the above-described corresponding point search method, if necessary, a new corresponding point search is performed using the corresponding point as the attention point. By repeating such processing, a point corresponding to an arbitrary point of interest in a time-series stereo image is obtained from a plurality of images. The three-dimensional image information calculation unit 7 sequentially obtains points corresponding to arbitrary attention points, and calculates three-dimensional image information using them.

  Here, the corresponding point search in the time-series stereo image will be briefly described with reference to the drawings. FIG. 2 is a diagram for explaining the corresponding point search in the time-series stereo image. In FIG. 2, an image L1 and an image R1, which are stereo images taken at time T1, are shown. In order to simplify the description, it is assumed that each camera is arranged in parallel in a stereo camera having a pair of left and right cameras that generate these images. Further, an image L2 and an image R2 taken at time T2, which is a time later than time T1, are shown. In the images L1, R1, L2, and R2, each square indicates one pixel. First, it is assumed that the point 15a in the image L1 at time T1 is input as a point of interest (start point). A point 15b on the image R1, which is a point corresponding to the point 15a, is obtained by a corresponding point search. In addition, when the point 15a is set as the attention point, the point 16a corresponding to the point 15a is obtained by the corresponding point search on the image L2 at the time T2. Then, with this point 16a as a point of interest, a point 16b corresponding to the point 16b in the image R2 at time T2 is obtained by the corresponding point search. Each point 15a, 15b, 16a, 16b is actually a point, but in view of ease of viewing, in FIG. Note that, for example, the corresponding point at the time when the time series image does not exist, such as the time between T1 and T2, is interpolated using the corresponding points at T1 and T2, which are the time before and after the time series image exists. What is necessary is just to obtain | require by etc. In addition, the corresponding point search is not limited to being performed on a subject image captured at a later time, but can also be performed on a subject image captured at a previous time.

  Next, a method for calculating the three-dimensional image information using the corresponding points obtained by the corresponding point search will be described. The coordinates of the point 15a are (p1x, p1y), the coordinates of the point 15b are (q1x, q1y), the coordinates of the point 16a are (p2x, p2y), and the coordinates of the point 16b are (q2x, q2y). Note that the vertical direction in the drawing is the Y direction of each image, and the horizontal direction is the X direction of each image. As described above, since the cameras are arranged in parallel, the Y coordinates of the points 15a and 15b are the same, and the Y coordinates of the points 16a and 16b are also the same.

  First, Δd1 which is a vector indicating the parallax in the images L1 and R1 is obtained from the coordinates of the point 15b obtained from the points 15a and 15a. Specifically, Δd1 is (q1x−p1x, 0). Further, Δf1, which is a vector indicating the motion in the images L1 and L2, is obtained from the coordinates of the point 16a obtained from the points 15a and 15a. Specifically, Δf1 is (p2x−p1x, p2y−p1y). Further, Δd2, which is a vector indicating parallax in the image at time T2, is obtained from the coordinates of the point 16b obtained from the points 16a and 16a. Specifically, Δd2 is (q2x−p2x, 0).

Note that, based on Δd1, the depth distance D1 of the image obtained from the image at time T1 is obtained. Here, the distance D1 is a coordinate in the direction perpendicular to the paper surface in FIG. 2, and this coordinate is a Z coordinate. Further, in the stereo camera that has generated the images L1, R1, L2, and R2, D1 is expressed by Equation 1 where f is the focal length of each camera and B is the baseline length between the cameras. In Equation 1, Δd1 is the magnitude of the vector.
D1 = fB / Δd1 (1)

Similarly, the distance D2 of the depth (Z coordinate direction) of the image obtained from the image at time T2 is expressed by Equation 2 using Δd2. In Equation 2, Δd2 is the magnitude of the vector.
D2 = fB / Δd2 (2)

  From these, the three-dimensional coordinates (X1, Y1, Z1) at the points 15a and 15b at time T1 can be expressed as (p1x · D1 / f, p1y · D1 / f, D1), and the points 16a and 16 at time T2 The three-dimensional coordinates (X2, Y2, Z2) in 16b can be expressed as (p2x · D2 / f, p2y · D2 / f, D2).

  A three-dimensional motion vector is obtained from these three-dimensional coordinates (X1, Y1, Z1) and (X2, Y2, Z2). Specifically, the three-dimensional motion vector is a vector represented by (X2-X1, Y2-Y1, Z2-Z1).

  In this way, the three-dimensional image information calculation unit 7 calculates the three-dimensional image information such as the three-dimensional coordinates and motion vectors of arbitrary points on the image of the subject imaged by the stereo camera 1. The three-dimensional coordinates (X1, Y1, Z1) calculated by the three-dimensional image information calculation unit 7 in this way are based on the imaging position of the stereo camera 1. Since the stereo camera 1 is mounted on the moving body, the reference (stereo camera) position of the three-dimensional coordinates of the image calculated by the three-dimensional image information calculation unit 7 is different.

  When the imaging unit is a monocular camera, a two-dimensional motion vector can be calculated from a time-series image generated by the monocular camera. In the case of a monocular camera, it is only necessary to consider an image captured by one camera among the images obtained by the stereo camera described above. For example, the images L1 and L2, which are time-series images, are acquired, searched for and obtained from the point 16a corresponding to the point 15a, and a two-dimensional motion vector may be obtained from the points 15a and 16a. Here, the two-dimensional motion vector is represented by Δf1. In this case, the three-dimensional information display device 100 is configured to include a three-dimensional information acquisition unit that is a device capable of performing three-dimensional measurement using a laser or millimeter wave, for example. Therefore, the three-dimensional image information calculation unit 7 may associate the stereoscopic information acquired by the stereoscopic information acquisition unit with the time-series image. Note that the three-dimensional image information calculation unit 7 may calculate the three-dimensional image information by a method other than the method described above.

  The moving object specifying unit 8 specifies a moving object corresponding image portion in the image. Here, the moving body refers to an object that is actually moving with respect to the ground, such as a vehicle such as an automobile or a motorcycle, a bicycle, or a pedestrian. The moving object corresponding image portion refers to a portion corresponding to the moving object displayed in the image. In the embodiment of the present invention, since the stereo camera 1 is mounted on a moving body such as a vehicle and performs imaging, even if the stereo camera 1 moves relative to the moving body, it is not necessarily a moving body. . Therefore, a method for specifying the moving object corresponding image portion in the time-series image generated by the stereo camera 1 will be described below. In specifying the moving object corresponding image portion, the moving object specifying unit 8 uses three-dimensional image information such as the three-dimensional coordinates, the two-dimensional motion vector, and the three-dimensional motion vector obtained by the three-dimensional image information calculating unit 7. Note that specifying the moving object-corresponding image portion on the image specifically refers to specifying the location where the moving object is displayed among the objects represented as the image, and acquiring the three-dimensional image information. .

  First, there is a method of specifying a moving object corresponding image portion using a vanishing point of movement. Here, the vanishing point of motion is a point where a straight line obtained by extending the motion vector at each point on the image along the direction intersects. This vanishing point is determined according to the moving direction of the object on the image. That is, when the camera is moving in the same direction, if it is the same object, it has moved in the same direction, so there is a vanishing point for that object. In addition, when the object on the image is a stationary object, the same vanishing point exists for all the objects that are stationary objects ("Examination of moving object recognition method using principal component analysis", Information Processing Society of Japan Report-Computer Vision and Image Media Vol. 1996, No. 31, 1995-CVIM-099, literature number: IPSJ-CVIM 9509008). It should be noted that most of the images of the subject imaged by the stereo camera 1 are considered to be occupied by a stationary object corresponding image portion corresponding to a stationary object such as a traffic light, a road surface, a pedestrian crossing, and a wall. Here, the stationary object-corresponding image portion refers to a portion displayed in the image and corresponding to the stationary object. Then, assuming that, the vanishing point for the most motion vectors is estimated to be the vanishing point of the stationary object corresponding to the stationary object corresponding image portion. Therefore, it can be estimated that each vanishing point existing after removing vanishing points for the most motion vectors among vanishing points existing in the image is a vanishing point of the moving object corresponding to the moving object corresponding image portion.

  Therefore, the moving object specifying unit 8 extends the motion vector calculated in the time-series image calculated by the three-dimensional image information calculating unit 7 along the direction, and determines a vanishing point on the image where they intersect. . Of these vanishing points, each vanishing point other than the vanishing points for the most motion vectors is estimated to be a vanishing point corresponding to the moving object corresponding image portion. Further, based on the vanishing point of the estimated moving object corresponding image portion in this way, the moving object corresponding image portion on the image is specified, and its three-dimensional image information is acquired. In this way, the moving object corresponding image portion in each time-series image can be specified. Since the motion vector is calculated by the three-dimensional image information calculation unit 7, it is not necessary to calculate a new motion vector to obtain the vanishing point, and the vanishing point can be easily calculated.

  Next, a method for specifying the moving object corresponding image portion by pattern recognition or template matching will be described. For example, for a moving object that is expected to exist as a subject, such as a vehicle such as an automobile or a motorcycle, a bicycle, or a pedestrian, the moving object corresponding image portion in the image may be specified using pattern recognition or template matching. Good. In pattern recognition, data for pattern recognition related to the moving object may be stored in the storage unit 12 in advance. And the moving body specific | specification part 8 specifies the moving body corresponding | compatible image part on an image by performing pattern recognition in an image using the data previously memorize | stored in the memory | storage part 12, and acquires the three-dimensional image information. Furthermore, in pattern recognition, for example, a moving object corresponding image portion can be identified more efficiently by learning pattern data using a method such as SVM (Support Vector Machine) or AdaBoost. . In template matching, a template relating to the moving object may be stored in the storage unit 12 in advance. Then, similarly to the corresponding point search described above, the moving object specifying unit 8 searches the image for a portion having a high correlation value with the template, thereby specifying the moving object corresponding image portion on the image, and obtaining the three-dimensional image information. get.

  Similarly to the above pattern recognition and template matching, as a method of specifying a moving object-corresponding image portion using moving object candidates, there is a method of specifying a vehicle on an image based on edge distribution and left-right symmetry in the image (for example, JP, 7-334800, A). By this method, the moving object specifying unit 8 may specify a moving object corresponding image portion such as a vehicle on the image and acquire the three-dimensional image information.

  Further, the three-dimensional motion vector obtained from the stereo time-series image is corrected by the moving speed of the stereo camera 1 that generated the stereo time-series image, so that the still-body-corresponding image portion and the moving-object-corresponding image on the image are corrected. There is also a method for discriminating a portion (see, for example, JP-A-2006-134035). When this method is used, the moving object specifying unit 8 receives speed information of the moving object on which the stereo camera 1 is mounted, and uses the three-dimensional motion vector calculated by the three-dimensional image information calculating unit 7 to move the moving object on the image. The corresponding image portion can be specified and the three-dimensional image information can be acquired.

  Alternatively, the moving object corresponding image portion may be specified by the operator selecting a moving object corresponding image portion from the image while viewing the image generated by the stereo camera 1. FIG. 3 is a diagram for explaining a case where the operator specifies a moving object-corresponding image portion, and FIG. 3A is a diagram showing a state in which the moving object-corresponding image portion is specified in the image at time T. 3 (B) is a diagram illustrating a state in which the moving object corresponding image portion in the image at time T + Δt is specified. For example, when the operator instructs the display device 5 and the image recording unit 6 using the input unit 4, the display device 5 displays an image generated by the stereo camera 1 and stored in the image recording unit 6. . The operator may select a part of the image displayed on the display device 5 by operating the mouse or the like that is the input unit 4. And the selected location is specified as a moving body corresponding | compatible image part. Specifically, an image for which 3D image information is calculated by the 3D image information calculation unit 7 is read from the image recording unit 6 and displayed on the display device 5. In addition to the displayed image, the display device 5 displays, for example, a cursor or the like that can operate the position on the screen of the display device 5 with a mouse. By selecting a portion on the image with the cursor, the selected portion is displayed. The three-dimensional image information is input to the moving object specifying unit 8 and the moving object corresponding image portion is specified.

  For example, as shown in FIGS. 3 (A) and 3 (B), from the image displayed on the display device 5 by the operator, the moving object corresponding image portions 21 and 22 including a car and the moving object corresponding image including a pedestrian. By selecting the portion 23 by the input unit 4, the moving object specifying unit 8 specifies the moving object corresponding image part on these images, and acquires the three-dimensional image information thereof. Also, in other time-series images, the operator may specify the moving object corresponding image portion. For example, FIG. 3B shows an image after Δt from FIG. 3A, and the operator may select the moving object corresponding image portion as described above.

  The moving object specifying unit 8 may specify the moving object-corresponding image portion for every time-series image by any of the above methods, or any one of the above for any one image, that is, one frame. The moving object corresponding image portion may be specified by the method described above, and the remaining time series images may be specified by tracking the moving object corresponding image portion using a corresponding point search or the like. Further, as a method of inputting a moving object corresponding image portion in an arbitrary image and tracking the moving object corresponding image portion in the time-series image of the image, not only a method by corresponding point search but also, for example, Lucas-Kanade There is a method using a calculation such as a method for calculating a motion vector. The moving object specifying unit 8 may specify a moving object corresponding image portion on the image for another time-series image using, for example, the Lucas-Kanade method and acquire the three-dimensional image information. The Lucas-Kanade method is a method for obtaining a motion vector between images, but by obtaining a motion vector, correspondence between images is possible, and tracking of a moving object corresponding image portion is also possible.

  Further, the moving object specifying unit 8 may specify the moving object corresponding image portion by one of the methods described above, or may selectively use any one of the methods to specify the moving object corresponding image portion. Good. For example, the moving object corresponding image portion is first identified by pattern recognition or template matching. If the moving object corresponding image portion cannot be identified by these methods, the operator identifies the moving object corresponding image portion using the input unit 4. It is good to do.

  The stationary object specifying unit 9 specifies a stationary object corresponding image part in each image based on the three-dimensional coordinates, the two-dimensional motion vector, the three-dimensional motion vector, and the like calculated by the three-dimensional image information calculating unit 7. Here, the stationary body is, for example, a traffic light, a road surface, a pedestrian crossing, a wall, etc., which is fixed to the ground, but the stereo camera 1 is mounted on a vehicle or the like that is a moving body. For this reason, the stereo camera 1 itself is also moving. Thereby, the stationary object corresponding image portion moves on the time-series image. As described above, there are the following methods for identifying a stationary object-corresponding image portion of a stationary object that is not fixed on the image but is not actually moved from the image. The stationary object specifying unit 9 uses these methods to specify the stationary object corresponding image portion from the image. Further, on the image, a part other than the moving object corresponding image part specified by the moving object specifying unit 8 may be specified as a stationary object. Note that specifying a still object-corresponding image portion on an image specifically specifies a portion where a stationary object is displayed among objects represented as an image, and acquires three-dimensional image information thereof. Say.

  First, for example, a method for specifying a stationary object corresponding image portion using a vanishing point of motion will be described. The stationary object specifying unit 9 obtains vanishing points in the time-series image calculated by the three-dimensional image information calculating unit 7, and among these vanishing points, the vanishing points for the most motion vectors correspond to the stationary object-corresponding image portion. Estimated to be a vanishing point in a stationary object. Further, based on the estimated vanishing point of the stationary object in this way, the stationary object corresponding image portion on the image is specified, and the three-dimensional image information is acquired. In this way, the stationary object corresponding image portion in each time-series image can be specified. Since the motion vector is calculated by the three-dimensional image information calculation unit 7, it is not necessary to calculate a new motion vector to obtain the vanishing point, and the vanishing point can be easily calculated.

  The stationary object specifying unit 9 detects a stationary object that is expected to exist, that is, a landmark, such as a traffic light, a sign, a signboard, or the like by pattern recognition or template matching, so that the stationary object corresponding image portion is detected. You may specify. Note that the pattern data and the template used at this time may be stored in the storage unit 12. In this way, the stationary body specifying unit 9 specifies the stationary body corresponding image portion on the image and acquires the three-dimensional image information. Similar to the moving object specifying unit 8, the stationary object corresponding image portion can be specified more efficiently by learning the pattern data.

  Alternatively, the still body corresponding image portion may be specified by the operator selecting a still body corresponding image portion from the image while viewing the image generated by the stereo camera 1. FIG. 4 is a diagram for describing a case where the operator specifies a stationary object-corresponding image, and FIG. FIG. 4B is a diagram showing a state in which a stationary object corresponding image in the image at time T + Δt is specified. For example, when the operator instructs the display device 5 and the image recording unit 6 using the input unit 4, the display device 5 displays an image generated by the stereo camera 1 and stored in the image recording unit 6. . Then, the operator may select a part of the image displayed on the display device 5 by operating the mouse or the like that is the input unit 4. And the selected location is specified as a stationary object corresponding | compatible image part. Specifically, an image for which 3D image information is calculated by the 3D image information calculation unit 7 is read from the image recording unit 6 and displayed on the display device 5. In addition to the displayed image, the display device 5 displays, for example, a cursor or the like that can operate the position on the screen of the display device 5 with a mouse. By selecting a part on the image with the cursor, the selected portion The three-dimensional image information is input to the stationary body specifying unit 9, and the stationary body corresponding image portion is specified.

  For example, as shown in FIGS. 4 (A) and 4 (B), from the image displayed on the display device 5 by the operator, the stationary object corresponding image portion 24 including the vicinity of the boundary between the road and the sidewalk and the wall surface, Select a stationary object corresponding image portion 25 including a traffic light and a road surface such as a pedestrian crossing, a stationary object corresponding image portion 26 including a sidewalk, a road surface and a wall surface, and a stationary object corresponding image portion 27 including a lane formed on the road surface and the road surface. Thus, the stationary object specifying unit 9 specifies the still object corresponding image portion on these images and acquires the three-dimensional image information.

  Note that the stationary object specifying unit 9 may specify the stationary object-corresponding image portion in all the time-series images by any of the above methods, or any one of the above-mentioned images for one image, that is, one frame. The stationary object-corresponding image portion may be specified by such a method, and the remaining time-series images may be identified by tracking the stationary object-corresponding image portion using a corresponding point search or the like. Further, as a method of inputting a still object corresponding image portion in an arbitrary image and tracking the still object corresponding image portion in the time-series image of the image, not only the method by the corresponding point search but also, for example, Lucas There is a method using an operation for calculating a motion vector such as the Kanade method. The stationary object specifying unit 9 may specify a stationary object-corresponding image portion on the other time series image using, for example, the Lucas-Kanade method and acquire the three-dimensional image information.

  For example, FIG. 4A shows the time T and FIG. 4B shows the image at time T + Δt, and FIG. 4B shows the image after Δt from FIG. 4A. Since the moving object on which the stereo camera 1 is mounted is moving, the positions of the stationary object corresponding image portions 24, 25, 26, and 27 are different in FIGS. 4 (A) and 4 (B). In addition, the stationary object corresponding image portion 27 cannot be searched by the corresponding point search because the vehicle which is a moving object is interrupted in FIG. 4B. Therefore, in such a case, the stationary object corresponding image portion 24 may be excluded from the candidates for the stationary object corresponding image portion.

  Note that the stationary object specifying unit 9 does not have to specify all the still object-corresponding image portions on the image. Further, the stationary object corresponding image portion does not need to have an area, and may be a point (pixel). Although several methods have been described above as the method for specifying the stationary object-corresponding image part, the stationary object specifying unit 9 may specify the stationary object-corresponding image part by one of these methods. This method may be selectively used to specify the stationary object corresponding image portion.

  In addition, when the 3D information display device 100 is used as a driving recorder for investigating the cause of a traffic accident such as a rear-end collision between automobiles, not only the time-series position change of the moving object but also the related matters. Therefore, the display of traffic lights is also important. Therefore, it is preferable to extract the information indicating which of the red, blue, and yellow lamps is lit in the traffic light in association with the time. For this reason, it is preferable that the stationary object specifying unit (signal specifying unit) 9 specifies the stationary object corresponding image portion of the signal.

  The calculation unit 10 uses the three-dimensional moving object-corresponding image portion specified in the images of the subject captured at different times with respect to the moving object-corresponding image portion specified by the moving object specifying unit 8 with reference to a time-series image in any arbitrary frame. Calculate the coordinates. As described above, since the stereo camera 1 captures images while moving, the reference of the three-dimensional coordinates of the moving object corresponding image part and the stationary object corresponding image part specified by the moving object specifying unit 8 and the stationary object specifying unit 9 is mutually Is different. Therefore, the calculation unit 10 acquires the three-dimensional coordinates of the moving object corresponding image portion in another time-series image based on an arbitrary image (integrated image). That is, the calculation unit 10 calculates three-dimensional coordinates with reference to the integrated image for moving object corresponding image portions in a plurality of temporally different images. Thereby, the temporal movement of the moving body moving relative to the stationary body can be specifically shown. In addition, it is possible to display a moving object whose position changes with respect to a stationary object in the reference image using the three-dimensional coordinates. Thereby, there is also an effect that it is possible to easily grasp the motion of the moving object easily and visually. Note that the three-dimensional coordinates in each image or the like based on the integrated image are hereinafter referred to as normalized three-dimensional coordinates.

  Therefore, a method for calculating the normalized three-dimensional coordinates of the moving object corresponding image portion in each image based on the integrated image will be described below. First, the calculation unit 10 selects an integrated image that is an image serving as an arbitrary reference, and selects arbitrary three points included in the stationary object corresponding image portion in the integrated image. For example, the calculation unit 10 selects an image of a subject captured at time T as an integrated image. In addition, since the stationary object corresponding | compatible image part is specified by the stationary body specific | specification part 9, the calculating part 10 should just select three points | pieces among them. In addition, since the three-dimensional coordinate for each image of each point is calculated, the calculation unit 10 can easily select three points that are not on the same straight line. Similarly, the calculation unit 10 acquires three points corresponding to the three points selected in the integrated image on an image of a different frame from the integrated image for calculating the standardized three-dimensional coordinates. For example, it is assumed that the calculation unit 10 converts the coordinates of the image at time T + Δt into calculation of standardized three-dimensional coordinates. For the corresponding three points, the data calculated by the three-dimensional image information calculation unit 7 may be used, or the calculation unit 10 may determine the corresponding points or the Lucas-Kaneda method.

  As described above, the calculation unit 10 selects three points that are not on the same straight line from the stationary object corresponding image portion of the image at time T, and obtains corresponding points on the image at time T + Δt. Then, the calculation unit 10 performs coordinate conversion of the three-dimensional coordinates of the three points at time T + Δt, which is necessary to match the surface constituted by the three points at time T + Δt with the surface constituted by the three points at time T. Calculate the necessary rotation and translation components. That is, the arithmetic unit 10 matches the normal vector of the surface composed of three points at time T with the normal vector of the surface composed of three points at time T + Δt, and any one of the three points at time T is obtained. A rotation component and a translation component are calculated to perform coordinate transformation such that any one of the three points at time T + Δt is matched with the point, or the three points centroid at time T + Δt are matched with the centroid at time T + Δt. The calculation unit 10 converts the three-dimensional coordinates of the specified moving object corresponding image portion in the image at time T + Δt by the calculated rotation component and translation component, thereby obtaining the standardized three-dimensional coordinates based on the image at time T. Can be calculated.

  Here, it is preferable that the three points selected in the integrated image are separated from each other in the three-dimensional coordinates. As a result, the stationary object-corresponding image portions match each other over a wide range in the stationary object-corresponding image portion, and the matching is more surely performed. And the calculating part 10 should just calculate the said rotation component and a translation component by the least square by these multiple sets. Thereby, the calculating part 10 can obtain | require the more stable solution (rotation component and translation component), and the conversion precision of a three-dimensional coordinate becomes high.

  In addition, another method will be described as a method of converting the three-dimensional coordinates of the specified moving object corresponding image portion on the basis of the integrated image. Specifically, a method using an ICP (Iterative Closest Points) algorithm will be described. Specifically, the calculation unit 10 sets, as initial values, three-dimensional coordinates at arbitrary points of the stationary object corresponding image portion in the integrated image specified by the stationary object specifying unit 9, and corresponds to the plurality of points. The point on the time series image of is acquired. As for the corresponding points, the data calculated by the three-dimensional image information calculation unit 7 may be used, or the calculation unit 10 may obtain the corresponding points by the search for corresponding points or the Lucas-Kaneda method. Then, the arithmetic unit 10 uses an ICP (Iterative Closest Points) algorithm to capture a plurality of points in the stationary object corresponding image portion of the integrated image that is imaged at the time T and is generated and the time corresponding to these points. It is possible to calculate a rotation component and a translation component necessary for coordinate conversion such that a plurality of points in a stationary object corresponding image portion of an image of T + Δt coincide with each other in three-dimensional coordinates. Then, the calculation unit 10 converts the three-dimensional coordinates of the specified moving object corresponding image portion in the image at the time T + Δt by the calculated rotation component and translation component, so that the image at the time T + Δt with the image at the time T as a reference. It is possible to calculate the normalized three-dimensional coordinates of the specified moving object corresponding image portion in FIG. In this way, by using the ICP algorithm, the arithmetic unit 10 can perform robust coordinate transformation that is less susceptible to noise with respect to a plurality of corresponding points.

  Note that the conversion of the three-dimensional coordinates of the moving object corresponding image portion specified in the image at the time T + Δt has been described with reference to the integrated image at the time T. However, the calculation unit 10 does not change the moving object corresponding image portion specified in another time-series image. Similarly, the conversion of the three-dimensional coordinates may be performed by calculating the rotation component and the translation component. Note that if the moving body on which the stereo camera 1 is mounted is going straight, the stationary body corresponding image portion corresponding to the stationary body at a distant place in the front exists in a plurality of time-series images, but the moving body turns left. Alternatively, when the vehicle is bent, such as turning right, the stationary object corresponding image portion existing in the subsequent time series image changes. Therefore, depending on each time-series image, there may be no corresponding point of the first selected point in the integrated image. Even in such a case, the arithmetic unit 10 changes the selected point to a new point. (Renewal). Then, it is possible to calculate standardized three-dimensional coordinates by performing coordinate transformation a plurality of times. As described above, the calculation unit 10 can calculate the standardized three-dimensional coordinates by using the three-dimensional image information of the stationary body without being limited by the movement of the moving body.

  Furthermore, the calculation unit 10 can calculate the coordinate position of the stereo camera 1 with respect to the reference integrated image. Specifically, the calculation unit 10 calculates the straight component and the rotation component of the stereo camera 1 (moving body) based on the motion vector of the road surface calculated by the three-dimensional image information calculation unit 7 and the three-dimensional image information such as parallax. Calculate (see Japanese Patent Application No. 2008-21950). Thereby, the three-dimensional coordinates of the stereo camera 1 in each image can be calculated. Furthermore, the three-dimensional coordinates of the stereo camera 1 for the integrated image may be calculated in the same manner as the calculation of the standardized three-dimensional coordinates of the specified moving object corresponding image portion for the integrated image.

As described above, the calculation unit 10 can calculate standardized three-dimensional coordinates for the identified moving object corresponding image portion. Since the calculated standardized three-dimensional coordinates in the specified moving object corresponding image portions have the same reference, it is possible to calculate motion information representing the movement of the moving object corresponding to the moving object corresponding image portion using these. The motion information is, for example, moving body speed, acceleration, speed vector, acceleration vector, and the like. The motion vector calculated by the three-dimensional image information calculation unit 7 is also one piece of motion information. Here, a calculation method for calculating the speed and acceleration of the moving object corresponding to the specified moving object corresponding image portion using the standardized three-dimensional coordinates will be described. Here, the standardized three-dimensional coordinates of three consecutive frames in the moving object corresponding image portion corresponding to the same moving object, calculated using the time-series images at intervals of t seconds, are (x1, y1, z1), respectively. Let (x2, y2, z2) and (x3, y3, z3). The velocity v1 of the moving body when these are imaged from (x1, y1, z1) and (x2, y2, z2) can be expressed by the following equation 3.
v1 = {(Vx1) ² + (Vy1) ² + (Vz1) ² } ⁽⁻²⁾ (3)
(Vx1, Vy1, Vz1)
= ((X2-x1) / t, (y2-y1) / t, (z2-z1) / t)

Moreover, the velocity v2 of the moving body when these are imaged from (x2, y2, z2) and (x3, y3, z3) can be expressed by the following equation 4.
^{v2 = {(Vx2) 2 +} (Vy2) 2 + (Vz2) 2} (-2) ··· (4)
(Vx2, Vy2, Vz2)
= ((X3-x2) / t, (y3-y2) / t, (z3-z2) / t)

Further, the acceleration a of the moving object obtained from the corresponding points of these three images can be expressed by the following formula 5.
^{a = {(Ax2) 2 +} (Ay2) 2 + (Az2) 2} (-2) ··· (5)
(Ax, Ay, Az)
= ((Vx2-Vx1) / t, (Vy2-Vy1) / t, (Vz2-Vz1) / t)

  The calculation unit 10 can calculate the speed and acceleration of the moving object corresponding to the specified moving object corresponding image portion by calculating using the above-described Expression 3 to Expression 5. In addition, since the calculating part 10 can also easily know the direction of speed and acceleration from the standardized three-dimensional coordinates, it can also calculate a speed vector and an acceleration vector.

  In addition, when there is a traffic light in the integrated image, the arithmetic unit 10 preferably calculates the color of the lit lamp in the traffic light for each frame. The stationary body specifying unit 9 specifies the three-dimensional image information of the traffic light. Then, the calculation unit 10 acquires brightness / color information, which is information that can determine the color of the lamp of the traffic light for each frame, from the time-series images stored in the image recording unit 6. It is assumed that the acquired information has the maximum luminance value. Thereby, the color of the lamp of the traffic light according to the imaging time is obtained. For example, the data may be stored in the storage unit 12.

  The image generation unit 11 generates an image signal for generating an image using various data calculated by the calculation unit 10 or the like. For example, the image generation unit 11 can generate an image signal that integrates a moving object corresponding image portion on each image into an integrated image by using the standardized three-dimensional coordinates. An image signal that causes the display device 5 to display an image that integrates a plurality of moving object corresponding image portions existing in other frames is generated with respect to the reference integrated image. Hereinafter, such display is referred to as integrated display. The three-dimensional coordinates of the integrated image are calculated by the three-dimensional image information calculation unit 7, and the standardized three-dimensional coordinates of the specified moving object corresponding image portion in a frame other than the integrated image with reference to the integrated image are calculated by the calculation unit 10. Since it is calculated, the image generation part 11 should just generate | occur | produce an image signal based on these. Thereby, the image generation part 11 can display the specified moving body corresponding | compatible image part to which the position changes temporally with respect to a stationary body corresponding | compatible image part in the integrated image on the display apparatus 5. FIG.

  In addition, when the moving object corresponding image part is integrated and displayed on the integrated image, there is no change in the stationary object corresponding image part and the unspecified moving object corresponding image part, but there may be a plurality of specified moving object corresponding image parts. Since the existence position of the moving object corresponding image portion varies depending on the time, the moving object corresponding image portion corresponding to the number of used time-series images (the number of frames) may exist on the image. Thereby, the motion of the moving object corresponding to the specified moving object corresponding image portion can be represented by one image. Note that in the integrated image after integration, there is a moving object-corresponding image part in a place where the moving object-corresponding image part does not exist, and a part of the original image is deleted in that part. become.

  The temporal display of the moving body moving with respect to the stationary body can be specifically shown by the integrated display image. Thereby, there is an effect that it is possible for the operator to grasp the movement of the moving object correctly and easily by viewing the image.

  Here, an image in the case where the other moving object corresponding image part is integrated into one integrated image using two time-series images shown in FIG. 5 will be described. FIG. 5 is a diagram illustrating an example of a time-series image used for image integration according to an embodiment of the present invention. In FIG. 5, the upper row is an image at time T, and the lower row is an image at time T + Δt, which is Δt after time T. In the image at time T, a traffic light 34a that is a still body corresponding image portion, a pedestrian 31a, a vehicle 32a, and a vehicle 33a that are moving body corresponding image portions are present on the image. In the image at time T + Δt, a traffic light 34b that is a stationary object corresponding image portion, a pedestrian 31b that is a moving object corresponding image portion, and a vehicle 32b are present on the image. From these images, the traffic light 34a and the traffic light 34b are images of the same traffic light, and the traffic light 34b is displayed larger than the traffic light 34a because the stereo camera 1 is closer to the traffic light 34b. Moreover, the pedestrian 31a and the pedestrian 31b are the images of the same pedestrian, and the pedestrian 31b has advanced to the roadway side (right direction in FIG. 5). Further, the vehicle 32a and the vehicle 32b are images of the same vehicle, and the vehicle 32b is closer to the traffic light 34b. The vehicle 32b is displayed larger than the vehicle 32a because the stereo camera 1 is approaching the vehicle. Further, the same vehicle as the vehicle 33a is not displayed in the image at time T + Δt. Since the vehicle 33a is moving away from the traffic signal 34a, the vehicle 33a is moving outside the image range at time T + Δt.

  For example, using the image shown in FIG. 5, the three-dimensional information display device 100 calculates the calculation of the standardized three-dimensional coordinates of the moving object corresponding image portion in each time-series image with reference to the integrated image. For example, when the image shown in FIG. 5 is used and the upper image in FIG. 5 is an integrated image, the image generation unit 11 of the three-dimensional information display device 100 displays the image shown in FIG. An image signal is generated to FIG. 6 is a diagram showing an integrated image in one embodiment of the present invention. FIG. 6 (A) is a diagram showing an integrated image based on an arbitrary image, and FIG. 6 (B) is an integrated image. It is a figure which shows the image converted into overhead view display. As shown in FIG. 6A, in addition to the traffic light 34a, the pedestrian 31a, and the vehicles 32a and 33a displayed in the image shown in the upper part of FIG. 5, the moving object corresponding image portion captured at time T + Δt. A certain pedestrian 31b and vehicles 32b and 33b are integrated. Although the vehicle 33b is not displayed in the image shown in the lower part of FIG. 5, for example, the calculation unit 10 estimates the speed of the vehicle 33a from the image before the time T, and the three-dimensional coordinates at the time T + Δt. It is possible for the image generation unit 11 to display the vehicle 33b by estimating and calculating.

  In addition, since the integrated image has three-dimensional coordinates, the image generation unit 11 can display the image on the display device 5 as an image viewed from different directions. Specifically, in the display device 5, as shown in FIG. For example, by using the bird's-eye view display in this way, a difference in speed between the vehicles 32a and 32b and the vehicles 33a and 33b can be well understood. The distance between the vehicle 33a and the vehicle 33b is longer than the distance between the vehicle 32a and the vehicle 32b. Therefore, it can be seen that the speed of the vehicles 33a and 33b is higher than that of the vehicles 32a and 32b. In FIG. 6B, points 34a and 34b indicating the position of the stereo camera 1 are also displayed. Since the calculation unit 10 also calculates the standardized three-dimensional coordinates of the stereo camera 1, the image generation unit 11 may generate an image signal indicating the position of the stereo camera 1 by using it. By doing in this way, the position of the mobile body carrying the stereo camera 1 can be grasped. As a result, the cause of an accident or the like may be easily investigated.

  Thus, the integrated image displaying the moving object corresponding image portions corresponding to the moving objects at different times makes it easy to understand the movement of the moving object and the operation status of the vehicle or the like at a glance. Therefore, by using this integrated image, it is possible to easily grasp the situation such as an accident.

  In addition, the image generation unit 11 displays an image so that the motion vector is superimposed on the image displayed on the display device 5 together with the moving object corresponding image portion corresponding to the moving object moving in time as described above, for example. A signal may be generated. The motion information is, for example, acceleration, speed, acceleration vector, speed vector, and the like, and these are calculated by the calculation unit 10. In addition, a motion vector that is motion information is calculated by the three-dimensional image information calculation unit 7. Therefore, the image generation part 11 should just generate | occur | produce an image signal so that these may be displayed on the display apparatus 5. FIG. FIG. 7 is a diagram showing an image on which motion information is superimposed and displayed in one embodiment of the present invention, and FIG. 7A is a diagram showing an image on which motion vectors are superimposed, and FIG. FIG. 7 is a diagram showing an image in which velocity vectors are superimposed and FIG. 7C is a diagram showing an image in which velocity is superimposed. The image generation unit 11 may display an image on the display device 5 as shown in, for example, FIGS. 7A to 7C. For example, as shown in FIG. 7A, a moving object corresponding image portion 41 corresponding to the same moving object in different frames is integrally displayed on the image 40, and further, the moving object corresponding image portion 41 corresponding to the moving object of each frame is displayed. The motion vector may be superimposed and displayed as an arrow 42. Further, as shown in FIG. 7B, a moving object corresponding image portion 41 corresponding to the same moving object in different frames is integrated and displayed in an image 40, and the moving object velocity vector for the moving object corresponding image portion 41 of each frame is further displayed. May be superimposed and displayed as an arrow 43. Further, as shown in FIG. 7C, a moving object corresponding image portion 41 corresponding to the same moving object in different frames is integrated and displayed on the image 40, and the moving object speed with respect to the moving object corresponding image portion 41 of each frame is set. The pop-up 44 representing may be displayed in a superimposed manner. The speed unit shown in the pop-up 44 may be km / h, for example. Thus, by displaying the motion vector superimposed on the image, the operator can easily grasp the motion of the moving body 41.

  Further, the image generation unit 11 displays, for example, the moving object corresponding image portions of the same moving object of a plurality of frames in a time series instead of simultaneously displaying on the image in the image displayed on the display device 5. Also good. FIG. 8 is a diagram showing an image in which moving object corresponding image portions are displayed in time series in one embodiment of the present invention, and FIG. 8 (A) is a diagram showing a first display image. FIG. 8B is a diagram showing the second display image, and FIG. 8C is a diagram showing the third display image. First, as illustrated in FIG. 8A, the display device 5 displays, for example, an image 40 in which an arrow 42 indicating a motion vector is superimposed and displayed on the moving object corresponding image portion 41a of the reference integrated image. Next, as shown in FIG. 8B, the display device 5 integrates the moving object corresponding image portion 41b of the next frame in time into the integrated image, and further superimposes and displays an arrow 42 indicating a motion vector. The moving object corresponding image portion 41a of the previous frame displays, for example, the blinked image 40. Next, as shown in FIG. 8C, the display device 5 integrates the moving object corresponding image portion 41c of the frame two frames later in time into the integrated image, and further superimposes and displays an arrow 42 indicating the motion vector. The moving object corresponding image portion 41b and the moving object corresponding image portion 41a of the previous frame and the previous frame display, for example, the blinking image 40. For example, the image generation unit 11 may switch the images shown in FIGS. 8A to 8C sequentially at predetermined time intervals and display them on the display device 5. Thus, the moving object corresponding image portions 41a to 41c are displayed so as to move with the passage of time. Therefore, it is possible to visually recognize the movement of the moving object corresponding image portions 41a to 41c, and the operator can easily grasp the movement of the moving object corresponding image portions 41a to 41c.

  Further, the image generation unit 11 may selectively display motion information in an image displayed on the display device 5. FIG. 9 is a diagram showing an image that selectively displays motion information of a moving object in an embodiment of the present invention. For example, as shown in FIG. 7A, the display device 5 integrally displays the moving object corresponding image portion 41 corresponding to the same moving object in different frames on the image 40, and further, the moving object corresponding image portion 41 of each frame. The motion vector of the moving object with respect to is superimposed and displayed as an arrow 42. In this state, as shown in FIG. 9, the operator operates the cursor 46 on the image by using the input unit 4 such as a mouse and overlays it on an arbitrary arrow 42 or clicks the mouse in the overlaid state. For example, a pop-up 44 representing the speed of the moving body 41 corresponding to the arrow 42 may be displayed. Since the movement information of the moving body 41 can be selectively displayed, the operator can easily grasp the movement of the moving body 41.

  In addition, in the image displayed on the display device 5, the image generation unit 11, when displaying a plurality of different moving objects in an integrated manner, makes it difficult to know which frame the displayed moving objects are from, so that they can be understood. Is preferably displayed. That is, in the displayed motion information, the motion information at the same time may be displayed in association with each other. For example, color display may be used, and moving objects in the same frame may be displayed in the same color. Further, motion information may be displayed, and the motion information may be displayed in the same color for the same frame (first display method). In addition, for example, in integrated display, motion information such as a motion vector may be superimposed and displayed together with a moving object, and a tag or the like may be displayed along with the motion vector (second display method). In this case, the tag may be pop-up displayed by selecting the motion vector by placing the cursor on the vector. FIG. 10 is a diagram showing an image displayed for a moving object corresponding image portion of a plurality of different moving objects in one embodiment of the present invention, and FIG. 10A is a diagram showing an image by the first display method. FIG. 10B is a diagram showing an image by the second display method. As shown in FIG. 10A, in the first display method, a plurality of different moving bodies 48 and 49 are integrated and displayed on the image 40, respectively. For example, arrows 50-1 and 51-1, which indicate motion vectors, arrows 50-2 and 51-2, and arrows 50-3 and 51-3 indicate motion vectors in the same frame, and these are the same color. It should be displayed. Thereby, the operator can visually recognize the movement information in the same frame, and it is easy to grasp the movements of a plurality of moving objects. Further, as shown in FIG. 10B, in the second display method, a plurality of different moving objects 48 and 49 are integrally displayed on the image 40, respectively. For example, an arrow 51 indicating a motion vector is also superimposed and a tag 52 attached to the arrow 51 is also superimposed and displayed. Each tag 52 represents a number or the like and points to a corresponding arrow 51. That is, the same number tag 52 may be attached to the arrow 51 indicating the motion vector of the moving object in the same frame. Thereby, the movement information in the same frame can be recognized visually, and the effects of easily grasping the movements of a plurality of moving objects are achieved. As described above, the tag 52 may be popped up by selecting a motion vector by overlaying or clicking the cursor 51 on the arrow 51 using the mouse or the like as the input unit 4. . In addition, instead of associating motion information at the same time, moving objects at the same time may be associated with each other, such as displaying moving objects in the same frame in the same color.

  In addition, since the image displayed by the display device 5 is based on an integrated image that is an arbitrary image, the lit lamp is not changed in the traffic light. However, when the three-dimensional information display device 100 is used as a driving recorder for investigating the cause of a traffic accident such as a rear-end collision between automobiles, any of red, blue, and yellow is used depending on the position of the moving object. Information about whether the lamp is lit is important. Therefore, using the information of the lamps that are lit in the traffic light for each frame calculated by the arithmetic unit 10, the image generation unit 11 is, for example, a lamp that is lit in the traffic lights together with the motion information that is superimposed and displayed. It is preferable to generate an image signal so as to display the color. As a result, not only the movement of the vehicle before and after the accident but also the state of the traffic light can be grasped from the image, so that it is useful for investigating the cause of the traffic accident.

  Next, an example of displaying an image on the display device 5 when the integrated image is an overhead view will be described. FIG. 11 is a diagram showing a display image example of each moving object corresponding image portion when the integrated display is an overhead view in one embodiment of the present invention, and FIG. 11A shows the moving object corresponding image portion depending on the frame rate interval. 11 (B) is a diagram showing an image in which moving object corresponding image portions are integrated and displayed at a desired time interval, and FIG. 11 (C) is an equidistant interval. FIG. 11D is a diagram showing an image in which each moving object corresponding image portion is displayed in an integrated manner, and FIG. 11D integrally displays the moving object corresponding image portion corresponding to the reference moving object at equal distance intervals, and corresponds to the remaining moving objects. As for the image portion, it is a diagram showing an image that is integrally displayed for a frame corresponding to a reference moving object.

  Since the image generation unit 11 generates an image signal for generating an integrated image using the normalized three-dimensional coordinates of each moving object for each frame calculated by the calculation unit 10, an image viewed from all angles can be obtained. It is possible to create an image signal to be displayed. FIG. 11A to FIG. 11D are diagrams showing images in a so-called overhead view viewed from above. By providing a bird's-eye view display, there is an effect that it is easy to grasp the movement of each moving object. In an image 55 shown in FIG. 11A, moving bodies 56 and 57 in all frames are displayed in an integrated manner. In addition, the image 55 shown in FIG. 11B integrally displays the moving objects 56 and 57 in frames at a desired interval without using all the frames. By doing in this way, the display number of the moving bodies 56 and 57 is reduced compared with FIG. 11 (A). Therefore, when there is no significant change in the movement of the moving bodies 56 and 57, it is possible to display an image that is easier to see by reducing the number of frames in this way. Note that the moving objects 56 and 57 at a desired time interval may be displayed in an integrated manner instead of the desired frame interval. If the desired time interval is used, there is a possibility that the standardized three-dimensional coordinates of the moving objects 56 and 57 at the time when imaging is not performed. In this case, the moving object 56 is interpolated as described above. , 57 can be calculated. For example, the three-dimensional image information calculation unit 7 calculates the three-dimensional image information of the moving bodies 56 and 57 at a desired time by interpolation using the previous and next frames, and the calculation unit 10 calculates the standardized three-dimensional coordinates. The image generation unit 11 may generate an image signal.

  In addition, the image 55 shown in FIG. 11C performs integrated display so that the moving object corresponding image portions 56 and 57 of the respective moving objects are equidistant from each other. That is, in the moving object corresponding image portions 56 and 57 of the moving object, the same moving object is displayed at a predetermined distance interval. Therefore, the moving object corresponding image portions 56 and 57 are not always displayed for each corresponding frame. In such a display, the standardized three-dimensional coordinates may be calculated by the above interpolation if necessary. In the image 55 shown in FIG. 11D, for example, the moving object corresponding image portion 56 of one moving object is integrated and displayed so as to be equidistant from each other, and the moving object corresponding image portion 57 of the remaining moving objects is displayed as the moving object corresponding image portion. The standardized three-dimensional coordinate position in the frame displayed for 56 is integrated and displayed.

  In addition to the above, various methods can be considered for the integrated display of each moving object. The display device 5 may perform preferable integrated display as appropriate according to the use state.

  The storage unit 12 stores data such as arithmetic processing and control processing in the arithmetic processing device 3. For example, the above-described data for template matching and pattern recognition is stored. In addition, the storage unit 12 includes the 3D image information calculated by the 3D image information calculation unit 7, the 3D coordinates of the moving object corresponding image portion specified by the moving object specifying unit 8, and the stationary object corresponding specified by the stationary object specifying unit 9. It is preferable to store the three-dimensional coordinates of the image portion and the standardized three-dimensional coordinates of each identified moving object-corresponding image portion converted using the integrated image calculated by the calculation unit 10 as a reference. Thereby, various calculations performed in the calculation processing unit 3 are efficiently performed, and the processing speed is shortened.

  The trigger 13 instructs the image recording unit 6 to stop erasing the time series image. For example, the trigger 13 may be configured to include an accelerometer. The three-dimensional information display device 100 is mounted on, for example, an automobile and can be used as a driving recorder. In this case, for example, when the automobile collides, images before and after that are required. In this way, when an automobile collides, the acceleration changes abruptly. Therefore, the acceleration is detected, and when the acceleration exceeds a predetermined threshold, images that are captured before and after that are generated. The trigger 13 may instruct the image recording unit 6 not to erase.

  Next, the operation of the 3D information display apparatus 100 according to an embodiment of the present invention will be described. The stereo camera 1 mounted on the vehicle (moving body) is installed, for example, with a lens facing in the traveling direction of the vehicle, and repeats imaging every predetermined time. The stereo camera 1 is a pair of left and right cameras, and these pair of cameras capture images simultaneously to obtain time-series stereo images. When a monocular camera is used as the imaging unit without using the stereo camera 1, the stereoscopic information is measured using a device capable of three-dimensional measurement, which is a stereoscopic information acquisition unit.

  A stereo image generated by the stereo camera 1 is stored by the image recording unit 6 as needed. The image recording unit 6 can store only a predetermined amount of stereo images. Then, when storing the stereo image, the image recording unit 6 determines whether or not the predetermined amount is exceeded when the next captured and generated stereo image is stored, and determines that the predetermined amount is exceeded. In this case, the image recording unit 6 deletes the earliest generated stereo image. Here, when the vehicle 13 is involved in an accident such as a collision, and the trigger 13 having an accelerometer is activated due to a rapid change in acceleration, for example, the image recording unit 6 is a predetermined time before the trigger 13 is activated. The stereo images up to are not deleted. Therefore, if the subject image captured after the trigger 13 is activated is generated in the predetermined amount that can be stored in the image recording unit 6, the stereo camera may stop the imaging operation. By doing so, images captured by the stereo camera 1 before and after the trigger 13 is activated are stored in the image recording unit 6, and analyzing these images may lead to investigation of the cause of the accident, etc. There is.

  The three-dimensional information display device 100 calculates the calculation of the standardized three-dimensional coordinates of the specified moving object corresponding image portion based on the integrated image that is an arbitrary image based on the image stored in the image recording unit 6. To do. This operation in the three-dimensional information display apparatus 100 will be described with reference to FIG. FIG. 12 is a flowchart showing the operation of the three-dimensional information display apparatus according to one embodiment of the present invention.

  When the operator inputs an instruction to calculate standardized three-dimensional coordinates, for example, using the input unit 4 or the like in a state where the images before and after the trigger 13 is activated are stored in the image recording unit 6, three-dimensional image information calculation is performed. The unit 7 calculates three-dimensional image information for each time series image using the stereo time series image stored in the image recording unit 6 (S101).

  And the moving body specific | specification part 8 specifies a moving body corresponding | compatible image part using those three-dimensional image information (S102). In specifying the moving object corresponding image portion, all moving object corresponding image portions may be specified, or any moving object corresponding image portion may be specified. For example, when the trigger 13 is actuated, only the moving object corresponding image portion corresponding to the moving object closest to the stereo camera 1 may be specified. Specifically, the moving object corresponding image portion corresponding to the moving object that exists at the coordinate position closest to the stereo camera 1 is specified from the three-dimensional image information in the generated image that is captured at the closest time when the trigger 13 is activated. . Thus, the moving object closest to the stereo camera 1 when the trigger 13 is activated is likely to cause the trigger 13 to be activated. Therefore, the cause of the accident may be easily identified by knowing the movement of the moving body before and after the trigger 13 is activated.

  In addition, the stationary object specifying unit 9 specifies a stationary object-corresponding image portion by using the 3D image information for each image calculated by the 3D image information calculating unit 8 (S103).

  Then, the calculation unit 10 calculates the three-dimensional image information for each image calculated by the three-dimensional image information calculation unit 7, the three-dimensional coordinates for each image of the moving object corresponding image portion specified by the moving object specifying unit 8, and the stationary object specifying unit 9. Based on the three-dimensional coordinates for each image of the stationary object-corresponding image part identified in step S3, the three-dimensional coordinates of the identified moving object-corresponding image part are calculated with reference to the integrated image that is an arbitrary image (S104). In addition, it is preferable that the calculating part 10 is imaged at the nearest time when the trigger 13 act | operates, and the produced | generated image is made into an integrated image. Thereby, the calculation of the standardized three-dimensional coordinates of the moving object corresponding image portion when the trigger is activated can be calculated with high accuracy. That is, as the time is further away from the reference image, the calculation error of the three-dimensional coordinates and the like of the moving object corresponding image portion increases. Therefore, by taking an image captured at the closest time when the trigger is activated and using the generated image as an integrated image, the three-dimensional coordinates of the moving object corresponding image portion before and after the trigger is activated are calculated with high accuracy.

  Further, the calculation unit 10 calculates the three-dimensional coordinates based on the integrated image in the stereo camera 1 generating these time-series images, that is, the normalized three-dimensional coordinates of the stereo camera 1 (S105). In addition, the calculation unit 10 may calculate motion information such as speed, acceleration, a speed vector, and an acceleration vector using the calculated standardized three-dimensional coordinates. Furthermore, the normalized three-dimensional coordinates of the stereo camera 1 may be calculated. In addition, when there is a traffic light in the integrated image, the arithmetic unit 10 preferably calculates the lighting color of the lamp in the traffic light for each time-series image.

  Then, as described above, the image generation unit 11 generates an image signal for generating an image using various data calculated by the calculation unit 10 or the like (S106). The image signal is transmitted to the display device 5, and the display device 5 displays an image that has been integratedly displayed (S107). The operator uses the input unit 4 to give an instruction to change the display method of the displayed image, so that the image generation unit 11 generates an image signal that provides the indicated display, It is preferable if the image displayed on the display device 5 can be changed. Accordingly, the operator can easily obtain necessary information from the image while viewing the image displayed on the display device 5.

  Here, an image generated by the stereo camera 1 in the three-dimensional information display device 100 and generated on the basis of a time series image when vehicles collide and displayed on the display device 5 will be described with reference to the drawings. FIG. 13 is a diagram showing an integrated image in a traffic accident in one embodiment of the present invention. Note that FIG. 13 is an overhead view display. As shown in FIG. 13, in the image 55, a vehicle 63 waiting for a right turn is stopped in an intersection. The vehicle 62 is traveling straight from the rear of the vehicle 63. There, it is estimated that the vehicle 61 traveling in the opposite lane turns right and enters the course of the vehicle 62, so that the vehicle 62 and the vehicle 61 are suddenly braked to avoid collision with each other. The trigger 13 is presumed to have been activated due to a sudden change in the acceleration of the vehicle 62 due to the sudden braking of the vehicle 62. In FIG. 13, when the trigger 13 is actuated, the moving object corresponding image portion in the generated image is displayed at the closest time (trigger point), and is displayed in a color different from other moving object corresponding image portions. I am doing so. In FIG. 13, vehicles 61a and 62a are moving object corresponding image portions at the trigger points. As a result, the operator can easily recognize the trigger point. In addition, a motion vector 64 is also superimposed on the image 55. Here, the magnitude of the motion vector 64 one frame before the vehicle 62a at the trigger point is significantly smaller than the motion vector 64 of the previous frame. You can see that it is over.

  As another display image, for example, an image shown in FIG. 14 will be described. FIG. 14 is a diagram showing another image integrated in a traffic accident according to an embodiment of the present invention. FIG. 14 is substantially the same as FIG. 13 except that the display shape of the motion vector 64a immediately before the trigger point is different from the other motion vectors 64 without changing the color of the vehicle at the trigger point. . As a result, the operator can easily recognize the trigger point.

  Further, for example, an image shown in FIG. 15 will be described as another display image. FIG. 15 is a diagram showing still another image integrated in a traffic accident in one embodiment of the present invention. FIG. 15 is almost the same as FIG. 13, but without changing the color of the vehicle at the trigger point, the motion vector 64 is displayed only when the speed or acceleration of the moving object changes greatly. The point is different. Thereby, the operator can easily determine a point to be noted.

  Further, in the three-dimensional information display device 100, when the stereo camera 1 is fixed, the reference positions to be imaged are the same in all time-series images. Therefore, since the reference of the three-dimensional coordinates for each image included in the three-dimensional image information is the same in all time-series images, the three-dimensional coordinates for each image are based on an arbitrary image without performing coordinate conversion. It can be said that this is the three-dimensional coordinates of the case. Therefore, it is not necessary to perform coordinate conversion for the three-dimensional coordinates for each image, and the calculation unit 10 may use the three-dimensional coordinates for each image calculated by the three-dimensional image information calculation unit 7 as a standardized three-dimensional image. . In this respect, in the three-dimensional information display apparatus 100, the case where the stereo camera 1 is fixed is different from the case where the stereo camera 1 is mounted on a moving body, but the other may be the same.

  As described above, the three-dimensional information display apparatus 100 uses the three-dimensional coordinates of the moving object corresponding image portion in each time-series image and the stereo camera 1 with reference to an integrated image that is an arbitrary image among a plurality of time-series images. The standardized three-dimensional coordinates that are the three-dimensional coordinates are calculated, and the moving object corresponding image portion of each time-series image is integrated and displayed on the reference integrated image. Thus, the operator can display an image that can easily grasp the movement of the moving object over time.

  The present specification discloses various aspects of the technology as described above, and the main technologies are summarized below.

  A three-dimensional information display device according to an aspect of the present invention includes an imaging unit that images a subject and generates a plurality of time-series images, a stereoscopic information acquisition unit that acquires stereoscopic information, the time-series image, and the stereoscopic information 3D image information calculation unit for calculating 3D image information in the time series image, a moving object specifying unit for specifying a moving object corresponding image part in the time series image, and the 3D image information. A calculating unit that calculates a three-dimensional coordinate based on any one of the time-series images for the identified moving object-corresponding image portion, and the identification in the time-series image based on the three-dimensional coordinates. The moving object corresponding image portion includes a display device that displays an image integrated with the reference image.

  As described above, the three-dimensional information display device can display an image that is integrated with respect to a moving object corresponding image portion in a plurality of temporally different images based on an arbitrary image. In this image, since a moving object whose position changes with respect to the stationary object is displayed in the reference image, the operator moves to the stationary object by looking at the image. It is possible to specifically grasp the temporal movement of the moving object. Therefore, it is possible to display an image that allows the operator to easily and correctly grasp the motion of the moving object, visually. In addition, since the three-dimensional coordinates are calculated, the display device can display an image viewed from an angle desired by the operator.

  In the above-described three-dimensional information display device, it is preferable that the display device superimposes and displays motion information corresponding to the moving object corresponding image portion displayed in an integrated manner on the displayed image.

  Thereby, the operator can grasp accurate movement information by viewing the image displayed on the display device. Therefore, it is possible to display an image that allows the operator to easily and correctly grasp the motion of the moving object, visually.

  In the above three-dimensional information display device, it is preferable that the motion information is at least one of a speed, an acceleration, a speed vector, an acceleration vector, and a motion vector of a moving object corresponding to the specified moving object corresponding image portion.

  Thereby, there exists an effect that the operator can grasp | ascertain the motion of a moving body easily correctly by these motion information.

  Moreover, in the above-described three-dimensional information display device, it is preferable that the display device displays the motion information at the same time in association with each other in the superimposed motion information.

  Thereby, the operator can easily understand the temporal correspondence of the motion information in different moving objects.

  In the 3D information display device described above, the imaging unit and the 3D information acquisition unit are stereo cameras, and the 3D image information calculation unit is configured to display a plurality of stereo time-series images generated by the stereo camera. Basically, it is preferable to calculate the three-dimensional image information using a corresponding point search.

  As described above, since the three-dimensional image information is calculated using the stereo time-series image, the highly accurate three-dimensional image information can be calculated. In addition, by using a stereo camera, it is not necessary to provide a three-dimensional information acquisition unit in addition to the camera, so that the cost and size of the three-dimensional information display device can be reduced.

  In the above-described three-dimensional information display device, it is preferable to use an image pattern of a window that is frequency-resolved and amplitude components are suppressed for the corresponding point search.

  As described above, in the corresponding point search, by suppressing the amplitude component from the frequency component, it is difficult to be affected by the luminance difference between the images and noise, and therefore it is possible to search for the corresponding point having robustness.

  In the above three-dimensional information display device, it is preferable that the imaging unit and the three-dimensional information acquisition unit are mounted on a moving body.

  Thereby, there exists an effect that an imaging location is not fixed to a specific location.

  The above three-dimensional information display device may further include a stationary body specifying unit that specifies a stationary body corresponding image portion in the time-series image, and the calculation unit uses the three-dimensional image information to For the stationary object corresponding image part specified by the specifying unit, a conversion component that matches the reference image is calculated for each time series image, and using the conversion component, for the specified moving object corresponding image part, It is preferable to calculate the three-dimensional coordinates in the reference image.

  As a result, the three-dimensional coordinates in the reference image can be easily calculated for the moving object corresponding image portion.

  In the above three-dimensional information display device, when the time-series images generated by the imaging unit are sequentially stored and a predetermined amount of time-series images are stored, the stored time-series images are stored. An image recording unit that erases the oldest image, and a trigger that instructs the image recording unit to stop erasing the time-series image after a predetermined time when activated, The three-dimensional image information calculation unit preferably calculates three-dimensional image information in the time-series image based on the time-series image and the stereoscopic information stored in the image recording unit.

  In this way, the image recording unit normally stores time-series images while erasing old images at any time, and when the trigger is activated, the image before and after the activation is not erased. Images for investigating the cause of operation can be stored. In other words, the image stored in the image recording unit is likely to contain a phenomenon that causes the trigger to operate, so analyzing these images makes it easy to determine the cause of the trigger operation. There is a high possibility of being able to investigate. Specifically, when the trigger is activated due to some accident, it is highly likely that the cause of the accident is easily investigated.

  Moreover, in the above-described three-dimensional information display device, it is preferable that the display device shows a trigger point on the displayed image.

  Thereby, the operator can easily recognize the trigger point that is a point to be noted in the displayed image. Since the trigger point is the closest time when the trigger is activated, it is highly possible that the cause of the trigger can be determined by analyzing the state of the trigger point.

  In the above-described three-dimensional information display device, it is preferable that the calculation unit sets an image of a subject captured at the closest time when the trigger is activated as the reference image.

  Thereby, the three-dimensional coordinates and the like of the moving object corresponding image portion when the trigger is activated are calculated with high accuracy. That is, as the time is further away from the reference image, the calculation error of the three-dimensional coordinates of the moving object corresponding image portion becomes larger. Therefore, by taking an image taken at the closest time when the trigger is activated and using the generated image as a reference, the three-dimensional coordinates of the moving object corresponding image portion before and after the trigger is activated are calculated with high accuracy. The

  Further, in the above-described three-dimensional information display device, the calculation unit uses the three-dimensional image information to determine, for the imaging unit, three-dimensional coordinates based on any one of the plurality of time-series images. Preferably, the display device displays an image in which the position of the imaging unit is integrated with the image based on the reference, based on the three-dimensional coordinates of the imaging unit calculated by the calculation unit.

  Thereby, the position of the imaging unit and the position of the moving body on which the imaging unit is mounted are calculated with high accuracy, and the position is displayed on the image. Therefore, the operator can grasp | ascertain the position by the time change of an imaging part concretely. Therefore, there is an effect that the operator can more specifically determine the situation from the displayed image.

  The three-dimensional information display device may further include a traffic light specifying unit that specifies a traffic light in the time-series image, and the calculation unit may determine a lamp color of the specified traffic light in the time-series image. The display device preferably displays the color of the lamp of the traffic light corresponding to the moving object corresponding image portion displayed in an integrated manner on the image to be displayed.

  As a result, it is possible to grasp from the image whether red, blue, or yellow was lit on the traffic light according to the movement of the moving object, so the operator can determine the situation more specifically from the displayed image. There is an effect that can be done.

In addition, a 3D information display method according to another aspect of the present invention includes a step of calculating 3D image information of a plurality of time-series images,
A step of specifying a moving object corresponding image portion in the time series image, and a three-dimensional image based on any one of the time series images for the specified moving object corresponding image portion using the three-dimensional image information. Based on the step of calculating coordinates and the three-dimensional coordinates, the identified moving object corresponding image portion in the time-series image is integrated and displayed on the reference image.

  As described above, the moving object corresponding image portions in a plurality of temporally different images are integrated and displayed based on an arbitrary image, so that a moving object whose position changes with respect to a stationary object is displayed in the image. The Thereby, the operator can easily and correctly grasp the motion of the moving object visually from this image.

  In order to express the present invention, the present invention has been properly and fully described through the embodiments with reference to the drawings. However, those skilled in the art can easily change and / or improve the above-described embodiments. It should be recognized that this is possible. Accordingly, unless the modifications or improvements implemented by those skilled in the art are at a level that departs from the scope of the claims recited in the claims, the modifications or improvements are not covered by the claims. It is interpreted that it is included in

  According to the present invention, it is possible to provide a three-dimensional information display apparatus and a three-dimensional information display method that display an integrated image of moving object corresponding image portions in a time-series image with an arbitrary image as a reference.

Claims (14)

An imaging unit that images a subject and generates a plurality of time-series images;
A three- dimensional information acquisition unit that acquires three-dimensional information with a three-dimensional measuring instrument ;
A three-dimensional image information calculation unit that calculates three-dimensional image information in the time-series image based on the time-series image and the stereoscopic information;
A moving object specifying unit for specifying a moving object corresponding image part in the time-series image;
An arithmetic unit that calculates three-dimensional coordinates based on any one of the time-series images for the identified moving object-corresponding image portion using the three-dimensional image information;
A three-dimensional information display device comprising: a display device that displays an image in which the identified moving object corresponding image portion in the time-series image is integrated with the reference image based on the three-dimensional coordinates.   The three-dimensional information display device according to claim 1, wherein the display device superimposes and displays motion information corresponding to the moving object corresponding image portion displayed in an integrated manner on the image to be displayed.   The three-dimensional information display device according to claim 2, wherein the motion information is at least one of a speed, an acceleration, a speed vector, an acceleration vector, and a motion vector of a moving object corresponding to the identified moving object corresponding image portion.   The three-dimensional information display device according to claim 2, wherein the display device displays the motion information at the same time in association with each other in the superimposed motion information. The imaging unit and the three-dimensional information acquisition unit are stereo cameras,
2. The 3D image information calculation unit according to claim 1, wherein the 3D image information calculation unit calculates the 3D image information using a corresponding point search based on a plurality of stereo time-series images generated by the stereo camera. Dimensional information display device.   The three-dimensional information display apparatus according to claim 5, wherein the corresponding point search uses a window image pattern in which frequency decomposition is performed and an amplitude component is suppressed.   The three-dimensional information display device according to claim 1, wherein the imaging unit and the three-dimensional information acquisition unit are mounted on a moving body. A still body specifying unit that specifies a still body corresponding image portion in the time-series image;
The calculation unit uses the three-dimensional image information to calculate, for each time-series image, a conversion component that matches the reference image with respect to the stationary object corresponding image portion identified by the stationary object identification unit. The three-dimensional information display device according to claim 7, wherein the three-dimensional coordinates in the image used as the reference are calculated for the identified moving object corresponding image portion using the conversion component. An image recording that sequentially stores the time-series images generated by the imaging unit and deletes the oldest image from the stored time-series images when a predetermined amount of time-series images is stored. And
A trigger that instructs the image recording unit to stop erasing the time-series image after a predetermined time when activated,
The 3D image information calculation unit calculates 3D image information in the time series image based on the time series image and the stereoscopic information stored in the image recording unit. Dimensional information display device.   The three-dimensional information display device according to claim 9, wherein the display device shows a trigger point in the image to be displayed.   The three-dimensional information display device according to claim 9, wherein the calculation unit uses an image of a subject captured at the closest time when the trigger is activated as the reference image. The computing unit uses the three-dimensional image information to calculate, for the imaging unit, three-dimensional coordinates based on any one of the plurality of time-series images,
3. The display device according to claim 1, wherein the display device displays an image in which the position of the imaging unit is integrated with the reference image based on the three-dimensional coordinates of the imaging unit calculated by the calculation unit. Dimensional information display device. Further comprising a traffic light identifying unit for identifying a traffic light in the time-series image;
The calculation unit determines a color of a lamp of the specified traffic light in the time series image,
The three-dimensional information display device according to claim 1, wherein the display device superimposes and displays a color of a lamp of a traffic light corresponding to the moving object corresponding image portion displayed in an integrated manner on the image to be displayed. Capturing a subject and generating a plurality of time-series images ;
Acquiring three-dimensional information with a three-dimensional measuring instrument ;
Calculating three-dimensional image information in the time-series image based on the time-series image and the stereoscopic information ;
Identifying a moving object corresponding image portion in the time-series image;
Using the three-dimensional image information, calculating a three-dimensional coordinate based on any one of the time-series images for the identified moving object-corresponding image portion;
A three-dimensional information display method for integrally displaying the specified moving object corresponding image portion in the time-series image on the reference image based on the three-dimensional coordinates.

Images