JP5409451B2 - 3D change detector - Google Patents

Description

  This invention relates to a change in observation object obtained by a disaster mapping helicopter equipped with a GPS (Global Positioning System) and a video camera, or a mobile mapping system that obtains a three-dimensional shape around a traveling path by a combination of GPS and a laser distance measuring device. The present invention relates to a three-dimensional change detection apparatus that detects

  There has been proposed a three-dimensional change detection device that detects a displacement of a three-dimensional structure such as the ground, a building, or a track using a fixed laser distance measuring device (for example, see Patent Document 1 and Patent Document 2). FIG. 18 is a diagram showing a configuration of a conventional three-dimensional change detection apparatus based on the invention described in Patent Document 1. In FIG.

JP 2008-076058 A JP 09-021636 A

Kenichi Kanaya, Mishima et al., “Three-dimensional reconstruction from two images with uncalibrated camera and its reliability evaluation” Information Processing Society Journal: Computer Vision and Image Media, Vol. 42, no. SIG6 (CVIM 2) (2001), pp. 1-8 "

  However, these conventional three-dimensional change detection devices operate only under the condition that means for observing a three-dimensional structure such as a laser range finder is fixed, such as a disaster prevention helicopter and a mobile mapping system. There was a problem that it was not possible to detect changes from observation results obtained using equipment that was observed while moving.

  The present invention has been made to solve the above-described problems. Even under the condition that the position of a video camera or a laser distance measuring device mounted on a moving body changes, the three-dimensional object to be observed is observed. It is an object of the present invention to obtain a three-dimensional change detection device capable of accurately specifying a change.

  A three-dimensional change detection apparatus according to the present invention is mounted on a moving body, and includes a photographing unit that photographs a predetermined observation target and outputs image data, a measuring unit that measures the position and orientation of the photographing unit, and the image Corresponding to the shooting range from known map data based on the ground height pattern calculation means for estimating the three-dimensional shape of the shooting range from the data and generating a ground height pattern having a ground height for each coordinate, and the position of the shooting means Elevation point data generation means for generating elevation point data having an elevation for each coordinate, and perspective projection conversion means for perspective projection conversion of the elevation point data based on predetermined parameters obtained from the position and orientation of the photographing means; Parameter calculating means for calculating optimum parameters of the perspective projection transformation so that the correlation between the ground height pattern and the elevation point data obtained by the perspective projection transformation is maximized; In which the ground clearance pattern and the optimum parameters in perspective projection converted elevation point data compared to each coordinate and a change detecting means for detecting a change of the predetermined observation target.

  According to the three-dimensional change detection apparatus according to the present invention, it is possible to accurately identify a change in an observation target even under a condition in which the position of a video camera or a laser distance measuring device mounted on a moving body changes. There is an effect.

It is a block diagram which shows the structure of the three-dimensional change detection apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the example of the image data image | photographed with the camera of the three-dimensional change detection apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the example of the information regarding the position and attitude | position of a camera measured by the measuring device of the three-dimensional change detection apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the example of the map data currently recorded on the hard disk of the three-dimensional change detection apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the ground high pattern by the three-dimensional change detection apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the relationship between the present position of the camera of the three-dimensional change detection apparatus which concerns on Embodiment 1 of this invention, a body vector, an optical axis vector, and a ground surface reference coordinate. It is a figure which shows the elevation point data by the three-dimensional change detection apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the elevation point data after perspective projection conversion by the three-dimensional change detection apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the elevation point data after perspective projection conversion by the three-dimensional change detection apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the elevation point data after perspective projection conversion by the three-dimensional change detection apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the elevation point data after the perspective projection conversion of the optimal parameter by the three-dimensional change detection apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the altitude point data after the perspective projection conversion of the image data and the optimal parameter by the three-dimensional change detection apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the altitude point data after the perspective projection conversion of the image data and the optimal parameter by the three-dimensional change detection apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the ground high pattern by the three-dimensional change detection apparatus which concerns on Embodiment 2 of this invention. It is a figure which shows the elevation point data by the three-dimensional change detection apparatus which concerns on Embodiment 2 of this invention. It is a figure which shows the elevation point data after the perspective projection conversion of the optimal parameter by the three-dimensional change detection apparatus which concerns on Embodiment 2 of this invention. It is a figure which shows the correlation coefficient for every area | region by the three-dimensional change detection apparatus which concerns on Embodiment 2 of this invention. It is a figure which shows the structure of the conventional three-dimensional change detection apparatus.

  Hereinafter, preferred embodiments of the three-dimensional change detection apparatus of the present invention will be described with reference to the drawings.

Embodiment 1 FIG.
A three-dimensional change detection apparatus according to Embodiment 1 of the present invention will be described with reference to FIGS. FIG. 1 is a block diagram showing the configuration of the three-dimensional change detection apparatus according to Embodiment 1 of the present invention. Note that the three-dimensional change detection apparatus according to Embodiment 1 of the present invention is premised on operation by a disaster prevention helicopter.

  In FIG. 1, a three-dimensional change detection apparatus according to Embodiment 1 of the present invention includes a pan head 10 for fixing a camera 1, and a measurement apparatus (measurement for performing measurement necessary for specifying the position and orientation of the camera 1). Means) 20, a computer 30 having a hard disk 31 and running a program 32-36, and a display 40 for displaying the calculation results of the computer 30 are provided. In addition, the whole apparatus may be mounted in the disaster prevention helicopter, or only a part of the apparatus, for example, the camera 1, the pan head 10, and the measuring apparatus 10 may be mounted.

  A camera (video camera or the like) 1 is fixed to a disaster prevention helicopter 100 or the like by a pan head 10 and corresponds to a photographing means. The ground height pattern calculation program 32 corresponds to ground height pattern calculation means for estimating the three-dimensional shape of the shooting range from the image data captured by the camera 1 and calculating the ground height pattern. The elevation point data generation program 33 corresponds to elevation point data generation means for generating elevation point data corresponding to the photographing range based on known map data or the like recorded in advance on the hard disk 31. The perspective projection conversion program 34 corresponds to a perspective projection conversion unit that performs perspective projection conversion of elevation point data using predetermined parameters. The parameter calculation program 35 calculates parameters based on information from the measuring device 20, and calculates the optimum parameters for perspective projection transformation that maximizes the correlation between the ground height pattern and the perspective projection transformed elevation point data. This corresponds to the parameter calculation means. The change detection program 36 corresponds to change detection means for detecting a change in the observation object by comparing the ground height pattern and the elevation point data based on the calculated perspective projection transformation parameter.

  Next, the operation of the three-dimensional change detection apparatus according to the first embodiment will be described with reference to the drawings.

  First, examples of various data used for explaining the operation will be described.

  FIG. 2 is a diagram showing an example of image data captured by the camera of the three-dimensional change detection apparatus according to Embodiment 1 of the present invention. In FIG. 2, (a) is image data 1A of a shooting range of 512 × 512 pixels (pixels) shot at time t, and (b) is 512 × 512 pixels (pixels) shot one second before time t. The image data 1B of the shooting range.

  FIG. 3 is a diagram showing an example of information on the position and orientation of the camera measured by the measurement device of the three-dimensional change detection device according to Embodiment 1 of the present invention. The measurement device 20 includes a GPS radio wave receiver, an angle sensor, and a laser range finder. As information 20A about the position and orientation of the camera 1, the current coordinates of the camera 1 at the time of shooting of image data taken by the camera 1 ( x, y, x), the distance D to the target point, the body vector (Hu, Hv, Hw), and the optical axis vector (Ou, Ov, Ow) are obtained.

  Here, the current coordinates (x, y, x) of the camera 1 are given as plane rectangular coordinates obtained by converting latitude, longitude, and altitude obtained based on GPS. Airframe vectors (Hu, Hv, Hw) are given as unit vectors in the lower direction of the airframe from the center of gravity of helicopter 100 (aircraft). Further, the optical axis vectors (Ou, Ov, Ow) are given as unit vectors when the current coordinates of the camera 1 are set as the origin.

  FIG. 4 is a diagram showing an example of map data recorded in advance on the hard disk of the three-dimensional change detection apparatus according to Embodiment 1 of the present invention. In FIG. 4, (a) is map data 31A indicating the shape of the building and the building ID (1001, 1002,...), And (b) is the correspondence between the building ID and the ground height (16 m, 38 m,...). Is table data 31B.

  First, the camera 1 mounted on the disaster prevention helicopter 100 captures the ground from above and outputs image data 1A and 1B in the capturing range shown in FIGS. 2 (a) and 2 (b).

  Next, as a first step, the ground height pattern calculation program 32 inputs the image data 1A and 1B, and estimates the three-dimensional shape of the shooting range by stereo image processing. Note that the stereo image processing is generally well-known computer processing, and thus description thereof is omitted here (for example, see Non-Patent Document 1).

  As the second step, the ground height pattern calculation program 32 quantizes the image data of the shooting range as a two-dimensional array with a resolution of 32 × 32 pixels (pixels) based on the estimated three-dimensional shape, and calculates the ground height for each pixel. A ground high pattern is generated by giving a light and shade value proportional to. An example of this ground high pattern is shown in FIG.

  Next, as a first step, the elevation point data generation program 33 is based on the information 20A relating to the position and orientation of the camera 1 shown in FIG. Calculate the coordinates. These coordinates are called ground surface reference point coordinates (Gx, Gy).

  FIG. 6 shows the relationship between the current position of the camera 1 mounted on the disaster prevention helicopter 100, the airframe vector, the optical axis vector, and the ground surface reference point coordinates. The UVW coordinate system indicating the airframe vector and the optical axis vector is a right-handed coordinate system in which the center of gravity of the helicopter 100 is the origin, the direction from the origin to the north is the U axis, and the direction from the origin to the nadir is the W axis. . The ground surface reference point coordinates (Gx, Gy) are calculated by the following equations (1) and (2).

  X and y are the current position of the camera 1 at the time of shooting, Ou and Ov are optical axis vectors, and D is the distance from the camera 1 to the target point.

  As the second step, the elevation point data generation program 33 captures the calculated ground surface reference point coordinates (Gx, Gy) from the map data 31A shown in FIG. A range of 200 m is taken out from the east, west, north, and south corresponding to the range, and quantized with a resolution of 10 m. From the table data 31B shown in FIG. 4B, the Z coordinate and the gray value corresponding to the ground height are obtained for each pixel. A given three-dimensional point group is generated. This point cloud is called elevation point data.

  An example of the elevation point data obtained in this way is shown in FIG. In this example, the elevation point data is 41 × 41 = 1681 points. For the sake of convenience, the elevation point data here is a right-handed coordinate system in which the ground surface reference point coordinates are the origin (0, 0, 0), and the X axis and the Y axis are parallel to the X and Y axes, respectively, of plane rectangular coordinates Normalized as data.

  Next, the perspective projection conversion program 34 first performs perspective projection conversion of the elevation point data based on the following equations (3) and (4) using the initial values of the parameters. Here, the perspective projection transformation is defined by six parameters of the X coordinate, Y coordinate, and Z coordinate of the projection center, and the rotation angle around the X axis, the rotation angle around the Y axis, and the rotation angle around the Z axis of the elevation point data. . The coordinates of the i-th elevation point data are (xi, yi, zi), the X value, Y value, and Z value of the projection center are Cx, Cy, and Cz, respectively. When the rotation angle and the rotation angle around the Z axis are ω, φ, and κ, respectively, perspective projection transformation is expressed by the following equations (3) and (4).

  In equation (4), f is the focal length of the projection center, and x ″ i and y ″ i are the coordinates of the i-th elevation point data after perspective projection conversion. Thereafter, the perspective projection conversion program 34 performs perspective projection conversion of the elevation point data using the parameters calculated (changed) by the parameter calculation program 35. Examples of perspective projection conversion for the elevation point data of FIG. 7 are shown in FIGS.

  FIG. 8A shows the elevation point data after perspective projection conversion when the six parameters are Cx = 0, Cy = 0, Cz = 850, ω = 0, φ = 0, and κ = 0. Note that the focal length f = 52 (m).

  FIG. 8B shows the elevation point data after perspective projection conversion when the six parameters are Cx = 0, Cy = 0, Cz = 750, ω = 0, φ = 0, and κ = 0. Note that the focal length f = 52 (m).

  FIG. 9A shows the elevation point data after perspective projection conversion when the six parameters are Cx = 0, Cy = −30, Cz = 800, ω = 0, φ = 0, and κ = 0. Note that the focal length f = 52 (m).

  FIG. 9B shows elevation point data after perspective projection conversion when the six parameters are Cx = 30, Cy = 0, Cz = 800, ω = 0, φ = 0, and κ = 0. Note that the focal length f = 52 (m).

  FIG. 10A shows the elevation point data after perspective projection conversion when the six parameters are Cx = 0, Cy = 0, Cz = 800, ω = 0, φ = 0, and κ = 10. Note that the focal length f = 52 (m).

  FIG. 10B shows the elevation point data after the perspective projection conversion when the six parameters are Cx = 0, Cy = 0, Cz = 800, ω = −30, φ = 0, and κ = 0. Note that the focal length f = 52 (m).

  Note that the initial values of the six parameters of the perspective projection conversion are given based on the information 20A relating to the position and orientation of the camera 1 shown in FIG. The parameter calculation program 35 calculates the X coordinate from the current position (x, y, z) of the helicopter 100, the distance D to the target point, the body vector (Hu, Hv, Hw), and the optical axis vector (Ou, Ov, Ow). Value, Y coordinate value and Z coordinate value (Cx, Cy, Cz), and parameters for perspective projection conversion of rotation angles ω, φ and κ are calculated based on the following equations (5)-(12) (optical axis) When the vector is in the first quadrant of the UV plane).

  Next, the parameter calculation program 35 obtains a correlation coefficient between the ground height pattern shown in FIG. 5 and the elevation projection data shown in FIG. Evaluate. Thereafter, between the elevation pattern shown in FIG. 5 and the perspective projection converted using the parameters obtained by changing the elevation point data shown in FIG. 7, for example, the elevation point data shown in FIGS. In the meantime, the correlation coefficient is obtained and the correlation is evaluated.

  Thereafter, the parameter coefficient is maximized while the parameter calculation program 35 repeatedly performs parameter changes, the perspective projection conversion of the elevation point data by the perspective projection conversion program 34, and the correlation evaluation by the parameter calculation program 35. Find the optimal parameters. A local search method can be used for this algorithm. Since this local search method is a widely known method, the description thereof is omitted here.

  Based on the equations (3) and (4), FIG. 11 shows the result of the perspective projection conversion program 34 performing the perspective projection conversion on the elevation point data of FIG. 7 using the optimum parameters.

  The change detection program 36 compares the ground height pattern of FIG. 5 calculated by the ground height pattern calculation program 32 with the elevation point data after the perspective projection conversion of FIG. Changes in arbitrary coordinates (pixels) of (the subject of the image data 1A in FIG. 2A) can be detected. Note that the change detection program 36 detects changes in all coordinates (pixels) of the ground height pattern and elevation point data, but skips coordinates having no data (value).

  12A corresponds to the image data 1A taken at time t in FIG. 2A, and FIG. 12B corresponds to the elevation point data after the perspective projection conversion with the optimum parameters in FIG. .

  As shown in FIG. 12A, for example, the coordinates (IX, IY) = (105, 121) on the image data 1A are on the elevation point data after perspective projection conversion, as shown in FIG. (PX, PY) = (− 3, 1), and further corresponds to the coordinate (PX, PY) = (− 3, 1) on the ground height pattern as shown in FIG. The elevation (shading) value of the coordinates (−3, 1) on the elevation point data after perspective projection conversion is “35 m or more”, and the ground height (shading) value of the coordinates (−3, 1) on the ground height pattern Is “35 m or more”, it can be confirmed that there is no change in the coordinates.

  13A corresponds to the image data 1A taken at time t in FIG. 2A, and FIG. 13B corresponds to the elevation point data after the perspective projection conversion with the optimum parameters in FIG. .

  Further, as shown in FIG. 13A, for example, coordinates (IX, IY) = (193, 161) on the image data 1A are elevation points after perspective projection conversion as shown in FIG. 13B. This corresponds to the coordinates (PX, PY) = (8, 4) on the data, and further corresponds to the coordinates (PX, PY) = (8, 4) on the ground height pattern as shown in FIG. Since the former value is “35 m or more” and the latter value is “5 m or more to less than 15 m”, it can be confirmed that the value has changed. That is, it is possible to detect a change such as a building whose ground clearance has increased or a building whose ground clearance has decreased. The change detection program 36 displays the result of detecting the change for each coordinate (pixel) on the display 40.

Embodiment 2. FIG.
A three-dimensional change detection apparatus according to Embodiment 2 of the present invention will be described with reference to FIGS. The configuration of the three-dimensional change detection apparatus according to Embodiment 2 of the present invention is the same as that of Embodiment 1 above.

  First, examples of various data used for explaining the operation will be described.

  In the second embodiment, each data is the image data 1A, 1B in FIG. 2, the information 20A on the position and orientation of the camera 1 in FIG. 3, and the map data 31A in FIG. It is the same as the table data 31B.

  Next, the operation of the three-dimensional change detection apparatus according to the second embodiment will be described with reference to the drawings.

  First, the operations of the camera 1 and the ground height pattern calculation program 32 are basically the same as those in the first embodiment. In the second embodiment, the ground height pattern having a resolution of 32 × 32 pixels (pixels) is used. In addition, the image data in the shooting range is quantized as a two-dimensional array with a resolution of 64 × 64 pixels (pixels), and a ground height pattern is generated by giving a gray value proportional to the ground height for each pixel. An example of this ground high pattern is shown in FIG.

  The operation of the altitude point data generation program 33 is basically the same as that in the first embodiment, but in this second embodiment, in addition to the altitude point data quantized at a resolution of 10 m, the altitude point data generation program 33 has a resolution of 5 m. Quantized elevation data is also generated. An example of elevation point data quantized with a resolution of 5 m is shown in FIG.

  The operations of the perspective projection conversion program 34 and the parameter calculation program 35 are basically the same as those in the first embodiment, but the elevation point data is quantized with a resolution of 10 m and quantized with a resolution of 5 m. However, altitude point data of two types of quantization levels can be used. The perspective projection conversion program 34 performs perspective projection conversion of elevation point data of two types of quantization levels. Therefore, in the second embodiment, a method for calculating the correlation coefficient between the ground height pattern and the elevation point data is defined as follows.

  First, the parameter calculation program 35 divides the ground height pattern of FIGS. 5 and 14 into 4 × 4 16 regions. Next, a correlation coefficient between each area of the ground height pattern having a resolution of 32 × 32 pixels and elevation point data having a resolution of 10 m is obtained for each area. Further, a correlation coefficient between each area of the ground height pattern having a resolution of 64 × 64 pixels and the elevation point data having a resolution of 5 m is obtained for each area. Then, the correlation coefficient is compared for each same region, and a high correlation coefficient is set as the correlation coefficient of the region. That is, using two types of ground height patterns of 32 × 32 pixels and 64 × 64 pixels and two types of elevation point data of 10 m and 5 m resolution, the correlation coefficient for each region of the ground height pattern. And the higher correlation coefficient is used as the correlation coefficient of the region. Further, the average of the calculated correlation coefficients of the 16 regions is obtained and determined as the correlation coefficient of the entire region, that is, the correlation coefficient of the ground height pattern and the elevation point data. And the parameter calculation program 35 calculates the optimal parameter when the correlation coefficient becomes the maximum value.

  FIG. 16 shows the result of perspective projection conversion of the elevation point data using the optimum parameters calculated by the parameter calculation program 35 in this way. In this example, a region quantized with a resolution of 10 m and a region quantized with a resolution of 5 m are mixed. The correlation coefficient (a) for each area of the elevation point data after perspective projection conversion in FIG. 16, the ground height pattern quantized with a resolution of 64 × 64 pixels and the elevation point data quantized with a resolution of 5 m FIG. 17 shows a comparison of the correlation coefficient (b) for each region in the case where the optimum parameter is calculated using only. In all regions, (a) has a higher correlation than (b). As in this example, a higher correlation coefficient can be obtained by adjusting the quantization resolution of the elevation point data for each region. Thereby, the correspondence between the ground height pattern and the altitude point data becomes accurate, and precise change detection becomes possible.

  In the second embodiment, the ground high pattern is divided into 4 × 4 16 regions, but may be divided into other division methods such as 4 × 3 and 5 × 5. .

  In the second embodiment, the ground height pattern is created with two types of resolutions of 32 × 32 pixels and 64 × 64 pixels. However, other resolutions such as 256 × 256 pixels and 160 × 120 pixels are used. For example, the image may be created at another resolution. Furthermore, in the second embodiment, the ground high pattern is created with two kinds of resolutions, but three or more kinds, for example, 3 such as 32 × 32 pixels, 64 × 64 pixels, and 128 × 128 pixels are used. You may create a ground high pattern with different resolutions.

Embodiment 3 FIG.
A three-dimensional change detection apparatus according to Embodiment 3 of the present invention will be described.

  In the first embodiment and the second embodiment described above, the case where the video camera 1 is mounted on the disaster prevention helicopter 100 has been described. However, the video camera 1 is mounted on a glider other than the disaster prevention helicopter 100 or a moving body such as a light aircraft. Also good.

  Further, although the video camera 1 has been described as the photographing means, a still camera may be used instead of the video camera 1. Further, a laser distance measuring device may be used as an imaging unit as in a mobile mapping system.

  1 camera, 10 platform, 20 measuring device, 30 computer, 31 hard disk, 32 ground height pattern calculation program, 33 elevation data generation program, 34 perspective projection conversion program, 35 parameter calculation program, 36 change detection program, 40 display, 100 Disaster prevention helicopter.

Claims (2)

An imaging means mounted on a moving body and imaging a predetermined observation target and outputting image data;
Measuring means for measuring the position and orientation of the photographing means;
A ground height pattern calculating means for estimating a three-dimensional shape of a shooting range from the image data and generating a ground height pattern having a ground height for each coordinate;
Elevation point data generation means for generating elevation point data having an elevation for each coordinate corresponding to the shooting range from known map data based on the position of the shooting means;
Perspective projection conversion means for perspective projection conversion of the elevation point data based on predetermined parameters determined from the position and orientation of the photographing means;
Parameter calculation means for calculating the optimum parameters of the perspective projection transformation so that the correlation between the ground height pattern and the elevation point data subjected to the perspective projection transformation is maximized;
A three-dimensional change comprising: a change detecting means for detecting the change of the predetermined observation object by comparing the ground height pattern and the elevation point data obtained by perspective projection conversion with the optimum parameter for each coordinate; Detection device. An imaging means mounted on a moving body and imaging a predetermined observation target and outputting image data;
Measuring means for measuring the position and orientation of the photographing means;
A ground height pattern calculating means for estimating a three-dimensional shape of a shooting range from the image data and generating ground height patterns of first and second resolutions having a ground height for each coordinate;
Elevation point data generating means for generating elevation point data of first and second resolutions corresponding to the shooting range from known map data and having an elevation for each coordinate based on the position of the imaging means;
Perspective projection conversion means for perspective projection conversion of the elevation point data of the first and second resolutions based on predetermined parameters obtained from the position and orientation of the photographing means;
The ground height pattern of the first and second resolutions is divided into a plurality of areas, and each area of the ground height pattern of the first resolution and the elevation point data of the first resolution subjected to perspective projection conversion A correlation coefficient is obtained, and a correlation coefficient between each area of the ground height pattern of the second resolution and the elevation point data of the second resolution obtained by perspective projection conversion is obtained, and two correlations of the same area are obtained. Select elevation point data of the region using the higher correlation coefficient as the correlation coefficient of the region, obtain the average of the correlation coefficients of the plurality of regions, and perform perspective projection transformation so that the average is maximized A parameter calculation means for calculating an optimum parameter;
The change in the predetermined observation object is detected by comparing the ground height pattern of the second resolution and the elevation point data in which the first and second resolutions, which are perspective projection transformed with the optimum parameter, are mixed for each coordinate. A three-dimensional change detection apparatus comprising: a change detection unit.

Images