JP4767578B2 - High-precision CV calculation device, CV-type three-dimensional map generation device and CV-type navigation device equipped with this high-precision CV calculation device - Google Patents

Description

The present invention relates to an image processing arithmetic device that calculates a camera position and a camera direction of a camera that takes a video of a moving image, and acquires position information of an arbitrary object based on the camera position information.
In particular, the present invention provides a high-precision CV computing device that automatically obtains CV (camera vector) data that indicates a camera position and a camera rotation angle with high accuracy from a plurality of frame images of a video image (moving image). A CV method that generates a 3D map in real time along with the traveling direction and vehicle posture of a moving body that travels and navigates based on the high-accuracy CV data, and outputs and displays the current status of the moving mobile body with high accuracy. The present invention relates to a three-dimensional map generation device and a CV navigation device capable of controlling a moving body such as a vehicle based on high-precision CV data.

In general, a car navigation system using a GPS geodetic satellite is known as means for acquiring position information of a moving vehicle or the like (see, for example, Patent Documents 1-3).
The GPS system reads the time and position data emitted from a plurality of geodetic satellites with a receiving device installed in the vehicle, calculates the three-dimensional coordinates of the receiving point from the difference in radio wave arrival time from each satellite, The current position is displayed.
According to such a GPS system, the three-dimensional position of the reception point can be measured in a global range.

Here, as the position accuracy obtained by the GPS system, conventionally, there is an influence of radio wave reflection and refraction in the ionosphere, and the error is 50 to 300 meters.
In recent years, a method has been added to measure the error of radio wave arrival time using known points of latitude, longitude, and altitude, and to transmit it as a correction signal to correct the error at the receiving point. Has been reduced to about a dozen meters.

Japanese Patent Laid-Open No. 11-304513 JP 2001-255157 A JP 2002-357430 A

However, it is difficult to accurately measure the three-dimensional position of a moving object with a conventional GPS system in which the positional accuracy error is in the range of several tens of meters. For example, it is continuous in real time with an accuracy of several centimeters. It was impossible to measure the position and output the position information.
Here, the inventor of the present application has developed a CV (camera vector) calculation technique for calculating a three-axis rotational attitude with respect to a three-dimensional position of a camera from an image movement caused by the movement of the camera. The CV calculation is a technique for obtaining a CV (Camera Vector) value, and the camera vector value is a value indicating a three-dimensional position and a three-axis rotation posture of a camera that acquires a video for measurement.

The CV value obtained by the CV calculation indicates the camera position of the acquired video, has an excellent characteristic that three-dimensional measurement in the video is possible, and has a higher accuracy than GPS. Information could be obtained.
However, since the CV value is not an absolute value but a relative value, the scale changes, and an error accumulates as the distance increases.

In this regard, position information acquired by the GPS system is an absolute value, and a three-dimensional position can be measured as an absolute value. However, the positional information obtained by GPS has a large error when viewed as a relative value, and the accuracy of GPS is that even if it is a DGPS (Differential GPS) system that performs positioning by receiving a correction signal from a fixed station. The accuracy is about 1 meter.
Furthermore, in the GPS system, as a matter of course, the position information obtained is only the coordinates of a certain measurement point and cannot be combined with a video image. Therefore, measurement in the image as in the CV calculation is impossible.

In this way, with conventional positioning systems, map creation devices, navigation devices, etc., there is a technology that can acquire the three-dimensional position information by measuring the absolute coordinates of the object with high accuracy (for example, an error of ± 15 cm) while moving. I did not.
Therefore, as a result of earnest research, the inventor of the present application automatically detects a sufficient number of feature points from a plurality of frame images of a moving image captured by a camera mounted on a moving body, and automatically tracks the feature points between frames. By doing this, it is possible to obtain CV data indicating the camera position and rotation angle with high accuracy by performing overlapping calculation on a large number of feature points, and calibrating the camera position information with absolute coordinates obtained by GPS or the like. It was conceived that the three-dimensional position coordinates of the moving object can be obtained and output with high accuracy.

  That is, the present invention has been proposed in order to solve the problems of the prior art, and the camera position of the moving object is determined based on the camera image captured when the moving object is moved using the image processing technique. 3D coordinates can be obtained with high accuracy from the GPS system, and the accumulation of 3D coordinate errors is corrected with absolute coordinates obtained by GPS, IMU, etc. It is an object of the present invention to provide a high-precision CV calculation device capable of acquiring accurate position information, a CV-type three-dimensional map generation device and a CV-type navigation device equipped with the high-precision CV calculation device.

To achieve the above object, a high-precision CV computing device of the present invention is fixed to the moving body, to acquire the video image by photographing the surrounding mobile together with the movement of the moving body, the position data of the mobile A moving body measuring unit for measuring position measurement data including movement amount data, and a video image acquired by the moving body measuring unit, and CV data indicating a three-dimensional position and a three-axis rotation position of the camera from the video image A data generation unit that obtains position measurement data measured by the mobile body measurement unit, and a CV data generated by the data generation unit are associated with a time synchronization CV by a reference time A time axis matching unit that generates the position measurement data as time-synchronized measurement data associated with a reference time, and generates the position measurement data as data. The CV data and the time synchronization measurement data are compared in association with each other on the same time axis, and the time synchronization CV data and the time synchronization measurement data are complementarily corrected based on the comparison result, and the final CV value of the CV data is corrected. A CV correction unit that generates a CV correction signal and a video image recorded in the data generation unit are output, and a high-accuracy CV value corrected by the CV correction signal is output corresponding to each frame of the video image. A high-accuracy CV data output unit, and in particular, the data generation unit automatically extracts a predetermined number of feature points from the image data of the moving image, and the extracted feature points A feature point correspondence processing unit that automatically tracks within each frame image of a moving image and obtains a correspondence between the frame images, and a three-dimensional position coordinate of the feature point for which the correspondence is obtained Therefore, a CV data calculation for performing the predetermined CV calculation is provided with a camera vector calculation unit for obtaining a camera vector consisting of the three-dimensional position coordinate and the three-dimensional rotation coordinate of the camera corresponding to each frame image from the three-dimensional position coordinate. It is set as the structure provided with a part.

Multiple addition, accurate CV operation apparatus of the present invention, the movable body measuring unit comprises a plurality of video cameras equipment, the data generation unit, with a parallax resulting video image by the plurality of video cameras equipment A parallax camera coordinate three-dimensionalization unit that generates a three-dimensional shape in the camera coordinate system from the image by the parallax between the cameras, and extracts a three-dimensional feature portion from the three-dimensional shape in the camera coordinate system, A part is tracked on a plurality of adjacent frames of a video image, or a three-dimensional shape corresponding to the frame is tracked, and a camera in a stationary coordinate system is obtained from the three-dimensional feature part and the camera position in the camera coordinate system. A multiple CV calculation unit that obtains coordinates and three-axis rotation angles by CV calculation, or a plurality of three-dimensional feature points extracted from a three-dimensional shape by the camera coordinate system The three-dimensional feature point is tracked on a plurality of adjacent frames of the video image or on the three-dimensional shape corresponding to the frame, and from the three-dimensional feature point and the camera origin coordinate acquired as camera coordinates, a stationary coordinate A multi-CV calculation unit that obtains the camera position coordinates and the three-axis rotation angle in the system by CV calculation.

The high-precision CV operation apparatus of the present invention, the movable body measuring portion is fixed to the moving body, the movement of the moving body, the whole circumference video camera by photographing the surrounding mobile acquires entire circumference video An apparatus unit, and a plurality of video camera device units that acquire images with parallax around the moving body by a plurality of cameras fixed to the moving body and having a known positional relationship, A plurality of video camera equipment units are installed with a known relationship between a three-dimensional position and a posture of each other, and the data generation unit records an all-round video image obtained by the all-around video camera device unit. A plurality of feature points are extracted from each frame image of the all-around video image recorded in the image recording unit and the all-around video image recording unit, and the feature points are traced to a plurality of adjacent frames of the all-around video image. A CV calculation unit that calculates CV data indicating the position coordinates of the all-around camera and the three-axis rotation angle in the stationary coordinate system based on the plurality of feature points in the coordinate system of the moving body and the camera positional relationship; A plurality of feature points are extracted in each frame image of the video image obtained by the plurality of video camera device units, the feature points are tracked in each adjacent frame image of the video image, and the distance between the cameras is known. A multi-CV calculation unit that obtains an absolute distance by tracking within each corresponding frame image of another camera, and calculates multi-CV data indicating a high-precision camera three-dimensional coordinate and a three-axis rotation posture; The time axis matching unit is configured to supply time data to the CV data and the multi-CV data, and based on the time data supplied from the clock unit. A time-synchronized CV data recording unit that records the CV data in association with each other, a time-synchronized multi-CV data recording unit that records the multi-CV data in association with the time data supplied from the clock unit, and a plurality of cameras A delay adjustment CV signal output unit that outputs a delay adjustment CV signal by correcting a delay in recording time between CV data acquired from the image by a time difference from the time data supplied from the clock unit; and a plurality of camera images A delay multi-CV signal output unit that outputs a delay-adjusted multi-CV signal by correcting a time difference between the multi-CV data acquired from the time data and the time data supplied from the clock unit, and A CV correction unit detects a coordinate shift due to a difference in installation position between the plurality of video camera device units and the all-round video camera device unit. And a device coordinate position correction unit that corrects the position coordinates of the plurality of video camera device units to the position of the all-round video camera device unit, the coordinate arrangement shape of the delay adjustment multi-CV signal, and the delay adjustment CV signal A coordinate array shape comparison unit that detects a coordinate array shape difference by comparing the coordinate array shape, the coordinate array shape of the delay adjustment multi-CV signal, and the coordinate array shape of the delay adjustment CV signal A correction signal generation unit that generates a correction signal for correcting a scale error in the three-axis direction from a difference in coordinate arrangement shape of the obtained sections, and a coordinate arrangement shape based on the CV data corresponding to the sections are coordinate arrangements based on the multi-CV data. A CV data scale correction unit that corrects CV data so as to match the shape, and correction processing in the CV data scale correction unit are repeated over the entire target section. By returning, the global scale correction unit that generates continuous position correction CV data, and the three-dimensional coordinates of the CV data corrected by the global scale correction unit are known, and recalculation of high-precision CV data including three-axis rotation A high-precision CV data output unit that outputs an all-round video image acquired by the all-round video camera device unit, and the all-round video image A recalculation CV data output unit that outputs a CV data signal that is position-corrected by the global scale correction unit, recalculated by the CV recalculation unit, and corrected for three-axis rotation in synchronization with each output image frame; It is set as the structure provided with.

The high-precision CV operation apparatus of the present invention, the data generating unit, the plurality video camera device unit plurality of images with a parallax, parallax system camera coordinate generating a three-dimensional shape in the camera coordinate system by the camera between parallax A plurality of three-dimensional feature parts are extracted from the three-dimensional shape in the camera coordinate system acquired by the three-dimensionalization part and the parallax camera coordinate three-dimensionalization part, and the three-dimensional feature parts are video Tracking between adjacent frames of an image, or tracking between three-dimensional shapes corresponding to the frame, and a stationary coordinate system of a camera provided in the moving body based on the three-dimensional feature portion and the camera positional relationship And a multi-CV calculation unit that obtains the camera position coordinates and the three-axis rotation angle by CV calculation.

The high-precision CV operation apparatus of the present invention, the movable body measuring portion is fixed to the moving body, the movement of the moving body, the whole circumference video camera by photographing the surrounding mobile acquires entire circumference video A device unit and a GPS measurement device unit that is fixed to the mobile body and acquires a three-dimensional position of the mobile body at a latitude and longitude altitude, and the all-around video camera device unit and the GPS measurement device unit mutually An all-round video image recording unit configured to record the all-round video image obtained by the all-round video camera device unit, the three-dimensional position and posture relationship being set as known, and the data generation unit; A plurality of feature points are extracted from each frame image of the all-round video image recorded in the video recording unit, the feature points are traced to a plurality of adjacent frames of the all-around video image, the plurality of feature points and the camera Based on positional relationship A CV data acquisition unit for calculating CV data indicating a three-dimensional position coordinate and a three-axis rotation angle in a stationary coordinate system of the all-round camera provided in the moving body, and the GPS measurement device from the GPS measurement device unit An absolute coordinate data acquisition unit that acquires absolute coordinate data indicating a three-dimensional position of the unit, and the time axis matching unit supplies time data to the CV data and the absolute coordinate data, and A time-synchronized CV data recording unit that associates and records the CV data based on time data supplied from the timepiece unit, and a time at which the absolute coordinate data is recorded in association with the time data supplied from the timepiece unit The recording time delay between the synchronous absolute coordinate recording unit and the CV data acquired from a plurality of camera images is set to the time difference between the time data supplied from the clock unit. A delay adjustment CV signal output unit that outputs a delay adjustment CV signal, and a delay adjustment absolute signal that outputs a delay adjustment absolute coordinate signal by correcting a time difference between the absolute coordinate signal output time and the measurement time of the absolute coordinate data. A coordinate signal output unit, and the CV correction unit generates a deviation of a coordinate and a three-axis rotation based on the CV data based on a difference in installation position between the GPS measurement device unit and the all-round video camera device unit. A coordinate position rotation correction unit that corrects the position coordinates of the GPS measurement device unit to the position of the all-round video camera device unit and outputs a delayed coordinate rotation correction signal; and the delay adjustment absolute value corrected by the coordinate rotation correction signal. A coordinate array shape comparison unit for comparing the coordinate array shape of the coordinate signal with the coordinate array shape of the delay adjustment CV signal, and comparing the coordinate array shape of the two coordinate arrays, the coordinate array shape of the CV value is excluded from the coordinate array shape. The absolute coordinate accuracy change point detection unit that detects the position of the absolute coordinate signal to be delayed as an absolute coordinate accuracy change point, and a high accuracy relative that selects a plurality of sections with high relative accuracy excluding the absolute coordinate accuracy change point CV data obtained by calculation from a value section selection unit and a point section corresponding to the high-precision relative value section selection unit based on the absolute coordinate data and CV data including a scale error due to accumulation of calculation errors A measurement interval corresponding unit that associates the interval of the point by using the time as a parameter, and when the measurement value does not exist at the coincidence point of the start point and the end point of time divided by the interval, The distance between the absolute coordinates of the start point and end point of the section in which the correspondence relationship is taken is used as the section distance to compensate for CV data including a scale error. A correction signal generation unit that generates a signal, a CV data scale correction unit that corrects the CV data by matching CV data corresponding to the section with section distance data based on absolute coordinate data with high relative accuracy, The high-precision CV data output unit includes a global scale correction unit that generates continuous high-precision CV data by repeating the correction process in the CV data scale correction unit a plurality of times over the entire area of the target section. The all-round video output unit that outputs the all-round video image acquired by the all-round video camera device unit, and the CV data signal that has been subjected to the all-scale scale correction in synchronization with each image frame of the all-round video image output. And a CV data output unit for outputting.

The high-precision CV operation apparatus of the present invention, the CV correction section, and the absolute accuracy changing point detecting unit distribution statistical processing unit for statistically processing the detected distribution of absolute accuracy change point in said absolute accuracy change point A temporary true value calculation unit that obtains a statistical center of the distribution of the absolute value as a temporary true value, and the absolute accuracy change point that indicates a section in which an error in the GPS measurement data is to be corrected based on the temporary true value A temporary true value correction data acquisition unit for acquiring a correction value of a section indicated by the start point and end point of the absolute coordinates, and absolute coordinates for obtaining a high-precision absolute coordinate array by correcting the absolute coordinate array based on the correction values of the section A high-accuracy CV data output unit that uses the high-accuracy absolute coordinate array as a reference for absolute coordinates, and converts the absolute coordinates to the scale-corrected CV data output from the high-accuracy CV data output unit. Give absolute A target additional CV data integration unit, synchronized with each image frame, is configured to include the absolute coordinate CV data output unit for outputting the CV data absolute coordinate is added, the.

The high-precision CV operation apparatus of the present invention, the movable body measuring unit is provided with a IMU measuring device for measuring the absolute triaxial rotational orientation, wherein the data generating unit, the IMU instrumentation unit absolute 3 measured by An absolute triaxial rotational attitude data acquisition unit that acquires axial rotation attitude data, and the time axis alignment unit records the absolute triaxial rotational attitude data in association with a time signal supplied from the clock unit A synchronous triaxial rotational attitude recording unit, and a delay adjustment triaxial rotational attitude output unit that outputs a point adjustment triaxial rotational attitude data signal by correcting a deviation between the recording time of the absolute triaxial rotational attitude data and a reference time. A three-axis rotation posture comparison unit that compares the three-axis rotation posture signal from the delay adjustment CV signal output unit and the absolute rotation three-axis posture signal from the delay adjustment three-axis rotation posture output unit. And 3 above A difference between the rotation attitude signal and the absolute rotation triaxial attitude signal is compared with a predetermined threshold value, and a difference between the three axis rotation attitude signal and the absolute rotation triaxial attitude signal is the threshold value. A three-axis rotation correction section setting unit that specifies a correction section when exceeding, and a three-axis rotation correction signal generation unit that generates a correction value in the correction section, and the high-precision CV data output unit includes the A triaxial rotation correction CV data output unit that outputs triaxial rotation correction data added to the triaxial rotation attitude data synchronized with each image frame of the CV data output from the CV data output unit is provided.

The high-precision CV operation apparatus of the present invention, the data generating unit, the entire circumference video images recorded by the spherical conversion coordinates, and schematic bipolar planar development unit for planar convert schematic beneath direction and directly above , Detecting the vertical direction on the basis of the vertical component included in the plane-converted image, and a general equatorial plane expansion unit for plane-converting the entire video image recorded in spherical transformation coordinates into a plurality of equatorial plane directions And a vertical direction detection unit that reflects the detected vertical direction in the calculation of the CV data.

The high-precision CV operation apparatus of the present invention, the movable body measuring portion, and the low scale all around video equipment unit for photographing an object with a low scale, high scale all around to shoot the object at a high scale A low-scale video signal acquisition unit for acquiring a low-scale video signal by the low-scale all-around video image device unit, and the high-scale all-around video image device unit. The high-scale video signal acquisition unit for acquiring the high-scale video signal, the low-scale all-around video image output from the low-scale all-around video image device unit, and the high-scale all-around video image device unit. A common feature point detection unit that detects a common feature point by comparing a high-scale all-round video image, extracting common feature points included in both images, and taking corresponding points, and each detected common feature For points, by weight function A common feature point weight setting unit that sets weighted common feature points, a low-scale feature point detection unit that detects low-scale feature points from the low-scale all-around video image, and detected low-scale features A low-scale inter-frame tracking unit that tracks a point to an adjacent frame, a high-scale feature point detection unit that detects a high-scale feature point from the high-scale all-around video image, and a detected high-scale feature point adjacent to each other Based on the low-scale feature points, the high-scale feature points, and the weight common feature points, the low-scale all-round video image and the high-scale all-round video image are collectively displayed. A CV calculation unit that performs CV calculation in an integrated manner and a CV data acquisition unit that outputs highly accurate CV calculation data obtained by the CV calculation unit are provided.

In addition, the high-precision CV calculation device of the present invention includes a two-dimensional map unit in which two-dimensional coordinates are given as known, and point coordinates of the two-dimensional map unit in a video image acquired by the moving body measurement unit. Acquired by a common point detection unit that detects a corresponding common point part, a common point two-dimensional coordinate detection unit that detects a two-dimensional coordinate on a two-dimensional map of the detected common point, and the mobile body measurement unit A feature point extraction unit that extracts a traceable feature point in a video image, a feature point tracking unit that tracks a corresponding point in a plurality of adjacent image frames, and a plurality of feature points A CV data acquisition unit for acquiring CV data, and a CV data acquisition unit for acquiring CV data, and a CV calculation unit for obtaining a three-dimensional position of the camera in a stationary coordinate system and a three-axis rotation posture based on the camera positional relationship Based on CV data from A common point three-dimensional coordinate calculation unit for acquiring a three-dimensional coordinate of the common point of the video image acquired by the moving body measurement unit, and a two-dimensional plane coordinate for converting the three-dimensional coordinate of the common point into a two-dimensional plane. A plurality of common points obtained by projecting the two-dimensional coordinates of the common point obtained from the data of the conversion unit, the two-dimensional map unit, and the three-dimensional coordinates of the common point obtained from the video image onto a two-dimensional plane. A common point two-dimensional coordinate comparison unit that compares the point two-dimensional coordinates, and detects the difference between the two two-dimensional coordinates compared, determines the position coordinates of the comparison signal that minimizes the difference, and at the same ratio A common point region CV value correction that corrects a coordinate in the direction of one coordinate axis, disperses the correction amount of the common point in a distributed manner, and replaces the position coordinate of the CV value for correcting the entire position coordinate of the CV value. And the common point portion and C position correction data is fixed, CV calculation is performed again based on the tracking result of the common point and the feature point, and more accurate three-axis rotation data is obtained. And a correction CV data output unit that outputs a correction CV value with increased.

The high-precision CV arithmetic device of the present invention is a three-dimensional map in which three-dimensional coordinates are given as known, a known three-dimensional point description map that shows a plurality of known three-dimensional coordinate points together with coordinates on a two-dimensional map, or Among the three-dimensional point list indicating a plurality of known three-dimensional coordinates, a map unit including at least one of the three-dimensional point lists, and a known three-dimensional point of the map unit among the video images acquired by the moving body measurement unit A common point detection unit for detecting a common point portion in an image frame of a video image corresponding to the same, and extracting and tracking the corresponding point in a plurality of image frames of the video image by using the common point portion as a common feature point A common feature point tracking unit; a common feature point weighting unit that weights the common feature point by distinguishing the common feature point from a general feature point and determining a priority in calculation; and a plurality of weighted common feature point tracking results. A CV calculation unit that calculates a three-dimensional position and a three-axis rotation posture of the camera in a stationary coordinate system based on a tracking result of the general feature points and a three-dimensional positional relationship of the camera; and the CV calculation unit A common point three-dimensional coordinate detection unit that obtains the three-dimensional coordinates of the common feature point that is output together with the CV value, a common point three-dimensional coordinate obtained as known from the data of the map unit, and a tertiary of the common feature point A common point three-dimensional coordinate comparison unit that compares original coordinates, a three-dimensional difference detection unit that detects a difference between the compared three-dimensional coordinates, and a three-dimensional general feature point output from the CV calculation unit A general feature point three-dimensionalization unit for acquiring coordinates, and two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the general feature point acquired by the general feature point three-dimensionalization unit onto a video plane in each frame of a video image; Of the video footage An image two-dimensional difference detection unit for detecting a two-dimensional difference between a general feature point in a frame and a two-dimensional coordinate of a video image plane; and a CV calculation so that the three-dimensional difference and the two-dimensional difference value are minimized. And a CV correction calculation unit that repeats the CV calculation in the CV calculation unit, and a CV data acquisition unit that outputs a corrected CV value whose accuracy has been improved by the correction.

Also, CV method three-dimensional map generation apparatus of the present invention, in a video image output from the high-precision CV data output section of the high-precision CV arithmetic unit, specifying an arbitrary object to be generated as a three-dimensional map An object specifying unit, a target three-dimensional measuring unit that three-dimensionally measures the target specified by the target specifying unit, and data that is three-dimensionally measured by the target specifying unit are described in a three-dimensional map. A 3D map part generator that generates 3D parts in a format, a 3D space definition part that defines a 3D space as a 3D map, and a position on the coordinates of the object, The object point surveying unit for obtaining coordinates, and the three-dimensional map component, in the three-dimensional space defined by the three-dimensional space definition unit, are three-dimensional by the three-dimensional coordinates obtained by the survey of the target point surveying unit. 3D coordinates that determine the original arrangement And tough, repeat the three-dimensional coordinates arrangement of the three-dimensional map part is configured to include a three-dimensional map output unit for outputting the three-dimensional map, a.

Further, the CV method 3D map generation apparatus of the present invention includes a 3D part database storing a plurality of 3D parts constituting a 3D map, and a 3D part stored in the 3D part database. A 3D component candidate selection unit for selecting a 3D component that is a candidate to be compared with the target specified by the target specifying unit; a part of the video image specified by the target specifying unit; A comparison unit that compares the three-dimensional component output from the component candidate selection unit, and a comparison result at the comparison unit that outputs a mismatch signal when there is a mismatch, and the three-dimensional component candidate selection based on the mismatch signal A mismatch signal output unit that outputs the next candidate three-dimensional part from the unit, a match signal output unit that outputs a match signal in the case of a match as a result of comparison in the comparison unit, and the third signal by the match signal Original parts When a matching part selecting unit that uses the three-dimensional part finally output from the complementary part selecting unit as a matching part, and the matching part is different from the corresponding video image, the matching part is corrected to generate a corrected part. A part correction unit, and the matching part or the correction part is sent to the three-dimensional map part generation unit.

Also, CV method three-dimensional map generation apparatus of the present invention, in a video image output from the high-precision CV data output section of the high-precision CV arithmetic unit, specifying an arbitrary target range to be generated as a three-dimensional map Based on the target range specification unit, the measurement point density specification unit for specifying the measurement point density in the target range specified by the target range specification unit, the measurement points based on the specified target range and the measurement point density Is output from a measurement point generation unit that generates a measurement point, a measurement point tracking unit that tracks the measurement point over a plurality of adjacent frames of the video image, and a high-precision CV data output unit of the high-precision CV arithmetic device Based on the CV value and the measurement point from which the tracking data is acquired by tracking the measurement point, a measurement point calculation unit for calculating the three-dimensional coordinates of the measurement point, and the three-dimensional of all the measurement points in the target range Coordinates A target range surveying unit for obtaining a three-dimensional shape created by a measurement point from a three-dimensional distribution of the measurement points, and a target composed of a straight line portion in the target range in the video image A specific measurement point designation registration unit that designates a part of an object as a specific measurement point and registers it together with the attribute, and the specific measurement point is specified, so that the attribute of the specific measurement point is sent to the measurement point generation unit And is tracked and calculated in the same manner as a general measurement point, is output to the measurement point calculation unit, and only the data of a specific measurement point is extracted from the three-dimensional coordinate data output from the measurement point calculation unit, A specific measurement point forming unit that gives a shape suitable for the attribute, and a three-dimensional shape formed from the specific measurement points output from the specific measurement point forming unit, obtained from the measurement points other than the specific measurement points With original shape A three-dimensional coordinate integration unit that case, it is constituted; and a three-dimensional map output section for generating and outputting a three-dimensional map on the basis of the integrated three-dimensional shape.

Also, CV method three-dimensional map generation apparatus of the present invention, the measurement point density specification section, on the basis of the CV value obtained from the high-precision CV data output section of the high-precision CV computing device, the rotational component of the video image An image stabilization unit that generates a stopped video, a source point determination unit that determines a source point that is a starting point of the motion of the video image in which the rotation component is stopped, and an image frame of an arbitrary designated range in a small area An image block dividing unit for dividing the image block into a set, a feature point extracting unit for extracting a block that can be a feature point for all of the image blocks, and a block that can be a part of a boundary line for all of the image blocks A boundary point extraction unit to extract, a region point extraction unit to extract points other than the feature points and boundary points of the image block as region points, and each point of the image block, A two-dimensional rubber band connection image that connects in a radial direction from the knot point and fixes the order so that the relationship between the adjacent blocks is maintained, and the distance between adjacent blocks is arbitrarily stretchable. A generation unit, wherein the measurement point generation unit generates a region created by the feature point selection unit that selects the feature point as a measurement point, and the region point and the feature point and the boundary point surrounding it. A measurement point determination unit that is a tracking measurement target using the feature point, the measurement point, and the region point as a measurement region, and the measurement point tracking unit includes the feature point, the boundary point, and the measurement region as the measurement point. Are tracked over a plurality of adjacent frames along the spring point direction, and the measurement point calculation unit outputs the tracking result of the measurement point and the high-precision CV data output unit of the high-precision CV calculation device. CV day And the measurement point surveying unit repeatedly calculates the three-dimensional coordinates of the boundary points surrounding the measurement point and the measurement area over the target range. If there is an image block that cannot be tracked by the measurement point tracking unit, the image block is obtained by interpolation based on the already calculated three-dimensional coordinates, and the target is estimated by predicting the three-dimensional position. Surveying the range, the three-dimensional shape generation unit determines a region to which the feature point and the boundary point belong from the three-dimensional coordinates of the feature point and the boundary point, and the feature point and the boundary point By affiliation, the rubber string cutting part that cuts one of the rubber band bonds of the block, cuts the bond between one area, and bonds to the other area, and all the blocks in the target range are feature points. , Boundary points, region points Classify and fix the feature points with the coordinates obtained as measurement points, interpolate the boundary points and area points where the coordinates are not fixed on the rubber string, give each 3D coordinate, 3D all blocks A three-dimensional rubber string coupling image unit for coupling rubber strings, and the three-dimensional coordinate integration unit determines the division density of the image block dividing unit every time the three-dimensional rubber string coupling image is generated. The image division density changing unit for changing the density from the first to the next, and performing the recalculation of the process from the previous image block division, and the three-dimensional rubber string combined image by the initially set density of the image block division 3D to generate a more detailed three-dimensional image by recording once and generating a more detailed three-dimensional shape sequentially in the next process, generating the three-dimensional shape sequentially by repeating the process until reaching a predetermined density Image recording It is constituted comprising a part, the.

Also, CV method three-dimensional map generation apparatus of the present invention includes a plurality camera imaging section having a plurality of video cameras field of view to obtain a duplicate video images having parallax, by the plurality of video cameras that moves with the moving body A parallax camera coordinate three-dimensionalization unit that obtains a camera coordinate system three-dimensional distance distribution data by calculating a three-dimensional shape of the camera coordinate system based on a captured video image having a parallax; a method three-dimensional device, and the high-precision CV calculation apparatus for obtaining CV data of a camera position including a plurality cameras, the camera coordinate three-dimensional distance distribution data obtained by the parallax method three-dimensional device, the A stationary coordinate system conversion unit for continuously converting into three-dimensional distance distribution data in a stationary coordinate system based on a high-precision CV value obtained by a high-precision CV arithmetic unit; and the video By combining the three-dimensional distance distribution data in the static coordinate system continuously obtained while overlapping with the progress of the image, it is constituted; and a stationary coordinate system synthetic binding unit to integrate the stationary coordinate system.

Also, CV method three-dimensional map generation apparatus of the present invention, the stationary object separation space a three-dimensional distance distribution data to be integrated in the stationary coordinate system by the static coordinate system synthetic binding unit, contain only the stationary object in the stationary coordinate system A moving object separation that separates and records three-dimensional data with temporal variation from the three-dimensional distance distribution data integrated into the stationary coordinate system by the composition unit and the stationary coordinate system combining and coupling unit as a moving object. And a section.

Also, CV method navigation system of the present invention includes: the CV method three-dimensional map generation apparatus, a vehicle capable of three-dimensional map device having a three-dimensional map information generated by the CV method three-dimensional map generation apparatus, three-dimensional From the travel route input device that inputs the required information including the data and attributes to the 3D map device in a changeable manner and the in-vehicle camera provided in the travel vehicle, the wide angle video data and the relatively low Based on the in-vehicle camera that outputs accurate CV data, the approximate CV calculation device, and the approximate CV data output from the approximate CV calculation device, obtains three-dimensional data indicating the current position and orientation of the traveling vehicle, The 3D data is roughly associated with the 3D map to detect the 3D spatial structure, and the obstacles that need emergency response before the recognition of the object are detected. The object is sent to the real-time control device, and the vehicle-mounted camera and the schematic CV calculation device are in the visible range from the road and detect the three-dimensional space configuration consisting of a plurality of objects related to the driving purpose and traffic The space construction device, the object space construction data detected by the object space construction device, and the roughly three-dimensional position data of the traveling vehicle are collated, are in the visible range from the traveling road, and meet the traveling purpose. An object recognition device for recognizing a three-dimensional shape of an individual object including a traffic signal, a road sign, and a road marking related to the object and identifying the shape and attribute by collating with the information of the three-dimensional map, and the object The three-dimensional shape or part shape of a plurality of objects by the recognition device is tracked three-dimensionally, or a characteristic shape is selected and tracked at the stage of the three-dimensional space configuration before the identification, and the tracking result is And the high-precision CV arithmetic unit for obtaining the time the CV data and CV data acquired by the high-precision CV computing device, an output of the object space configuration device, and the object recognition result of the object recognition device A current state determination device that outputs the current state determination result of the traveling vehicle based on the output of the on-vehicle measuring instrument including the on-vehicle radar, and a predetermined state including an accelerator, a brake, and a handle of the traveling vehicle by the output of the current state determination device And a real-time control device that controls the operation unit or changes data of the travel route input device.

Furthermore, the CV type navigation device of the present invention provides the output of the high-precision CV calculation device, the recognition result of the object recognition device, and the current state determination result of the current state determination device on a two-dimensional map or a three-dimensional map in real time. It is a structure provided with the real-time display device displayed on the.

According to the high-accuracy CV calculation device of the present invention having the above-described configuration, a CV (camera vector) indicating a camera position, a camera rotation angle, etc. with high accuracy is automatically calculated from a plurality of frame images of a moving image. Can be requested. Then, by correcting the obtained CV data with position information obtained by GPS or the like, absolute coordinates are given to the CV data that is a relative value, and scale calibration is performed, so that high accuracy and high sampling can be achieved. An accuracy CV arithmetic unit is realized.
In order to calibrate absolute coordinates, it is usually necessary to calibrate at the position of the instrument etc., but by using CV calculation, the position of the instrument etc. is of course reflected in the video image. There is an excellent advantage that it can be calibrated in the coordinates of the point.

According to the CV calculation, a relative accuracy of about ± 15 cm can be realized. By combining this with the absolute accuracy of GPS statistical processing, GPS absolute coordinate calibration is performed by CV calculation, and at the same time GPS The relative coordinates of the CV calculation can be calibrated with the absolute coordinates.
Thereby, highly accurate position information can be generated and acquired, and a three-dimensional map having an absolute accuracy of approximately ± 15 cm can be generated using this position information.
By implementing a three-dimensional map generation device and a navigation device using such a high-precision CV arithmetic device of the present invention, it is possible to control a traveling vehicle or the like with an accuracy of approximately ± 15 cm.

According to the present invention, a CV (camera vector) indicating a camera position, a camera rotation angle, etc. with high accuracy is automatically obtained from a plurality of frame images of a moving image by calculation, and the obtained CV data is obtained by GPS or the like. By correcting with the obtained position information, absolute coordinates can be given to CV data that is a relative value, and scale calibration can be executed.
As a result, high-accuracy position information can be generated and acquired, and a navigation device capable of generating a three-dimensional map using the obtained high-accuracy position information and controlling a traveling vehicle or the like is realized. can do.

Hereinafter, preferred embodiments of a high-precision CV calculation device according to the present invention, and a CV-type three-dimensional map generation device and a CV-type navigation device including the high-precision CV calculation device will be described with reference to the drawings.
Here, the high-precision CV arithmetic device, CV system three-dimensional map generation device, and CV system navigation device of the present invention described below are realized by processing, means, and functions executed by a computer in accordance with instructions of a program (software). . The program sends commands to each component of the computer, and performs predetermined processing and functions as shown below, such as automatic extraction of feature points, automatic tracking of extracted feature points, calculation of three-dimensional coordinates of feature points, camera Perform vector operations. Thus, each process and means in the present invention are realized by specific means in which the program and the computer cooperate.
Note that all or part of the program is provided by, for example, a magnetic disk, optical disk, semiconductor memory, or any other computer-readable recording medium, and the program read from the recording medium is installed in the computer and executed. The The program can also be loaded and executed directly on a computer through a communication line without using a recording medium.

[High-precision CV processor]
First, the basic concept of the high-precision CV arithmetic device according to the present invention will be described.
The high-accuracy CV calculation device of the present invention automatically obtains CV (camera vector) indicating the camera position and camera rotation angle with high accuracy from a plurality of frame images of a video image by calculation and obtains the obtained CV data. Is corrected with position information obtained by GPS or the like, thereby giving absolute coordinates to CV data that is a relative value, and performing scale calibration.

[Basic concept]
When a navigation device such as an image measuring device, a surveying device, or a traveling vehicle is used, an image and a CV value obtained from the image can be used. However, when a CV value is obtained from a moving image, the scale is also a relative value. Therefore, in the CV calculation data in the long section, the error accumulates and the error increases. Accordingly, if the CV calculation is used as it is, accumulation of errors occurs particularly in measurement in a long distance section, but by correcting the CV value with absolute value data from a GPS or an IMU (Inertial Measuring Unit). The accumulated error can be corrected and calibrated.
Therefore, in the present invention, by combining measurement data with a small absolute error such as GPS in the CV calculation, it is possible to realize a highly accurate apparatus such as CV image measurement, image surveying, image positioning, and image navigation. It is a thing.

Here, the CV calculation means obtaining a CV value, and the obtained result is referred to as CV value or CV data. The notation CV is an abbreviation for camera vector, which is a value indicating a three-dimensional position and a three-axis rotation posture of a camera such as a video camera that obtains an image for measurement or the like.
The CV calculation acquires a moving image (video image), detects a feature point in the image, tracks it in a plurality of adjacent frames, and creates a triangle formed by the camera position and the tracking locus of the feature point in the image. The three-dimensional position of the camera and the three-axis rotational posture of the camera are obtained by generating a large number of images and analyzing the triangle.

In the CV calculation, it is an important characteristic that, in the process of obtaining the CV value, simultaneously, the three-dimensional coordinates are obtained simultaneously for the feature points in the video.
Further, the CV value obtained by calculation from the moving image is obtained simultaneously with the three-dimensional camera position and the three-dimensional camera posture corresponding to each frame of the moving image. Moreover, in principle, the characteristic that a CV value is obtained corresponding to an image with a single camera is an excellent feature that can be realized only by CV calculation.
For example, in other measurement means (such as GPS and IMU), in order to simultaneously acquire each frame of a moving image and its three-dimensional camera position and three-dimensional camera posture, an image frame and a measurement sampling time are used. Since it has to be highly accurate and completely synchronized, it becomes a huge device, which is practically difficult to realize.

CV data obtained by calculation from a moving image is a relative value when not processed, but if it is a short section, three-dimensional position information and three-axis rotation angle information can be acquired with high accuracy.
In addition, since the CV data is acquired from the image, the acquired data is a relative value, but it can be realized by other methods that can measure the positional relationship with an arbitrary object in the image. With characteristics.
Further, since the CV value corresponding to the image can be obtained, the CV calculation capable of obtaining the camera position and its three-axis rotation posture directly from the image in the in-image measurement or survey is suitable for the in-image measurement or the in-image survey.

In contrast, the GPS system has an excellent characteristic of having absolute coordinates, but the sampling frequency in acquiring GPS data is about 10 times per second, and it is difficult to synchronize with a moving image.
Therefore, it is extremely difficult to synchronize GPS with the image of the camera for surveying and the measuring instrument with the camera device and GPS device attached to the vehicle that moves at high speed. The surveying device synchronized with the image and the positioning with high accuracy are difficult. Devices, navigation devices, etc. are not realized.
In addition, it is more difficult to detect the three-axis attitude of the camera and the three-axis rotation attitude acquired from the GPS or the like in synchronization with each other than the position synchronization.

As a cause of errors in GPS positioning and three-axis rotational attitude positioning, GPS satellites near the horizon are strongly affected by the ionosphere, causing errors due to multipath, and causing large errors in measurement.
Further, when the receivable satellite is switched near the horizon, the accuracy of the measurement value changes and a relative error that causes the measurement position to become discontinuous is generated.
On the other hand, since GPS is an absolute distance measurement, various errors may occur at the measurement point. The error does not accumulate, and the longer the distance measurement, the smaller the ratio of the error to the measurement distance. .

Therefore, in the high-precision CV arithmetic device of the present invention, it is noted that CV measurement and GPS measurement are relative coordinates and absolute coordinates, respectively, and each has a characteristic of complementing each other's disadvantageous surface. A device capable of mutually complementing and correcting position information is realized.
That is, according to the high-precision CV arithmetic device of the present invention, the acquired CV (camera vector) is a relative value with respect to the scale, and thus includes an accumulated error. However, absolute coordinates are given by GPS data or the like. , Cumulative error can be greatly reduced. By correcting the CV data with GPS, scale calibration can be performed by giving absolute coordinates, and a high-accuracy CV arithmetic unit capable of high-accuracy and high-sampling is realized.
In order to calibrate the absolute coordinates, it is usually necessary to calibrate at the position of the measuring instrument or the like. However, by using CV calculation, the position of the measuring instrument as well as the coordinates of the point shown in the image are displayed. Can also be calibrated.

Since the position of the video camera can be accurately measured using such a high-precision CV arithmetic device, it is possible to perform in-image surveying using this, and a CV surveying device can be realized. A highly accurate three-dimensional map can be generated.
Note that a 3D map may be an actual map that covers a wide range of kilometers, just like a normal map, but based on the same principle, a 3D map or a very narrow range of tertiary under a microscope. The original shape may be shown.

Furthermore, a CV-type navigation device can be realized by performing high-precision CV calculation in real time by CV calculation based on a three-dimensional map and an onboard camera image.
Here, the CV navigation device is an automatic driving device or the like that recognizes and understands traffic signs, road markings, and signals, and controls the vehicle to travel to a predetermined location. However, the CV navigation device is used. Therefore, it is possible to realize the robot's eyes and brain that can recognize and judge the surroundings, and can be used and applied to robot technology by miniaturization.

[Basic configuration]
Hereinafter, an embodiment of a high-precision CV arithmetic device according to the present invention will be described with reference to FIG. FIG. 1 is a block diagram showing a basic configuration of a high-precision CV arithmetic device according to an embodiment of the present invention. FIG. 2 is a block diagram showing details of the high-precision CV arithmetic apparatus shown in FIG.
As shown in these drawings, the high-precision CV calculation device 1 includes a moving body measurement unit 10, a data generation unit 20, a time axis matching unit 30, a CV correction unit 40, and a high-precision CV data output unit 50. In addition, a CV video composition display unit 60 is selectively provided.

The moving body measurement unit 10 is fixed to a moving body such as a traveling vehicle for the purpose of acquiring CV data, and includes a video camera device unit 11 that captures images around the moving body by moving the moving body.
In addition, the moving body measuring unit 10 includes a video camera device 11 for the purpose of correcting CV data, and a position measuring device unit 12 that can measure data relating to its own position and movement amount data. As will be described later, the position measuring instrument base 12 is configured by a GPS device, an IMU device, or the like that acquires absolute coordinates. However, it is possible to acquire a plurality of video cameras with different viewpoints and images with parallax. It can also be constituted by a plurality of cameras.

Here, as shown in FIGS. 3 and 4, the moving body corresponds to a traveling vehicle such as an automobile, and a video camera device section 11 for obtaining CV data is provided on the ceiling of the moving body 2.
In the example shown in FIGS. 3 and 4, as the video camera device unit 11 mounted on the moving body 2, an all-around camera 11 a capable of shooting a wide range image around the traveling moving body 2 is provided. Cameras that capture and acquire a wide range of images include cameras with wide-angle lenses and fish-eye lenses, moving cameras, fixed cameras, cameras with multiple cameras fixed, cameras that can rotate around 360 degrees, etc. An all-around camera 11a that is fixed integrally and that captures a wide-range image as the moving body 2 moves can be used.

According to the all-around camera 11a, as shown in FIG. 4, by being installed on the ceiling of the moving body 2, a 360-degree all-around image of the camera can be simultaneously captured by a plurality of cameras. As 2 moves, a wide-range video can be acquired as moving image data.
Here, the omnidirectional camera is a video camera that can directly acquire the omnidirectional video of the camera, but can be used as the omnidirectional video if more than half of the entire circumference of the camera can be acquired as video. Even in the case of a normal camera with a limited angle of view, the accuracy of the CV calculation is reduced, but it can be handled as a part of the entire peripheral video.

Note that the wide-range video imaged by the all-around camera 11a can be pasted as a single image on a virtual spherical surface that matches the angle of view at the time of imaging. The spherical image data pasted on the virtual spherical surface is stored and output as spherical image (360 degree image) data pasted on the virtual spherical surface. The virtual spherical surface can be set to an arbitrary spherical shape centered on the camera unit that acquires a wide range image.
FIG. 5A is an appearance image of a virtual spherical surface to which a spherical image is pasted, and FIG. 5B is an example of a spherical image pasted to the virtual spherical surface. FIG. 4C shows an example of an image obtained by developing the spherical image of FIG.

The data generation unit 20 includes a video video recording unit 21 that records video from the video camera device unit of the moving body measurement unit 10, and from the recorded video video, represents a three-dimensional position and a three-axis rotation position of the camera. A CV data calculation unit 23 for performing calculation is provided. Details of the CV calculation in the CV data calculation unit 23 will be described later (see FIGS. 6 to 15).
Further, the data generation unit 20 includes a measurement data acquisition unit 22 that acquires measurement data from the position measurement device unit of the moving body measurement unit 10.

The time axis alignment unit 30 includes a clock unit 31 that generates a reference time.
Then, the time axis alignment unit 30 associates the CV data obtained by the CV calculation unit of the data generation unit 20 with the reference time to generate time synchronization CV data, and the data generation unit 20. The time-synchronized measurement data generating unit 33 generates time-synchronized measurement data by associating the measurement data from the measurement data acquisition unit with a reference time.
Here, the time axis alignment unit 30 can also be established by adjusting to any time of the device provided in the moving body measurement unit 10. For example, in the video camera device section of the moving body measuring unit 10, the time axis can be adjusted even if the data of the other device is displayed as a video or the time display synchronized with the data is acquired as a video. .

The CV correction unit 40 includes a time axis comparison unit 41 that compares the time-synchronized CV data generated by the time-axis matching unit 30 and the time-synchronized measurement data on the same time axis and associates the two on the same time axis.
Then, the CV correction unit 40 complements and corrects the time-synchronized CV data and the time-synchronized measurement data based on the result of the time axis comparison unit 41, and finally generates a CV correction signal for correcting the CV value. A correction signal generation unit 42 is provided.

The high-precision CV data output unit 50 includes a video image output unit 52 that outputs the video image of the video image recording unit 21, and a CV correction signal generated by the CV correction unit 42 corresponding to each frame of the video image. Is provided with a CV data output unit 51 that outputs a CV value with higher accuracy.
Furthermore, the video image output and the high-accuracy CV data can be provided with a CV image composition display unit 60 for displaying on the same screen (see FIG. 1).

The CV video composition display unit 60 displays the CV data of the frame and the CV data of the preceding and succeeding frames in a place corresponding to the camera position of the video video. The CV data indicates the three-dimensional position and the three-axis rotation, and the CV value can be observed simultaneously in each frame of the video image by displaying it over the video image. An example of an image displayed by superimposing CV data on a video image is shown in FIG.
If the camera position is correctly displayed in the video image, the position in the video image indicated by the CV value is the center of the image, and if the camera movement is close to a straight line, the CV values of all frames are displayed overlapping. Therefore, for example, as shown in FIG. 15, it is appropriate to display the position of 1 meter right below the camera position. Alternatively, it is more appropriate to display the CV value at the height of the road surface based on the distance to the road surface.

[CV data calculation unit]
Next, details of the basic CV calculation in the high-precision CV calculation device 1 will be described with reference to FIGS.
As described above, the CV calculation is performed by the CV data calculation unit 23 of the data generation unit 20. As shown in FIG. 6, the CV data calculation unit 23 includes a feature point extraction unit 23a, a feature point correspondence processing unit 23b, a camera vector calculation unit 23c, an error minimization unit 23d, and a three-dimensional information tracking unit 23e. A high-precision camera vector calculation unit 23f is provided.

The feature point extraction unit 23a automatically extracts a sufficient number of feature points from the moving image data captured by the video camera device unit 11 of the moving body measurement unit 10 and temporarily recorded in the video image recording unit 21.
The feature point correspondence processing unit 23b automatically obtains the correspondence relationship by automatically tracking the feature points automatically extracted in each frame image between the frames.
The camera vector calculation unit 23c automatically calculates a camera vector corresponding to each frame image from the three-dimensional position coordinates of the feature points for which the correspondence relationship has been determined.
The error minimizing unit 23d performs statistical processing so that the solution distribution of each camera vector is minimized by overlapping calculation of a plurality of camera positions, and automatically determines the camera position direction subjected to the error minimizing process. To do.

The three-dimensional information tracking unit 23e positions the camera vector obtained by the camera vector calculation unit 23c as an approximate camera vector, and based on the three-dimensional information sequentially obtained as part of the image in the subsequent process, a plurality of frame images 3D information is automatically tracked along an image of an adjacent frame. Here, three-dimensional information (three-dimensional shape) is mainly three-dimensional distribution information of feature points, that is, a collection of three-dimensional points, and this collection of three-dimensional points constitutes a three-dimensional shape. To do.
The high-precision camera vector calculation unit 23f generates and outputs a camera vector with higher accuracy than the camera vector obtained by the camera vector calculation unit 23c based on the tracking data obtained by the three-dimensional information tracking unit 23e.
Then, the camera vector obtained as described above is input to the time axis matching unit 30, and CV correction is performed using the position measurement data.

There are several methods for detecting a camera vector from feature points of a plurality of images (moving images or continuous still images). In the CV data calculation unit 23 of the present embodiment shown in FIG. These feature points are automatically extracted and automatically tracked to obtain a three-dimensional vector and a three-axis rotation vector of the camera by epipolar geometry.
By taking a sufficient number of feature points, camera vector information is duplicated, and an error can be minimized from the duplicated information to obtain a more accurate camera vector.

Here, the camera vector refers to a vector of degrees of freedom possessed by the camera.
In general, a stationary three-dimensional object has six degrees of freedom of position coordinates (X, Y, Z) and rotation angles (Φx, Φy, Φz) of the respective coordinate axes. Therefore, the camera vector refers to a vector of six degrees of freedom of the camera position coordinates (X, Y, Z) and the rotation angles (Φx, Φy, Φz) of the respective coordinate axes. When the camera moves, the direction of movement also enters the degree of freedom, which can be derived by differentiation from the above six degrees of freedom.
As described above, the detection of the camera vector by the camera vector computing device 1 of the present embodiment is that the camera takes six degrees of freedom for each frame and determines six different degrees of freedom for each frame. It is.

Hereinafter, a specific camera vector detection method in the CV data calculation unit 23 will be described with reference to FIG.
First, image data acquired by the video camera device unit 11 of the moving body measurement unit 10 is input to the feature point extraction unit 23a of the CV data calculation unit 23 via the video image recording unit 21 (or directly), and feature point extraction is performed. The point 23a is automatically extracted from the appropriately sampled frame image by the unit 23a, or the small region image to be the feature point is automatically extracted, and the feature point correspondence processing unit 23b automatically selects the feature point correspondence between the plurality of frame images. Is required.
Specifically, more than a sufficient number of feature points that are used as a reference for detecting a camera vector are obtained. An example of feature points between images and their corresponding relationships are shown in FIGS. In the figure, “+” is a feature point that is automatically extracted, and the correspondence is automatically tracked between a plurality of frame images (see correspondence points 1 to 4 shown in FIG. 9).
Here, as shown in FIG. 10, it is desirable to specify and extract feature points in each image (see circles in FIG. 10), for example, about 100 feature points are extracted. Extract points.

Subsequently, the camera vector calculation unit 23c calculates the three-dimensional coordinates of the extracted feature points, and calculates the camera vector based on the three-dimensional coordinates. Specifically, the camera vector calculation unit 23c includes a sufficient number of feature positions that exist between successive frames, a position vector between moving cameras, a three-axis rotation vector of the camera, and each camera position and feature. In this embodiment, the relative values of various three-dimensional vectors, such as vectors connecting points, are calculated continuously. In this embodiment, for example, camera motion (camera position) is solved by solving an epipolar equation from the epipolar geometry of a 360-degree all-round image. And camera rotation).

Images 1 and 2 shown in FIG. 9 are images obtained by expanding a 360-degree all-round image into a Mercator, and assuming latitude φ and light θ, points on image 1 are (θ1, φ1), and points on image 2 are ( θ2, φ2). The spatial coordinates of each camera are z1 = (cos φ1 cos θ1, cos φ1 sin θ1, sin φ1), z2 = (cos φ2 cos θ2, cos φ2 sin θ2, sin φ2). When the camera movement vector is t and the camera rotation matrix is R, z1 ^T [t] × Rz2 = 0 is the epipolar equation.
By providing a sufficient number of feature points, t and R can be calculated as a solution by the method of least squares by linear algebra calculation. This calculation is applied to a plurality of corresponding frames.

Here, it is preferable to use a 360-degree all-round image as an image used for the calculation of the camera vector.
The image used for the camera vector calculation may be any image in principle, but a wide-angle image such as a 360-degree all-round image shown in FIG. 9 makes it easier to select many feature points. Therefore, in the present embodiment, the 360-degree all-round image is used for the CV calculation, whereby the tracking distance of the feature points can be increased, and a sufficiently large number of feature points can be selected. It is possible to select feature points that are convenient for each distance and short distance. In addition, when correcting the rotation vector, the calculation process can be easily performed by adding the polar rotation conversion process. As a result, a calculation result with higher accuracy can be obtained.
Note that FIG. 9 is a Mercator projection of a 360-degree spherical image obtained by combining images taken by one or a plurality of cameras in order to facilitate understanding of the processing in the CV data calculation unit 23. Although an expanded image is shown, in an actual CV calculation, it is not always necessary to be an expanded image by the Mercator projection.

Next, the error minimizing unit 23d calculates and calculates a plurality of vectors based on each feature point according to a plurality of calculation equations based on a plurality of camera positions corresponding to each frame and the number of feature points. Statistical processing is performed so that the distribution of the position of each feature point and the camera position is minimized to obtain a final vector. For example, the optimal solution of the least square method is estimated by the Levenberg-Marquardt method for multiple frame camera positions, camera rotations, and multiple feature points, and errors are converged to determine the camera position, camera rotation matrix, and feature point coordinates. .
Further, feature points having a large error distribution are deleted, and recalculation is performed based on other feature points, thereby improving the accuracy of computation at each feature point and camera position.
In this way, the position of the feature point and the camera vector can be obtained with high accuracy.

FIGS. 11 to 13 show examples of the three-dimensional coordinates of feature points and camera vectors obtained by CV calculation. FIG. 11 to FIG. 13 are explanatory diagrams showing a vector detection method by CV calculation according to the present embodiment, showing the relative positional relationship between the camera and the object obtained from a plurality of frame images acquired by the moving camera. FIG.
FIG. 11 shows the three-dimensional coordinates of the feature points 1 to 4 shown in the images 1 and 2 in FIG. 9 and the camera vector (X, Y, Z) that moves between the images 1 and 2.
12 and 13 show a sufficiently large number of feature points, the positions of the feature points obtained from the frame image, and the position of the moving camera. In the figure, a circle mark that continues in a straight line at the center of the graph is the camera position, and a circle mark located around the circle indicates the position and height of the feature point.

Here, as shown in FIG. 14, a plurality of CV calculations in the CV data calculation unit 23 are performed in accordance with the distance from the camera to the feature point in order to obtain more accurate three-dimensional information of the feature point and the camera position. The feature points are set and a plurality of calculations are repeated.
Specifically, the CV data calculation unit 23 automatically detects feature points that have image characteristics in the image and uses them for camera vector calculation when obtaining corresponding points of the feature points in each frame image. The unit calculation is repeated by paying attention to the two frame images Fn and Fn + m of the nth and n + mth, and the unit calculation with n and m appropriately set is repeated.
m is the frame interval, and the feature points are classified into a plurality of stages according to the distance from the camera to the feature point in the image. The distance from the camera to the feature point is set so that m becomes larger. It is set so that m is smaller as the distance to is shorter. This is because the change in position between images is less as the distance from the camera to the feature point is longer.

Then, while sufficiently overlapping the classification of the feature points by the m value, a plurality of stages of m are set, and as n progresses continuously with the progress of the image, the calculation proceeds continuously. Then, the overlap calculation is performed a plurality of times for the same feature point in each step of n and m.
In this way, by performing unit calculation focusing on the frame images Fn and Fn + m, a precise camera vector is calculated over a long time between each frame sampled every m frames (frames are dropped). However, in m frames (minimum unit frames) between the frame images Fn and Fn + m, a simple calculation that can be performed in a short time can be performed.

If there is no error in the precision camera vector calculation for every m frames, both ends of the camera vector of the m frames overlap with the Fn and Fn + m camera vectors that have been subjected to the high precision calculation. Accordingly, m minimum unit frames between Fn and Fn + m are obtained by a simple calculation, and both ends of the camera vector of the m minimum unit frames obtained by the simple calculation are Fn and Fn + m obtained by high precision calculation. The scale adjustment of m consecutive camera vectors can be made to match the camera vectors.
In this way, as n progresses continuously with the progress of the image, the scale adjustment is performed so that the error of each camera vector obtained by calculating a plurality of times for the same feature point is minimized, and integration is performed. A camera vector can be determined.
Accordingly, it is possible to speed up the arithmetic processing by combining simple arithmetic operations while obtaining a highly accurate camera vector having no error.

Here, there are various simple calculation methods depending on the accuracy. For example, when (1) many feature points of 100 or more are used in high-precision calculation, the minimum number of simple calculation is about ten. (2) Even if the number of the same feature points is the same as the feature points and camera positions, innumerable triangles are established there, and equations for that number are established. By reducing the number of equations, it can be simplified.
In this way, integration is performed by adjusting the scale so that the error of each feature point and camera position is minimized, distance calculation is performed, and feature points with large error distribution are deleted, and other features are added as necessary. By recalculating the points, the calculation accuracy at each feature point and camera position can be improved.

  In addition, by performing high-speed simple calculation in this way, camera vector real-time processing becomes possible. In real-time processing of camera vectors, calculation is performed with the minimum number of frames that can achieve the target accuracy and the minimum number of feature points that are automatically extracted, the approximate value of the camera vector is obtained and displayed in real time, and then the image is accumulated. Accordingly, the number of frames can be increased, the number of feature points can be increased, camera vector calculation with higher accuracy can be performed, and approximate values can be replaced with camera vector values with higher accuracy for display.

Furthermore, in this embodiment, it is possible to track three-dimensional information (three-dimensional shape) in order to obtain a more accurate camera vector.
Specifically, first, the three-dimensional information tracking unit 23e positions the camera vector obtained through the camera vector calculation unit 23c and the error minimization unit 23d as an approximate camera vector, and the image generated in the subsequent process Based on the three-dimensional information (three-dimensional shape) obtained as a part, partial three-dimensional information included in a plurality of frame images is continuously tracked between adjacent frames to automatically track the three-dimensional shape.
Then, from the tracking result of the three-dimensional information obtained by the three-dimensional information tracking unit 23e, a high-precision camera vector calculation unit obtains a more accurate camera vector.

The feature point extraction unit 23a and the feature point correspondence processing unit 23b described above automatically track feature points in a plurality of inter-frame images, but the number of feature point tracking frames is limited due to disappearance of feature points. May come. In addition, since the image is two-dimensional and the shape changes during tracking, there is a certain limit in tracking accuracy.
Therefore, the camera vector obtained by the feature point tracking is regarded as an approximate value, and the three-dimensional information (three-dimensional shape) obtained in the subsequent process is traced on each frame image, and a high-precision camera vector is obtained from the trajectory. Can do.
The tracking of 3D shapes is easy to obtain matching and correlation accuracy, and the 3D shape does not change in size and size depending on the frame image, so it can be tracked over many frames. The accuracy of the camera vector calculation can be improved. This is possible because the approximate camera vector is known by the camera vector calculation unit 23c and the three-dimensional shape is already known.

  When the camera vector is an approximate value, the error of 3D coordinates over a very large number of frames is small because there are few frames related to each frame by feature point tracking. The error of the three-dimensional shape when a part of the image is cut is relatively small, and the influence on the change and size of the shape is considerably small. For this reason, the comparison and tracking in the three-dimensional shape is extremely advantageous over the two-dimensional shape tracking. In tracking, when tracking with 2D shape, tracking changes in shape and size in multiple frames are unavoidable, so there are problems such as large errors and missing corresponding points. However, in tracking with a three-dimensional shape, there is very little change in shape, and in principle there is no change in size, so accurate tracking is possible.

Here, as the three-dimensional shape data to be tracked, there are, for example, a three-dimensional distribution shape of feature points, a polygon surface obtained from the three-dimensional distribution shape of feature points, and the like.
It is also possible to convert the obtained three-dimensional shape from a camera position into a two-dimensional image and track it as a two-dimensional image. Since the approximate value of the camera vector is known, projection conversion can be performed on a two-dimensional image from the camera viewpoint, and it is also possible to follow a change in the shape of the object due to movement of the camera viewpoint.

The camera vector obtained as described above can be displayed superimposed on the video image shot by the video camera device unit 11.
For example, as shown in FIG. 15, the video from the in-vehicle camera is developed in a plane, the corresponding points on the target plane in each frame image are automatically searched, and the corresponding points are combined to match the target plane. A combined image is generated and displayed in the same coordinate system.
Furthermore, the camera position and camera direction can be detected one after another in the common coordinate system, and the position, direction, and locus can be plotted.

And the CV data calculated | required as mentioned above will be correct | amended based on the position measurement data acquired in the position measurement apparatus part 11 of the mobile body measurement part 10. FIG.
As described above, when the CV value is obtained from the moving image, the scale also becomes a relative value. Therefore, the error is accumulated in the CV calculation data in the long section, and the error increases particularly in the measurement in the long distance section. .
On the other hand, in GPS, GPS satellites near the horizon are strongly affected by the ionosphere and generate errors due to multipath. Further, when the receivable satellite is switched near the horizon, the accuracy of the measurement value changes and a relative error that causes the measurement position to become discontinuous is generated.
However, since GPS is an absolute distance measurement, various errors occur at measurement points. However, errors do not accumulate, and the longer the distance measurement, the smaller the ratio of the error to the measurement distance.

FIG. 16 shows a comparison between error modes of CV data and GPS data.
As shown in the figure, if the movement of the moving body is a short section, the accumulated error is larger in the GPS data than in the CV data. Therefore, it is advantageous to correct the GPS data with the CV data in this section.
On the other hand, when the movement of the moving body reaches a long distance section, the accumulated error of CV data increases, but the error of GPS data does not accumulate, and as a result, the ratio of the error to the measured distance decreases. Therefore, it is advantageous to correct the CV data with the GPS data in the long distance section.

As described above, in the present invention, the CV data obtained by the CV calculation and the measurement data with a small absolute error such as GPS are combined and complemented and corrected to achieve higher accuracy without error accumulation. Position information is generated.
Hereinafter, a more specific embodiment of a high-precision CV arithmetic device that corrects CV data based on absolute coordinates will be described.

[CV calculation from 3D parallax with multiple cameras]
First, an embodiment of a high-precision CV calculation device 171 that performs CV calculation from a three-dimensional parallax shape by a plurality of cameras will be described with reference to FIG.
As shown in the figure, the high-accuracy CV arithmetic device 171 of this embodiment has almost the same configuration as the high-precision CV arithmetic device 1 having the basic configuration shown in FIG. A system camera coordinate three-dimensionalization unit 1724 is provided.

The moving body measurement unit 1710 includes a plurality of video camera devices 1771 for capturing a plurality of images with parallax.
The data generation unit 1720 includes a parallax coordinate coordinate three-dimensionalization unit 1724 that generates a three-dimensional shape in the camera coordinate system from a plurality of images with parallax of video images obtained from the plurality of video camera devices 1771 using inter-camera parallax.
In addition, the data generation unit 1720 includes a multi-CV data calculation unit 1723.

The multi-CV data calculation unit 1723 extracts a three-dimensional feature part from the three-dimensional shape in the camera coordinates generated by the parallax method coordinate three-dimensionalization unit 1724, tracks the three-dimensional feature part in an adjacent frame, By solving a triangle composed of a three-dimensional feature part and a camera position, it has a function of calculating (CV calculation) the relative movement trajectory of the camera installed on the moving body and its three-axis rotation angle.
Alternatively, the multi-CV calculation unit 1723 extracts a plurality of three-dimensional feature points from the three-dimensional shape in the camera coordinates generated by the parallax method coordinate three-dimensionalization unit 1724, and adjoins the three-dimensional feature points. It has the function of tracking the frame and finding the relative movement trajectory of the camera installed on the moving body and its three-axis rotation angle by calculation (CV calculation) by solving the triangle composed of the three-dimensional feature point and the camera position. .

The CV calculation described above uses a two-dimensional point in the image as a feature point. However, in the multi-CV calculation according to the present embodiment, the calculation can be performed by using a three-dimensional feature point whose three-dimensional shape is already known as a camera coordinate. Accuracy is increased, calculation speed is improved, and error accumulation is reduced.
In a two-dimensional feature point, the CV value can be obtained from 6 to 8 or more independent two-dimensional feature points, but in a three-dimensional feature point, 3 to 4 independent three-dimensional feature points. A CV value can be determined.

Further, if a part of the three-dimensional shape is used as a feature part as it is, the three-dimensional shape is regarded as a collection of feature points, and therefore, innumerable (maximum number of pixel) feature points are included in a part of the three-dimensional shape. If there is at least one three-dimensional feature part, CV calculation is possible. In practice, it is desirable to improve accuracy by using a large number of three-dimensional parts and treating them statistically.
As described above, in the present embodiment, one or more three-dimensional feature parts are included in the three-dimensional shape obtained by the parallax method, which is a known technique, while using the high-precision CV arithmetic device 1 shown in FIG. Alternatively, a method for obtaining a CV value by extracting three or more three-dimensional feature points and tracking them is adopted.
Note that the accuracy can be improved by extracting a large number of three-dimensional feature points or three-dimensional feature parts and performing statistical processing, similar to the CV calculation of two-dimensional feature points.

[CV data correction by absolute length CV value of multiple cameras]
Next, an embodiment of a high-precision CV arithmetic device 181 that performs CV data correction based on the absolute length CV values of a plurality of cameras will be described with reference to FIGS.
As shown in the figure, the high-accuracy CV arithmetic device 181 of this embodiment has substantially the same configuration as the high-accuracy CV arithmetic device 1 having the basic configuration shown in FIG. A plurality of video camera device units 1812 are provided as position measurement device units.

FIG. 18 shows a basic form of the high-precision CV arithmetic unit 181 of this embodiment.
The moving body measuring unit 1810 includes an omnidirectional video camera device unit 1811 that is fixed to the moving body and includes a omnidirectional video camera (see FIGS. 3 and 4) for photographing the periphery of the moving body as the moving body moves. .
In addition, the moving body measuring unit 1810 includes a plurality of video camera device units 1812 that are fixed to the moving body and for obtaining a three-dimensional position of the moving body.
The devices constituting the all-round video camera device unit 1811 and the multiple video camera device unit 1812 are installed such that the relationship between the three-dimensional position and the posture is known.

The data generation unit 1820 includes an omnidirectional video image recording unit 1821 that records the omnidirectional video image obtained by the omnidirectional video camera device unit 1811.
In addition, the data generation unit 1820 extracts a sufficiently large number of feature points from each frame image of the all-around video image recorded in the all-around video image recording unit 1821, tracks the feature points in adjacent frames, and features By solving a sufficiently large number of triangles composed of points and camera positions, a CV calculation unit 1823 is provided that calculates the relative movement trajectory of the all-around camera and the three-axis rotation angle of the all-around camera by calculation.

In addition, the data generation unit 1820 performs CV computation on the multiple video camera device unit 1812 in the same manner as the all-round video camera device unit 1811, extracts a sufficient number of feature points in each adjacent frame image, and Unlike the case of a single omnidirectional camera, the absolute distance is obtained by tracking and tracking not only within adjacent frames, but also with each adjacent frame of another camera with a known distance. A multi-CV calculation unit 1824 is provided that acquires accurate camera three-dimensional coordinates and a three-axis rotation posture.
The multi-CV calculation unit 1824 cuts out a plurality of three-dimensional feature parts from the three-dimensional shape in the camera coordinate system, tracks the three-dimensional feature parts in adjacent frames, and determines the three-dimensional feature parts and the camera position. Is obtained by calculating (CV calculation) the relative movement trajectory of the camera installed on the moving body and its three-axis rotation angle.

In the present embodiment, the three-dimensional coordinates and the CV value of the three-axis rotation angle of the high-precision representative camera constituting the multiple video camera device unit 1812 are compared with the CV value obtained from the all-round video camera device unit 1811. Thus, the CV value of the all-round video camera device unit 1811 is calibrated.
Furthermore, the three-dimensional shape around the camera can be easily and directly obtained from the plurality of video camera equipment units 1812 by a known technique. This three-dimensional shape is a three-dimensional shape in the camera coordinate system viewed from the camera position. For this reason, a moving camera cannot achieve its purpose because the coordinate system is not fixed. Therefore, it can be converted into a stationary coordinate system by combining with the CV value, and the accuracy can be further improved by combining known techniques.

The time axis matching unit 1830 includes a clock unit 1831 that supplies time data to the signal from the CV data calculation unit 1823 and the signal from the multi-CV calculation unit 1824.
The time axis alignment unit 1830 also associates and records the CV data with the time signal supplied from the clock unit 1831 and records the absolute coordinate data with the time signal supplied from the clock unit 1831. A time-synchronized multi-CV data recording unit 1833 for recording in association is provided.
In addition, the CV data includes a delay adjustment CV signal output unit 1834 that outputs a delay adjustment CV signal by correcting a time difference from the recording time in consideration of a delay due to a moving image processing time or the like.
The absolute coordinate data includes a delayed multi-CV signal output unit that corrects a time difference between the absolute coordinate signal output and the measurement time and outputs a delay adjustment absolute coordinate signal.

The CV correction unit 1840 generates a shift in coordinates due to the difference in installation position between the plurality of video camera device units 1812 and the all-round video camera device unit 1811 from the CV signal, and puts it at the position of the all-round video image device unit 1811. A device coordinate position correction unit 1841 for correcting the position of a plurality of video camera device units (multi-CV data device unit) 1812 is provided.
The CV correction unit 1840 includes a coordinate array shape comparison unit that detects a coordinate array shape difference by comparing the coordinate array shape of the delay adjustment multi-CV signal with the coordinate array shape of the delay adjustment CV signal.
Also, the CV correction unit 1840 determines the scale error in the three-axis direction from the coordinate array shape of the delay adjustment multi-CV signal, the coordinate array shape of the delay adjustment CV signal, and the difference in the coordinate array shape of the section with the corresponding relationship between them. A correction signal generation unit 1843 that generates a correction signal for correction is provided.

In addition, the CV correction unit 1840 includes a CV data scale correction unit that corrects the CV data so that the coordinate arrangement shape based on the CV data corresponding to the section matches the coordinate arrangement shape based on the multi-CV data with high relative accuracy. Prepare.
The CV correction unit 1840 includes a global scale correction unit that generates a series of position correction CV data by repeating the above process over the entire target section.
Furthermore, the CV correction unit 1840 includes a CV recalculation unit 1846 that performs the recalculation of high-accuracy CV data including the three-axis rotation again with the three-dimensional coordinates of the CV data that have been globally scale corrected.

  The high-precision CV data output unit 1850 includes an all-around video image output unit 1852 that outputs an all-around video image, and the entire scale is corrected in synchronization with each image frame of the all-around video image output, and further recalculated. And a CV data output unit 1851 for outputting the CV data signal subjected to the three-axis rotation correction.

19 to 20, two synchronized cameras constituting the multiple video camera device unit 1812 of this embodiment and an asynchronous all-round camera constituting the all-around video camera device unit 1811 are fixed, and the vehicle traveling method is fixed. An example in the case of attaching toward is shown.
According to this example, the object in the vehicle traveling direction is captured by two cameras, and the distance is obtained from the correspondence. The feature points in the video obtained from the two cameras form a polyhedron as shown in FIGS. 20 (a) and 20 (b).
By tracking feature points constituting the polyhedron, the camera position is obtained by CV calculation. The accuracy can be improved under the condition that the polyhedral shape is not deformed. In addition, there is an advantage that feature points can be tracked over a long distance by attaching a camera in the traveling direction of the vehicle (or the reverse direction of traveling). In order to further improve the accuracy, the camera direction can be set to all directions as shown in FIG.

At this time, the obtained CV value is not a relative value but an absolute length having a scale.
It should be noted that the feature points for the CV calculation relating to the two cameras and the feature points relating to the CV calculation relating to the all-around camera generally do not need to match.
As shown in FIG. 19, using two cameras as a unit is advantageous in terms of operability and the like, but tracking a common feature point with a plurality of cameras is also preferable because it contributes to improving accuracy. Therefore, two cameras are used as a set, and the feature points can be extracted and tracked independently, and the feature points may be common.
In this embodiment, it is important to use the all-around camera asynchronously, and only two video cameras are synchronized.
In addition, in the example shown in FIG. 20, one omnidirectional camera and two multiple cameras are used, but other than this, for example, by using a wide-angle image for multiple cameras, It is also possible to change the camera. In that case, the all-around camera can be omitted, and the moving object measuring unit 1810 of this embodiment can be configured with only a plurality of cameras.
FIG. 21 shows a variation of the combination of the omnidirectional camera and the plurality of cameras.

As described above, the CV value is obtained simultaneously with the three-dimensional coordinates of the image of the camera coordinates by a plurality of cameras. You can get it.
Note that the CV calculation and the parallax method of a plurality of cameras have conflicting properties.
That is, in the parallax method using a plurality of cameras, since the three-dimensional measurement is performed based on the parallax, the accuracy is improved when the focal length of the lens is long, that is, when the angle is narrow. However, the measurement range is narrow due to the narrow angle, and the three-dimensional coordinates in the camera coordinate system (not the stationary coordinate system) are obtained.

On the other hand, in the CV calculation method, the accuracy of the wide-angle lens with a short focal length is higher, and the wide-angle lens has a wider field of view, and the coordinates are obtained in a stationary coordinate system. Therefore, it is advantageous in terms of accuracy to use a wide-angle lens for CV calculation and a narrow-angle lens for a parallax method.
However, since the apparatus and the like are large, in this embodiment, a wide-angle lens is also used for a plurality of cameras so that the requirements for both the CV calculation method and the parallax method are satisfied simultaneously.

Of course, when it is not necessary to consider the fact that the device becomes large or the amount of data increases, all of the cameras are all-round cameras, wide angle lenses for CV calculation, and parallax calculation. By preparing any of the standard lenses for use and separating the functions, higher accuracy can be achieved.
For multiple cameras, use a monochrome camera to increase the accuracy of distance measurement in the camera coordinate system, and install a single color camera in addition to the 3D data acquired by the monochrome camera. By displaying, it is possible to observe the measured three-dimensional data as a color image (see FIG. 21D).

Furthermore, it is also possible to provide a parallax-type camera coordinate three-dimensionalization unit 2225 that generates a camera coordinate three-dimensional shape from a plurality of images with parallax from a plurality of cameras, as in the high-precision CV arithmetic device 221 shown in FIG.
The parallax type camera coordinate three-dimensionalization unit 2225 is the same as the parallax type camera coordinate three-dimensionalization unit 1724 of the high-precision CV calculation device 171 in FIG. 17, and the other components are the high-precision CV calculation shown in FIG. The configuration can be similar to that of the device 181.
As described above, the high-precision CV arithmetic device according to the present embodiment can adopt various variations and combinations including the combination of the omnidirectional camera and a plurality of cameras, taking into consideration the scale and cost of the device. Flexible response.

[CV data correction by GPS]
Next, with reference to FIG. 23, an embodiment of a high-precision CV arithmetic unit 231 that performs CV data correction by GPS will be described.
The high-precision CV arithmetic device 231 of this embodiment shown in the figure is characterized by including a GPS device unit 2312 as a position measurement device unit of the moving body measurement unit 2310.
The moving body measuring unit 2310 is fixed to the roof of a moving body such as an automobile or an aircraft, and moves around the moving body to acquire images of an all-around video camera or a normal video camera for photographing the periphery of the moving body. A video camera device unit 2311 and a GPS measurement device unit 2312 that is fixed to a moving body such as an automobile or an aircraft and that acquires absolute coordinates such as GPS and IMU for acquiring a three-dimensional position of the moving body at a latitude and longitude altitude are provided. .
The devices constituting the all-round video camera device unit 2311 and the GPS measurement device unit 2312 are installed with a known relationship between the three-dimensional position and orientation so that the positional relationship can be understood from each other.

The data generation unit 2320 includes an omnidirectional video image recording unit 2321 that records the omnidirectional video image obtained by the omnidirectional video camera device unit 2311.
In addition, the data generation unit 2320 extracts a sufficiently large number of feature points from each frame image of the all-around video image recorded in the all-around video image recording unit 2321, tracks the feature points in adjacent frames, and features A CV data acquisition unit (CV data) for calculating a relative movement locus of a camera installed on a moving body and a three-axis rotation angle of the camera by calculation (CV calculation) by solving a sufficiently large number of triangles composed of points and camera positions An arithmetic unit) 2323 and an absolute coordinate data acquisition unit 2322 for acquiring three-dimensional position data by a GPS measurement device unit 2310 such as GPS or IMU.

The time axis matching unit 2330 includes a clock unit 2331 that supplies time data with high relative accuracy generated from the same clock to the signal output from the CV data acquisition unit 2323 and the signal output from the absolute coordinate data acquisition unit 2322. Prepare.
In addition, the time axis alignment unit 2330 associates and records the CV data with the time signal supplied from the clock unit 2331 and records absolute coordinate data with the time signal supplied from the clock unit 2331. A time-synchronized absolute coordinate recording unit 2333 for recording in association is provided.
In addition, regarding CV data, in consideration of a certain time delay derived from image processing time of a moving image that exists in principle in some cameras, a delay for correcting a deviation from a recording time and outputting a CV signal An adjustment CV signal output unit 2334 is provided.
Further, the absolute coordinate data includes a delay adjustment absolute coordinate signal output unit 2335 that corrects a delay time between the absolute coordinate signal output time and the measurement time and outputs a signal.
Note that the delay adjustment CV signal and the delay adjustment absolute coordinate signal are matched to each other regardless of the delay time of each time.

The CV correction unit 2340 generates, from the CV signal, a coordinate and a three-axis rotation shift caused by a difference in installation position between the GPS measurement device unit 2312 and the all-around video camera device unit 2311, and the all-around video image device unit 2311. Is provided with a coordinate rotation correction unit 2341 that corrects the position of the GPS measuring device to output a delayed coordinate rotation correction signal.
Also, the CV correction unit 2340 compares the coordinate arrangement shape of the delay adjustment absolute coordinate signal with high absolute accuracy with the coordinate arrangement shape of the CV value with high relative accuracy, and the delay adjustment absolute coordinate signal is generated by the coordinate rotation correction signal. A coordinate array shape comparison unit 2342 is provided for comparing the coordinate array shape with the coordinate array shape of the delay adjustment CV signal.

In addition, the CV correction unit 2340 detects a GPS array shape point greatly deviating from the coordinate array shape of the delay adjustment CV signal with high relative accuracy by comparing the coordinate array shapes of the two coordinate arrays as the accuracy change point of the absolute coordinate on the GPS side. An absolute coordinate accuracy change point detection unit 2343 is provided.
Also, the CV correction unit 2340 may improve the absolute accuracy when the absolute coordinate data such as GPS is changed from the delay adjustment absolute coordinate signal, for example, at the satellite switching point. However, since the relative error is significantly reduced, the section where the relative accuracy is reduced is deleted by comparing with the CV value, or the section where the GPS signal in the basement or the building cannot be received is deleted. Thus, a high-accuracy relative value section selection unit 2344 that eliminates the absolute coordinate accuracy change point and selects a plurality of sections having high relative accuracy is provided.

Also, the CV correction unit 2340 uses the section of the point corresponding to the high-precision relative value section selection unit based on the absolute coordinate data and the CV data obtained by calculation from the moving image for the CV data including the scale error due to the accumulation of the calculation error. Corresponding points and sections using time as a parameter, or the sampling frequency of CV data is generally higher than the sampling frequency of GPS, so there is a measured value at the coincidence point of the start point and end point divided by the above section If not, a measurement section corresponding unit 2345 is provided for finding and matching the matching points by interpolation.
In addition, the CV correction unit 2340 corrects CV data including a scale error caused by accumulation of errors by CV calculation, using the distance between the absolute coordinates of the start point and the end point of both sections having the above correspondence as the section distance. A correction signal generation unit 2346 that generates a correction signal for performing the correction.
Also, the CV correction unit 2340 corrects the CV data by correcting or adjusting the CV data corresponding to the above section by matching or adjusting the section distance data based on the absolute coordinate data with high relative accuracy. Part 2347 is provided.
Furthermore, the CV correction unit 2340 includes a global scale correction unit 2348 that generates a series of high-precision CV data by repeating the above process a plurality of times over the entire target section.

  The high-precision CV data output unit 2350 is an all-around video output unit 2352 that outputs an all-around video image, and a CV that outputs a CV data signal that is globally scaled in synchronization with each image frame of the all-around video image output. A data output unit 2351 is provided.

The high-precision CV calculation processing by the high-precision CV calculation device 231 of the present embodiment configured as described above is performed as follows.
First, a GPS device and a camera device for moving image acquisition are loaded on the same vehicle or the like. At this time, the three-dimensional positional relationship between the GPS device and the camera device is known by actual measurement.
The GPS measurement point and the camera shooting point can be acquired in relation to each other by time, and the GPS measurement data and the camera CV data are expressed and recorded as a function of the same clock time.

For data acquired from GPS, set multiple appropriate sections from the data corresponding to the time and distance obtained from GPS, and select only multiple sections with high GPS relative accuracy from the above sections. Thus, the time and movement distance data constituting the section are acquired.
Next, regarding the CV data obtained from the moving image, the CV data obtained from the image is made to correspond to the measurement data in the same time section of the same place corresponding to the three-dimensional distance data by GPS. When the measured value does not exist at the coincidence point of the start point and the end point time divided by the above section, the coincidence point is obtained by interpolation, and the distance obtained from the GPS data of the start point and end point of the above section is obtained from the CV data. The CV data is corrected by replacing the obtained distance. By repeating the above operation in a plurality of selected sections, it is possible to generate a series of highly accurate CV data corrected by GPS data.

[GPS data correction]
Next, with reference to FIG. 24, an embodiment of a high-precision CV arithmetic unit 241 that performs GPS data correction will be described.
The high-precision CV calculation device 241 of the present embodiment shown in the figure has the same basic configuration as the high-precision CV calculation device 231 shown in FIG. 23, and further, correction means for correcting an error in GPS measurement data It is characterized by having.
In FIG. 24, elements indicated by chain lines are portions common to the above-described embodiment, and redundant description is omitted as appropriate. The same applies to the following embodiments.

As shown in FIG. 24, in the high-precision CV calculation device 241 of the present embodiment, the CV correction unit 2440 includes a distribution statistics 2441, a temporary true value calculation unit 2442, a temporary true value correction data acquisition unit 2443, An absolute coordinate correction unit 2444 is provided.
The distribution statistics 2441 statistically processes the distribution of the absolute accuracy change points detected by the absolute accuracy change point detection unit (2343 in FIG. 23) due to GPS satellite switching points or the like.
The temporary true value calculation unit 2442 performs statistical processing on the distribution of absolute accuracy change points such as GPS detected by the absolute accuracy change point detection unit, and obtains a temporary true value of GPS. Specifically, the provisional true value calculation unit 2442 obtains the centroid of the distribution by using the least square method or the like for the statistical center of the distribution of the absolute accuracy change points by GPS, and obtains the provisional true value.

With the provisional true value correction data acquisition unit 2443 and the provisional true value, the absolute error correction value of the section of the start point and end point of the absolute accuracy change point caused by the satellite switching point or the like is acquired.
The absolute coordinate correction unit 2444 first obtains a high-accuracy GPS absolute coordinate array in which the absolute coordinate array is corrected based on the correction values of the above-described sections.
The absolute coordinate array corrected by the absolute coordinate correction unit 2444 is output to the high-precision CV data output unit 2450.

The high-precision CV data output unit 2450 of this embodiment further includes an absolute coordinate addition CV data integration unit 2451 and an absolute coordinate CV data output unit 2452 in addition to the high-precision CV data output unit 2350 of FIG.
The absolute coordinate addition CV data integration unit 2451 gives absolute coordinates to the scale-corrected CV data output from the CV data output unit (2351 in FIG. 23) using the absolute coordinate data obtained from the GPS data as a reference. .
The all-around video image is output from the all-around video output unit (2352 in FIG. 23), and the absolute coordinate CV data output unit 2452 outputs CV data in which absolute coordinates are added to each image frame.

The correction processing of GPS data by the high-precision CV arithmetic unit 241 of the present embodiment configured as described above is performed as follows.
First, the GPS device and the camera device for moving image acquisition are loaded on the same vehicle or the like, and the three-dimensional positional relationship and the three-dimensional three-axis rotation relationship between the GPS device and the camera device are known by actual measurement.
Next, the GPS measurement point and the camera photographing point can be acquired in relation to each other by time, and the GPS measurement data and the camera CV data are expressed and recorded as a function of the time of the same clock.

Then, the GPS data and CV data are compared at the same time, and the absolute error due to GPS changes at the point where the CV data is continuously arranged with respect to the time and the GPS acquisition data becomes discontinuous. As a result, the data is detected as an error change point, and a sufficiently large number of error change point data is acquired from sufficiently long time data to obtain a sufficiently large distribution of error change points.
Statistical processing is performed from the above distribution by operations such as least squares, and the provisional correction true value of GPS is obtained. Adopted as a GPS correction value.
Furthermore, by replacing some GPS data in the section starting from the above error change point with CV data with high linearity, the absolute accuracy of GPS is improved, and the GPS data with the increased absolute accuracy makes absolute CV values Coordinates can be given.

[3-axis rotation accumulated error correction by IMU]
Next, with reference to FIG. 25, an embodiment of a high-precision CV arithmetic unit 251 that performs three-axis rotation cumulative error correction by the IMU will be described.
The high-precision CV calculation device 251 of this embodiment shown in the figure has the same basic configuration as the high-precision CV calculation device 231 shown in FIG. 23, and further, a correction for correcting a 3-axis rotation accumulated error by the IMU. It is characterized by providing a means.

Specifically, first, the moving body measurement unit 2510 includes an IMU measurement device unit 2511 that can measure an absolute three-axis rotation posture such as an IMU.
The data generation unit 2520 includes an absolute three-axis rotation posture data acquisition unit 2521 that acquires data of an absolute three-axis rotation posture by the IMU measurement device unit 2511.
The time axis alignment unit 2530 includes a time-synchronized triaxial rotational attitude recording unit 2531 that records absolute triaxial rotational attitude data in association with a time signal newly supplied from the clock unit (2331 in FIG. 23).
In addition, the time axis alignment unit 2530 corrects a deviation from the recording time and outputs a signal by correcting a deviation from the recording time in consideration of a delay derived from the processing time of the device unit with respect to the absolute triaxial rotational posture data. An output unit 2532 is provided.

The CV correction unit 2540 compares the three-axis rotation posture signal from the delay adjustment CV signal output unit (2334 in FIG. 23) with the absolute rotation three-axis posture signal from the delay adjustment three-axis rotation posture output unit 2532. A posture comparison unit 2541 is provided.
The CV correction unit 2540 includes a triaxial rotation threshold value comparison unit 2542 that compares the difference between the triaxial rotation posture signal and the absolute rotation triaxial posture signal with a predetermined threshold value. A triaxial rotation correction section setting unit 2543 is provided that specifies a correction section when the threshold value is exceeded.
Further, the CV correction unit 2540 includes a triaxial rotation correction signal generation unit 2544 that generates a correction value in the correction section.

The high-precision CV data output unit 2550 includes a triaxial rotation correction CV data output unit 2551 and a triaxial rotation absolute coordinate addition CV data output unit 2552.
The triaxial rotation absolute coordinate addition CV data output unit 2552 corresponds to the CV data from the delay adjustment CV signal output unit (34 in FIG. 23) or the CV data from the CV data output unit (2351 in FIG. 23). Or, for the CV data from the absolute coordinate CV data output unit (2452 in FIG. 24), triaxial rotation correction and absolute coordinates are added to the triaxial rotation attitude data from the absolute coordinate CV data output unit to each image frame. Output.
In this case, the corrected triaxial rotation data becomes absolute coordinates.
The triaxial rotation correction CV data output unit 2551 outputs triaxial rotation correction CV data obtained by adding only the triaxial rotation correction to the output of the CV data output unit.

A more specific example of the high-precision CV arithmetic unit 251 including the GPS and the IMU according to the present embodiment will be described with reference to FIGS.
As shown in FIG. 26, the omnidirectional camera, GPS and IMU, and two wide-angle cameras are fixed and fixed on the roof of a small vehicle. As shown in FIG. 27 (a), each device is connected to an information processing device (PC) mounted on a vehicle and performs arithmetic processing in real time.

Here, a Ladybug made by PGR Co., which can shoot the entire 360 ° circumference is used as the all-around camera.
The IMU preferably has three axes, but a single axis or axis is sufficient.
One GPS and one IMU are attached to each vehicle. The three-dimensional positional relationship between the GPS, the IMU, and the two cameras and the relationship between the vehicle and each device are obtained by accurately measuring in advance (see FIG. 26).
In addition, by setting it as the moving body measurement part which integrated GPS and the camera, it can attach or detach and detach while integrated, and can also be easily replaced with other vehicles.

By calculating the absolute coordinates of the vehicle by GPS and acquiring IMU data, the three-axis absolute rotation amount of the vehicle can be obtained by calculation.
However, since the accuracy of each data by GPS and IMU is low, it corrects it with the CV value by two wide angle cameras.
Further, the GPS can obtain a highly accurate positional relationship by obtaining the stationary position over a long period of time.
The relationship between the omnidirectional camera, GPS data, IMU data, and each frame image of the images acquired from the two wide-angle cameras is related to the time by a common clock as shown in FIGS. . For example, the relationship between the GPS data, the IMU data, and the frame images of the images acquired from the all-around camera and the wide-angle camera are recorded in complete synchronization with the time generated from the PC clock.

The absolute coordinates are acquired by GPC, the absolute length is acquired by two wide-angle cameras, and the absolute triaxial rotation angle is acquired from the IMU.
Then, the scale calibration of the CV value is performed with the GPS data by the high-precision CV arithmetic device 231 shown in FIG.
In general, GPS can acquire absolute coordinates, but also includes errors regarding absolute errors. Therefore, first, the high-accuracy feature of the relative scale of the GPS is used, and the high-accuracy GPS section distance is obtained by focusing on only the GPS relative scale.
There are points and sections where the absolute error accuracy of GPS suddenly changes due to satellite switching, multipath, etc., and the relative accuracy is significantly reduced there, so the GPS data becomes discontinuous. A point or section where the relative error has decreased due to the change in the absolute error is defined as an “error change point” (see FIG. 29).

First, the error change point is deleted from the relative error correction interval in the interval including the error accuracy change point. By the above section selection, it is possible to select only a highly accurate part as GPS relative coordinates.
Therefore, GPS data corresponding to a certain time interval of GPS and CV data are compared with a section having a high relative error, and the relative distance is reflected in the CV data to generate a highly accurate CV value. Thereby, accumulation of errors peculiar to CV data can be corrected.

The CV data automatically extracts feature points in the all-round video image, tracks them to a plurality of adjacent frames, and analyzes the position of the moving camera and a plurality of triangles formed by each feature point, so that the three-dimensional camera A position and a triaxial rotational attitude are required. The coordinates and scale of CV data are relative coordinates.
Here, although the GPS data includes an absolute error, here, the relative value of the CV data is corrected by focusing only on the relative value. As shown in FIG. 30, the accumulated error of the CV data can be corrected by the relative value correction, and the linearity of the scale is corrected better.
For sections that are not scale-corrected due to deletion of sections from GPS data, the accumulated error in adjacent sections is considered to be very small. Do.

Next, the absolute length of the CV value of the all-around camera is corrected by the CV value (see FIG. 18) by the two wide-angle cameras, and the scale is calibrated.
Further, the GPS discontinuity is corrected using the linearity of the CV value.
Hereinafter, a method of giving absolute coordinate values to CV data based on the absolute coordinates while correcting GPS data with CV data will be described with reference to FIGS. 29 to 31.
The GPS is output including an absolute error, but if the error accuracy changes suddenly due to satellite switching or the like, the GPS data becomes discontinuous and the relative error increases. At this time, since the position of CV data is detected by a method completely different from that of GPS, data is continuously acquired without being affected by satellite switching or the like.

It can be said that the relative error increases and the absolute error changes due to the discontinuity of the GPS, but the absolute error does not always increase at this time, and the absolute error may decrease. Therefore, by extracting the error change point, and detecting the GPS discontinuity using the high sampling characteristics of the CV value and the high accuracy of the relative value of the short interval, the GPS is linearly corrected. (See FIGS. 29 to 31).
Thus, by replacing the relative coordinates of the detected discontinuous points with the CV data, the continuity of GPS can be recovered, and the accuracy of the relative coordinates is improved.

However, the relative accuracy is improved only by the above processing, but the absolute accuracy is not necessarily improved.
Therefore, by statistically processing the movement amount of the error change point, an average movement amount can be obtained, thereby improving the absolute accuracy.
As shown in FIG. 32, since it can be assumed that an error occurs around the correct coordinates, a provisional corrected true value can be obtained by statistical processing.

Specifically, first, as shown in FIG. 32A, the discontinuous portion of the GPS data is extracted while moving, instead of fixed point observation. It is considered that the satellite reception is switched at the discontinuous portion, and a number of differences “δns” are obtained from the GPS between the two points of the predicted value and the observed value of the discontinuous portion, and FIG. A distribution map as shown is generated. “+” Shown in the figure indicates each observation point.
Here, “δns” is a difference value vector between an expected value and an actual measurement value at a discontinuous point, as shown in FIG. Further, as shown in FIG. 32 (a), the predicted value means the predicted value of the predicted point obtained when the continuous observation data up to the previous time including the error is continued with the error as it is.

Next, the center of gravity of the distribution map is obtained by the method of least squares. In FIG. 32 (b), the distribution is shown two-dimensionally, but in reality it is a three-dimensional distribution.
Specifically, the distance from the true value of the unknown (■ shown in FIG. 32 (b)) to the observation point difference value “+” is δns, and the sum of the squares (** 2) is Σδns ** 2. The minimum value is obtained as a true value. Σns ** 2 = 0 means that the sum of squares of the distance from the true value to the observation point difference value becomes zero.
If the true value δ0 is obtained, the formula for obtaining the true value from the actually measured value is as shown in FIG.
Formula: [G0Tn = GmTn + (δns−δ0s)]
(Δns−δ0s) indicates a true value difference distance that is a distance from the obtained true value to the observation execution value. By adding this true value difference distance to the measured raw data GmTn, the effective value G0Tn to be obtained is obtained.
As described above, the absolute coordinates of GPS with high accuracy can be obtained by substituting and replacing the temporary correction true values at the respective error change points.

Then, as shown in FIG. 33, CV data with high accuracy is obtained by correcting CV data at all points corresponding to GPS sampling points and CV frames by using GPS data with increased absolute accuracy. Data can be acquired.
Further, the IMU deletes the accumulation of errors of the triaxial rotation angle of the CV data by the absolute triaxial rotation angle, and acquires the absolute triaxial rotation angle from the CV data.
Since each sampling of the image frame and the GPS data is performed independently, it is preferable to acquire the absolute three-axis rotation angle by deleting the error accumulation at the timing at which each sampling is very close.
As described above, highly accurate CV calculation can be realized.

[Single axis rotation accumulated error correction by vertical direction detection]
Next, with reference to FIG. 34, an embodiment of a high-accuracy CV arithmetic unit 341 that performs one-axis rotation accumulated error correction by vertical direction detection will be described.
The high-precision CV arithmetic unit 341 of the present embodiment shown in the figure is characterized by comprising correction means for detecting the vertical direction from the acquired video image and thereby correcting the CV data.
Specifically, in the high-precision CV arithmetic device 341 of the present embodiment, the data generation unit 3420 includes a general bipolar plane expansion unit 3422, a general equator plane expansion unit 3423, and a vertical direction detection unit 3424.

As described above, the all-round image is generally recorded in spherical transformation coordinates (see FIG. 5).
Therefore, the schematic bipolar plane expansion unit 3422 performs plane conversion (perspective conversion) in the generally upward direction and the downward direction from the omnidirectional video image recorded in the omnidirectional video image recording unit 3421 (see FIGS. 35 and 36). .
In addition, the schematic equator plane development unit 3423 performs plane conversion (perspective conversion) on all-round video images in a plurality of equatorial plane directions (see FIG. 37).
The vertical direction detection unit 3424 detects the vertical direction by image processing from the image plane-converted into the directly above / downward direction and the equatorial plane direction.
The detected vertical direction is output / reflected to the CV data acquisition unit 3425, and the vertical direction is added to the CV calculation.

As shown in FIGS. 35 to 37, the plane-converted image generally includes a lot of vertical components if an artificial structure is included. Therefore, paying attention to the vertical components, the plane-converted image is included in the plane-converted video. The vertical line of the artificial structure to be detected is detected as a group of radiation intersecting at the pole (zenith) on the bipolar plane, and as a vertical line group on the equator plane.
For example, if the images of both polar planes are superimposed, if the axis of plane conversion matches the vertical in the image, the poles created by the respective ship groups in the vertical direction match. If they do not match, there are two poles.
Therefore, by moving the plane conversion axis so that the poles formed by the intersections of the vertical lines of the plane conversion planes coincide with each other, the conversion axis and the vertical direction in the video can be matched.

Further, on the approximate equator conversion plane, a group of vertical lines included in the video is expressed as parallel perpendicular lines in the image. If the conversion axis matches the vertical ship group in the video, all perpendicular lines become parallel vertical lines. If they do not match, the parallelism will be lost and only a part of the perpendicular will remain.
Therefore, the conversion axis in which the vertical line group increases most indicates the vertical direction.

Hough transform is effective for detecting a straight line portion in an image. By Hough transform, linear components can be expressed by equations.
In the bipolar plane, if the intercept of each straight line from the coordinate axis is obtained from each straight line that can be expressed by an equation, it can be determined how much it deviates from the coordinate origin. By obtaining the coordinate origin such that each straight line passes through the origin, the conversion axis and the vertical component in the image can be matched.
Further, in the equator conversion plane, the position where the vertical in the video is parallel / vertical may be obtained by obtaining the transformation axis in which many straight lines of the linear component obtained by the Hough transformation are vertical.
As described above, the vertical direction obtained from the bipolar plane transformation and the vertical direction obtained from the equator plane transformation are in principle the same. Accordingly, both may be used, but only one of them may be used. It is preferable to use both in order to cope with various situations of the image.

[High-precision CV calculation using aerial and ground images]
Next, an embodiment of a high-precision CV calculation device 381 that performs high-precision CV calculation using an aerial image and a ground image will be described with reference to FIG.
The high-accuracy CV arithmetic device 381 of this embodiment shown in the figure is characterized by comprising correction means for detecting the vertical direction from the acquired video image and thereby correcting the CV data.
Specifically, the high-accuracy CV calculation device 381 of the present embodiment mainly acquires the shape of the object and the adjacent shape by the CV value in the low-scale CV calculation, and mainly uses the spatial arrangement of the object in the high-scale CV calculation. By acquiring the shape as a CV value and combining the two, high-precision CV data consistent with the two is acquired.

As shown in FIG. 38, the moving body measurement unit 3810 includes two video video equipment units 3811 and 3812 of a low scale image and a high scale image.
The low-scale all-round video image equipment unit 3811 is composed of a video image equipment mounted on a vehicle for photographing a ground object with a wide-angle image for the purpose of CV calculation.
The high-scale all-round video image equipment unit 3812 is composed of a video image equipment mounted on an aircraft or the like for aerial photography that similarly captures an object on the ground with a wide-angle image for the purpose of CV calculation. .
The low-scale all-around video image equipment unit 3811 and the high-scale all-around video image equipment unit 3812 are composed of two different types of video equipment units, each of which may perform shooting, or the same video equipment unit. It may be used at different times for shooting on the ground and in the air.

As shown in FIG. 38, the data generation unit 3820 is configured to process low-scale and high-scale video images corresponding to the low-scale all-around video image device unit 3811 and the high-scale all-around video image device unit 3812. It has become.
The low-scale video image recording unit 3821a acquires the low-scale video signal from the low-scale all-around video image device unit 3811 and records the video.
The high scale video image recording unit 3821b acquires a high scale video signal from the high scale all-around video image equipment unit 3812 and records the video.

The common feature point detection unit 3823 receives a low-scale all-around video image output from the low-scale all-around video image unit 3811 and a high-scale all-around video image output from the high-scale all-around video image unit 3812. The comparison is performed to find a common feature point included in the two images, take the corresponding point, and detect the common feature point.
The common feature point weight setting unit 3824 gives a weight to each common feature point by a weight function for later CV calculation.

The low-scale feature point detection unit 3822a detects low-scale feature points from the low-scale all-around video image.
The low-scale inter-frame tracking unit 3825a tracks low-scale feature points in adjacent frames.
The high-scale feature point detection unit 3822b detects high-scale feature points from the high-scale all-around video image.
The high-scale inter-frame tracking unit 3825b tracks high-scale feature points in adjacent frames.

The CV calculation unit 3826 generates a low-scale ground video and a high-scale aerial video from a low-scale feature point, a high-scale feature point, and a weight common feature point. Perform CV calculation by integrating all together.
The CV data acquisition unit 3827 acquires and outputs highly accurate CV calculation data obtained by the CV calculation unit 3826. The CV calculation data is recorded in the time synchronization CV data recording unit 3828 as necessary.

As described above, in the present embodiment, a plurality of images with different scales showing the destination point are prepared, and each of them is subjected to CV calculation, and a large-scale image (an image with a large scale, for example, an aerial image) By calibrating the entire small scale image (a small scale image, for example, a ground image) with the CV value, it is possible to eliminate accumulation of errors in the CV value of the small scale image.
In aerial images, if the altitude is sufficiently high relative to the measurement range, the camera position can be regarded as infinite. This is because the 2D map is treated as an aerial image from altitude infinite. It shows that a two-dimensional map can be used instead of a captured image.

[High-precision CV calculation with map]
Next, with reference to FIG. 39, an embodiment of a high-precision CV arithmetic device 391 having a map will be described.
The high-precision CV arithmetic device 391 of the present embodiment shown in the figure includes a two-dimensional map unit 3921 in the data generation unit 3920.
The two-dimensional map unit 3921 is a map in which the two-dimensional coordinates of the described points are given as known.

The data generation unit 3920 includes a CV data acquisition unit 3922 including a feature point extraction unit 3922a, a feature point tracking unit 3922b, and a CV calculation unit 3922c.
The feature point extraction unit 3922a extracts a feature point serving as a clue to tracking from the video of the all-around video video recording unit.
The feature point tracking unit 3922b tracks feature points in a plurality of adjacent image frames.
The CV calculation unit 3922c analyzes the corresponding points of the plurality of feature points and the triangle formed by the camera position coordinates, and obtains the three-dimensional position and the three-axis rotation posture of the camera.
The CV data acquisition part which acquires CV data is comprised.

The time axis alignment unit 3930 includes a common point detection unit 3931.
The common point detection unit 3931 detects a common point portion in the all-around video image corresponding to the point two-dimensional coordinates of the two-dimensional map unit. This common point detection process may be automatic or manual.
Here, the common point portion may be a common “point”, but a shape having a shape is easier to detect. Therefore, the common point portion is formed by a road shape, an intersection shape, a three-dimensional structure, or a part of a plurality of structures. A three-dimensional space is advantageous. Therefore, it is suitable to detect a common shape of the image and the map and use a plurality of point groups constituting the shape as a common point.

The CV correction unit 3940 includes a common point three-dimensional coordinate calculation unit 3941, a common point two-dimensional coordinate detection unit 3944, a two-dimensional plane coordinate conversion unit 3944, a common point two-dimensional shape comparison unit 3944, a common point region CV value correction unit 3945, A CV correction calculation unit 3946 is provided.
The common point three-dimensional coordinate detection unit 3941 obtains the three-dimensional coordinates of the common point detected by the common point detection unit 3931 from the CV calculation data by calculation.
A two-dimensional plane coordinate conversion unit (ortho conversion unit) 3944 projects and outputs a plurality of common point two-dimensional coordinates by projecting the three-dimensional coordinates of the common points obtained from the all-around video image onto the two-dimensional plane.

The common point two-dimensional coordinate detection unit 3942 detects the two-dimensional coordinates on the two-dimensional map of the common point detected by the common point detection unit 3931.
The common point two-dimensional shape comparison unit 3944 uses the two-dimensional common point two-dimensional coordinates (comparison signal 1) obtained from the data of the two-dimensional map unit 3921 and the three-dimensional coordinates of the common points obtained from the all-around video image. A plurality of common point two-dimensional coordinates (comparison signal 2) obtained by projecting on a plane are compared.
The common point region CV value correction unit 3945 detects the difference between the two-dimensional coordinates to be compared, determines the position coordinates of the comparison signals of the two so that the difference is minimized, and further, another point with the same ratio. The coordinates of the CV value are corrected by correcting the coordinates in the coordinate axis direction, distributing the correction amount of the common point in a distributed manner, and distributing the influence of the correction of the common point in the periphery. Replace with

The CV correction calculation unit 3946 fixes the camera position coordinate data that matches the map at the common point portion, performs the CV calculation again using the common point and the feature point tracking result, and obtains more accurate three-axis rotation data. get.
The corrected CV value whose accuracy has been improved by the above correction / recalculation is output by the CV data output unit 3951 of the high-precision CV data output unit 3950.
39, each element is classified according to the basic configuration of the high-precision CV arithmetic device 1 shown in FIGS. 1 and 2, but the high-precision CV arithmetic device 401 shown in FIG. As described above, the elements 4021 to 4027 according to the present embodiment can be regarded as high-precision processing up to the CV data acquisition unit 4028. In this case, all the elements can be positioned as the data generation unit 4020. Is possible.

[High-precision CV calculation using known 3D points]
Next, with reference to FIG. 41, an embodiment of a high-precision CV calculation device 411 that performs high-precision CV calculation using known three-dimensional points will be described.
If you can find a known 3D point instead of the camera position in the acquired image, you can calibrate using the 3D coordinates in the image without moving the measuring instrument to the known point. In the present embodiment, high-precision CV calculation is performed using such known three-dimensional points.

Specifically, the high-precision CV calculation device 411 of the present embodiment includes a map unit 4121 having known three-dimensional point information.
The map unit 4121 includes a three-dimensional map 4121a in which two-dimensional coordinates are given as known, a known three-dimensional point description map 4121b in which a plurality of known three-dimensional coordinate points are described together with coordinates on the two-dimensional map, Although it does not form, the map part by any one of the three-dimensional spot list | wrist 4121c which marked the three-dimensional coordinate of the known several points, or its combination is comprised.
This corresponds to three-dimensional data obtained by measuring GPS at a fixed point for a long time.
This also corresponds to the case where an accurate three-dimensional map exists in advance.

The common point detection unit 4123a detects a common point portion in a known three-dimensional point of the map unit 4121 and an image frame output from the corresponding all-around video image recording unit 4122a.
The common feature point tracking unit 4123b uses the common point portion as a common feature point to track by finding corresponding points in the plurality of image frames of the all-around video image recording unit 4122a.
The common feature point weighting unit 4123c distinguishes the common feature points from the general feature points and performs weighting for determining the priority order during the calculation. Here, how much priority is given to the common feature points over the general feature points is determined.

The CV calculation unit 4122d analyzes a plurality of triangles formed by the tracking result of the plurality of weighted common feature points, the tracking result of the general feature points by the general feature point tracking unit 4122c, and the three-dimensional position coordinates of the camera. Then, the three-dimensional position of the camera and the three-axis rotation posture are obtained by calculation.
The CV value has a three-dimensional position coordinate and a three-axis rotation posture. Here, attention is paid to the three-dimensional coordinates of the general feature point and the common feature point obtained simultaneously with the CV value. That is, here, acquiring the CV value is treated as the same meaning as acquiring the three-dimensional coordinates of the feature point.
The common point three-dimensional coordinate detection unit 4124a acquires the three-dimensional coordinates of the common feature point that is output simultaneously with the CV value from the CV calculation unit 4122d. In principle, since the CV value is obtained simultaneously with the three-dimensional coordinates of the feature point, the CV value and the three-dimensional output of the feature point can be output simultaneously.

The common point three-dimensional coordinate comparison unit 4124b compares the common point three-dimensional coordinates obtained as known from the data of the map unit 4121 with the three-dimensional coordinates of the common feature points.
The three-dimensional difference detection unit 4124c detects a difference between both three-dimensional coordinates to be compared.
The general feature point three-dimensionalization unit 4125 acquires the three-dimensional coordinates of the general feature points output from the CV calculation unit 4122d.
The image two-dimensional difference detection unit 4126 projects the three-dimensional coordinates of the general feature points acquired by the general feature point three-dimensionalization unit 4125 onto the video plane in each frame output from the all-around video image recording unit 4122a. A two-dimensional difference between the two-dimensional coordinates and the two-dimensional coordinates of the general feature points in each frame output from the all-around video image recording unit 4122a on the image plane of the all-around video image is detected.

Then, the CV calculation parameters are changed by the CV correction calculation unit 4127 so that the three-dimensional difference and the two-dimensional difference are minimized, and the CV calculation in the CV calculation unit 4122d is repeated.
The corrected CV value whose accuracy has been improved by the above correction / recalculation is acquired and output by the CV data acquisition unit 4128.

[3D map generator]
Next, a specific embodiment of the three-dimensional map generation apparatus provided with the high-precision CV arithmetic apparatus of the present invention as described above will be described.
First, with reference to FIG. 42, a basic configuration of an embodiment of a three-dimensional map according to the present invention will be described.
FIG. 42 is a block diagram showing a basic configuration of a 3D map generation apparatus 4200 provided with a high-precision CV arithmetic apparatus according to the present invention.
The three-dimensional map generation device 4200 includes the high-precision CV calculation device 421 according to the present invention described above, and also includes an object designation unit 4210, an object three-dimensional measurement unit 4220, a three-dimensional map component generation unit 4320, and a tertiary Each unit includes an original space definition unit 4240, an object spot surveying unit 4250, a 3D coordinate determination unit 4260, and a 3D map output unit 4270.

The object specifying unit 4210 automatically or automatically selects an object to be expressed as a three-dimensional map in the all-around video image from the all-around video image and the output from the CV data acquisition unit corresponding to each frame. Specify manually.
The target object three-dimensional measuring unit 4220 specifies a target object to be taken in as a map by the target object specifying unit 4210, and three-dimensionally measures the target object.
The three-dimensional map component generation unit 4230 generates a three-dimensional component in the format described in the three-dimensional map from the data three-dimensionally measured by the object specifying unit 4210.
The three-dimensional space definition unit 4240 defines a three-dimensional space as a three-dimensional map with three-dimensional coordinates.
The object spot surveying unit 4250 surveys the position of the object on the real space coordinates to obtain the three-dimensional coordinates.
The three-dimensional coordinate determination unit 4260 determines the three-dimensional arrangement of the three-dimensional map component in the virtual three-dimensional space defined by the three-dimensional space definition unit 4240 based on the three-dimensional coordinates obtained by surveying.
Then, the 3D map output unit 4270 repeats the 3D coordinate arrangement of the 3D map component and outputs a 3D map.

Next, with reference to FIGS. 43 and 44, a specific processing operation in the 3D map generation apparatus 4200 of the present embodiment will be described.
First, in order to prevent the error accumulation of the CV value, as shown in FIG. 43, a surveying traveling vehicle for generating a map is caused to travel so as to intersect in a certain section.
By doing in this way, since a common feature point can be used, accumulation of errors can be considerably prevented.
For scale calibration, aerial photography and ground photography are combined by CV calculation, and the CV value is obtained at the same time, so that error accumulation can be reduced.
The ground shooting CV calculation and the aerial shooting CV calculation are appropriately combined, a coupling coefficient is set, and the short distance error and the long distance error are separated and adjusted.
CV coupling is performed first, and then partial scale adjustment is performed.
The short distance is calculated with a triangle from the CV value, and the long distance is calculated from the CV value by integrating the camera coordinates to obtain the interframe distance. Aerial CV is used for the distance between frames.

The actual work is performed as shown in FIG. The following operations can be performed according to the display contents displayed on the viewer (display means).
As an operation in the object specifying unit 4410, the following object unit specifying operation is performed.
-Point specification by mouse operation-Line specification by mouse operation-Outline specification by mouse operation-Surface specification by mouse operation

Next, in the object three-dimensional measurement unit 4420, the shape of the object is acquired as follows. This is an operation in the case of a short distance from the object centering on the ground photography CV.
・ Cut the specified point part ・ Cut the image around the specified point ・ Extract the feature point ・ Acquire the positional relationship between the feature point and the specified point ・ Track the surrounding image of the specified point to the adjacent frame ・ Track the feature point to the adjacent frame ・ The specified point part Specified point 3D measurement by positioning and CV value

Further, in the object point surveying unit 4430, the spatial arrangement of the object is acquired as follows. This is an operation in the case where the aerial CV video is mainly used.
・ Specify the start point ・ Partial image tracking of the coordinates of the start point → 3D position measurement ・ Specify the end point ・ Partial image of the coordinates of the end point Tracking → 3D position measurement ・ Track the surrounding image of the specified point to the adjacent frame Thus, 3D map generation can be performed.

[PRM 3D map generator]
Next, with reference to FIG. 45, an embodiment of a 3D map generation apparatus to which the PRM technology is added will be described.
The 3D map generation apparatus 4500 shown in the figure is characterized by adding the PRM technology to the 3D map generation apparatus having the basic configuration shown in FIG.
Here, PRM is an abbreviation for Parts Reconstruction Method (3D space recognition method), and is a technique for recognizing an object developed by the present inventor (see International Application PCT / JP01 / 05387). Specifically, the PRM technology prepares all the shapes and attributes of the object to be predicted in advance as parts (operator parts), compares these parts with actual live-action images, and selects matching parts. Technology that recognizes objects. For example, “parts” of objects required for traveling vehicles and the like are lanes as road markings, white lines, yellow lines, crossing roads, speed signs as road signs, guide signs, etc. Therefore, the recognition can be easily performed by the PRM technology. Further, even when searching for an object in the CV video, it is possible to limit the expected three-dimensional space in which the object exists to a narrow range, and to improve the efficiency of recognition.

Specifically, in this embodiment, as the PRM means, a 3D component database 4510, a 3D component candidate selection unit 4520, a comparison unit 4530, a mismatch signal output unit 4540, a match signal output unit 4550, a match component selection unit 4560, Each component correction unit 4570 is provided.
In the three-dimensional parts database 4510, a plurality of three-dimensional map parts constituting a three-dimensional map are prepared in advance.
The three-dimensional component candidate selection unit 4520 extracts a three-dimensional component that is a candidate for comparison with the target specified by the target specifying unit (4210 in FIG. 42) from the database and selectively prepares it.
The comparison unit 4530 projects a part of the all-around video image specified by the object specifying unit and the three-dimensional component output from the three-dimensional component candidate selection unit 4520 three-dimensionally or on the same plane, Compare.

The mismatch signal output unit 4540 outputs a mismatch signal when there is a mismatch as a result of the comparison performed by the comparison unit 4530. The mismatch signal causes the 3D component candidate selection unit 4520 to sequentially output the next candidate 3D component. Sent out.
The coincidence signal output unit 4550 outputs a coincidence signal when the comparison unit 4530 matches the result of the comparison.
The matching part selection unit 4560 outputs the part finally sent from the three-dimensional part candidate part selection unit 4520 as a matching part in accordance with the matching signal.
The component correction unit 4570 corrects and generates a corrected component when the matching component is slightly different from the actual video.
As described above, a corrected part or a matching part from the matching part selection unit 4560 is sent to the three-dimensional map part generation unit (4230 in FIG. 42) when there is no correction, and a three-dimensional map is generated. become.

This PRM method is very effective not only for generating a three-dimensional map, but particularly for automatically updating a three-dimensional map.
PRM 3D maps as well as general 3D maps are already part-by-object and expressed as a set of these parts, so the coordinates are already set in the same way as for new generation. By comparing the shape or attribute of a known part of the old 3D map with the same coordinates in the latest video, it is possible to confirm whether there is a difference from the old data.
In addition to confirming new and old 3D map parts, it is also possible to confirm old and new data between old and new images (2D).

An example of specific implementation as an automatic update device for a three-dimensional map is shown below.
The old three-dimensional map to be updated is included in the three-dimensional component database 4510 in FIG.
Thus, in the automatic updating apparatus, the old three-dimensional map is used as the three-dimensional database 4510.
Since the old 3D map consists of a set of 3D parts with coordinates, it is possible to update each part of the old 3D map in the order of the arrangement of the parts of the old 3D map. This is much more efficient than creating a new 3D map.

In the 3D part candidate selection unit 4520, each part constituting the old 3D map is sent in order and compared with a part of the new video having the same coordinates.
The new video may be converted into 3D parts and compared between 3D parts, but the new video is compared with the partial video in the new video that compares the shape or attributes of the 3D parts with the video (2D). The method is simpler, and this method is adopted here.
The comparison unit 4530 confirms whether or not the parts of the old 3D map and the coordinate-corresponding portion of the new video match, and if there is a match, no change is made, and the old 3D map is directly used as the new 3D map.

On the other hand, in the case of a mismatch, the mismatch signal output unit (4540) is sent to the three-dimensional component candidate selection unit (4520), the candidate components are repeatedly compared to find a matching component, and the old component is automatically used by the new component. If updated, the 3D map is automatically updated in parts, and the old 3D map is automatically updated by automatically repeating this operation.
Here, since a 3D map can be converted to a 2D map at any time by automatically updating the 3D map, the same applies to the automatic update of a 2D map.
New parts that are newly added but not in the old 3D map are the same as the above-described PRM method for generating a new 3D map.

[3D map generation from aerial images]
Next, with reference to FIGS. 46 to 48, an embodiment of a 3D map generation apparatus that generates a 3D map using an aerial image will be described.
The 3D map generation device 4600 shown in FIGS. 46A and 46B is characterized by generating a 3D map based on aerial images.
Specifically, as shown in FIG. 46A, the 3D map generation apparatus 4600 includes the following elements.

The target range designation unit 4610 designates a target range to be generated as a three-dimensional map in the all-around video image from the output of the CV data acquisition unit corresponding to the all-round video image and each frame.
The measurement point density specifying unit 4620 specifies the measurement point density in the target range.
The measurement point generator 4630 generates a measurement point at an arbitrary position automatically or manually with the above target range and measurement point density.
The measurement point tracking unit 4640 tracks the measurement points over a plurality of adjacent frames of the all-around video image.
The measurement point calculation unit 4650 calculates the three-dimensional coordinates of the measurement points from the CV value output from the CV data acquisition unit and the measurement points obtained from the measurement data and tracking data obtained by tracking.
The target range surveying unit 4660 obtains a three-dimensional distribution of all measurement points in the target range.

On the other hand, the specific measurement point designation registration unit 4680 has a corner, an edge, a surface, a solid, or a part of the object composed of a straight line portion often seen in an artificial object within the target range in the all-round video image. Is designated as a specific measurement point and registered together with the attribute.
Since the specific measurement point is specified, it is sent to the measurement point generation unit 4630 with the attribute registered, and is tracked and calculated in the same manner as a general measurement point, and the measurement point calculation unit 4650 is output. Is done.
The specific measurement point formation unit 4690 extracts only specific measurement points from the output from the measurement point calculation unit 4650, and gives a shape suitable for the attribute.
The three-dimensional shape generation unit 4670 generates a three-dimensional shape formed by the specific measurement points output from the specific measurement point formation unit 4690.
The three-dimensional coordinate integration unit 46100 integrates the three-dimensional shape generated by the three-dimensional shape generation unit 4670 together with the three-dimensional shape obtained from measurement points other than the specific measurement point.
Then, the 3D shape generated and integrated as described above is generated and output as a 3D map by the 3D map output unit 46110.

Furthermore, as shown in FIG. 46B, the 3D map generation apparatus 4600 of the present embodiment can also include a 3D shape generation unit 4601 and a combination condition setting unit 4602 described below.
The three-dimensional shape generation unit 4601 includes a wire frame generation unit 4601a that represents a three-dimensional distribution of measurement points in a wire frame, and a texture pasting unit 4601b that pastes the texture of the target range of the all-around video image on the wire frame. .
Further, the combination condition setting unit 4602 changes the combination condition of the wire frame for generating an appropriate three-dimensional shape when the wire frame is not uniquely determined from the above three-dimensional coordinate distribution.

47 and 48 show a specific example of a three-dimensional generation apparatus that generates a three-dimensional map from the aerial image as described above.
One GPS and five wide-angle video cameras are attached to a part of the aircraft as shown in FIG. 47 (a) as shown in FIG. 47 (b).
Each camera has a field of view of about 50 ° × 70 ° as shown in FIGS. 48 (a) and 48 (b), and images the ground with a partial field of view overlapping.
The distance between the cameras is accurately measured and reflected in later calculations. Further, the camera mounting angle is obtained by calculation later as a rough setting in consideration of fluctuation due to vibration.

Video images taken by the five video cameras having wide-angle lenses are stored in synchronization with the demoted video image recording unit.
Image data and GPS data from each wide-angle camera are recorded simultaneously with the time by the same clock.
From the wide-angle video image, absolute coordinate calibration is performed by GPS, CV calculation is performed, fluctuation of GPS data is corrected by CV data, and CV data is acquired as CV data by the CV data acquisition unit.

[Generation of 3D video from 2D video]
Next, an embodiment of a 3D map generation apparatus that generates a 3D video from a 2D video image will be described with reference to FIGS. 49 to 57.
A 3D map generation apparatus 4700 shown in FIG. 49 generates a 3D video from a 2D video image.
Specifically, as shown in FIG. 49, the 3D map generation apparatus 4700 includes elements that perform the following processing operations.

A target range specifying unit 4710 specifies a target range to be generated as a three-dimensional video in the all-around video video that is output from the high-precision CV arithmetic device 471.
FIGS. 50 and 51 show an example of making the operator a simple block (see FIG. 51A), and an example in which the operator is specified for a general image (see FIGS. 50 and 51B). Indicates.
Here, there is an advantage that the calculation time is shortened by determining the range and making it three-dimensional using an operator. Of course, it is possible to make the entire area three-dimensional at a time.
Further, by limiting the target range and specifying as narrow as one target object, the PRM three-dimensional map generation device is used as a pre-stage of target recognition in the three-dimensional map generation by PRM shown in FIG. It is also possible.

In the measurement point density designation unit 4720, the image stabilization unit 4720a generates a video in which the rotation component of the moving image is stopped from the CV value acquired from the CV data acquisition unit. At this time, it is possible to perform coordinate transformation so as to fix the three-axis rotation component. This stabilizes the image, and the movement of the image due to camera movement is defined in the direction of the source point.
Next, the spring point determination unit 4720b determines the spring point that is the origin of the motion of the moving image when the rotation component is stopped.

Next, the image block dividing unit 4720c divides the designated range of the video into image blocks. Here, image block division is, for example, a rectangular operator based on simple block division in units of rectangles, or a region formed by radiation extending radially from the source point along the epipolar line and a curve or straight line intersecting with the radiation. This can be done using a source point direction operator.
Since the rectangular operator can treat each block on the epipolar line as a substantially one-dimensional object, the same result as the source point direction operator can be obtained.
There are various types of source direction operators depending on the image development method.

The radiation along the epipolar line becomes a straight line in a plane-converted image (perspective image), but becomes a curve in a Mercator image. Also, a spherically developed image (fish-eye lens projection method) is curved, but it is possible to make it close to a straight line by taking the source point near the center of the image, which is advantageous for subsequent tracking calculations.
Note that the operator can expand the entire image.
FIG. 50 shows an example in which the source point direction operator is applied to the traffic sign portion.
FIG. 52 shows the relationship between the source point position in the plane conversion image and the source point direction division operator.

FIG. 53 shows a source point position and a suction point position in a Mercator image, and examples of a rectangular operator and a source point operator as image division examples.
As shown in the figure, in a Mercator image, each divided region obtained by image division is generated from a spring point, moves on the epipolar line in a curved manner, and moves so as to disappear at the suction point. In a plane development image corresponding to the perspective method, the region moves linearly.

In addition, the actual motion of each divided area changes as the direction of the camera changes, and as the movement direction changes, the position of the source point and the suction point changes every moment, so a part of the above motion is continuously stacked. Become an exerciser.
In a general image in which the direction of rotation changes, the above state is established differentially.
By stabilizing the rotation of the image and the camera direction by post-processing, it is possible to realize the above state continuously. That is, the process of stabilizing the rotation of the image and the camera direction fixes the source point and the suction point, eliminates the rotation, and maintains the above state for a long time.
The source point direction operator is divided into regions divided by this epipolar line and the line intersecting it. In FIG. 53, a vertical line is used, but there is no particular limitation.

The above points are the same in the spherical development image shown in FIG.
FIG. 54 shows an example of image division, an example of an operator, and a source point position in a spherically developed image (fisheye lens projection method).
Specifically, an example of an image spring point position and a suction point position by a fish-eye lens projection method is shown. Although there are mainly two types of fish-eye lens projection methods, the same can be said about the spring point and the suction point.
The image segmentation area moves in a curve on the epipolar line connecting the two points of the spring point and the suction point. Although it is a curve, it can be treated as a one-dimensional vector substantially because it follows a uniquely determined path once the spring point is determined.

With respect to all the image blocks divided by the image block dividing unit 4720c as described above, blocks that can be feature points are extracted by the feature point extracting unit 4720d. The feature points can be tracked as they are, and the CV calculation can be performed again by adding the feature points.
In addition, boundary points that can be part of the boundary line are extracted from the image block by the boundary point extraction unit 4720e.
Further, the region point extraction unit 4720f extracts the region points using the points other than the feature points and the boundary points as region points.
These feature points, boundary points, and region points are called three types of point elements.
FIG. 55 shows the classification and classification of feature points, boundary points, and area points for all blocks.

Then, each point of the image block is combined by a two-dimensional rubber string combined image generation unit 4720g so that the points are radially connected from the spring point, the order is fixed, and the distance relationship is free. Is done.
FIG. 56 shows the two-dimensional rubber string connection state.
In the above measurement density designating unit 4720, the image stabilizing unit 4720a is not necessarily required, and the source point determining unit 4720b is not necessarily required. In this case, the image block dividing unit 4720c is simply a rectangle. It is also possible to classify feature points, boundary points, and region points by dividing the block into blocks.
If the image block division is a simple division like a rectangle without depending on the starting point, the rubber string coupling requires the rubber string coupling in both the vertical and horizontal directions without changing the order of the blocks. .

Next, in the measurement point generation unit 4730, a feature point is first selected as a measurement point by the feature point selection unit 4730a.
In addition, the region generation unit 4730b generates a region created by the region points and the feature points and boundary points that surround them.
The final image block classification is shown in FIG. In the figure, a region where closed region points surrounded by boundary points or feature points are classified is a hatched portion, and an incomplete region is classified by a filled portion.
Then, the measurement point determination unit 4730c determines the feature point as a measurement point and the region as a measurement region as a tracking measurement target.

The measurement point tracking unit 4740 tracks feature points, boundary points, and measurement areas as measurement points in a plurality of adjacent frames in the direction of the spring point.
The feature points can be traced to a plurality of adjacent frames as they are.
By tracking the boundary points in the direction of springing, the corresponding points may be determined, but some of them cannot be determined. In addition, although a region point cannot be found by tracking itself, there are some points that can be tracked if tracking is performed using the shape of the entire closed region.
However, even if the points cannot be traced, they are connected by rubber straps and the arrangement order is determined, and since the distance and speed from the source point can be expressed by a linear function, unknown blocks can be represented. It is possible to assume tracking points of some unknown blocks by decorating them in proportion.

When tracking over adjacent frames, if the tracking block is near the source point, as the tracking to the adjacent frame increases, the change in the corresponding image block increases. Just taking it makes the shape different and incomplete correspondence.
The change in the size of the image block depends on the distance from the spring point, can be calculated, and can be predicted in advance. Therefore, it is possible to perform tracking by reflecting the expectation and changing the size and shape of the image block as a function of the distance from the spring point. In this way, more appropriate tracking is possible, and tracking accuracy is improved. Note that, when the image block dividing unit 4720c simply divides the image without using the source point, the corresponding calculation is simplified, but the measurement point tracking unit does not use the tracking direction in two dimensions. The demerits of increased efficiency and increased computation speed occur.

In the measurement point calculation unit 4750, the three-dimensional coordinates of the measurement point and the three-dimensional coordinates of the region are acquired from the tracking result of the measurement point and the CV data output from the CV data acquisition unit.
Here, three-dimensional measurement is possible for the feature points, traceable boundary points, and traceable closed region point sets, and the three-dimensional coordinates can be obtained by calculation.

In the target range surveying unit 4760, the three-dimensional coordinates of the measurement points and the boundary points surrounding the measurement region are repeatedly calculated over the target range, and if there is an image block that cannot be tracked yet in the tracking unit, The image block is obtained by interpolation from the already calculated three-dimensional coordinates, the three-dimensional position is predicted, and the target range is surveyed.
Even if the points cannot be traced, they are connected with rubber straps, and the order of arrangement is determined, and the distance and speed from the source point can be expressed by a linear function, so the three-dimensional coordinates For the unknown block, the three-dimensional coordinates of the unknown block can be inferred from the adjacent image blocks of the three-dimensional coordinates in a proportional relationship.
Here, the processing is limited to the target range, but the entire image can be converted once over the entire image by the same principle and method, but there is a demerit that it takes a calculation time.

In the three-dimensional shape generation unit 4770, the boundary line belonging classification unit 4770a determines the region to which the feature point and the boundary point belong from the three-dimensional coordinates of the feature point and the boundary point.
Next, the rubber cord cutting portion 4770b cuts one of the rubber cord connections by the affiliation of the feature point and the boundary point, and establishes the connection between the one region and connects it to the other region. A boundary point generally forms a boundary line of a plurality of regions in succession, but it is necessary to determine which region of the plurality of regions the boundary point belongs to. It is also the same that the boundary point is determined first and the region belonging to the boundary point is determined.
Furthermore, all blocks in the target range are classified into feature points, boundary points, and area points by the three-dimensional rubber band combination video unit 4770c, and the feature points are fixed at the coordinates obtained as measurement points, and the last remaining still For image blocks whose coordinates are not fixed, by interpolating on the rubber string and giving each three-dimensional coordinate, all blocks are given three-dimensional coordinates, and finally all blocks are three-dimensionally connected with the rubber string. To do.

Only the feature points are determined by tracking. Some boundary points can be traced by tracking the source direction, which can limit the tracking direction, and some of the boundary points are required to have three-dimensional coordinates. In addition, in the case of segmentation with simple image blocks, tracking each block on the epipolar line makes it difficult to track other than the feature points. Thus, even when the tracking position can be predicted, the three-dimensional coordinates can be inferred from the surrounding feature point distribution where the three-dimensional coordinates are known. Furthermore, with respect to what creates a region, the shape of the region itself is traced to a plurality of adjacent frames, whereby the three-dimensional coordinates of the region unit can be obtained. Therefore, finally, all image blocks can acquire three-dimensional coordinates.
The example shown in FIG. 57 (b) is a two-dimensional rubber string connection, but three-dimensional rubber string connection is obtained by giving a three-dimensional coordinate to each block.

In the three-dimensional coordinate integration unit 47100, the image division density changing unit 47100a changes the division density of the image combined with the three-dimensional rubber string from a rough density to a higher density sequentially.
Also, the three-dimensional image recording integration unit 47100b once records the three-dimensional rubber string-coupled image by setting the density of the image block division first, and sequentially generates more detailed three-dimensional shapes by the next process. Then, it is repeated and sequentially generated up to the target density to generate a more detailed three-dimensional image.
The three-dimensional rubber string connection image first generates a rough three-dimensional shape, and then gradually generates a dense three-dimensional shape by correcting it.
Based on the three-dimensional shape generated as described above, a three-dimensional map is generated and output by the three-dimensional map output unit.

If the video is simple, it is sufficient to generate a single three-dimensional rubber string combined image.
However, in the case of a complicated shape, it is necessary to change the density a plurality of times and make it gradually finer. Also, depending on the specified range of the target range specifying unit, changing the density and changing the repetition requires a density change suitable for the image and the range.

Hereinafter, a more specific example of the area division by the area division described above will be described.
Here, the area division by the area division according to the present invention is equally effective in the aerial image and the ground image, and a generalized example is shown in FIGS. 52 and 55 to 57. .
In the area division, first, a target survey target range is designated in a wide-angle video image. In the example shown in FIG. 55A, 110% of the entire area is designated.
Next, the target portion is blocked. When making a block, a spring point is obtained from the CV data, and is blocked so as to be arranged on an extension line from the spring point (see FIG. 52A).

Also, it is divided into blocks that are concentric with the radial line centered at the spring point. Even if the position of this block is changed by subsequent tracking or the like, the relationship with the adjacent block is maintained so that the strip connected by the rubber string is expanded and contracted. This is a two-dimensional rubber string coupling image (see FIG. 52B).
This two-dimensional rubber string video generation process facilitates subsequent tracking work. The tracking may be performed on this radiation line.
Block the entire area of the target specification range, and each block is subjected to autocorrelation, and the block with a peak inside is used as a feature point. And

The area of the figure and the ground is filled in as shown in FIG.
At this stage, there is no distinction between the figure and the ground, but the figure surrounded by the feature points and the contour candidate points is the figure, and the one not enclosed is the ground. FIG. 57 (b) shows a separated version.
At this stage, the video is divided into blocks, and each block is divided into four types of feature points, boundary lines, diagrams, and grounds, and each block belongs to one of the four types. However, in the all-round video, there is no distinction between the figure and the ground.
Furthermore, from the three types excluding the ground, it is possible to classify into points that form contours, regions that are illustrated, and regions that are ground.

A portion that overlaps a short distance from a feature point or an inappropriate part such as a moving body is excluded, and is determined as a measurement point, and the measurement point is tracked in the video.
From the measurement points and the already obtained CV data, the three-dimensional coordinates of each measurement point are obtained by calculation.
From this measurement point calculation output, the contradictory measurement results are excluded, and the three-dimensional coordinate distribution of the target range is obtained.
Then, a three-dimensional rubber string combined image reflecting all blocks of the image is generated by classification of feature points, region boundaries, diagrams, grounds, and the like.

On the other hand, specific measurement points that are particularly important in the survey target range are manually specified and registered. The specific measurement point specifies a part derived from an artificial structure such as a vertical part, a horizontal part, or a right-angled part in the image, thereby improving the accuracy of the subsequent three-dimensional map. Further, the attribute is stored, and the vertical and horizontal portion is later reflected on the three-dimensional map. From the specific measurement points, those that are not suitable for calculation are excluded, and appropriate specific measurement points are determined.
Then, the specific measurement point is tracked in the wide-angle video image, and the specific measurement point is calculated from the tracking result and the CV data. At this time, the vertical and horizontal attributes of the specific measurement point are reflected.

The 3D image combined with the 3D rubber string is already a 3D map, and can be used as a 3D map at this stage as well. can do.
Since the 3D image formed as a 3D rubber string has already been pasted as a texture on the polygonal skeleton, in this case, the texture pasting portion (4601b in FIG. 46B) is not necessary.
Polygon nodal points are set with polygon generation conditions for adjusting the relationship according to the actual situation. This is because the connection relationship between the polygons is not uniquely determined and needs to be matched to reality.

Coordinate integration of 3D rubber band combined image or polygonized 3D image and output of specific measurement point survey, and in the process, integrate the attributes of specific feature points and integrate 3D coordinates.
Then, the integrated one is output as a three-dimensional map.
In this way, it is possible to generate a three-dimensional map directly from an aerial image.
At the time of shooting, it is an excellent advantage that a 3D map can be acquired directly from a moving image without worrying about the camera posture, that is, the posture of the aircraft, and without making it orthorectified.

[Multi-camera parallax 3D map generator]
Next, with reference to FIGS. 58 to 59, an embodiment of a parallax combined type three-dimensional map generation apparatus using a plurality of cameras will be described.
As shown in FIG. 58, the 3D map generation apparatus 5800 of the present embodiment includes a multi-camera photographing unit 5801 having a plurality of video cameras for acquiring video images with overlapping visual fields and parallax. Features.
Note that the high-precision CV calculation device provided in this embodiment corresponds to the high-precision CV calculation device 171 in which the all-around camera shown in FIG. Of course, this does not preclude application to a high-precision CV arithmetic unit equipped with an all-round camera.

The 3D map generation apparatus of the present embodiment first includes a parallax 3D conversion device 5801 having a multi-camera imaging unit 5801a and a parallax camera coordinate 3D conversion unit 5801b.
The parallax method three-dimensional apparatus 5801 obtains the distance from the camera position to the target object by calculation over the overlapping visual field range by using video images with parallax captured by a plurality of video cameras moving on a vehicle or the like. The camera coordinate 3D distance distribution data is acquired.

Note that this three-dimensional parallax method is an existing technology and also exists as an apparatus. However, although a three-dimensional image can be obtained, it is a camera coordinate system image, which is a three-dimensional image viewed from the camera position, and it is meaningful if the camera is stationary, but if the camera moves In this case, the position and orientation of the camera itself in the stationary coordinate system are unknown, and thus the acquired three-dimensional coordinates are also unknown.
Therefore, by adopting the configuration as in the present embodiment, the camera position is known by obtaining the CV value of the three-dimensional coordinates of the camera itself. Can be converted to.
A specific example of the parallax camera device is shown in FIG.

The present embodiment includes the high-precision CV arithmetic device 5802 according to the present invention as described above.
The high-precision CV calculation device 5802 includes an image acquisition unit 5802a and a CV calculation unit 5802b.
The image acquisition unit 5802a acquires and temporarily records video images from cameras representing a plurality of cameras.
The CV calculation unit 5802b converts CV values acquired by other cameras installed at the same time into camera positions, and acquires CV data of representative camera positions. This CV calculation processing is the same as that of the high-precision CV calculation device 171 shown in FIG.

In this CV calculation, a CV value is not directly obtained from the video from the camera, but another camera having an angle of view suitable for the CV calculation is also provided, and the high angle is obtained regardless of the angle of view and resolution of the camera. An accuracy CV value is obtained. Moreover, it can be asynchronous, which is advantageous.
Note that CV data of representative camera positions can also be acquired from the video of the camera. In this case, since the camera is used, it is not a camera dedicated to CV calculation. Therefore, since the angle of view cannot be selected as a wide angle, the angle of view is narrowed or the resolution is lowered. Will fall.

Furthermore, the three-dimensional map generation apparatus 5800 of this embodiment includes a stationary coordinate system conversion unit 5803 and a stationary coordinate system composition combining unit 5806.
The stationary coordinate system conversion unit 5803 continuously converts the three-dimensional distance distribution data in the camera coordinate system obtained by the parallax method three-dimensional device 5801 into the stationary coordinate system using the high-precision CV value obtained by the high-precision CV arithmetic device 5802. Convert to 3D distance distribution data in coordinate system.
Since the CV value indicates the three-dimensional position and the three-axis rotation posture of the camera, the camera coordinate three-dimensional distance distribution data can be accurately converted into an arbitrary coordinate system.
The stationary coordinate system combining / combining unit 5806 combines the three-dimensional distance distribution data in the stationary coordinate system obtained continuously while overlapping with the progress of the video image, and integrates them into the stationary coordinate system.

In addition, since 3D map generation continuously obtains converted data as a moving image, the data is duplicated. Therefore, the three-dimensional map is generated by combining and superimposing them, but the processing of the moving object remains.
Therefore, if necessary, the moving object separation unit 5805 and the stationary object separation space constituting unit 5808 shown below are provided to separate the moving object and separate it into a moving object and a stationary object for display. can do.
Since stationary objects overlap or moving objects do not overlap, they can be separated. In addition, the moving object can be separated from the CV value of each region.

Therefore, the moving object separation unit 5805 separates the three-dimensional data with time variation and separates and records it as the moving object.
Then, the stationary object separation space forming unit 5808 is configured to include only the stationary object in the stationary coordinate system when it is integrated into the stationary coordinate system.
As described above, the CV calculation and the parallax method have contradictory properties, and variations in selection and combination of a camera for CV data and a plurality of cameras for the parallax method depend on the scale and cost of the apparatus. Can be set optimally (see FIG. 21).

[Navigation equipment]
Next, with reference to FIGS. 60 to 62, a specific embodiment of the CV navigation system equipped with the high-precision CV arithmetic device according to the present invention will be described.
As shown in FIG. 60, a CV system navigation device 6000 according to this embodiment includes a CV system 3D map generation device 6001, a 3D map device 6002, a travel route input device 6003, an in-vehicle camera, and a schematic CV calculation device 6004. , An object space constructing device 6005, an object recognizing device 6006, a high-precision CV computing device 6007, a current status judging device 6008, a real-time control device 6009, and a real-time display device 6010.

The three-dimensional map generation apparatus 6001 is a CV system three-dimensional map generation apparatus 6001 including a high-precision CV arithmetic apparatus that generates the above-described three-dimensional map according to the present invention.
The 3D map device 6002 includes 3D map information that is generated by the CV 3D map generation device 6001 and can be mounted on a vehicle.
The travel route input device 6003 is an input means that can input information necessary for travel to the 3D map device 6002 in advance as 3D data and attributes, and change the information on the way.
The in-vehicle camera and the approximate CV calculation device 6004 output wide-angle video data and relatively low-precision CV data at the time of traveling from the in-vehicle camera.

The object space constructing device 6005 acquires the current position and orientation of the traveling vehicle in three dimensions from the output approximate CV data, roughly corresponds to the three-dimensional map, detects the three-dimensional space structure, Prior to recognition, an obstacle requiring emergency response is recognized, and emergency control signals for avoidance and stop are sent to the real-time control device with the highest priority. Then, the object space constituting apparatus 6005 detects a three-dimensional space structure composed of a plurality of objects that are in a visible range from the traveling road and are related to the purpose of traveling and traffic, by using the in-vehicle camera and the schematic CV calculating apparatus 6004.
In the navigation device 6000 of the present embodiment, the three-dimensional space configuration can be easily detected by overlapping the fields of view of the two cameras shown in FIG. Real-time processing is possible.
As an emergency response to accident avoidance, humans may jump out in front of the vehicle. Before recognizing whether or not, as an emergency decision, an action is taken to prevent an accident by avoiding or stopping the vehicle.

The object recognition device 6006 compares the object space configuration data with the approximate three-dimensional position data of the vehicle, is in the visible range from the road, matches the driving purpose, and relates to traffic, traffic lights, road signs, The three-dimensional shape of each object such as a road marking is compared with the information of the three-dimensional map to identify and recognize the shape and attributes.
FIG. 61 shows more specific processing contents of the object space construction device 6005 and the object recognition device 6006.
As shown in the figure, the object space constructing apparatus 6005 divides the image on the vehicle-mounted video output at the coarse density by the coarse density image division 6101, and the video space from the CV value by the coarse density three-dimensional space arrangement recognition 6102. Recognize a rough three-dimensional arrangement.

Next, in the object approximate position detection 6103, paying attention to the expected position of a traffic light, a road sign, etc. from the image for which the approximate 3D position is obtained, the 3D of the approximate 3D position of the object is obtained from the CV data. The target range is specified by narrowing the specified range of the target range specifying unit to a range including about one target object, and the range is image-divided. This image division processing is performed in the same manner as in the embodiment shown in FIG.
Further, a position in an adjacent frame is predicted by the object lock-on 6104, and the object is tracked by the object lock-on tracking 6105.

Next, in the object recognition apparatus 6006, first, the object area image block division 6106 is decomposed into three types of point elements including feature points, boundary points, and area points constituting the object, and the points and areas are tracked. .
From this tracking result, the object region three-dimensional shape generation 6107 obtains the three-dimensional shape.
On the other hand, as in the PRM method shown in FIG. 45, a database part is prepared as 3D map information, and the part or the part is previously decomposed into three types of point elements. A part or three types of point elements and regions of the part are compared, the part is identified by the three-dimensional part identification 6109 by PRM, and the object is recognized by the object final recognition 6110.

As shown in FIG. 60, the high-accuracy CV calculation device 6007 three-dimensionally tracks the three-dimensional shapes or part shapes of a plurality of objects by the object recognition device 6006, or is a stage of three-dimensional space configuration before specification. Then, a characteristic shape is selected, cut and tracked, and highly accurate CV data is acquired from the tracking result.
With this two-stage structure, high-speed real-time computation is performed at the stage prior to recognition, obstacles are detected with the highest priority, obstacles are avoided, and some recognition time is spent on advanced recognition. Low-speed real-time processing can be performed.
The current state determination device 6008 comprehensively determines the high-accuracy CV data, the output of the object space configuration, the object recognition result, and the output of the in-vehicle radar or other in-vehicle measuring instrument, and concludes the traveling condition. Installed a current status judgment device that leads

The real-time control device 6009 appropriately controls the accelerator, brake, steering wheel, etc. of the vehicle or changes the data of the travel route input device according to the output of the current state determination device 6008.
In other words, by analyzing the CV value obtained in real time and the camera image, the target position, the target direction, the target speed, the target posture of the vehicle or aircraft are directly adjusted according to the schedule, and appropriately corrected according to the situation of the site, Can be controlled.
The real-time display device records the control signal output from the travel route input device 6003 in advance in the travel vehicle with respect to the position and orientation of the travel vehicle loaded with the camera, and the high-precision CV arithmetic device 6007 loaded on the travel vehicle. The current position and orientation are obtained from the output, and the vehicle is controlled to travel on the planned travel route by the recorded control signal. Further, it is judged from the information obtained during the travel, and is adjusted to the situation around the current position. In addition, the output of the high-precision CV arithmetic unit 6007, the recognition result, and the current state determination result are displayed in real time on a two-dimensional map or a three-dimensional map as necessary, and the driver or the like can be controlled. Inform.
FIG. 62 shows an example in which the navigation device 6000 according to the present embodiment as described above is mounted on a traveling vehicle.

[Pedestrian Navigation System]
Furthermore, with reference to FIGS. 63 to 69, an embodiment of a pedestrian navigation system will be described as an application example of the CV navigation system provided with the high-precision CV arithmetic device according to the present invention.
The pedestrian navigation system is realized as a navigation device to which the high-precision CV arithmetic device according to the present invention is applied, and is placed in a state where the pedestrian navigation device is connected to the car navigation device when the vehicle travels. In the car navigation device, normal guidance (car navigation) is performed. On the other hand, when the guided person leaves the vehicle, the pedestrian navigation device can be used as a single pedestrian navigation device independently of the car navigation device. It is.
Because the pedestrian navigation device needs to share information with the car navigation device while the vehicle is running, the function of the pedestrian navigation device is combined with the function of the car navigation device when placed in a state connected to the car navigation device, It has the function of displaying and guiding on the large display of the car navigation system in the car.

As the main function of pedestrian navigation, car navigation device and pedestrian navigation device can basically have the same function, but in pedestrian navigation device, it is necessary to also guide inside underground streets and buildings, In addition, it is necessary to determine the self position by a function other than GPS measurement.
The GPS function is mainly used in car navigation devices, but the GPS function is complementary in pedestrian navigation devices, and the position is obtained from CV images acquired from other than pedestrian navigation devices or camera images added to pedestrian navigation devices. CV function is important.
The pedestrian navigation device can exchange data with the information center, and the information center provides a CV video map distribution service and a point specifying service to the pedestrian navigation device.
The pedestrian navigation device and the information center are communicably connected via a line. As the line, a mobile phone line or a dedicated line can be used, but at least a line that enables data transmission of still images and moving images is required.

Hereinafter, a specific configuration of the pedestrian navigation system 6300 according to the present embodiment will be described.
As shown in FIG. 63, the pedestrian navigation system 6300 includes an information center facility 6310 and a pedestrian navigation device 6320 as a basic configuration.
The information center facility 6310 has a function of recording a CV video having the camera position and orientation data of the shooting point and its attributes, searching for coordinates and attributes, and distributing the CV video and attributes of the specified point. Have. This information center facility 6310 is installed in advance.
The pedestrian navigation device 6320 is composed of a terminal device that can be carried by a pedestrian who is a guided person. In the device body, by specifying the destination point, communication with the information center facility 6310 leads to the destination point. Acquire CV video and coordinates, attributes, etc., or change destinations and routes during walking, acquire new data by communicating with the information center, and use these acquired data to bring pedestrians to the destination invite.

And the pedestrian navigation system 6300 which consists of the above basic structures is used with a combination with a car navigation apparatus, as shown in FIG.
Specifically, the car navigation device 6330 constitutes a car navigation device having a normal car navigation function, and is connected to the pedestrian navigation device 6320 via the coupling device 6340. That is, the pedestrian navigation device 6320 is detachably connected to the car navigation device 6330.
In the coupling device 6340, the car navigation device 6330 and the pedestrian navigation device 6320 are detachable, have an integrated function in the connected state, and function independently in the separated state. This is a device for connecting a navigation device and exchanging data.

More specifically, the pedestrian navigation system 6300 according to one embodiment has a configuration as shown in FIG.
In the information center facility 6310, first, the center reception unit 6311 receives route data and the like from the pedestrian navigation device 6320.
In the CV video database unit 6312, the CV video including the destination point is stored, recorded together with its attributes, and stored in a database so that it can be searched with the received coordinates or attributes.
The proximity position CV video selection unit 6313 selects the CV video most appropriate for the received data from the data in the CV video database unit.
The center transmitting unit 6314 transmits the selected most appropriate CV video to the pedestrian navigation device 6320.

In the pedestrian navigation apparatus 6320, first, a destination point for guidance of a pedestrian is designated by a destination point designation unit 63201.
In addition, an arbitrary passage point is designated by the passage point designation unit 63202.
The route calculation unit 63203 automatically determines a route on the way by setting a local point and a destination point.
The route determination unit 63204 selects an appropriate route from the plurality of routes calculated by the route calculation unit 63203.
The route transmission unit 63205 transmits route data indicating the route, the local point, and the destination determined by the route determination unit 63204 to the information center facility.

The CV video reception unit 63206 receives the CV video distributed from the center transmission unit 6314 of the information center facility 6310.
The point video selection unit 63207 selects a local point video from the CV video distributed by the information center facility 6310.
The point image page turning unit 63208 selects CV images one after another as the pedestrian moves.
The destination point arrival guidance unit 63209 notifies the pedestrian of the destination when the destination point is reached.

The current location coordinate acquisition unit 63210 acquires the position at an arbitrary point where the pedestrian is moving, and transmits the acquired data to the information center facility 6310 as a signal of the elapsed route transmission unit 63205 based on the acquired data. The target CV video is received from the information center facility 6310 through the CV video receiver 63206.
The display guide unit 63211 switches the point video selection unit 63207, the point video page turning unit 63208, the destination point arrival guide unit 63209, and the CV video distributed as needed, respectively, or displays them simultaneously.
The voice guidance unit 63212 guides the guidance content displayed on the display guidance unit 63211 by voice as necessary.

As shown in FIG. 66, the pedestrian navigation system 6300 configured as described above includes a photographing unit in the pedestrian navigation apparatus 6320, and navigation based on the video acquired by the information center facility 6310 together with information from the information center facility 6310. Means for performing can be provided.
Specifically, as shown in the figure, the pedestrian navigation device 6320 first shoots a video around the pedestrian using a wide-angle video imaging unit 63213.
The image processing unit 63214 primarily records captured wide-angle video, and extracts feature points in adjacent frames.
The CV calculation unit 63215 calculates a camera vector (CV) by calculation based on the tracked feature points. This CV calculation is as described above.
The absolute coordinate conversion unit 63216 acquires absolute coordinates based on an object whose length, distance, and coordinates are known.

The navigation position CV image generation unit 63217 generates a moving image image to which the CV value of the pedestrian navigation device 6320 is added.
The two-type CV video comparison unit 63218 compares the CV video delivered from the information center facility 6310 with the CV video acquired by the pedestrian navigation device 6320 and the two types of CV video, and the positional relationship between the two types of CV video. To decide.
The self-position determining unit 63219 determines a self-position in the CV video distributed from the information center facility 6310, and further generates a signal for automatically selecting the CV video in the point video page turning unit 63208 and performing automatic page turning.
The self-position marker generation unit 63220 generates a self-position marker and sends a signal indicating the self-position on the guide video or the map to the display guide unit 63211.

In addition, as described above, it is possible to provide a means for transmitting photographing data acquired on the pedestrian navigation device 6320 side to the information center facility 6310 and reflecting it in distribution information.
Specifically, as shown in FIG. 67, the pedestrian navigation apparatus 6320 has an overall data transmission unit 63221 that outputs from the route determination unit 63204, an output from the wide-angle video imaging unit 63212, and an absolute coordinate conversion unit 63216. Is output to the information center facility 6310.
In the information center facility 6310, the center reception unit 6311 receives a signal transmitted from the general data transmission unit 63221 of the pedestrian navigation apparatus 6320.
The CV video database unit 6312 stores the CV video including the destination point, records it together with its attributes, and stores it in a database so that it can be searched with the received coordinates or attributes.

The two-type CV video comparison unit 6315 obtains the approximate position of the pedestrian navigation device 6320 from the received position signal, selectively extracts the surrounding CV video from the CV video database unit 6312, and receives the received walking as necessary. The video acquired by person navigation device 6320 is compared with the CV video.
The self-position determining unit 6316 determines its own position on the coordinates or on the video by the CV video comparison of the two-type CV video comparing unit 6315.
The point CV video selection unit 6317 selects a corresponding CV video from the CV video database unit 6312 acquired by the self-position determination unit 6316 determining the self-position.
The CV video or the like of the selected point is transmitted from the center transmission unit 6314 to the pedestrian navigation device 6320.

Furthermore, in the pedestrian navigation system 6300 having the above-described configuration, more accurate navigation can be performed using GPS together.
Specifically, as shown in FIG. 68, the pedestrian navigation apparatus 6320 can include a GPS unit 63222.
The GPS unit 63222 acquires absolute coordinates by GPS.
Thereby, when GPS radio waves can be received satisfactorily and there is sufficient accuracy, it is possible to improve the accuracy by acquiring absolute coordinates by GPS and further complementing each other with the GPS and CV values. In addition, when GPS radio waves cannot be received or when the GPS error is large, self-position determination can be performed based on the CV value that has already been corrected for correction.

Next, an example of a specific operation in one embodiment of the pedestrian navigation system 6300 having the above configuration will be described with reference to FIG.
In the operation example shown in the figure, it is assumed that the vehicle is driven to the vicinity of the destination by the car navigation device, the vehicle is stopped at the parking lot near the destination, and then guided to the final destination by pedestrian navigation from there on foot. ing.
First, the destination point can be set on the map of the car navigation device 6330 or on the map of the pedestrian navigation device 6320.
The destination point can be set on any display of the pedestrian navigation device 6320 and the car navigation device 6330. In general, the display of the car navigation device 6330 has a larger screen, and the pedestrian navigation device 6320 has the car navigation device. In the state connected to 6330, it is advantageous to set on the large screen display of the car navigation device 6330.

Only the car navigation device 6330 loaded on the vehicle can guide the guided person to the parking lot near the destination, but by keeping the pedestrian navigation device 6320 connected to the car navigation device 6330 in the middle, It is possible to receive delivery from the information center 6310 of main intersections, branch points, and CV video maps near the destination.
In addition, important points may be overlooked only by the accuracy of the car navigation device 6330, and a location specifying service can be received from the information center 6310 when necessary.
The information center 6310 and the pedestrian navigation device 6320 are connected by a line so that not only the CV video map but also other information can be exchanged as will be described later.
In addition, the position accuracy of the vehicle can be obtained with high accuracy and reflected in the car navigation device 6330 by the CV function as a function of the pedestrian navigation device 6320 alone, regardless of the information center 6310.

The pedestrian navigation device 6320 connected to the car navigation device 6330 receives the distribution of the CV video map from the information center 6310, and acquires the CV video map to which the attribute is added. For example, it is possible to obtain an all-around video of an intersection or an important passing point around the destination or on the way, which is scheduled to turn right or left, as seen from the vehicle position.
In the CV video, the names of the main objects in the video, such as the position of traffic lights necessary for traffic, road signs, road markings, destination markings, street names, building names and important structure names, are marked. Has been.

A destination is set on the car navigation device 6330 before the vehicle travels. If the destination is clearly marked on the video around the destination distributed from the information center 6310, it is set as the destination. If only the periphery is marked, it is registered and set as a passing destination.
The vehicle is highly accurate enough to correct the position measurement by GPS and identify the lane position and stop line by the CV function of the pedestrian navigation device 6320 (integrated with the function of the connected car navigation device during travel). The position can be measured and displayed.
Here, the CV function is a function of acquiring position coordinates by a wide-angle camera by CV calculation and specifying the position.

The function of the pedestrian navigation device 6320 is not only to show road signs, road markings, and destination markings by the attribute display function of the distributed CV video map, but also to direct lines to buildings, stores and hotels registered in the map There is a service function that can be connected with.
In addition, the camera equipped for the CV function can automatically recognize the traffic light.
In addition, the location specifying service can be received by transmitting the video of the local store to the information center 6310. This is effective when the guided person gets lost. This is particularly important when the pedestrian navigation device 6320 is alone.

The car travels mainly with the car navigation device 6330 to the vicinity of the destination and the pedestrian navigation device 6320 as a slave, and a nearby parking lot is selected from the car navigation device 6330 and parked.
If the engine is stopped or the connection between the car navigation device 6330 and the pedestrian navigation device 6320 is disconnected, the CV video map of the important points required from the parking lot received while traveling by the car navigation device 6330 to the vicinity of the destination is walking. It can be automatically copied to the person navigation device 6320.
After the guided person leaves the vehicle, the pedestrian navigation device 6320 is disconnected from the car navigation device 6330 and used while being worn by the guided person.
Thereafter, the pedestrian navigation device 6320 separated from the car navigation device guides the guided person to the destination by a map, video, and voice.

In the pedestrian navigation apparatus 6320, the destination direction is indicated in the all-round image by the CV video map previously distributed from the information center 6310. The guided person can confirm his / her position and the destination direction by comparing the entire circumference image with the actual one.
The guided person walks and proceeds while watching the CV video map in the direction of the destination or watching the two-dimensional map by the pedestrian navigation device 6320.
If the pedestrian navigation device 6320 includes the GPS unit 63222 (see FIG. 68), the GPS location information can be displayed on the map.
When a pedestrian navigation device 6320 that does not include the GPS unit 63222 or a place where GPS radio waves cannot be received, the CV function of the pedestrian navigation device 6320 allows the guided person to know his / her position.

Further, the pedestrian navigation device 6320 includes a wide-angle camera, and can have a CV calculation function that calculates and calculates its own moving direction and position from the camera video.
Specifically, an image around the place where the guided person stands is photographed automatically or manually by a camera provided in the pedestrian navigation device 6320 and transmitted to the information center 6310. The information center 6310 compares the received video with the CV video recorded in the database to determine the shooting position of the received video.
The result of the position determination is sent back to the pedestrian navigation device 6320 side as three-dimensional coordinates, and the received coordinates can be displayed on the map or on the CV video map on the pedestrian navigation device 6320 side.
In this way, even if the guided person gets lost, he can check his / her position at any time by this position specifying service function.

In the case where the user's position cannot be specified only by the delivered CV video, the peripheral video acquired by the camera of the pedestrian navigation device 6320 can be transmitted to the information center 6310 to receive the location specifying service.
There are two types of methods for specifying the position by image recognition. As a first method, a moving image capable of CV calculation is transmitted from the pedestrian navigation apparatus 6320, and the position is specified by three-dimensional comparison with the CV video of the information center 6310. This method has high accuracy but is expensive in terms of transmitting moving images.
As a second method, it is possible to specify the position by transmitting a still image from the pedestrian navigation device 6320 and comparing it with the CV video on the information center 6310 side. This method transmits a still image of a two-dimensional image, but since the original is a moving image, the CV value is obtained. Therefore, by transmitting the CV value of a series of movements simultaneously, the information center It is a good clue when comparing on the 6310 side, and accuracy is obtained. In the two-dimensional map display of the basic function, the user's position and destination direction can be displayed on the two-dimensional map.

If the GPS unit 63222 is attached to the pedestrian navigation device 6320, guidance in a range where GPS radio waves can be received is possible. However, if the GPS unit 63222 is not attached, an error occurs even if it is attached, Furthermore, appropriate guidance cannot be performed due to radio wave conditions, or in buildings, underground malls, or places where radio waves do not reach.
On the other hand, in the pedestrian navigation device 6320 with the CV function, it is possible to obtain the amount of movement by a vector using a CV video and one specific service even in a place where radio waves do not reach, and thus guidance is possible.

Although it is possible to specify the position to some extent by the CV function even in an actual space without processing for the purpose of guidance, if a certain rule and a convention marker are attached to a passage or nearby utility pole or wall, etc. The location can be specified by automatic recognition.
The accuracy of the position can be improved even if there is a marker for automatically specifying the location, or simply by adding a marker as a feature point for facilitating CV calculation. Various markers can be marked on the ceiling as feature points in buildings and underground malls. For the pedestrian navigation apparatus 6320, the building, the ceiling of the underground shopping area, etc. are excellent information addition parts.
The marker may be captured with the naked eye, but is basically captured with a camera and identified by image processing. Therefore, the marker is not visible to the naked eye, but can be a marker that reflects near-infrared light with sensitivity of the CCD of the camera so as to be captured by the camera.
Also, registering a characteristic shape in the real space as a three-dimensional feature facilitates CV calculation, contributing to accuracy and reliability.

The final guidance to the destination is convenient if there is image information of the destination, but if there is a marker indicating the final destination, the guidance is completed there. Whether it is a marker added for position recognition or an original attribute, it contributes to position identification.
If the final destination is not registered, guidance is made to the passing destination, but since the route to the final destination can be recorded thereafter, the final destination can be set from the next time.
The final destination can be set by acquiring a marker or CV video of the final destination from another. Even if the coordinates of the final destination are unclear, if there is a marker that can be easily specified, it can be set as the final destination.
Note that since the guidance based on the CV video is mainly a video, not a two-dimensional map, the accuracy of coordinates is not important.

As described above, the high-precision CV arithmetic device of the present invention, the CV method three-dimensional map generation device and the CV method navigation device provided with the high-precision CV arithmetic device have been described with reference to preferred embodiments. Needless to say, the present invention is not limited to the embodiments, and various modifications can be made within the scope of the present invention.
For example, a mobile body to which a CV system 3D map generation apparatus or a CV system navigation apparatus provided with the high-precision CV arithmetic apparatus of the present invention can be applied is not limited to a vehicle traveling on the ground, but an airplane traveling in a 3D space. It may be.

  The present invention can be suitably used for, for example, a three-dimensional map generation device for generating a three-dimensional map provided in a car navigation device mounted on an automobile.

It is a block diagram which shows the basic composition of the high precision CV arithmetic unit which concerns on one Embodiment of this invention. It is a block diagram which shows the detail of the high precision CV arithmetic unit shown in FIG. It is the schematic which shows the state which mounted the video camera apparatus part of the high precision CV arithmetic unit which concerns on one Embodiment of this invention in a moving body, and is a perspective view of the vehicle which mounts the perimeter camera on the roof part. BRIEF DESCRIPTION OF THE DRAWINGS It is the schematic which shows the state which mounted the video camera equipment part of the high precision CV arithmetic unit which concerns on one Embodiment of this invention in a moving body, (a) is a front view of the vehicle which mounts the perimeter camera on a roof part, (B) is also a plan view. It is explanatory drawing which shows the conversion image obtained from the image | video image | photographed with a omnidirectional camera, (a) is a virtual spherical surface to which a spherical image is affixed, (b) is an example of a spherical image affixed to a virtual spherical surface , (C) shows an image obtained by developing the spherical image shown in (b) on a plane according to the Mercator projection. It is a block diagram which shows the basic composition of the CV data calculating part of the high precision CV calculating apparatus which concerns on one Embodiment of this invention. It is explanatory drawing which shows the specific detection method of a camera vector in the CV data calculating part which concerns on one Embodiment of this invention. It is explanatory drawing which shows the detection method of the specific camera vector in the CV data calculating part which concerns on one Embodiment of this invention. It is explanatory drawing which shows the detection method of the specific camera vector in the CV data calculating part which concerns on one Embodiment of this invention. It is explanatory drawing which shows the designation | designated aspect of the desirable feature point in the detection method of the camera vector by the CV data calculating part which concerns on one Embodiment of this invention. It is a graph which shows the example of the three-dimensional coordinate and camera vector of the feature point obtained by the CV data calculating part which concerns on one Embodiment of this invention. It is a graph which shows the example of the three-dimensional coordinate and camera vector of the feature point obtained by the CV data calculating part which concerns on one Embodiment of this invention. It is a graph which shows the example of the three-dimensional coordinate and camera vector of the feature point obtained by the CV data calculating part which concerns on one Embodiment of this invention. It is explanatory drawing which shows the case where a some feature point is set according to the distance of a feature point from a camera, and a some calculation is repeated in the CV data calculating part which concerns on one Embodiment of this invention. It is a figure at the time of displaying the locus | trajectory of the camera vector calculated | required by the CV data calculating part which concerns on one Embodiment of this invention in a video image | video. It is explanatory drawing which shows the comparison of the error of CV data and GPS data. It is a block diagram which shows the structure of the high-precision CV calculating apparatus which performs CV calculation from the parallax three-dimensional shape by the some camera which concerns on one Embodiment of this invention. It is a block diagram which shows the structure of the high precision CV arithmetic unit which performs CV data correction | amendment by the absolute length CV value of the several camera which concerns on one Embodiment of this invention. It is explanatory drawing which shows the camera apparatus with which the high-precision CV arithmetic unit shown in FIG. 18 is equipped, (a) is a top view of the two synchronized cameras which comprise a several video camera apparatus part, (b) is (a). ) And a plan view of a vehicle in which an asynchronous all-around camera constituting the all-around video camera device unit is fixed to the roof, and (c) is a right side view of the vehicle. (A) is a side view conceptually showing an image taken by the camera device shown in FIG. 19, (b) is a plan view, and (c) is a camera provided in the high-precision CV arithmetic unit shown in FIG. It is a top view of the vehicle which shows the other structural example of an apparatus. (A)-(c) is a top view which shows the example of a combination structure of the several camera with which the high precision CV calculating apparatus shown in FIG. 1 shows a configuration in a case where a high-precision CV calculation device that performs CV data correction based on absolute length CV values of a plurality of cameras according to an embodiment of the present invention includes a parallax type camera coordinate three-dimensionalization unit that generates a three-dimensional camera coordinate. It is a block diagram. It is a block diagram which shows the structure of the high precision CV arithmetic unit which performs CV data correction by GPS which concerns on one Embodiment of this invention. It is a block diagram which shows the structure of the high precision CV arithmetic unit which performs GPS data correction concerning one Embodiment of this invention. It is a block diagram which shows the structure of the high precision CV arithmetic unit which performs 3 axis | shaft rotation accumulation error correction by IMU which concerns on one Embodiment of this invention. It is explanatory drawing which shows the camera apparatus with which the high-precision CV arithmetic unit shown in FIG. 25 is equipped, (a) is a top view of two cameras for absolute length measurement, (b) is the camera shown to (a), The top view of the vehicle which fixed the perimeter video camera and IMU apparatus to the roof part, (c) is a right view of a vehicle similarly. (A) is explanatory drawing which shows typically the camera apparatus and measurement apparatus with which the high-precision CV calculating apparatus shown in FIG. 25 is equipped, (b) is acquired with the camera apparatus and measurement apparatus shown in (a). It is explanatory drawing which shows the correspondence of the data value to be performed. FIG. 26 is a plan view of a vehicle traveling direction schematically showing a correspondence relationship between data acquired by a camera device and measurement devices provided in the high-precision CV arithmetic device shown in FIG. 25. FIG. 26 is a plan view of the vehicle traveling direction schematically showing error change points at which GPS data becomes discontinuous in the high-precision CV arithmetic device shown in FIG. 25. FIG. 26 is a plan view of a vehicle traveling direction schematically showing correction of CV data by an absolute value of GPS data in the high-precision CV arithmetic device shown in FIG. 25. FIG. 26 is a plan view of a vehicle traveling direction schematically showing correction of GPS discontinuity points by linearity of CV values in the high-precision CV arithmetic device shown in FIG. 25. In the high-accuracy CV calculation device shown in FIG. 25, an explanatory diagram for explaining GPS absolute coordinate correction by a temporary correction true value at an error change point, (a) schematically shows a discontinuous point of GPS data. (B) is a plan view showing the distribution of difference values at the discontinuous points shown in (a), and (c) is an arithmetic expression for obtaining the center of gravity of the distribution shown in (b). is there. FIG. 26 is an explanatory diagram for explaining correction of CV data by GPS data in the high-accuracy CV arithmetic device shown in FIG. 25, and (a) corresponds to GPS sampling points and CV frames by GPS data with increased absolute accuracy. The top view which shows typically the state which correct | amended CV data and gave the absolute value in all the points, (b) is a figure which shows the specific numerical example of CV data before correction | amendment and after correction | amendment. . It is a block diagram which shows the structure of the high precision CV calculating apparatus which performs 1 axis | shaft rotation accumulation error correction by the vertical direction detection which concerns on one Embodiment of this invention. It is a figure which shows the example of an image by which plane conversion (perspective conversion) was carried out substantially right below by the high-precision CV arithmetic unit shown in FIG. 34, (a) is a video image, (b) is the image which extracted only the outline from the video image. It is. It is a figure which shows the example of an image by which plane conversion (perspective conversion) was carried out substantially right above by the high-precision CV arithmetic unit shown in FIG. 34, (a) is a video image, (b) extracted only the outline from the video image. It is an image. It is a figure which shows the example of an image by which plane conversion (perspective conversion) was carried out to the equatorial plane direction roughly by the high-precision CV arithmetic unit shown in FIG. 34, (a) is a video image, (b) extracted only the outline from the video image. It is an image. It is a block diagram which shows the structure of the high precision CV calculating apparatus which performs the high precision CV calculation by the aerial image | video and ground image based on one Embodiment of this invention. It is a block diagram which shows the structure of the high precision CV calculating apparatus provided with the two-dimensional map which concerns on one Embodiment of this invention. It is a block diagram which shows the structure of the example of a change of the high precision CV calculating apparatus provided with the two-dimensional map which concerns on one Embodiment of this invention. It is a block diagram which shows the structure of the high precision CV calculating apparatus which performs high precision CV calculation using the known three-dimensional point which concerns on one Embodiment of this invention. It is a block diagram which shows the basic composition of the CV type | system | group 3D map production | generation apparatus which concerns on one Embodiment of this invention. (A)-(b) is a top view which shows the driving | running | working pattern of the traveling vehicle for surveying in the three-dimensional map production | generation apparatus shown in FIG. It is explanatory drawing explaining the specific processing operation in the three-dimensional map production | generation apparatus shown in FIG. 42, and is the block diagram which simulated the display surface which can be operated with a viewer. It is a block diagram which shows the structure of the CV type | system | group PRM three-dimensional map production | generation apparatus which concerns on one Embodiment of this invention. It is a block diagram which shows the structure of the three-dimensional map production | generation apparatus which performs the three-dimensional map production | generation by the aerial image based on one Embodiment of this invention. It is a block diagram which shows the structure of the example of a change of the three-dimensional map production | generation apparatus which performs the three-dimensional map production | generation by the aerial image which concerns on one Embodiment of this invention. (A) is a perspective view which shows an example of the aircraft which mounts the three-dimensional map production | generation apparatus shown in FIG. 46, (b) is the example of arrangement | positioning of the wide-angle video camera and GPS apparatus with which the aircraft shown to (a) is equipped. FIG. It is explanatory drawing which showed typically the visual field of the wide angle video camera shown in FIG. 47, (a) is a front view, (b) is a top view. It is a block diagram which shows the structure of the 3D map production | generation apparatus which produces | generates a 3D image | video from the 2D moving image image | video which concerns on one Embodiment of this invention. FIG. 50 is a diagram illustrating an example of an operator for image division in the three-dimensional map generation device illustrated in FIG. 49, and is an image example in which a source point direction operator is applied to a general image and a range is designated. 49A is a diagram illustrating an example of an operator for image division in the three-dimensional map generation apparatus illustrated in FIG. 49. FIG. 49A is a simple block image example of a rectangular operator, and FIG. 49B is a diagram illustrating the rectangular operator illustrated in FIG. It is an example of an image that specifies a range in conformity with a general image. It is. 49 is a diagram showing an example of an operator for image division in the three-dimensional map generation device shown in FIG. 49, (a) is an image example showing the relationship between the source point position in the plane conversion image and the source point direction division operator; (B) is an enlarged view thereof. It is a figure which shows an example of the operator for an image division in the three-dimensional map production | generation apparatus shown in FIG. 49, It is an example of an image which shows the starting point position and suction point position in a Mercator image, and image division. FIG. 50 is a diagram illustrating an example of an operator for image division in the three-dimensional map generation apparatus illustrated in FIG. 49, and is an image example illustrating a source point position, a suction point position, and an image division in a spherically developed image. (A)-(b) is explanatory drawing which shows the image division process sequence in the three-dimensional map production | generation apparatus shown in FIG. (A)-(b) is explanatory drawing which shows the image division process sequence in the three-dimensional map production | generation apparatus shown in FIG. (A)-(b) is explanatory drawing which shows the image division process sequence in the three-dimensional map production | generation apparatus shown in FIG. It is a block diagram which shows the structure of the three-dimensional map production | generation apparatus of the parallax combined method by multiple cameras which concerns on one Embodiment of this invention. FIG. 59 is an explanatory diagram illustrating a configuration example of a parallax camera in the three-dimensional map generation device illustrated in FIG. 58. It is a block diagram which shows the structure of the CV type navigation apparatus which concerns on one Embodiment of this invention. FIG. 61 is a block diagram illustrating specific processing operations in the navigation device shown in FIG. 60. FIG. 61 is a right side view of a vehicle schematically showing a vehicle on which the navigation device shown in FIG. 60 is mounted. It is a block diagram which shows the basic composition of the pedestrian navigation system which concerns on one Embodiment of this invention. It is a block diagram which shows another example of the basic composition of the pedestrian navigation system which concerns on one Embodiment of this invention. It is a block diagram which shows an example of the specific structure of the pedestrian navigation system which concerns on one Embodiment of this invention. It is a block diagram which shows another example of the specific structure of the pedestrian navigation system which concerns on one Embodiment of this invention. It is a block diagram which shows another example of the specific structure of the pedestrian navigation system which concerns on one Embodiment of this invention. It is a block diagram which shows another example of the specific structure of the pedestrian navigation system which concerns on one Embodiment of this invention. It is a block diagram which shows an example of operation | movement of the pedestrian navigation system which concerns on one Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 High precision CV arithmetic unit 10 Mobile body measurement part 20 Data generation part 30 Time axis alignment part 40 CV correction part 50 High precision CV data output part 60 CV image composition display part

Claims (17)

A moving body measuring unit fixed to the moving body, capturing a video image by photographing the periphery of the moving body along with the movement of the moving body, and measuring position measurement data including position data and movement amount data of the moving body;
A video image acquired by the moving body measuring unit is recorded, a predetermined CV calculation is performed to generate CV data indicating a three-dimensional position and a three-axis rotation position of the camera from the video image, and the moving body measuring unit A data generation unit for acquiring measured position measurement data;
A time axis matching unit that generates CV data generated by the data generation unit as time-synchronized CV data associated with a reference time and generates the position measurement data as time-synchronized measurement data associated with a reference time When,
The time-synchronized CV data and the time-synchronized measurement data are compared in association with each other on the same time axis, and the time-synchronized CV data and the time-synchronized measurement data are complementarily corrected according to the comparison result, and the final CV value of the CV data A CV correction unit for generating a CV correction signal for correcting
A high-precision CV data output unit that outputs a video image recorded in the data generation unit and outputs a high-precision CV value corrected by the CV correction signal corresponding to each frame of the video image;
With
The data generator is
A feature point extraction unit that automatically extracts a predetermined number of feature points from image data of a moving image, and the extracted feature points are automatically tracked within each frame image of the moving image to obtain a correspondence between the frame images. The feature point correspondence processing unit and the three-dimensional position coordinates of the feature point for which the correspondence relationship is obtained are obtained, and the camera composed of the three-dimensional position coordinates and the three-dimensional rotation coordinates of the camera corresponding to each frame image from the three-dimensional position coordinates. A high-precision CV computing device comprising a camera vector computing unit for obtaining a vector and a CV data computing unit for performing the predetermined CV computation ;
An object designating unit for designating an arbitrary object to be generated as a three-dimensional map in the video image output from the high-precision CV data output unit of the high-precision CV arithmetic unit;
A target three-dimensional measurement unit that three-dimensionally measures the target specified by the target specification unit;
From the data three-dimensionally measured by the object designating unit, a three-dimensional map component generation unit that generates a three-dimensional component in a format described in a three-dimensional map,
A three-dimensional space definition section for defining a three-dimensional space as a three-dimensional map;
Surveying the position of the object on the coordinates, and obtaining the three-dimensional coordinates of the object point surveying unit,
A three-dimensional coordinate determination unit that determines a three-dimensional arrangement of the three-dimensional map component in the three-dimensional space defined by the three-dimensional space definition unit based on the three-dimensional coordinates obtained by the survey of the object point surveying unit. When,
A three-dimensional map output unit for outputting a three-dimensional map by repeating the three-dimensional coordinate arrangement of the three-dimensional map component;
With
The video image is acquired by moving so that a moving body for generating a map intersects in a certain section, and includes images of aerial photography and ground photography,
The CV method three-dimensional map generation device , wherein the CV calculation performed by the high-precision CV calculation device combines the aerial and ground images by CV calculation to obtain a CV value at the same time . A moving body measuring unit fixed to the moving body, capturing a video image by photographing the periphery of the moving body along with the movement of the moving body, and measuring position measurement data including position data and movement amount data of the moving body;
A video image acquired by the moving body measuring unit is recorded, a predetermined CV calculation is performed to generate CV data indicating a three-dimensional position and a three-axis rotation position of the camera from the video image, and the moving body measuring unit A data generation unit for acquiring measured position measurement data;
A time axis matching unit that generates CV data generated by the data generation unit as time-synchronized CV data associated with a reference time and generates the position measurement data as time-synchronized measurement data associated with a reference time When,
The time-synchronized CV data and the time-synchronized measurement data are compared in association with each other on the same time axis, and the time-synchronized CV data and the time-synchronized measurement data are complementarily corrected according to the comparison result, and the final CV value of the CV data A CV correction unit for generating a CV correction signal for correcting
A high-precision CV data output unit that outputs a video image recorded in the data generation unit and outputs a high-precision CV value corrected by the CV correction signal corresponding to each frame of the video image;
With
The data generator is
A feature point extraction unit that automatically extracts a predetermined number of feature points from image data of a moving image, and the extracted feature points are automatically tracked within each frame image of the moving image to obtain a correspondence between the frame images. The feature point correspondence processing unit and the three-dimensional position coordinates of the feature point for which the correspondence relationship is obtained are obtained, and the camera composed of the three-dimensional position coordinates and the three-dimensional rotation coordinates of the camera corresponding to each frame image from the three-dimensional position coordinates. A high-precision CV computing device comprising a camera vector computing unit for obtaining a vector and a CV data computing unit for performing the predetermined CV computation;
A target range designating unit for designating an arbitrary target range to be generated as a three-dimensional map in the video image output from the high precision CV data output unit of the high precision CV arithmetic unit;
A measurement point density designating unit for designating a measurement point density in the target range designated by the target range designating unit;
Based on the specified target range and the measurement point density, a measurement point generation unit that generates measurement points;
A measurement point tracking unit that tracks the measurement points over a plurality of adjacent frames of the video image;
Measurement that calculates the three-dimensional coordinates of the measurement point based on the CV value output from the high-precision CV data output unit of the high-precision CV calculation device and the measurement point from which the tracking data is acquired by tracking the measurement point A point calculator,
A target range surveying unit for obtaining a three-dimensional coordinate distribution of all measurement points in the target range;
A three-dimensional shape generator for generating a three-dimensional shape created by the measurement points from the three-dimensional distribution of the measurement points;
A specific measurement point designation registration unit that designates a part of an object composed of a straight line portion as a specific measurement point in the target range in the video image, and registers the attribute together with the attribute;
By specifying the specific measurement point, the attribute of the specific measurement point is sent to the measurement point generation unit, tracked and calculated in the same manner as a general measurement point, and output to the measurement point calculation unit, From the three-dimensional coordinate data output from the measurement point calculation unit, only the data of the specific measurement point is extracted, and the specific measurement point formation unit that gives a shape suitable for the attribute,
A three-dimensional coordinate integration unit that integrates a three-dimensional shape formed by specific measurement points output from the specific measurement point forming unit with a three-dimensional shape obtained from the measurement points other than the specific measurement points;
A 3D map output unit for generating and outputting a 3D map based on the integrated 3D shape;
With
The measurement point density designation unit is
An image stabilization unit that generates a video in which a rotation component of a video image is stopped based on a CV value acquired from a high-precision CV data output unit of the high-precision CV arithmetic unit;
A source point determination unit for determining a source point that is a starting point of the motion of the video image in which the rotation component is stopped;
An image block dividing unit that divides an image frame of an arbitrary designated range into a set of image blocks of a small area;
A feature point extraction unit that extracts a block that can be a feature point for all of the image blocks;
For all the image blocks, a boundary point extraction unit that extracts a block that can be a part of the boundary line;
A region point extraction unit that extracts points other than the feature points and boundary points of the image block as region points;
Each point of the image block is coupled in a direction extending radially from the spring point, and is fixed so as to maintain a relationship adjacent to the order, and the distance relationship between the adjacent blocks can be arbitrarily expanded and contracted. A two-dimensional rubber string combined image generating unit to be combined,
The measurement point generator is
A feature point selection unit for selecting the feature points as measurement points;
An area generation unit for generating an area created by the area points and the feature points and boundary points surrounding the area points;
A measurement point determination unit that is a tracking measurement target with the feature point, measurement point, and region point as a measurement region, and
The measurement point tracking unit is
The feature points, boundary points, and measurement areas as the measurement points are tracked over a plurality of adjacent frames along the spring point direction,
The measurement point calculation unit is
Based on the tracking results of the measurement points and the CV data output from the high-precision CV data output unit of the high-precision CV arithmetic device, the three-dimensional coordinates of the measurement points and the three-dimensional coordinates of the region are acquired,
The measurement point surveying unit
3D coordinates of boundary points surrounding the measurement point and measurement area are repeatedly calculated over the target range, and if there is an image block that cannot be tracked by the measurement point tracking unit, the image block has already been calculated. Calculated by interpolation based on the known three-dimensional coordinates of, and predicting the three-dimensional position, surveying the target range,
The three-dimensional shape generation unit
From the three-dimensional coordinates of the feature point and the boundary point, a boundary line belonging classification unit that determines a region to which the feature point and the boundary point belong;
With the affiliation of the feature point and the boundary point, cut one side of the rubber cord coupling of the block, cut off the coupling between one region, and a rubber cord cutting part to be coupled to the other region,
All blocks in the target range are classified into feature points, boundary points, and area points. The feature points are fixed at the coordinates obtained as measurement points, and the boundary points and area points whose coordinates are not fixed are interpolated on a rubber string. A three-dimensional rubber string coupling image unit that gives the respective three-dimensional coordinates and three-dimensionally couples all blocks with a rubber string;
The three-dimensional coordinate integration unit
Image division for changing the division density of the image block division unit from coarse density to higher density sequentially every time the three-dimensional rubber string combined video is generated, and performing recalculation of the process from the previous image block division A density changing section;
The image combined with the three-dimensional rubber string is temporarily recorded according to the initially set density of the image block division, and in the next process, more detailed three-dimensional shapes are sequentially generated, and the process is repeated to a predetermined density. 3D image recording integration unit that generates 3D shapes sequentially and generates more detailed 3D images,
A CV method three-dimensional map generation apparatus comprising: The moving body measuring unit is
With multiple video camera equipment,
The data generator is
A parallax camera coordinate three-dimensionalization unit that generates a three-dimensional shape in a camera coordinate system from a plurality of images with parallax of video images obtained by the plurality of video camera devices;
A three-dimensional feature part is extracted from the three-dimensional shape in the camera coordinate system, and the three-dimensional feature part is tracked on a plurality of adjacent frames of the video image, or is traced on a three-dimensional shape corresponding to the frame. A multi-CV calculation unit that obtains the camera coordinates and the three-axis rotation angle in the stationary coordinate system from the three-dimensional feature part and the camera position in the camera coordinate system by CV calculation;
Alternatively, a plurality of three-dimensional feature points are extracted from a three-dimensional shape based on the camera coordinate system, and the three-dimensional feature points are tracked on a plurality of adjacent frames of the video image or on a three-dimensional shape corresponding to the frame. A multi-CV calculation unit for obtaining a camera position coordinate and its three-axis rotation angle in a stationary coordinate system from the three-dimensional feature point acquired as the camera coordinate and the camera origin coordinate by CV calculation;
The CV type | system | group three-dimensional map production | generation apparatus of Claim 1 or 2 provided with these. The moving body measuring unit is
A video camera device unit that is fixed to the moving body and that captures a video image of the video around the moving body along with the movement of the moving body, and a plurality of video cameras that are fixed to the moving body and have a known positional relationship A plurality of video camera device units that acquire images with parallax around a moving object by a camera, and the all-around video camera device unit and the plurality of video camera device units have a known relationship between a three-dimensional position and a posture. Installed as
The data generator is
An all-around video image recording unit for recording all-around video images obtained by the all-around video camera device unit;
Extracting a plurality of feature points from each frame image of the all-round video image recorded in the all-around video image recording unit, tracking the feature points to a plurality of adjacent frames of the all-around video image, A CV calculation unit that calculates CV data indicating the position coordinates and the three-axis rotation angle of the all-around camera in the stationary coordinate system based on the plurality of feature points in the coordinate system and the camera positional relationship;
A plurality of feature points are extracted in each frame image of the video image obtained by the plurality of video camera device units, the feature points are tracked in each adjacent frame image of the video image, and the distance between the cameras is known. A multi-CV calculation unit that obtains an absolute distance by tracking within each corresponding frame image of another camera, and calculates multi-CV data indicating a high-precision camera three-dimensional coordinate and a three-axis rotation posture; With
The time axis alignment unit is
A clock unit for supplying time data to the CV data and the multi-CV data;
A time-synchronized CV data recording unit that records the CV data in association with each other based on time data supplied from the clock unit;
A time-synchronized multi-CV data recording unit that records the multi-CV data in association with each other based on time data supplied from the clock unit;
A delay adjustment CV signal output unit for correcting a delay in recording time between CV data acquired from a plurality of camera images by a time difference from the time data supplied from the clock unit and outputting a delay adjustment CV signal;
A delayed multi-CV signal output unit that outputs a delay-adjusted multi-CV signal by correcting a delay in recording time between multi-CV data acquired from a plurality of camera images by a time difference from the time data supplied from the clock unit; With
The CV correction unit is
A coordinate shift due to a difference in installation position between the plurality of video camera device units and the all-round video camera device unit is generated based on the CV data, and the position coordinates of the plurality of video camera device units are A device coordinate position correction unit for correcting the position;
A coordinate array shape comparison unit that detects a coordinate array shape difference by comparing the coordinate array shape of the delay adjustment multi-CV signal with the coordinate array shape of the delay adjustment CV signal;
From the coordinate arrangement shape of the delay adjustment multi-CV signal, the coordinate arrangement shape of the delay adjustment CV signal, and the difference in the coordinate arrangement shape of the section in which the corresponding relationship is established, a correction signal for correcting the scale error in the three-axis direction is generated. A correction signal generation unit to perform,
A CV data scale correction unit that corrects the CV data so that the coordinate array shape by the CV data corresponding to the section matches the coordinate array shape by the multi-CV data;
A global scale correction unit that generates continuous position correction CV data by repeating the correction process in the CV data scale correction unit over the entire target section;
A CV recalculation unit that performs recalculation of high-precision CV data including three-axis rotation with the three-dimensional coordinates of the CV data corrected by the global scale correction unit being known,
High-precision CV data output unit
An all-around video image output unit for outputting all-around video images acquired by the all-around video camera device unit;
A recalculation CV that outputs a CV data signal that is position-corrected by the global scale correction unit, recalculated by the CV recalculation unit, and corrected for three-axis rotation in synchronization with each image frame of the all-round video image output. A CV type three-dimensional map generation apparatus according to any one of claims 1 to 3 , further comprising a data output unit. The data generator is
A parallax-type camera coordinate three-dimensionalization unit that generates a three-dimensional shape in a camera coordinate system from a plurality of images with parallax of the plurality of video camera device units, and a parallax between cameras
A plurality of three-dimensional feature parts are extracted from the three-dimensional shape in the camera coordinate system acquired by the parallax method camera coordinate three-dimensionalization unit, and the three-dimensional feature parts are extracted between adjacent frames of the video image. Or between the three-dimensional shapes corresponding to the frame, and based on the three-dimensional feature portion and the camera positional relationship, the camera position coordinates in the stationary coordinate system of the camera provided in the moving body and its The CV type | system | group 3D map production | generation apparatus of Claim 4 provided with the multi-CV calculating part which calculates | requires a 3-axis rotation angle by CV calculation. The moving body measuring unit is
An all-round video camera device unit that captures an all-around video image by photographing the periphery of the moving body along with the movement of the moving body, and a fixed three-dimensional position of the moving body. GPS measurement device unit that acquires at latitude and longitude altitude, these all-around video camera device unit and GPS measurement device unit is installed with a known relationship between the three-dimensional position and orientation,
The data generator is
An all-around video image recording unit for recording all-around video images obtained by the all-around video camera device unit;
Extracting a plurality of feature points in each frame image of the all-around video image recorded in the all-around video image recording unit, tracking the feature points to a plurality of adjacent frames of the all-around video image, A CV data acquisition unit that obtains CV data indicating a three-dimensional position coordinate and a three-axis rotation angle in a stationary coordinate system of an all-around camera provided in the moving body based on a feature point and a camera positional relationship;
An absolute coordinate data acquisition unit that acquires absolute coordinate data indicating a three-dimensional position of the GPS measurement device unit from the GPS measurement device unit,
The time axis alignment unit is
A clock unit for supplying time data to the CV data and the absolute coordinate data;
A time-synchronized CV data recording unit that records the CV data in association with each other based on time data supplied from the clock unit;
A time-synchronized absolute coordinate recording unit that records the absolute coordinate data in association with the time data supplied from the clock unit;
A delay adjustment CV signal output unit for correcting a delay in recording time between CV data acquired from a plurality of camera images by a time difference from the time data supplied from the clock unit and outputting a delay adjustment CV signal;
A delay adjustment absolute coordinate signal output unit that corrects a time difference between the absolute coordinate signal output time of the absolute coordinate data and the measurement time and outputs a delay adjustment absolute coordinate signal, and
The CV correction unit is
A coordinate due to a difference in installation position between the GPS measurement device unit and the all-round video camera device unit and a deviation of three-axis rotation are generated based on the CV data, and a position coordinate of the GPS measurement device unit is calculated from the all-round video camera device. A coordinate position rotation correction unit that outputs a delayed coordinate rotation correction signal by correcting to the position of the unit;
A coordinate array shape comparison unit that compares the coordinate array shape of the delay adjustment absolute coordinate signal corrected by the coordinate rotation correction signal with the coordinate array shape of the delay adjustment CV signal;
An absolute coordinate accuracy change point detection unit that detects a point of the delay adjustment absolute coordinate signal deviating from the coordinate array of the CV values as a coordinate change point of the absolute coordinate by comparing the coordinate array shape of the two coordinate arrays;
A high-accuracy relative value section selection unit that selects a plurality of sections having a high relative accuracy excluding the absolute coordinate accuracy change point;
For CV data including a scale error due to accumulation of calculation errors, a section of a point corresponding to the high-precision relative value section selection unit based on the absolute coordinate data and a section of a point based on CV data obtained by calculation from a video image, Corresponding time as a parameter, when there is no measurement value at the coincidence point of the start point and the end point time divided by the interval, a measurement interval corresponding unit to obtain and correspond to the matching point by interpolation,
A correction signal generation unit that generates a correction signal for correcting CV data including a scale error, with the distance between the absolute coordinates of the start point and end point of the section in which the correspondence relationship is taken as a section distance;
A CV data scale correction unit for correcting the CV data by matching the CV data corresponding to the section with the section distance data based on the absolute coordinate data having high relative accuracy;
A global scale correction unit that generates continuous high-precision CV data by repeating the correction process in the CV data scale correction unit a plurality of times over the entire area of the target section,
High-precision CV data output unit
An all-around video output unit for outputting all-around video images acquired by the all-around video camera device unit;
Wherein in synchronization with each image frame all around the video image output, the entire scale corrected CV CV data output unit for outputting data signals and, CV method three-dimensional according to any one of claims 1 to 5 comprising a Map generator . The CV correction unit is
A distribution statistical processing unit that statistically processes the distribution of the absolute accuracy change points detected by the absolute accuracy change point detection unit;
A temporary true value calculation unit for determining a statistical center of the distribution of the absolute accuracy change points as a temporary true value;
Temporary true value correction data acquisition for acquiring a correction value of a section indicated by a start point and an end point of the absolute accuracy change point indicating a section in which an error of the GPS measurement data is to be corrected based on the temporary true value And
An absolute coordinate correction unit for obtaining a high-accuracy absolute coordinate array obtained by correcting the absolute coordinate array based on the correction value of the section;
The high-precision CV data output unit is
An absolute coordinate-added CV data integration unit that gives absolute coordinates to the scale-corrected CV data output from the high-precision CV data output unit using the high-precision absolute coordinate array as an absolute coordinate reference;
The CV type three-dimensional map generation apparatus according to claim 6 , further comprising: an absolute coordinate CV data output unit that outputs CV data to which absolute coordinates are added in synchronization with each image frame. The moving body measuring unit is
An IMU measuring instrument that measures absolute 3-axis rotation posture
The data generator is
An absolute triaxial rotational attitude data acquisition unit that acquires absolute triaxial rotational attitude data measured by the IMU measuring instrument unit;
The time axis alignment unit is
A time-synchronized triaxial rotational attitude recording unit that records the absolute triaxial rotational attitude data in association with a time signal supplied from the clock unit;
A delay adjustment 3-axis rotation attitude output unit that corrects a deviation between the recording time of the absolute 3-axis rotation attitude data and a reference time and outputs a point adjustment 3-axis rotation attitude data signal;
The CV correction unit is
A three-axis rotation posture comparison unit that compares the three-axis rotation posture signal from the delay adjustment CV signal output unit and the absolute rotation three-axis posture signal from the delay adjustment three-axis rotation posture output unit;
A triaxial rotation threshold value comparison unit that compares a difference between the triaxial rotation posture signal and the absolute rotation triaxial posture signal with a predetermined threshold value;
A three-axis rotation correction section setting unit that specifies a correction section when a difference between the three-axis rotation attitude signal and the absolute rotation three-axis attitude signal exceeds the threshold;
A triaxial rotation correction signal generation unit that generates a correction value in the correction section,
The high-precision CV data output unit is
Wherein the three-axis rotational position data synchronized to each image frame of the CV data outputted from the CV data output unit, according to claim 4 to 7 comprises a 3-axis rotation correction CV data output unit for outputting by adding 3-axis rotational correction The CV type | system | group three-dimensional map production | generation apparatus in any one. The data generator is
A general bipolar plane expansion unit for plane-converting the entire circumference video image recorded in spherical transformation coordinates in a substantially straight down direction and a straight up direction;
A general equatorial plane development unit that plane-transforms the entire video image recorded in spherical transformation coordinates into a plurality of equatorial plane directions;
Wherein detecting a vertical direction based on the vertical component contained in the plane converted image, the detected vertical direction CV method of claims 1 to 8 wherein and a vertical direction detector to be reflected in the calculation of the CV data 3D map generator . The moving body measuring unit is
A low-scale all-around video equipment unit that captures an object at a low scale,
A high-scale all-around video image equipment unit for photographing the object at a high scale,
The data generator is
A low-scale video signal acquisition unit for acquiring a low-scale video signal by the low-scale all-around video image equipment unit;
A high-scale video signal acquisition unit that acquires a high-scale video signal by the high-scale all-around video image equipment unit;
The low-scale all-around video image output from the low-scale all-around video image equipment unit is compared with the high-scale all-around video image unit output from the high-scale all-around video image device unit, and are included in both images A common feature point detection unit that detects common feature points by extracting common feature points and taking corresponding points;
A common feature point weight setting unit that sets a weight common feature point weighted by a weight function for each detected common feature point;
A low-scale feature point detector for detecting a low-scale feature point from the low-scale all-around video image;
A low-scale inter-frame tracking unit that tracks detected low-scale feature points in adjacent frames;
A high-scale feature point detection unit that detects a high-scale feature point from the high-scale all-round video image;
A high-scale inter-frame tracking unit that tracks detected high-scale feature points in adjacent frames;
A CV computing unit that performs CV computation by collectively integrating the low-scale all-round video image and the high-scale all-around video image based on the low-scale feature point, the high-scale feature point, and the weight common feature point;
The CV system three-dimensional map production | generation apparatus of Claim 1 thru | or 9 provided with the CV data acquisition part which outputs the CV calculation data with the high precision obtained by the said CV calculation part. A two-dimensional map part given two-dimensional coordinates as known;
A common point detection unit for detecting a common point portion corresponding to the point coordinates of the two-dimensional map unit in the video image acquired by the moving body measurement unit;
A common point two-dimensional coordinate detection unit for detecting a two-dimensional coordinate on a two-dimensional map of the detected common point;
A feature point extracting unit for extracting a traceable feature point in the video image acquired by the moving body measuring unit, and a feature for tracking the extracted feature point in a plurality of adjacent image frames CV data including a point tracking unit, and a CV calculation unit that obtains a three-dimensional position and a three-axis rotation posture of the camera in the stationary coordinate system based on the corresponding points of a plurality of feature points and the camera positional relationship, and acquires CV data An acquisition unit;
Based on CV data from the CV data acquisition unit, a common point three-dimensional coordinate calculation unit that acquires a three-dimensional coordinate of a common point of video images acquired by the moving body measurement unit;
A two-dimensional plane coordinate converter that converts the three-dimensional coordinates of the common point into a two-dimensional plane;
A plurality of common point two-dimensional coordinates obtained by projecting the two-dimensional coordinate of the common point obtained from the data of the two-dimensional map unit and the three-dimensional coordinate of the common point obtained from the video image onto a two-dimensional plane. A common point two-dimensional coordinate comparison unit for comparing
The difference between the two two-dimensional coordinates compared is detected, the position coordinate of the comparison signal that minimizes the difference is determined, the coordinate in the other coordinate axis direction is corrected at the same ratio, and the correction amount of the common point is calculated. A common point region CV value correcting unit that dispersively distributes to the periphery thereof and replaces the position coordinates of the CV value for correcting the entire position coordinates of the CV value;
CV that fixes the position coordinate data of the camera that matches the map at the common point portion, performs CV calculation again based on the tracking result of the common point and the feature point, and obtains more accurate three-axis rotation data A correction calculation unit;
The CV system three-dimensional map production | generation apparatus in any one of Claims 1 thru | or 10 provided with the correction | amendment CV data output part which outputs the correction | amendment CV value which the precision improved by the said correction | amendment. 3D map given as known 3D coordinates, a known 3D point description map showing multiple known 3D coordinate points together with coordinates on a 2D map, or a 3D point showing multiple known 3D coordinates A map unit comprising at least one of the lists;
Among the video images acquired by the moving body measurement unit, a common point detection unit that detects a common point part in an image frame of the video image corresponding to a known three-dimensional point of the map unit,
A common feature point tracking unit that extracts and tracks corresponding points in a plurality of image frames of the video image using the common point part as a common feature point;
A common feature point weighting unit that distinguishes the common feature point from a general feature point, and performs weighting for determining a priority order at the time of calculation;
Based on the weighted tracking results of the common feature points, the tracking results of the general feature points, and the three-dimensional positional relationship of the cameras, the three-dimensional position and three-axis rotation posture of the camera in the stationary coordinate system are calculated. A CV calculation unit obtained by:
A common point three-dimensional coordinate detection unit for obtaining a three-dimensional coordinate of the common feature point output together with a CV value from the CV calculation unit;
A common point three-dimensional coordinate comparison unit that compares the common point three-dimensional coordinates obtained as known from the data of the map unit and the three-dimensional coordinates of the common feature points;
A three-dimensional difference detector for detecting a difference between the compared three-dimensional coordinates;
A general feature point three-dimensionalization unit for obtaining a three-dimensional coordinate of the general feature point output from the CV calculation unit;
Two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the general feature points acquired by the general feature point three-dimensionalization unit onto an image plane in each frame of the video image, and a video of the general feature points in each frame of the video image An image two-dimensional difference detection unit for detecting a two-dimensional difference from the two-dimensional coordinates of the image plane;
A CV correction calculation unit that changes CV calculation parameters and repeats CV calculation in the CV calculation unit so that the values of the three-dimensional difference and the two-dimensional difference are minimized;
The corrected by, CV method three-dimensional map generator according to any one of claims 1 to 11 comprising the CV data acquisition unit for outputting a correction CV value of increased accuracy, the. A 3D parts database storing a plurality of 3D parts constituting a 3D map;
A 3D part candidate selection unit that selects a 3D part that is a candidate to be compared with the target specified by the target specifying unit from among the three-dimensional parts stored in the three-dimensional part database;
A comparison unit that compares a part of the video image designated by the object designating unit with a three-dimensional component output from the three-dimensional component candidate selection unit;
As a result of comparison in the comparison unit, a mismatch signal output unit outputs a mismatch signal in the case of mismatch, and outputs a next candidate three-dimensional part from the three-dimensional part candidate selection unit based on the mismatch signal;
As a result of comparison in the comparison unit, a coincidence signal output unit that outputs a coincidence signal in the case of coincidence
A matching part selecting unit that uses the three-dimensional part finally output from the three-dimensional part candidate part selecting unit as a matching part by the matching signal;
A component correction unit that corrects the matching part to generate a corrected part when the matching part is different from the corresponding video image, and
The matching component or CV method three-dimensional map generation apparatus according to claim 1, wherein sending the modified component on the three-dimensional map component generation unit. A multi-camera shooting unit equipped with a plurality of video cameras for acquiring video images with parallax and parallax;
A parallax camera that obtains the camera coordinate system three-dimensional distance distribution data by calculating the three-dimensional shape of the camera coordinate system based on video images with parallax photographed by the plurality of video cameras moving with the moving body A coordinate three-dimensional section,
A parallax type three-dimensional apparatus comprising :
The camera coordinate three-dimensional distance distribution data obtained by the parallax method three-dimensional device is continuously converted into a three-dimensional distance distribution in a stationary coordinate system based on the high-precision CV value obtained by the high-precision CV arithmetic device. A stationary coordinate system conversion unit for converting data,
A static coordinate system combining unit that combines the three-dimensional distance distribution data in the static coordinate system obtained continuously while overlapping with the progress of the video image, and integrates it into the static coordinate system;
The CV type | system | group 3D map production | generation apparatus in any one of Claims 1 thru | or 13 provided with these . A three-dimensional distance distribution data integrated into a stationary coordinate system by the stationary coordinate system combining unit, a stationary object separation space configuration unit configured only with a stationary object in the stationary coordinate system;
From the three-dimensional distance distribution data integrated into the stationary coordinate system in the stationary coordinate system combining and coupling unit, separating the three-dimensional data with temporal variation, and separating and recording as a moving object,
The CV type | system | group three-dimensional map production | generation apparatus of Claim 14 provided with. Said CV method three-dimensional map generation apparatus according to claim 1 to 15, wherein,
An in-vehicle 3D map device having 3D map information generated by the CV 3D map generator;
A predetermined information necessary for traveling including three-dimensional data and attributes, a traveling route input device that inputs to the three-dimensional map device in a changeable manner, and
An in-vehicle camera that outputs wide-angle video data and relatively low-accuracy CV data at the time of traveling from the in-vehicle camera provided in the traveling vehicle, and a schematic CV computing device;
Based on the rough CV data output from the rough CV calculation device, three-dimensional data indicating the current position and posture of the traveling vehicle is acquired, and the three-dimensional data is roughly associated with the three-dimensional map to form a three-dimensional space configuration. , Detecting obstacles that need emergency response before recognition of the object, sending emergency control signals of avoidance and stop to the real-time control device with the highest priority, by the in-vehicle camera and the approximate CV calculation device, An object space composing device that detects a three-dimensional space structure composed of a plurality of objects that are in a visible range from the travel path and are related to the travel purpose and traffic;
The object space configuration data detected by the object space configuration device is collated with the approximate three-dimensional position data of the traveling vehicle, is in a visible range from the traveling path, matches the purpose of traveling, and is related to traffic, An object recognition device for recognizing a three-dimensional shape of an individual object including a road sign and a road marking by comparing the information of the three-dimensional map with the shape and attributes;
The three-dimensional shape or part shape of a plurality of objects by the object recognition device is three-dimensionally tracked, or a characteristic shape is selected and tracked at the stage of the three-dimensional space configuration before the specification, and the tracking result is used. The high-precision CV calculation device according to claim 1, wherein high-precision CV data is acquired;
Based on CV data acquired by the high-precision CV arithmetic device, the output of the object space constituting device, the object recognition result of the object recognition device, and the output of an in-vehicle measuring instrument including an in-vehicle radar, A current state determination device for outputting a current state determination result of the traveling vehicle;
A real-time control device that controls a predetermined operation unit including an accelerator, a brake, and a handle of the traveling vehicle, or changes data of the traveling route input device, according to an output of the current state determination device;
A CV type navigation device comprising: The precision output of CV computing device, the recognition result of the object recognition apparatus, and claim 16 comprises a real-time display device that displays the current status determination result of said current determination device, in real time on a two-dimensional map or three-dimensional map The CV navigation system described.