JPWO2005038402A1 - Navigation device - Google Patents

Abstract

Provided is a navigation system that can indicate a current position of a moving body within an error of several centimeters. The navigation device 100 obtains predetermined three-dimensional information including three-dimensional coordinates of video feature points in a range observed from the moving body based on a real image obtained by a camera provided on the moving body such as a vehicle. A feature point 3D map generation apparatus 110 to be generated in advance, a recording medium 120 on which the predetermined 3D information is recorded, and a real image obtained by a camera provided on the moving body are recorded on the recording medium 120. Compared with the 3D information of the camera, the position and direction on the 3D coordinates that match the actual image are obtained, and the position, speed, acceleration, viewpoint direction, and 3-axis rotation of the camera provided in the moving body And a point search navigation device 130 for outputting and displaying predetermined items such as posture, triaxial rotational speed, triaxial rotational acceleration, and the like.

Description

  The present invention relates to a navigation device for guiding a traveling direction, a current situation, and the like of a moving body such as a vehicle, a ship, and an aircraft. In particular, the present invention searches for a current position of a moving body that travels and sails in a three-dimensional space and displays it on a three-dimensional map together with the traveling direction of the moving body, the vehicle posture, and the like, thereby moving the moving body. The present invention relates to a navigation device that can output and display the current status of the system with high accuracy.

In general, a car navigation system using a GPS geodetic satellite is known as a navigation device for navigating the movement of a vehicle or the like (see, for example, Patent Documents 1-3).
The GPS navigation system reads the time and position data emitted from a plurality of geodetic satellites with a receiving device installed in the vehicle, calculates the three-dimensional coordinates of the receiving point from the difference in radio wave arrival time from each satellite, The current position of is displayed. According to such a GPS navigation system, the three-dimensional position of the reception point can be measured in a global range.

Here, as the position accuracy obtained by the GPS navigation system, conventionally, there is an influence of radio wave reflection and refraction in the ionosphere, and the error is 50 to 300 meters.
In recent years, a method has been added to measure the error of radio wave arrival time using known points of latitude, longitude, and altitude, and to transmit it as a correction signal to correct the error at the receiving point. Has been reduced to about a dozen meters.

JP-A-11-304513 JP 2001-255157 A JP 2002-357430 A

However, in the conventional navigation system in which the positional accuracy error is in the range of several tens of meters as described above, the error is too large to be applied to automatic driving or the like. For example, in order to realize automatic driving of a vehicle on a road, it is necessary to increase the vehicle position accuracy on the road to an error of about several centimeters. This accuracy is close to the accuracy of surveying and uses a conventional navigation system. As long as this is done, it is impossible to measure the position continuously in real time with a precision of several centimeters and output it.
In addition to automatic driving, it is necessary to obtain a positional accuracy of about several centimeters in real time even when used for, for example, car garage entry, aircraft takeoff and landing, robot navigation, etc. No navigation system has been realized so far.

  Therefore, as a result of earnest research, the inventor of the present application automatically detects a sufficient number of feature points from a plurality of frame images of a moving image captured by a camera mounted on a moving body, and automatically tracks the feature points between frames. Then, it has been conceived that the camera position and the rotation angle can be obtained with high accuracy by overlapping calculation for a large number of feature points, and the three-dimensional position coordinates of the moving object can be displayed with high accuracy by the camera position information.

  That is, the present invention has been proposed in order to solve the problems of the prior art, and by using an image processing technique, the three-dimensional coordinates of the feature points are accurately measured in advance for the moving path of the moving object. In addition, by comparing the three-dimensional coordinates with the camera image taken when the actual moving body moves, the three-dimensional coordinates indicating the camera position of the moving body can be obtained with high accuracy from the GPS system, and the error It is an object of the present invention to provide a navigation device that can indicate the current position of a moving body within a range of several centimeters.

  In order to achieve the above object, a navigation device according to the present invention includes a recording medium in which video feature points in a range observed from a moving object to be navigated are recorded in three-dimensional coordinates, and a moving object to be navigated. Compare the 3D coordinates of the video feature points obtained by reproducing the recording medium with the actual video obtained by the camera, find the point and direction on the 3D coordinates that match the actual video, and move 3D coordinate position, speed, acceleration, viewpoint direction, 3-axis rotation posture, 3-axis rotation speed, 3-axis rotation acceleration of the camera provided on the body, or any combination thereof And a point search navigation device that outputs the item.

  In addition, the navigation device of the present invention provides information recorded on a recording medium in a small area including a type of video feature point, its three-dimensional coordinates, and a video feature point in a range observed from a moving body. Three-dimensional arrangement of two-dimensional images, their three-dimensional coordinates, shapes of objects including video feature points, their three-dimensional coordinates, and peripheral images necessary for movement of moving objects other than video feature points, CG The shape and three-dimensional coordinates, etc., an image of a road on which the moving body moves, a vehicle traveling path, a planned route, etc., CG, its three-dimensional shape, and its three-dimensional coordinates, A combination thereof, all of them, or their attribute information is recorded together with the three-dimensional map.

  In the navigation device of the present invention, the point search navigation device designates the feature point 3D map reproduction unit for reproducing the recording medium and the approximate current position of the moving object, and the approximate current for limiting the search range at the initial setting. A location specifying unit, a current location surrounding feature point designating unit that reads out a plurality of feature points around the current location of the moving object from a three-dimensional map recorded on a recording medium, and designates it as a search target for the current location, and a moving object that is a navigation target In the video recorded in the video recording unit, a camera video acquisition unit that acquires the video around the moving body from the camera provided in the camera, a video temporary recording unit that records the video acquired by the camera video acquisition unit, The feature point search unit in the image for finding candidate feature points that should be the same as the search target is compared with the feature point candidate obtained in the feature point search unit in the image and the search target around the current location. The corresponding relationship as the same object is obtained by collation, a predetermined number of corresponding points are determined from the candidates, and the in-video feature point corresponding unit that receives the three-dimensional coordinates of the determined corresponding points from the recording medium is determined. Using the corresponding points and their three-dimensional coordinates, the camera coordinate calculation unit that determines the three-dimensional data such as the camera position, direction, and posture indicating the current state of the moving object by calculation, and the tertiary determined by the camera coordinate calculation unit The present invention includes a current location display unit that displays a combination of original data or all of them on a screen alone or together with information such as a map, a video, and an attribute recorded on a recording medium.

  In addition, the navigation device of the present invention records video feature points in a range observed from a moving body in three-dimensional coordinates based on an actual image obtained by a camera provided in the moving body for generating a recording medium. In addition, a feature point three-dimensional map generation device that generates information to be recorded on the recording medium is provided.

  Further, the navigation device of the present invention includes a camera video acquisition unit in which the feature point 3D map generation device acquires a surrounding image of the mobile body from a camera provided in the mobile body for recording medium generation, and a camera video acquisition unit A video recording unit that records the image acquired in Step 1, a feature point extraction unit that automatically extracts a predetermined number of feature points from the image data recorded in the video storage unit, and a feature point extracted by the feature point extraction unit A feature point correspondence processing unit that automatically tracks within each frame image to obtain a correspondence relationship between the frame images, and obtains a three-dimensional position coordinate of the feature point for which the correspondence relationship is obtained by the feature point correspondence processing unit, A feature point / camera vector calculation unit for obtaining a camera vector corresponding to each frame image from the three-dimensional position coordinates, and a three-dimensional position coordinate of each feature point obtained by the feature point / camera vector calculation unit Error minimizing unit that automatically determines the 3D coordinates of the feature points and the camera vector that have been subjected to statistical processing to minimize the distribution of the camera vector and the error minimizing process, and the error minimizing unit The three-dimensional shape and the three-dimensional coordinates of the image of the camera vector and the feature point or the small region including the feature point and the distribution thereof as a three-dimensional map together with the path of the moving object to be navigated A 3D map generation / recording unit that is arranged and recorded on a recording medium together with an object or the like including a feature point is provided.

  In addition, the navigation device of the present invention generates, based on an actual image obtained by a camera provided in a moving body to be a navigation target, a video feature point in a range observed from the moving body in three-dimensional coordinates. In addition, while generating a camera vector from the three-dimensional coordinates and generating a three-dimensional map based on the generated three-dimensional coordinates, the three-dimensional distribution of feature points and the three-dimensional coordinates of the camera provided in the moving body Feature point 3D map generation that outputs any one or a combination of predetermined items including position, speed, acceleration, viewpoint direction, 3-axis rotation posture, 3-axis rotation speed, and 3-axis rotation acceleration A configuration including a display device is provided.

  The feature point 3D map generation / display apparatus of the present invention records a camera video acquisition unit that acquires a surrounding image of the moving body from a camera provided in the moving body, and an image acquired by the camera video acquisition unit. A video recording unit, a feature point extraction unit that automatically extracts a predetermined number of feature points from image data recorded in the video storage unit, and feature points extracted by the feature point extraction unit are automatically tracked in each frame image And a feature point correspondence processing unit for obtaining a correspondence relationship between frame images, and obtaining a three-dimensional position coordinate of the feature point for which the correspondence relationship has been obtained by the feature point correspondence processing unit. The feature point / camera vector calculation unit for obtaining the camera vector corresponding to the image, and the three-dimensional position coordinates and camera vector distribution of each feature point obtained by the feature point / camera vector calculation unit are minimized. Error minimization unit that automatically determines the three-dimensional coordinates and camera vectors of the feature points that have been subjected to statistical processing and error minimization processing, and error minimization processing was performed by the error minimization unit A three-dimensional map of the three-dimensional shape of a camera vector and a feature point or a small area image including the feature point, its three-dimensional coordinates, and its distribution, along with the movement trajectory of the mobile object to be navigated or, if necessary, the planned movement path And a 3D map generation / display unit for displaying together with an object including a feature point.

  In the navigation device of the present invention, the feature point / camera vector calculation unit uses any two frame images Fn and Fn + m (m = frame interval) used for the three-dimensional coordinate calculation of the camera vector and the feature point as unit images. The unit calculation for obtaining the three-dimensional coordinates of the desired feature point and the camera vector is repeated, and for the frame image between the two frame images Fn and Fn + m, the three-dimensional coordinates of the camera vector and the feature point are obtained by a simplified calculation. The error minimizing unit scales so that the error between the three-dimensional coordinates of each camera vector and the feature point obtained by calculating a plurality of times for the same feature point is minimized as n advances continuously with the progress of the image. The final three-dimensional coordinate is determined by adjusting and integrating.

  In addition, the feature point / camera vector calculation unit of the present invention sets the frame interval m in accordance with the distance from the camera to the feature point such that m increases as the distance from the camera to the feature point increases. It is configured to perform calculation.

  Further, the feature point / camera vector calculation unit of the present invention deletes a feature point having a large error distribution of the obtained camera vector or feature point in three-dimensional coordinates, and recalculates another feature point if necessary. To increase the accuracy of the three-dimensional coordinate calculation.

  In the navigation device of the present invention, the recording medium and the point search navigation device are provided separately from each other, and predetermined three-dimensional information recorded in the recording medium provided in the base station or other mobile object is transmitted through the communication line. Via one or two or more point search navigation devices.

  Further, the navigation device of the present invention is configured such that the point search navigation device designates the approximate current position of the moving body by the approximate current position designation unit based on the latitude and longitude altitude data obtained by GPS.

  In addition, the point search navigation device of the present invention corrects the GPS by converting the three-dimensional data such as the camera position, direction, and posture indicating the current moving body status obtained by the camera coordinate calculation unit into the latitude and longitude altitude. The correction signal is output as an auxiliary signal for obtaining position data from GPS when a video feature point cannot be obtained.

  The navigation device of the present invention is configured such that the moving object to be navigated is an automobile, an aircraft, a ship, a person, a robot, a heavy machine, a spacecraft, a deep sea exploration ship, a machine having a moving part, or the like.

According to the navigation apparatus of the present invention as described above, a sufficient number of feature points are automatically detected from a plurality of frame images of a moving image captured by a camera mounted on a moving body such as a vehicle, and each frame is detected. By automatically tracking feature points, a large number of feature points can be overlapped and the camera vector (camera position and rotation angle) and the three-dimensional position coordinates of the feature points can be obtained with high accuracy.
Then, the three-dimensional coordinates of the obtained feature points are stored in advance in a recording medium, and the three-dimensional coordinates are compared with a camera image taken from a moving body that actually moves, or an image obtained from the camera From the above, it is possible to directly generate the three-dimensional coordinates of the camera position in real time and obtain highly accurate three-dimensional information indicating the current camera position, thereby being used as a navigation system for a mobile object.

Specifically, in the navigation device of the present invention, in order to obtain the current position coordinates of a moving body such as a vehicle in real time with higher accuracy than GPS, the image processing technique is used to obtain a plurality of features in the image. Pay attention and measure the three-dimensional coordinates of the feature points in advance with high accuracy. Then, a map (3D map) in which the feature points are described in three-dimensional coordinates is stored in a recording medium, and the recording medium is reproduced on the mobile body side, whereby the three-dimensional coordinates of the feature points can be read out. Further, the feature point in the video is extracted from the camera image obtained at the current location of the moving object, and the direction of the feature point and the direction of the feature point whose three-dimensional coordinates recorded in advance on the recording medium are known. In comparison, by obtaining the coordinates of points where the directions of a plurality of feature points coincide with each other, three-dimensional coordinates indicating the camera position, that is, three-dimensional coordinates indicating the current position of the moving body can be obtained.
Also, it is not equipped with a recording medium, and feature points are automatically extracted and tracked automatically from the video acquired by the camera of the moving body, and 3D coordinates are obtained directly without comparing with 3D maps. The camera position can also be obtained.

As a result, the current position of a moving object such as a traveling vehicle is accurately indicated directly from a camera image or by a three-dimensional map generated and recorded in advance, which is impossible with a conventional GPS system. A highly accurate navigation system with an error range of about several centimeters can be realized.
In order to generate a 3D map indicating the three-dimensional coordinates of feature points to be recorded on a recording medium (or generated in real time), a road that is expected to travel and its surroundings are photographed and recorded in advance. Extracting multiple feature points from these images automatically or manually, finding these feature points in the image, tracking the trajectory of each feature moving in each frame of the video, and epipolar geometry It is possible to generate a three-dimensional map of each feature point by solving linear simultaneous equations by science.

FIG. 1 is a block diagram showing a schematic configuration of a navigation device according to a first embodiment of the present invention.
FIG. 2 is a block diagram showing a schematic configuration of a feature point three-dimensional map generation apparatus according to the first embodiment of the present invention.
FIG. 3 is a block diagram showing a schematic configuration of the point search navigation device according to the first embodiment of the present invention.
FIG. 4 is an explanatory view schematically showing the correspondence between the three-dimensional coordinates recorded on the recording medium and the camera image in the navigation device according to the first embodiment of the present invention.
FIG. 5 is an explanatory diagram showing a specific camera vector detection method in the feature point three-dimensional map generation device according to the first embodiment of the present invention.
FIG. 6 is an explanatory diagram showing a specific camera vector detection method in the feature point three-dimensional map generation device according to the first embodiment of the present invention.
[FIG. 7] It is explanatory drawing which shows the specific detection method of a camera vector in the feature point three-dimensional map production | generation apparatus which concerns on 1st embodiment of this invention.
[FIG. 8] It is explanatory drawing which shows the designation | designated aspect of the desirable feature point in the detection method of the camera vector by the feature point 3D map production | generation apparatus concerning 1st embodiment of this invention.
FIG. 9 is a graph showing an example of 3D coordinates and camera vectors of feature points obtained by the feature point 3D map generation apparatus according to the first embodiment of the present invention.
FIG. 10 is a graph showing an example of three-dimensional coordinates of feature points and camera vectors obtained by the feature point three-dimensional map generating apparatus according to the first embodiment of the present invention.
FIG. 11 is a graph showing an example of 3D coordinates and camera vectors of feature points obtained by the feature point 3D map generation apparatus according to the first embodiment of the present invention.
FIG. 12 is an explanatory diagram showing a case where a plurality of feature points are set according to the distance of the feature points from the camera and a plurality of calculations are repeatedly performed in the feature point three-dimensional map generation device according to the first embodiment of the present invention It is.
FIG. 13 is an explanatory diagram showing a specific example of shaking component detection in a shaking component detection unit provided in the navigation device according to the first embodiment of the present invention.
FIG. 14 is an explanatory diagram showing an example of a stabilized image that is corrected based on the shaking component detected by the shaking component detection unit according to the first embodiment of the present invention.
FIG. 15 is a graph showing a trajectory of a camera vector that is corrected based on a shake component detected by a shake component detection unit according to the first embodiment of the present invention.
FIG. 16 is a diagram when the trajectory of the camera vector obtained by the feature point 3D map generation device according to the first embodiment of the present invention is displayed in the generated 3D map.
FIG. 17 is an explanatory view showing a display example of a three-dimensional shape (three-dimensional map) generated and displayed by the navigation device according to the first embodiment of the present invention.
FIG. 18 is an explanatory diagram showing a 3D map generation method in the feature point 3D map generation apparatus according to the first embodiment of the present invention.
FIG. 19 is an explanatory diagram showing a 3D map update method in the feature point 3D map generation apparatus according to the first embodiment of the present invention.
FIG. 20 is a view showing an example of a three-dimensional map generated by the feature point three-dimensional map generating apparatus according to the first embodiment of the present invention, and (a) is a cross-sectional view of a road represented by the three-dimensional map. (B) is an example of a three-dimensional map of the road shown in (a), which is a projected image taken from above the road, and (c) is for acquiring three-dimensional coordinates in the three-dimensional map shown in (b). It is a figure which shows the operator components used for.
[FIG. 21] A three-dimensional view of the road shown in FIG. 20, in which a road sign operator part (CG part) is synthesized.
[FIG. 22] It is a figure explaining the case where the attribute of a target object is manually acquired and registered in the CV video shown in FIG. 21, (a) is CV video, (b) Arbitrary point and straight line in CV video. (B) shows a three-dimensional map generated and displayed by registering the designated points and straight lines.
FIG. 23 is an explanatory diagram showing an example of an outline of the operation of the entire navigation device according to the first embodiment of the present invention.
[FIG. 24] It is explanatory drawing which shows another example of the outline | summary of operation | movement of the whole navigation apparatus concerning 1st embodiment of this invention.
FIG. 25 is an explanatory diagram showing another example of the outline of the operation of the entire navigation device according to the first embodiment of the present invention.
FIG. 26 is an explanatory diagram showing another example of the outline of the operation of the entire navigation device according to the first embodiment of the present invention.
FIG. 27 is a block diagram showing a schematic configuration of an optional device added to the navigation device according to the second embodiment of the present invention.
FIG. 28 is a block diagram showing a schematic configuration of a navigation device according to a third embodiment of the present invention.
FIG. 29 is a block diagram showing a schematic configuration of another embodiment of the navigation device according to the third embodiment of the present invention.
FIG. 30 is a block diagram showing a schematic configuration of a navigation device according to a fourth embodiment of the present invention.
FIG. 31 is a block diagram showing a schematic configuration when the navigation devices according to the first to fourth embodiments of the present invention are combined.
FIG. 32 is an explanatory diagram showing the three-dimensional coordinates of feature points generated and displayed by the real-time navigation method according to the fourth embodiment of the present invention and the current position of the moving object.
[FIG. 33] It is explanatory drawing which shows the three-dimensional coordinate of the feature point produced | generated and displayed by the real-time navigation method in 4th embodiment of this invention, and the present position of a moving body.
FIG. 34 is a block diagram showing a specific configuration of the navigation device according to the fourth embodiment of the present invention.
FIG. 35 is a block diagram showing the contents of processing operations in the navigation device according to the fourth embodiment of the present invention.
FIG. 36 is an explanatory view schematically showing a specific example using the navigation device according to the fourth embodiment of the present invention.

Hereinafter, a preferred embodiment of a navigation device according to the present invention will be described with reference to the drawings.
Here, the navigation apparatus of the present invention described below is realized by processing, means, and functions executed by a computer in accordance with instructions of a program (software). The program sends a command to each component of the computer, and performs predetermined processing and functions as shown below, such as automatic extraction of feature points, automatic tracking of extracted feature points, calculation of three-dimensional coordinates of feature points, The camera vector is calculated. Thus, each process and means in the navigation apparatus of the present invention are realized by specific means in which the program and the computer cooperate.
Note that all or part of the program is provided by, for example, a magnetic disk, optical disk, semiconductor memory, or any other computer-readable recording medium, and a program read from the recording medium is installed in the computer and executed. Is done. The program can also be loaded and executed directly on a computer through a communication line without using a recording medium.

[First embodiment]
First, a first embodiment of a navigation device according to the present invention will be described with reference to FIGS.
[Basic configuration]
FIG. 1 is a block diagram showing a schematic configuration of a navigation device according to the first embodiment of the present invention.
A navigation device 100 according to this embodiment shown in the figure includes a feature point three-dimensional map generation device 110, a recording medium 120, and a point search navigation device 130.

In this embodiment, in order to be able to acquire the current position coordinates of a moving object such as a vehicle, an aircraft, a ship, etc. in real time with higher accuracy than GPS, the image processing technique is used in this embodiment to have a plurality of features in the image. Paying attention, the feature point three-dimensional map generation device 110 is used to accurately measure the three-dimensional coordinates of the feature points in advance, and a map (3D map) describing the feature points in the three-dimensional coordinates is generated. The generated 3D map is recorded on a recording medium 120 such as a DVD, a hard disk, or a CD.
Then, on the side of a mobile object such as a vehicle to be navigated using this navigation device, the point search navigation device 130 uses the camera image of the current location obtained by the camera provided on the mobile body that actually moves to within the video. The feature point is extracted, and the direction of the feature point is compared with the direction of the feature point whose three-dimensional coordinates recorded in advance on the recording medium are known. Calculate by calculation.
Accordingly, it is possible to obtain three-dimensional coordinates indicating the camera position provided in the moving body, that is, three-dimensional coordinates indicating the current position of the moving body.
Note that examples of the moving body navigated by the navigation device include automobiles, ships, aircraft, robots, moving machines, moving people, and the like. ), Deep-sea exploration vessels, machines with moving parts, and even spacecraft.

Specifically, in a mobile object that is a navigation target, an image having three-dimensional coordinates of a plurality of video feature points read from the recording medium 120 by the point search navigation device 130 is displayed on a traveling vehicle, an aircraft, or the like. Search in the video obtained from the attached camera and find the correspondence. Corresponding points with a plurality of feature points described as a three-dimensional map obtained from the recording medium 120 are obtained by image recognition in the two-dimensional video obtained from the camera.
Next, a point where the directions of the corresponding points coincide with each other is searched for in a three-dimensional map in the recording medium and obtained by calculation. The position becomes the current position of the camera, that is, the current position of the moving object. Then, in the three-dimensional map recorded in the recording medium 120, the three-dimensional current position, speed, acceleration, direction, rotational speed, and rotational acceleration of the vehicle equipped with the camera can be displayed in real time.

In this way, in the navigation device of the present embodiment, the current position of the moving object is accurately indicated by the three-dimensional coordinates generated and recorded in advance, and an error range that is impossible with the conventional GPS system. Can realize a highly accurate navigation system of about several centimeters.
In this embodiment, a 3D map indicating the three-dimensional coordinates of feature points is recorded on a recording medium, so that mass production and distribution can be performed. Thereby, it becomes possible to read out the three-dimensional coordinates of the feature points by acquiring the user of the navigation device and the recording medium and reproducing it.
In order to generate a 3D map indicating the three-dimensional coordinates of the feature points to be recorded on the recording medium, the road and its surroundings that are expected to travel are photographed in advance with a camera mounted on a vehicle for 3D map generation, etc. A plurality of feature points are automatically or manually extracted from the video by the feature point 3D map generation device 110, and a plurality of the feature points are obtained in the image, and they are included in each frame of the moving image. By tracking the trajectory that moves and solving the linear simultaneous equations by epipolar geometry, a three-dimensional map (3D map) showing the camera position and the three-dimensional coordinates of each feature point can be generated.

Here, in order to generate a 3D map with high accuracy, it is preferable to use a technique of detecting a video feature point in an image and tracking its movement. By automating the detection of feature points and automating tracking, manual work can be greatly omitted.
In order to obtain the three-dimensional coordinates and the camera position (camera vector) from the feature points in the image, for example, the feature points are tracked in the image so that there are 6 to 7 or more feature points at the same time. Then, using epipolar geometry for these feature points, the three-dimensional coordinates of the feature points and the camera position can be obtained by calculation. The position accuracy is insufficient.

Therefore, in this embodiment, as will be described later, the number of feature points to be extracted and tracked is sufficiently increased, and multiple parallaxes are acquired by using a sufficient number of frames, and the excess feature points and the number of frames are determined. To get. Using multiple parallaxes with excess feature points and excess frames, statistical processing is performed, repeated calculation is repeated, the error distribution of the camera position is obtained, and then the camera position with high accuracy is obtained by statistical processing therefrom. . In this way, the camera position of each frame can be obtained with high accuracy, and if the camera position is obtained with high accuracy, then the third order for all pixels in the image can be obtained with a technique for obtaining three-dimensional coordinates from the parallax. Original coordinates can be obtained.
Details of 3D map generation will be described later.
Note that the moving body for generating the 3D map to be recorded on the recording medium includes, for example, an automobile, a ship, an aircraft, a robot, a moving machine, a moving person, and the like.

As described above, in the navigation device 100 according to the present embodiment, the 3D map itself is generated in advance by the feature point 3D map generation device 110, and the generated 3D map is recorded in the recording medium 120, which is then used as the point search navigation device 130. By playing on the side of a moving body such as a vehicle equipped with a vehicle, it is possible to search for a spot by comparing an actual image with a 3D map. Therefore, on the user side, only the recording medium 120 can be obtained (purchased), and can be reproduced and used by a vehicle equipped with the point search navigation device 136, and the navigation device 100 can be used easily and inexpensively. .
In this sense, the feature point 3D map generation device 110 does not need to be provided on the user side, and can be provided separately from the recording medium 120 and the point search navigation device 130. In addition, if a predetermined 3D map can be generated and recorded in the recording medium 120, it is also possible to generate and record a 3D map by a configuration other than the feature point 3D map generation device 110.

[Specific configuration]
Hereinafter, the feature point three-dimensional map generation device 110, the recording medium 120, and the point search navigation device 130 that constitute the navigation device 100 of the present embodiment will be described more specifically.
[Feature point 3D map generator]
FIG. 2 is a block diagram showing a schematic configuration of the feature point three-dimensional map generation apparatus 110 according to the present embodiment.
The feature point 3D map generation apparatus 110 is based on a real image obtained by a camera provided in a moving body such as a vehicle, and includes predetermined 3D coordinates including video-like feature points in a range observed from the moving body. Generate 3D information.

Specifically, as shown in FIG. 2, a camera image acquisition unit 111, a video recording unit 112, a feature point extraction unit 113, a feature point correspondence processing unit 114, a feature point / camera vector calculation unit 115, and an error minimization unit 116. , A shake component detection unit 117, an absolute coordinate acquisition unit 118, and a 3D map generation recording unit 119.
The camera image acquisition unit 111 acquires a surrounding image of the moving body from a camera provided in the moving body such as an in-vehicle camera of the moving vehicle.
The video recording unit 112 records the image acquired by the camera video acquisition unit 111.
The feature point extraction unit 113 manually and automatically specifies and extracts a small area image to be a feature point from the recorded image.

The feature point correspondence processing unit 114 automatically obtains the correspondence relationship by automatically tracking the feature points automatically extracted in each frame image between the frames.
The feature point / camera vector computing unit 115 obtains the three-dimensional position coordinates of the feature point for which the correspondence relationship is obtained, and automatically obtains the camera vector corresponding to each frame image from the three-dimensional position coordinates.
The error minimizing unit 116 performs statistical processing so that the distribution of the positions of the camera vectors and the feature points is minimized by performing a plurality of overlap operations, detects feature points having a larger error, and deletes them. Thus, the entire error is minimized.

  The shake component detection unit 117 has a one-to-one correspondence with a predetermined vehicle position (camera position) from the camera vector (three-dimensional position coordinates and three-axis rotation coordinates of the camera) obtained by the feature point / camera vector calculation unit 115. ) And a scheduled camera vector which is a vehicle rotation posture (corresponding to the camera posture on a one-to-one basis). Then, a positional deviation component signal and a rotational deviation component signal are generated from the difference between the scheduled camera vector and the current camera vector or the difference between the camera vector at the time of evaluation and all or one of these deviation component signals is generated. The values of the parts and their selection and combination are converted into a coordinate system to be evaluated appropriately according to the purpose, and the shaking of the camera (a fixed object such as a vehicle to which the camera is fixed) is evaluated and output. If necessary, it can be displayed. Details of this shaking component detection will be described later.

Then, based on the obtained camera vector and its shake component, irregular blurring caused by camera shake of an image acquired in moving image shooting such as a video image is corrected, and there is no blurring from a blurry image. An image can be generated (image stabilization processing). Further, based on the obtained camera vector and its shake component, the position and orientation of the camera itself can be driven and controlled, and the image can be stabilized as in the image stabilization processing (position and orientation stabilization processing).
Furthermore, based on the obtained camera vector, the object specified in the image is measured in the live-action coordinate system to obtain its three-dimensional coordinates, and the specified object for which the three-dimensional coordinates are obtained is always the center of the image frame. Image display or the position and orientation of a camera (a fixed object to which the camera is fixed) can be controlled so as to be displayed at a position (or any predetermined position) (target object lock-on process). At this time, the target image to be subjected to lock-on control may be an original image that includes a shaking component, or may be an image that has been stabilized by image stabilization processing.

The absolute coordinate acquisition unit 118 converts the obtained three-dimensional relative coordinates into the absolute coordinate system from the known absolute coordinates of a predetermined reference point determined in advance, and for all points of the feature points or necessary predetermined points Gives absolute coordinates.
When absolute coordinates such as latitude and longitude are not required, the length can be calibrated with each image using the length reference point indicating the length reference, the scale can be adjusted, and the correct scale coordinates can be obtained. In this case, the feature point / camera vector calculation unit 115 calculates the three-dimensional coordinates at both ends of the length reference point, and calculates the distance between the two points of the length reference point from the obtained three-dimensional coordinates. Then, in the error minimizing unit 116, the distance between two points of the length reference point obtained by the calculation by the feature point / camera vector calculating unit 115 is overlapped so as to match the known length of the length reference point. Repeat the calculation and perform statistical processing.
Of course, the coordinate reference point and the length reference point can be used simultaneously, and in that case, the accuracy can be further improved.

Here, as will be described later, the reference point is a point used as a reference when converting the three-dimensional relative coordinate into the absolute coordinate, and a known reference coordinate (three-dimensional absolute coordinate) is measured in advance by an arbitrary method. Point (coordinate reference point). Further, the reference point can include a reference point (length reference point) having a known length together with a reference point having a known three-dimensional absolute coordinate, or in place of a reference point having a known three-dimensional absolute coordinate.
The length reference point is a reference point that consists of two or more points and treats the distance between the two points as a known one. For example, the distance between the length reference points is set to 1 meter, It can be obtained by installing a large number of 1-meter bars. Then, photographing is performed so that at least one length reference point overlaps each image. By providing such a length reference point, the scale can be calibrated for each image with reference to the known length of the length reference point, and the accuracy can be greatly improved.

The length reference point can be considered to be the same as setting multiple coordinate reference points, but setting a large number of length reference points that are “lengths” means that the coordinate reference points that are “points” are It is more effective than setting a large number. That is, if only two coordinate reference points are set in the entire measurement range, they can be converted into absolute coordinates, and the coordinate reference points are not always observed from all images, and a plurality of coordinate reference points are set. It is more advantageous in terms of cost and labor to provide a plurality of length reference points. Therefore, for example, in the entire measurement range, only two coordinate reference points are provided, and a large number of bars having a predetermined length (for example, 1 meter) indicating the reference of the length are arranged in the measurement range and at random, the present invention is applied. Automatic surveying can be carried out, and the labor and cost of measurement work can be greatly reduced.
Note that any method may be used for measuring the three-dimensional coordinates and the length of the reference point (coordinate reference point or length reference point). For example, absolute coordinates or You can get the length.

  The 3D map generation / recording unit 119 determines the three-dimensional shape, the three-dimensional coordinates, and the distribution of the camera vector and the feature point that have been subjected to the error minimization process, or the small region image including the feature point, the vehicle, etc. Are arranged as a three-dimensional map together with the path of the moving body (traveling route, navigation route, etc.) and recorded on the recording medium 120 together with the object including the feature points.

  In the feature point three-dimensional map generation apparatus 110 as described above, a three-dimensional map of feature points to be recorded on the recording medium 120 is obtained by obtaining a plurality of corresponding points from a two-frame image by epipolar geometry, as will be described later. Is generated. In the present embodiment, epipolar geometry capable of calculating the three-dimensional coordinates of corresponding points is used, and furthermore, the corresponding points are automatically detected, and all calculations that are sufficient if there are about seven corresponding points and two-frame images are all required. A high-precision 3D map is generated by minimizing errors by calculating and statistically processing over the frames. That is, using a sufficiently large amount of information such as the number of feature points and the number of frames, the error of each feature point is reduced, and the feature point including the error is deleted to generate a highly accurate feature point three-dimensional map. It is like that.

[recoding media]
FIG. 3 is a block diagram showing a schematic configuration of the recording medium 120 and the point search navigation device 130 according to the present embodiment.
The recording medium 120 is a medium that can record data such as a DVD, a hard disk, and a CD, and stores predetermined information including 3D map information generated by the feature point 3D map generation apparatus 110.
The information recorded on the recording medium 120 includes (1) the types of features of video-like feature points that can be observed from a moving body and their three-dimensional coordinates (three-dimensional maps), and (2) the video-like feature points. 3D arrangement and 3D coordinates of 2D image of small area, (3) Shape of object (2D or 3D shape) including video feature points and 3D coordinates, (4) Feature points It is not, but the shape (two-dimensional or three-dimensional shape) and three-dimensional coordinates such as peripheral images and CG necessary for traveling and navigation, (5) moving path of the moving body, for example, road, vehicle traveling path, scheduled navigation path, etc. Image, CG and its shape (two-dimensional or three-dimensional shape), and its three-dimensional coordinates. And any one of these information, those combinations, or all of them is recorded with a three-dimensional map in the form including those attributes as needed.

As described above, the feature points that can be observed from the moving object are described in the recording medium 120. By recording including the image of the small area around the feature point, the local point video and the map of the moving object are recorded. The above feature points can be easily handled, which is preferable.
Further, as will be described later, the point search navigation device 130 obtains the three-dimensional coordinates as a result of the calculation, and the extracted feature points do not need to be the feature points seen from human vision. In order to output easy-to-understand information, it is desirable to record a map of the vehicle traveling path, surrounding buildings, and the like.
Furthermore, even if the information is not directly related to the recognition of the current location, as information that assists the user's travel etc., for example, images such as traffic signs and road displays, CG and its attributes are recorded, It is preferable because it is easy to understand and easy to operate.

[Point search navigation device]
The point search navigation device 130 is a device installed on the side of a moving body such as a vehicle, and compares an actual image obtained by a camera provided on the moving body with predetermined three-dimensional information recorded on the recording medium 120. Find the point and direction on the 3D coordinates that match the actual video. Accordingly, any one of the predetermined items including the position, speed, acceleration, viewpoint direction, three-axis rotation posture, three-axis rotation speed, and three-axis rotation acceleration of the camera provided on the moving body on the three-dimensional coordinates Output multiple items that are combined.
Specifically, as shown in FIG. 3, a feature point 3D map reproduction unit 131, an approximate current position designation unit 132, a current location surrounding feature point designation unit 133, a camera image acquisition unit 134, a video temporary recording unit 135, an in-video feature A point search unit 136, an in-video feature point correspondence unit 137, a camera coordinate calculation unit 138, and a current location display unit 139 are provided.

The feature point 3D map reproduction unit 131 reproduces the recording medium 120 and reads predetermined three-dimensional information recorded. As described above, since the recording medium 120 is provided as a DVD or a CD, the user loads the recording medium 120 on a navigation system provided in his / her vehicle or the like and reproduces it. The feature point 3D map playback unit 131 plays back the feature point 3D map recorded on the recording medium 120. The reproduced 3D map describes the three-dimensional coordinates of the feature points and their attributes.
The approximate current position designation unit 132 determines and designates the approximate current position of the moving body by some means, and limits the search range at the time of initial setting. The designation of the current position can be designated manually by the user, for example, or the approximate current position of the moving object can be designated by latitude and longitude altitude data obtained by GPS. By specifying and inputting the approximate position information of the moving body, it becomes a big clue to search for feature points around the current location. And GPS can be utilized as a means for that. Although GPS is less accurate than the navigation device 100 of the present invention, it can be said that it has adequate accuracy as approximate position information, and can be effectively used as a means for specifying approximate position information.

The current location surrounding feature point designating unit 133 reads a plurality of feature points around the current location from the 3D map of the recording medium 120, designates them as search targets for the current location, and outputs them to the in-video feature point search unit 136. Since the approximate position is known by the specification of the approximate current position specifying unit 132, the current location surrounding feature point instruction unit 133 takes in the feature point data around the current location from the recording medium 120, and distributes these feature points as three-dimensional coordinates.
Similar to the camera image acquisition unit 111 of the feature point 3D map generation device 110, the camera image acquisition unit 134 acquires an ambient image of the moving object from the camera provided in the moving object such as the in-vehicle camera of the moving vehicle by the in-vehicle camera. To do.
Similar to the video recording unit 112 of the feature point 3D map generation apparatus 110, the video temporary recording unit 135 records the image acquired by the camera video acquisition unit 134.

The in-video feature point search unit 136 searches the video recorded in the video temporary recording unit 135 for several feature point candidates that should be the same as the search target specified by the current location peripheral feature point specification unit 133.
The in-video feature point correspondence unit 137 compares and matches the feature point candidates searched by the in-video feature point search unit 136 with the search target around the current location to find a matching point, and obtains the correspondence as the same object. . Then, a sufficient number of corresponding points for the calculation are determined from the candidates for which the correspondence relationship is obtained.
Here, in order to obtain the coincidence point between the three-dimensional coordinates recorded on the recording medium 120 and the camera video, for example, image processing techniques such as matching and correlation can be used.
FIG. 4 shows two-dimensionally the correspondence between the three-dimensional coordinates recorded on the recording medium 120 and the camera image. In the figure, ● represents a feature point in which the correspondence between the camera image and the three-dimensional coordinates was taken, and x represents a feature point in which the correspondence was not taken.

The camera coordinate calculation unit 138 receives the determined three-dimensional coordinates of the corresponding points from the recording medium 120, and uses the determined corresponding points and the three-dimensional coordinates to indicate the camera position, direction, posture, and the like indicating the current vehicle situation. The three-dimensional data is calculated and determined. A point where a large number of feature points recorded on the recording medium 120 and a three-dimensional array of feature points of the captured image coincide with each other is a three-dimensional coordinate of the camera position to be obtained. The navigation system is completed by displaying data such as the obtained three-dimensional coordinates of the camera position, speed, acceleration, and rotation posture.
In other words, the current location display unit 139 displays several items of 3D data obtained by the camera coordinate calculation unit 138 that indicate the current state of the moving body, or a combination item thereof alone, or a map, video, attribute In addition to the information recorded on the recording medium 120, some or all of them are displayed in a target format on a map such as a travel map and a planned travel route.

In this way, the point search navigation device 130 obtains a plurality of feature points recorded in the recording medium 120 and corresponding feature points in the video captured in real time, and their observation directions match. The viewpoint on the three-dimensional map can be obtained by calculation.
In addition to the three-dimensional information of feature points, a map and various information are recorded on the recording medium 120, and can be displayed together with the information.
In the present embodiment, although not particularly illustrated, the point search navigation device 130 further directly controls a moving body such as a vehicle on which the navigation device is mounted based on the three-dimensional data determined by the camera coordinate calculation unit 138. A control device can be provided. That is, a moving body such as a vehicle can be automatically controlled based on highly accurate position information required by the navigation device, and an automatic driving system can be realized.

Further, the point search navigation device 130 can update the data of the recording medium 120 by adding the function of the feature point three-dimensional map generation device 110 described above. That is, in the point search navigation device 130, the video by the camera mounted on the user's vehicle or the like is accumulated, and the feature points of the recording medium over a plurality of frames and the corresponding points with the small area image including the feature points are captured by the camera. Similar to the feature point 3D map generation apparatus 110, the feature point of the recording medium over a plurality of frames, or a small area image including the feature point, and a small image in the video acquired by the camera are tracked in the acquired image. It is possible to obtain coordinate update data such as movement of an existing feature point by calculation from the correspondence with the image of the region, and add the result as a feature point from the next time.
Alternatively, a new feature point is detected from the video by the camera and added to the three-dimensional coordinates, so that it can be added as a feature point from the next time and added as a feature point from the next time.

  In this way, by providing the point search navigation device 130 provided on the user side with a device corresponding to the feature point 3D map generation device 110, it is possible to search while making a map, and to search for local points. At the same time, feature point detection and three-dimensional coordinate calculation can be simultaneously performed and recorded, and data on the recording medium can be updated and used as data from the next time. Thereby, a data update device and a real-time navigation device can be configured. The data update device will be described later with reference to FIG. 27, and the real-time navigation device will be described later with reference to FIG.

[Camera vector calculation and 3D map generation method]
Next, a camera vector calculation and 3D information (3D map) generation method in the feature point 3D map generation apparatus 110 (the point search navigation apparatus 130 as necessary) of the above-described embodiment will be described.
There are several methods for obtaining 3D information of camera vectors and feature points from feature points of a plurality of images (moving images or continuous still images). In the feature point 3D map generation apparatus 110 of this embodiment, A sufficiently large number of feature points are automatically extracted and automatically tracked, and the epipolar geometry is used to obtain the 3D vector of the camera, the 3 axis rotation vector, and the 3D coordinates of the feature points. is there. By taking a large number of feature points, camera vector information is duplicated, and errors can be minimized from the duplicated information to obtain more accurate camera vectors and feature point three-dimensional coordinates.

First, an image is acquired by an in-vehicle camera or the like, and a camera vector is obtained with high accuracy by using a sufficiently large number of points corresponding to each other between frames. In principle, if there are 6 to 7 feature points, the three-dimensional coordinates can be obtained, but in this embodiment, for example, a sufficiently large number of points such as about 100 points are used to obtain the distribution of the solution. Each vector is obtained from the distribution by statistical processing, and a camera vector is obtained as a result.
From the three-dimensional position of the camera thus obtained and the three-axis rotation of the camera, it is added as data to each frame image, and a plurality of parallaxes obtained from a plurality of frame images, ie, multiple parallaxes, have already been acquired. The three-dimensional coordinates of the feature points of the target object can be obtained by calculation from the three-dimensional position of the camera being operated.

Note that the above processing is not limited to the on-vehicle camera, for example, a person freely shakes the camera in his / her hand to shoot an object, calculates a camera vector from the image after shooting, and from the camera vector, The three-dimensional shape of the photographed object can be obtained.
By repeating the above processing, a wide range of three-dimensional shapes, that is, a three-dimensional map (3D map) is generated.

[Camera vector calculation]
The camera vector is a vector of degrees of freedom possessed by the camera.
In general, a stationary three-dimensional object has six degrees of freedom of position coordinates (X, Y, Z) and rotation angles (Φx, Φy, Φz) of the respective coordinate axes. Therefore, the camera vector refers to a vector of six degrees of freedom of the camera position coordinates (X, Y, Z) and the rotation angles (Φx, Φy, Φz) of the respective coordinate axes. When the camera moves, the direction of movement also enters the degree of freedom, which can be derived by differentiation from the above six degrees of freedom.
As described above, the detection of the camera vector in the feature point 3D map generation apparatus 110 according to the present embodiment is such that the camera takes six degrees of freedom for each frame, and six different degrees of freedom for each frame. Is to decide.

Hereinafter, a more specific method of detecting the three-dimensional coordinates of the camera vector and the feature point in the feature point three-dimensional map generation apparatus 110 will be described with reference to FIG.
In the feature point three-dimensional map generation apparatus 110, first, the feature point extraction unit 113 automatically extracts a point or a small area image that should be a feature point from a frame image appropriately sampled. Then, the correspondence between feature points is automatically obtained between a plurality of frame images. Specifically, more than a sufficient number of feature points that are used as a reference for detecting a camera vector are obtained. An example of feature points between images and their corresponding relationships are shown in FIGS. In the figure, “+” is a feature point that is automatically extracted, and the correspondence is automatically tracked between a plurality of frame images (see correspondence points 1 to 4 shown in FIG. 7).
Here, for feature point extraction, as shown in FIG. 8, it is desirable to specify and extract a sufficiently large number of feature points in each image (see circles in FIG. 8). For example, about 100 feature points are extracted. Extract points.

Subsequently, the feature point / camera vector calculation unit 115 calculates the three-dimensional coordinates of the extracted feature points, and calculates the camera vector based on the three-dimensional coordinates. Specifically, the feature point / camera vector calculation unit 115 includes a sufficient number of feature positions that exist between successive frames, a position vector between moving cameras, a three-axis rotation vector of the camera, and each camera. Relative values of various three-dimensional vectors such as vectors connecting positions and feature points are continuously calculated by calculation.
In this embodiment, in principle, a 360-degree all-round image is used as a camera image, and the camera motion (camera position and camera rotation) is calculated by solving the epipolar equation from the epipolar geometry of the 360-degree all-round image. ing.

  The 360-degree omnidirectional video is, for example, a panoramic image, an omnidirectional image, or a 360-degree omnidirectional image captured by a wide-angle lens, a camera with a fisheye lens, a plurality of cameras, or a rotating camera, and is captured by a normal camera. Since a wider range than the image is shown, it is preferable because highly accurate camera vector calculation can be calculated more easily and quickly. Note that the 360-degree all-round video is not necessarily a video including the entire 4π space, but a part of the 360-degree all-round can also be handled as a camera vector calculation video. In that sense, an image captured by a normal camera can also be regarded as a part of a 360-degree all-around image, and although there are few excellent effects as in the present embodiment, there is essentially no difference. 360 degree all-around video (4π video).

Images 1 and 2 shown in FIG. 7 are images obtained by Mercator expansion of 360-degree all-round video. When latitude φ and longitude θ are given, points on image 1 are (θ1, φ1) and points on image 2 are ( θ2, φ2). The spatial coordinates of each camera are z1 = (cos φ1 cos θ1, cos φ1 sin θ1, sin φ1), z2 = (cos φ2 cos θ2, cos φ2 sin θ2, sin φ2). If the camera movement vector is t and the camera rotation matrix is R, z1 ^T [t] × Rz2 = 0 is the epipolar equation.
By providing a sufficient number of feature points, t and R can be calculated as a solution by the method of least squares by linear algebra calculation. This calculation is applied to a plurality of corresponding frames.
Note that FIG. 7 uses a map projection to describe a 360-degree all-round spherical image obtained by synthesizing images taken by one or a plurality of cameras in order to facilitate understanding of the processing in the feature point 3D map generation apparatus 110. Although the image developed by the Mercator projection is shown, the actual processing of the feature point three-dimensional map generation apparatus 110 does not necessarily need to be a developed image by the Mercator projection.

Next, the error minimizing unit 116 calculates and calculates a plurality of vectors based on each feature point according to a plurality of calculation equations based on a plurality of camera positions corresponding to each frame and the number of feature points. Statistical processing is performed so that the distribution of the position of each feature point and the camera position is minimized to obtain a final vector. For example, the optimal solution of the least square method is estimated by the Levenberg-Marquardt method for a plurality of frames of camera positions, camera rotations, and a plurality of feature points, and the error is converged to obtain the camera position, camera rotation matrix, and feature point coordinates. .
Further, feature points having a large error distribution are deleted, and recalculation is performed based on other feature points, thereby improving the accuracy of computation at each feature point and camera position.
In this way, the position of the feature point and the camera vector can be obtained with high accuracy.

9 to 11 show examples of the three-dimensional coordinates of feature points and camera vectors obtained by the feature point / camera vector calculation unit 115. 9 to 11 are explanatory diagrams showing the vector detection method in the present embodiment, and showing the relative positional relationship between the camera and the object obtained from a plurality of frame images acquired by the moving camera. .
FIG. 9 shows the three-dimensional coordinates of the feature points 1 to 4 shown in the images 1 and 2 in FIG. 7 and the camera vector that moves between the images 1 and 2.
10 and 11 show a sufficiently large number of feature points, the positions of the feature points obtained from the frame image, and the position of the moving camera. In the figure, a circle mark that continues in a straight line at the center of the graph is the camera position, and a circle mark located around the circle indicates the position and height of the feature point.

Here, the calculation in the feature point 3D map generation apparatus 110 is performed according to the distance from the camera to the feature point as shown in FIG. A plurality of feature points are set, and a plurality of calculations are repeated.
Specifically, the vector detection unit automatically detects a feature point that has a video feature in the image, and obtains a corresponding point of the feature point in each frame image. Focusing on the n + m-th two frame images Fn and Fn + m, unit calculation is performed, and unit calculation with n and m appropriately set is repeated.
m is the frame interval, and the feature points are classified into multiple stages according to the distance from the camera to the feature points in the image, and set so that m increases as the distance from the camera to the feature points increases. It is set so that m is smaller as the distance to is shorter. This is because the longer the distance from the camera to the feature point, the smaller the change in position between images.

Then, while sufficiently overlapping the classification of the feature points by the m value, a plurality of stages of m are set, and as n progresses continuously with the progress of the image, the calculation proceeds continuously. Then, the overlap calculation is performed a plurality of times for the same feature point at each step of n and m.
In this way, by performing unit calculation focusing on the frame images Fn and Fn + m, a precise camera vector is calculated over a long time between frames sampled every m frames (frames are dropped). However, in the m frames (minimum unit frame) between the frame images Fn and Fn + m, a simple calculation that can be performed in a short time can be performed.

If there is no error in the precision camera vector calculation for every m frames, both ends of the camera vector of the m frames overlap with Fn and Fn + m camera vectors that have been subjected to the high precision calculation. Accordingly, m minimum unit frames between Fn and Fn + m are obtained by a simple calculation, and both ends of the camera vector of the m minimum unit frames obtained by the simple calculation are Fn and Fn + m obtained by high precision calculation. Scale adjustment of m consecutive camera vectors can be made to match the camera vectors.
Thereby, it is possible to speed up the calculation process by combining simple calculations while obtaining a highly accurate camera vector without error.

  Here, there are various methods for the simple calculation depending on the accuracy. For example, (1) When a large number of feature points of 100 or more are used in the high-precision calculation, the minimum number of simple calculation is about ten. (2) Even if the number of the same feature points is the same, if the feature points and the camera positions are considered equally, innumerable triangles are formed there, and equations corresponding to that number are formed. By reducing the number of equations, it can be simplified. In this way, integration is performed by adjusting the scale so that the error of each feature point and camera position is minimized, distance calculation is performed, and feature points with large error distribution are deleted, and other features are added as necessary. By recalculating the points, the calculation accuracy at each feature point and camera position can be improved.

  In addition, by performing high-speed simple calculation in this way, camera vector real-time processing becomes possible. In real-time processing of camera vectors, calculation is performed with the minimum number of frames that can achieve the target accuracy and the minimum number of feature points that are automatically extracted, the approximate value of the camera vector is obtained and displayed in real time, and then the image is accumulated. Accordingly, the number of frames can be increased, the number of feature points can be increased, camera vector calculation with higher accuracy can be performed, and approximate values can be replaced with camera vector values with higher accuracy for display.

  Furthermore, it is preferable to track three-dimensional information (three-dimensional shape) in order to obtain a more accurate camera vector. Specifically, the tracking of the three-dimensional information is performed by positioning the camera vector obtained through the feature point / camera vector calculation unit 115 and the error minimization unit 116 as an approximate camera vector, and is generated by a subsequent process. Based on the three-dimensional information (three-dimensional shape) obtained as part of the image, the partial three-dimensional information contained in multiple frame images is continuously tracked between adjacent frames to automatically track the three-dimensional shape. I do. Then, from the tracking result of the three-dimensional information obtained by automatic tracking, a highly accurate camera vector is obtained in the highly accurate camera vector calculation unit.

  The feature point extraction unit 113 and the feature point correspondence processing unit 114 described above automatically track feature points in a plurality of inter-frame images, but the number of feature point tracking frames is limited due to disappearance of feature points. May come. In addition, since the image is two-dimensional and the shape changes during tracking, there is a certain limit in tracking accuracy. Therefore, the camera vector obtained by the feature point tracking is regarded as an approximate value, and the three-dimensional information (three-dimensional shape) obtained in the subsequent process is traced on each frame image, and a high-precision camera vector is obtained from the trajectory. Can do. Such tracking of the three-dimensional shape is easy to obtain the accuracy of matching and correlation, and since the three-dimensional shape does not change its size and size depending on the frame image, it can be tracked over many frames, This can improve the accuracy of the camera vector calculation. This is possible because the approximate camera vector is known by the feature point / camera vector calculation unit 115 and the three-dimensional shape is already known.

  When the camera vector is an approximate value, the error of 3D coordinates over a very large number of frames is small because there are few frames related to each frame by feature point tracking. The error of the three-dimensional shape when a part of the image is cut is relatively small, and the influence on the change and size of the shape is considerably small. For this reason, the comparison and tracking in the three-dimensional shape is extremely advantageous over the two-dimensional shape tracking. In tracking, when tracking with 2D shape, tracking changes in shape and size in multiple frames are unavoidable, so there are problems such as large errors and missing corresponding points. However, in tracking with a three-dimensional shape, there is very little change in shape, and in principle there is no change in size, so accurate tracking is possible.

  Here, as the three-dimensional shape data to be tracked, there are, for example, a three-dimensional distribution shape of feature points, a polygon surface obtained from the three-dimensional distribution shape of feature points, and the like. It is also possible to convert the obtained three-dimensional shape from a camera position into a two-dimensional image and track it as a two-dimensional image. Since the approximate value of the camera vector is known, projection conversion can be performed on a two-dimensional image from the camera viewpoint, and it is also possible to follow a change in the shape of the object due to movement of the camera viewpoint.

[Swing component detection]
The camera component obtained as described above is extracted in the shake component detection unit 117 as a deviation component between the predetermined camera position and the scheduled camera vector indicating the camera posture.
In the shake component detection unit 117, for example, δX, δY, which are shake components due to vehicle positions (that is, camera positions) X, Y, Z and vehicle rotation postures (that is, camera postures) Φx, Φy, Φz on which an in-vehicle camera is mounted. , ΔZ, δΦx, δΦy, and δΦz are all subject to evaluation. Here, δX, δY, δZ, δΦx, δΦy, and δΦz do not necessarily mean differential values or difference values, but mean deviations from a predetermined position and a predetermined posture. In many cases, the vibration component can be detected by substituting with a differential value. However, if a predetermined position and a predetermined posture are determined in advance, the difference between them is δX, δY, δZ, δΦx, δΦy, and δΦz.

Specifically, in a train traveling on a track, the scheduled camera vector is close to the average value measured during traveling, but when navigating in a three-dimensional space like an aircraft, On average, it does not agree with the one when driving.
As the shake component output, a total of 12 parameters of X, Y, Z, Φx, Φy, Φz and δX, δY, δZ, δΦx, δΦy, δΦz can be output.
However, depending on which shake evaluation is intended, the number of parameters can be selectively combined from these, and can correspond to the evaluation object.

That is, when the outputs from the feature point / camera vector calculation unit 115 and the shake component detection unit 117 are combined, X, Y, Z, Φx, Φy, Φz, δX, δY, δZ, δΦx, δΦy, δΦz Although there are twelve parameters, only three parameters δΦx, δΦy, and δΦz are required for normal image stabilization processing. On the other hand, when a plurality of cameras are used at the same time, the three-dimensional position of the image can be corrected. Therefore, it is necessary to prepare parameters δX, δY, and δZ. In general, posture control requires six parameters in total including δX, δY, and δZ in addition to this if it includes control of δΦx, δΦy, δΦz, and position in the case of rotation control. Furthermore, if situation judgment is included, the output from the feature point 3D map generator 110 is X, Y, Z, and Φx, Φy, Φz.
Therefore, it can be selectively combined from the 12 obtained parameters and used for image processing and attitude control.
In addition to the twelve variables, other coefficients depending on the shooting conditions used for image stabilization and posture stabilization include a limitation on the swing width of the image frame as a reference posture of the camera.

FIG. 13 shows a specific example of shaking component detection in the shaking component detection unit 117. The example shown in the figure is a case where a camera is attached to a vehicle for traveling, and the shaking component detection unit 117 detects shaking from a moving image taken at that time.
In the figure, a thick arrow indicates the traveling direction of the vehicle to which the camera is attached, and the position and orientation of the camera with the optical axis of the camera as the origin is the camera coordinate system (Xc, Yc, Zc) (shown in the figure). A broken line is a vehicle coordinate system (Xt, Yt, Zt) that is mounted in a semi-fixed state (solid line shown in the figure), and a coordinate system that always changes the coordinate axis in the vehicle traveling direction is a rotating world coordinate system (Xwr , Ywr, Zwr) (two-dot chain line shown in the figure), and a coordinate system representing the static system of the outside world is a world coordinate system (Xw, Yw, Zw) (one-dot chain line shown in the figure). Then, the relationship between the four coordinate systems is obtained and converted into a coordinate system necessary for evaluation to express the vehicle shake.

The camera vector obtained by the feature point / camera vector calculation unit 115 is the camera coordinate system (Xc, Yc, Zc) itself. Since the camera coordinate system is generally set in an arbitrary direction, the camera coordinate system is temporarily converted into a vehicle coordinate system (Xt, Yt, Zt) in order to detect vehicle shake. This conversion is a simple rotation conversion, and is generally semi-fixed. Once set, there is no change until the measurement is finished.
By selecting the vehicle traveling direction as one of the three axes of the vehicle coordinate system (Xt, Yt, Zt), a coordinate system suitable for evaluating the shaking can be obtained.

Further, it is appropriate to express the trajectory of the vehicle movement in the world coordinate system (Xw, Yw, Zw) which is a stationary coordinate system. To express velocity, it can be expressed simply in the rotating world coordinate system (Xwr, Ywr, Zwr), but to express it as a vector, it is appropriate to express it in the world coordinate system (Xw, Yw, Zw). .
In shake evaluation, evaluation is performed in a coordinate system suitable for shake evaluation.
The shaking signal is detected as a deviation from the planned course, but in the example shown in FIG. 13, the shaking is evaluated using the average course of the vehicle as the planned course. Therefore, the movement trajectory of the camera is obtained on the world coordinate system, the average course is obtained, and this is set as the planned course.

In the shake component detection unit 117 of this embodiment, the shake component can be detected only by a camera that acquires image data without using a gyro that is a reference for the posture. The obtained camera vector is a relative value, and since there is no calibration device with a world coordinate system such as a gyro, accumulation of errors occurs. For this reason, in order to always evaluate the swing with respect to the vehicle, it is necessary to give an average vertical horizontal direction. Therefore, if one camera coordinate system is installed so as to coincide with the horizontal axis with respect to the vehicle at the time of camera installation, the horizontal posture can be easily calibrated later. Accordingly, the camera coordinate system (Xc, Yc, Zc) may be converted into the vehicle coordinate system (Xt, Yt, Zt), and the shake may be measured and evaluated.
As the shake to be evaluated, there are a positional deviation component Xt, Yt, Zt, a rotational component Φxt, Φyt, Φzt, a positional deviation difference δXt, δYt, δZt, etc. Because it is an acceleration component, the meaning of shaking is different from other components).

In the evaluation of the shaking component as described above, the following variables and display should be evaluated.
-Vehicle position display in the world coordinate system:
(Xw, Yw, Zw)
・ Speed and acceleration display in the rotating world coordinate system rotated in the vehicle traveling direction:
(ΔXwr, δYwr, δZwr) (ΔδXwr, ΔδYwr, ΔδZwr)
-Shake display in the vehicle coordinate system:
(ΔXt, ΔYt, (ΔZt)) (ΔΦxt, ΔΦyt, ΔΦzt)
・ Rotary display of vehicle coordinate system and camera coordinate system (semi-fixed):
(Xc, Yc, Zc) = F (Xt, Yt, Zt)
・ Direction display in world coordinate system:
(Xw, Yw, Zw) = G (Xt, Yt, Zt)
・ Direction display in camera coordinate system:
(Xc, Yc, Zc) = H (Xt, Yt, Zt)
・ Vehicle coordinate system origin movement and rotation posture display with respect to the world coordinate system:
(Xw, Yw, Zw) (δXw, δYw, δZw)

According to the shaking component detection unit 117 of the present embodiment as described above, for example, in the case of a camera attached to a train, the shaking component detection unit 117 analyzes and analyzes the shaking of the train to detect abnormalities in the vehicle or the track. It becomes possible to discover. Normally, the vibration component is measured by using an expensive device such as a mercury acceleration watch, but by using the vibration component detector 117 of this embodiment, the vibration component can be easily detected and displayed. Can do.
By using such a shaking component detection unit 117, it is possible to realize the above-described image stabilization processing, camera position / posture stabilization processing, and further, target object lock-on processing.

  FIG. 14 shows an example of an image that is converted into a stabilized image by a correction signal based on the shake component detected by the shake component detection unit 117. For example, as shown in FIGS. The image having the fluctuation is output and displayed as a stabilized image corrected as shown in FIGS. FIG. 15 is a graph showing the locus of the corrected camera vector. In FIG. 15, the locus of camera movement is arranged in a straight comb shape at the center of the graph, and shows the position and height of the moving camera. ing.

  When the three-dimensional relative position coordinates of each point are obtained as described above, the absolute coordinate acquisition unit 118 gives the known coordinates of the reference point whose absolute coordinates have been measured in advance to each three-dimensional relative coordinate. The three-dimensional relative coordinates are converted into an absolute coordinate system, and absolute coordinates are assigned to all the measurement points, reference points, and feature points (or necessary predetermined points). As a result, final absolute coordinates for a desired measurement point and a designated point arbitrarily specified in the feature point are obtained, and the data is recorded in the 3D map generation / recording unit 119, and as required as 3D map information. Are output and displayed.

  In the above description, the feature point, camera coordinates, and rotation (camera vector) have been described as being simultaneously obtained by the feature point / camera vector rendering unit 115. However, once a camera vector is obtained, a new feature point or feature is obtained. Arbitrary designated points can be easily calculated from the already obtained camera vectors as a single vertex of two images, that is, the bases of the two camera positions, without recalculation with the camera vector. Can do. Since the accuracy of the camera vector does not change, the accuracy of new feature points and arbitrary designated points does not change. However, if the camera vector is obtained again and recalculated, the accuracy generally improves.

The camera vectors and the three-dimensional coordinates of the feature points obtained as described above can be displayed in the generated three-dimensional map.
For example, as shown in FIG. 16, the image from the in-vehicle camera is developed in a plane, the corresponding points on the target plane in each frame image are automatically searched, and the corresponding points are combined so as to coincide with each other. A combined image is generated and displayed in the same coordinate system. Then, the camera position and the camera direction can be detected one after another in the common coordinate system, and the position, direction, and locus can be plotted.
FIG. 17 shows a display example of a 3D map generated in the present embodiment.

In the above method, one camera or a plurality of cameras may be used to acquire video. By installing multiple cameras at the same time and using short distance measurement with multiple camera distances together, it is possible to obtain the reference length of 3D absolute distance, and also enables detection of moving objects, moving object coordinates, distance, The speed can be measured, and the moving body portion can be deleted.
By using two cameras, absolute measurement can be performed, and since absolute measurement can be performed, it is possible to give a reference length to the calculation result of one camera, and this enables detection of moving objects. . Therefore, unnecessary moving objects can be deleted from the feature points.
Furthermore, by using a plurality of cameras, two or more feature points whose absolute coordinates are known can be obtained in the image, and the absolute coordinates can be given to the feature point map. Relative values are obtained when measured with a single camera, but absolute distance is given when multiple cameras are used. If the known absolute coordinates are given as two or more feature points, the absolute coordinates can be obtained.

[3D map generation]
Next, a method for generating a 3D map generated based on the camera vectors and the three-dimensional coordinates of feature points obtained as described above will be described with reference to FIGS.
FIG. 18 is an explanatory diagram illustrating a method for generating a 3D map generated by the feature point 3D map generation apparatus 110 according to the present embodiment, and FIG. It is explanatory drawing which shows the update method of a 3D map to be called.

In this embodiment, two types of 3D maps can be generated as 3D maps (three-dimensional maps) used in the navigation device (automatic traffic guidance device), one is a CV video map, and the other is a CV map. It is a three-dimensional map by the CG map produced | generated from an image | video.
A CV video is a camera vector video, that is, a camera position and camera orientation (camera vector) obtained from a 360-degree all-round video is acquired over all frame images, and a camera vector (CV value) is associated with each frame. 360 degree all-around video. As described above, even a normal video that is not a 360-degree all-round video can be positioned as a part of a 360-degree all-round video.

A process for generating a CG map from a CV video map will be described below with reference to FIG.
As shown in the figure, first, a panoramic image of the travel space is captured by a camera mounted on a vehicle traveling on a road surface, and a 360-degree all-round image is acquired (S1801: ground 4π captured image).
Based on the acquired 360-degree all-around video, the above-described camera vector calculation is performed (S1802: CV calculation) to acquire a camera vector.
Then, the acquired camera vector is calibrated based on a live-action image, an existing map (S1807), etc. (S1803: calibration), and further, since the CV value is a relative value, an existing map (S1807) is obtained. ) To obtain latitude / longitude / altitude and obtain absolute coordinates (S1808).

In order to observe the road surface and the like from a bird's-eye view, an aerial image (S1804) is taken, and a camera vector is obtained by performing CV calculation (S1805) on the aerial image, and similarly using an existing map (S1807) or the like. Calibration is performed (S1806), and absolute coordinates are acquired (S1808).
The error of each obtained CV value is corrected and the coordinates are integrated (S1809), thereby constructing a CV video map database (S1810).
The CV video map database (S1810) can be used as it is for a traffic guide device as a three-dimensional map, but a three-dimensional CG map can be generated based on the CV video.

Hereinafter, a method for generating a three-dimensional CG map that is generally accepted in a car navigation apparatus or the like from a CV video map will be described.
As shown in FIG. 18, the CV video map database (S1810) is stabilized (S1811) by the following process.
In the stabilization process, as shown in S1811, first, a part of the video is cut out from the range to be converted to CG, and image stabilization is eliminated by image stabilization. Next, a virtual camera moving direction is determined, and an error in the image is corrected by specifying the traveling direction and rolling correction. Furthermore, time axis correction for traveling at a constant speed in a certain direction is performed. In this stabilization process, the image is locked on a predetermined fixed plane and the stabilization process is performed.
The image stabilized as described above is displayed on the MRSS viewer (S1812). Here, the MRSS viewer (product name of Iwane Laboratory Co., Ltd.) is an abbreviation for the Mixed Reality Simulation System viewer, and is a display device that can acquire three-dimensional coordinates on the viewer.

Since the image is stabilized by the stabilization process (S1811) described above, the virtual traveling surface moves exactly parallel to the road surface. There, the moving speed of the plane of the object is inversely proportional to the vertical distance from the traveling camera, and the closer the object is, the faster the object is, and the slower the object is. That is, since the moving speed of the object plane has a one-to-one relationship with the distance, only the target object plane can be selectively extracted by background separation and object extraction (S1813: vector selection extraction). In addition, the coordinates of the object can be acquired by performing three-dimensional measurement (S1813) on the MRSS viewer.
Moreover, a certain range of target object can be selectively taken out by giving a width | variety to a moving vector. Since the object is acquired from multiple viewpoints by the feature of the 360-degree video, the background and the texture of the object can be extracted by direction and pasted on the CG by the multi-view texture extraction cut (S1814).
Furthermore, the shape of the object can be extracted with the wire frame by specifying the three-dimensional coordinates (manual) of the object and acquiring the wire frame CG (S1815).

Next, the representative point is manually clicked on the displayed target object, and the three-dimensional coordinates of the target object are replaced with the CG creation tool (S1816). Thereby, it can input into CG generation tools, such as 3DSMAX, and can generate CG on the tool (S1817). A simple figure can be generated on the MRSS viewer. That is, CG generation (S1818) of vertical lines, horizontal lines, vertical planes, horizontal planes, vertical cylinders, etc., and figure rotation, figure cutting, movement, and simple processing (S1819) can be performed.
Then, the attribute of the object is manually added to the generated CG (S1820), the coordinates are integrated (S1821), and a three-dimensional CG map database (S1822) is constructed.

  In addition, about the three-dimensional map to which the above CG was added, CG can be simplified or deleted suitably. The role played by CG in 3D maps is because one sees it. Therefore, if the 3D map can be viewed (recognized) by the machine without being viewed by humans, the shape of the CG itself can be accurately determined according to the purpose of use or according to the required accuracy. There is no need to generate the CG, and the registration of the name (name of the object), the approximate position coordinates, and the approximate size is sufficient as the attribute of the CG. Further, it may be omitted, the approximate size can be ignored, and it may be possible to express and register only with points, straight lines, and surfaces.

For example, when a machine sees, color and value are not necessary, and road signs, utility poles, guardrails, etc. need only be position coordinates on the road side, and no shape is necessary, for example, guardrails are approximated by lines or planes . Therefore, guard races, etc. only acquire a rough existence range, road signs only need to register the existence position coordinates as points, and the center line of the road approximates with continuous lines and registers to achieve the purpose. Can do.
Thus, the generation of the CG three-dimensional map can be simplified as much as possible, thereby reducing the cost.

Next, the case of updating the CV video map generated as described above will be described with reference to FIG.
As shown in FIG. 19, when updating the CV video map (S1901), first, automatic update preprocessing is performed.
Specifically, an important point is manually designated for an object suitable for comparison between old and new images, such as an object suitable as a mark in the image (S1902). The important point is an image of a fragment having a certain area. Further, a characteristic part as an image is automatically extracted (S1903: feature point automatic extraction), and the three-dimensional coordinates of the important point and the feature point are acquired (S1904).

Important points and feature points can be recorded together with three-dimensional coordinates (S1906) by making them partly three-dimensional parts (S1905) corresponding to the video as required, thereby enabling automatic updating for updating. A CG video map database (S1907) having an update function is constructed.
After the pre-processing as described above, the CV video map data is updated.
In the update process, first, a 4π-captured video (360-degree all-round video) is acquired as an update video (S1908). That is, in order to update the CV video map data, it is necessary to acquire an update video and compare the old and new video.

In the comparison of new and old videos, first, initial alignment is performed for important points (S1909), and only the first frame of the updated video is manually matched with old videos of important points (S1910). Next, by acquiring new image coordinates (S1911), the three-dimensional coordinates and rotation coordinates of the important point of the updated image are acquired.
Similarly, feature points are automatically extracted by feature point extraction (S1912), new and old images are handled by feature point correspondence (S1913), and new image coordinates are acquired (S1914). Get coordinates and rotation coordinates.

By obtaining three or more important points, or adding feature points as necessary, new image camera coordinate point acquisition (S1915), the camera vector of the updated image can be obtained by calculation for solving the triangle. Note that feature point extraction / new / old correspondence / coordinate acquisition (S1912 to S1914) may be omitted.
When the updated video and its camera vector are acquired in this way (S1915), the CV video map database (S1907) having an automatic update function can be updated with the camera vector.
Further, in order to update the CG map data, the above-described new image coordinate acquisition (S1911) acquires the three-dimensional coordinates of the important point, compares the new and old coordinates (S1916), and specifies the update object ( S1917), it is automatically determined whether or not they are at the same coordinates. If the coordinates do not match, it is determined that the object needs to be replaced, and the CG for update is automatically or manually generated by manual processing (S1918) by CG generation shown in FIG. The updated data is recorded in the three-dimensional CG map database (S1919).

FIG. 20 shows an example of generating a three-dimensional map based on a video taken from the sky of the road. The road image shown in the figure is a 360-degree CV image, and is not a complete plan view but a road surface observed from several meters above the ground.
When generating a three-dimensional map of a road, the shape near the road surface is important, and high measurement accuracy is required. In general, since it is known in advance that the road structure has a structure as shown in the cross-sectional view of FIG. 20A, the shape of the road structure can be predicted and three-dimensional measurement can be performed.

Furthermore, matching and gripping over a wide area is possible by making use of the feature of the 360-degree video and setting the road surface display so that the viewing direction is directly below the road surface. Specifically, matching and grip in an area of about 15 * 15 pixels is usually the limit in an arbitrary direction, but in the direct display, the viewpoint and the road surface are nearly perpendicular, and the inter-frame image changes its shape. Therefore, image distortion caused by each frame can be ignored.
Thereby, for example, matching and grip (M & G) can be performed in a wide area of 50 * 50 pixels or more, matching and grip can be performed even on a road surface with few features, and measurement accuracy is improved.
Furthermore, since road markings (center line, shoulder line, etc.) are drawn on the road pavement surface according to a predetermined standard, the pattern is prepared in advance as a part of a PRM operator (PRM Operator). It is possible to detect the three-dimensional position by comparing the image with the operator part.

  Here, PRM is an abbreviation for Parts Reconstruction Method (3D space recognition method), and is a technique for recognizing an object for which the applicant has already applied for a patent (International Application PCT / JP01 / 05387). reference). Specifically, the PRM technology prepares all the shapes and attributes of the object to be predicted in advance as parts (operator parts), compares these parts with actual live-action images, and selects matching parts. Technology that recognizes objects. The `` parts '' of the objects required for automatic guided driving and automatic driving of vehicles are lanes, white lines, yellow lines, crossing roads as road markings, speed signs as road signs, guidance signs, etc. Since these are fixed shapes, they can be easily recognized by the PRM technology. Further, even when searching for an object in the CV video, it is possible to limit the expected three-dimensional space in which the object exists to a narrow range, and to improve the efficiency of recognition.

Specifically, as a road surface operator, there is a pattern as shown in FIG. Although many other patterns not shown are assumed as operator parts, it is not necessary to measure the entire road surface in a three-dimensional map, and it is sufficient to complete the road sectional view by sampling the road surface at appropriate intervals. Therefore, it can be said that the degree shown in FIG. 20 is sufficient.
Furthermore, by preparing a three-dimensional PRM operator part (PRM 3D Operator) and matching it in three dimensions, for example, the step of the curb portion of the road can be accurately reproduced.

FIG. 21 shows a three-dimensional map in which the road shown in FIG. 20 is stereoscopically viewed.
As shown in the figure, in the video of the outfitted road, the PRM operator exhibits its effectiveness in the recognition of a three-dimensional road sign rather than the road surface display such as the center line shown in FIG. That is, regarding the recognition of the road sign, as shown in FIG. 21A, assuming the road sign expected space on the CV video, the type, position, shape and coordinates of the target road sign in the limited space. Can be recognized.
In the CV video, the road sign prediction space can be synthesized and arranged as a CG on the photographed image, and the target road sign can be searched only within the limited range.
In addition, since the shape and size of the road sign are usually determined, as a three-dimensional operator part of each road sign prepared in advance (see FIG. 21 (b)), the road sign prediction space is three-dimensionally displayed. It is possible to search for and find signs of a certain size. Then, the type, position, coordinates and shape of the found sign are recognized.

In this way, the CV video can be handled in the same way as an object having three-dimensional coordinates, which is extremely advantageous for searching. For road signs that have already been determined in shape, the apparent size at the 3D position can be obtained by calculation, so it is advantageous to use the PRM operator. By preparing various signs, it is possible to recognize a target sign by searching for a matching part from the prepared sign parts.
Furthermore, a 360 degree live-action CV video displayed in the MRSS viewer can be obtained by (1) acquiring the three-dimensional coordinates of the designated point by clicking an arbitrary point in the CV video with a mouse. Connect any two specified points to a straight line with the mouse, measure the distance between the two points, and (3) measure the area of the polygon by specifying and entering any polygon with the mouse be able to. Therefore, by using this function, just by clicking the position of the target object in the CV video with a mouse click, (1) an attribute registration point can be specified in the live-action video, and (2) road shape And road markings can be registered as straight lines, and (3) road surfaces and signboard surfaces can be registered.

FIG. 22 is a diagram showing a procedure for manually acquiring and registering an object attribute in a CV video. When the CV video shown in FIG. 22A is displayed in the MRSS viewer, FIG. As shown in (b), a desired point or straight line can be designated in the video using the mouse. The designated points, straight lines, and surfaces can be registered, and can be output and displayed as a three-dimensional map as shown in FIG.
In this way, a 3D map can be generated by designating an arbitrary point of a CV video and registering only a straight line or a surface. If attributes are classified at the time of registration, attribute extraction is completed at the same time. 3D maps can be easily generated and acquired.

[Navigation behavior]
Next, an overview of the overall operation of the navigation device according to the present embodiment as described above will be described with reference to FIGS.
As shown in FIG. 23, the operation of the navigation device of this embodiment is roughly divided into two, one is pre-processing performed on the CV video map generation device (feature point three-dimensional map generation device 110) side, and One is post-processing performed on the side of a navigation device (point search navigation device 130) loaded on a vehicle or the like.

[Preprocessing]
Pre-processing is performed according to the following procedure.
First, as shown in FIG. 23, a 360-degree all-round video is acquired by an in-vehicle camera or the like (S2201: 369-degree video shooting), and a camera vector of all frames is acquired for the 360-degree video by CV calculation (S2202: CV data addition). In principle, it is desirable that the video captured by the camera is a 360-degree all-round video, but it is not necessarily required to be a 360-degree all-round video.
Since the CV data obtained by the CV calculation is a relative value, it is converted into absolute coordinates (latitude / longitude / height) based on actual measurement data, GPS, and the like (S2203: absolute coordinate acquisition).

Further, in the CV video, an image part that can be a measurement reference later is cut out and added and recorded together with the attribute of the part (S2204: designation reference part coordinate and attribute addition). Here, coordinates may be further given. The specified point is cut out as a partial video to reduce the data amount. The dedicated mark can also be acquired as an image of a stationary object.
Important points, signboards, and the like in the CV video are converted into CG and combined with or associated with the CV video together with attributes (S2205: addition of designated CG). Since traffic signs and the like are common, only the ID and coordinates are added as a common CG.
Through the above procedure, CV data and attributes corresponding to each frame are generated (S2206: CV video generation). In addition, when the image is not observed only by measurement, the image is only the characteristic part. Then, the generated CV video is distributed to the navigation device side that performs post-processing by WEB, HDD, DVD or the like (S2207: CV video distribution).

[Post-processing]
The post-processing is performed according to the following procedure.
First, as shown in FIG. 23, the distributed CV video is received via WEB or the like, or read from a purchased DVD or the like (S2301: CV video reception).
In addition, the current 360-degree video is acquired by a camera loaded on the vehicle (S2302: real-time video acquisition). Also here, the acquired video does not necessarily have to be a 360-degree video.
Then, the current position of the vehicle is calculated by comparing the received CV image and the real-time image captured by the in-vehicle camera (S2303: real-time vehicle position measurement). The calculation of the current position of the vehicle includes (1) a method of calculating by CV calculation, (2) a method of calculating the coordinates of the vehicle by calculation from a designated point whose coordinates are known, and (3) an intermediate between the combination of both. There is a method.

Thereafter, the obtained three-dimensional position of the host vehicle is displayed on the map with high accuracy (for example, accuracy more than 10 times GPS), and items necessary for navigation are automatically displayed (S2304: navigation item automatic selection, S2305: Multiview display). At this time, the navigation purpose attribute is displayed. Further, the coordinates of the vehicle and the map can be compared, and signs, road markings, guide plates, etc. necessary for traveling of the host vehicle can be sequentially selected according to the traveling. Furthermore, the attribute can be displayed by clicking on the object in the display image.
Note that displaying the three-dimensional position of the host vehicle is sufficient for navigation, and the CV video (recorded video) itself may not be displayed.
Further, when displaying the position of the host vehicle on a three-dimensional map, the GPS can be used as auxiliary information for obtaining the approximate position (see the second embodiment described later).

Although the entire operation of the navigation device is completed as described above, as shown in FIG. 24, in the post-processing on the navigation device side, the CV video around the host vehicle is displayed at an arbitrary viewing angle, and from an arbitrary viewpoint depending on the purpose. It can be displayed (S2306 in FIG. 24: arbitrary viewpoint video and attribute display). At this time, a traffic sign etc. can be taken out from a common CG part and expressed. It is also possible to display daytime images at night or summer images in the snowy winter.
In addition, as shown in FIG. 25, in post-processing, information on the driving situation such as the inter-vehicle distance, the direction and speed of the vehicle running around, the road surface condition, etc. is judged in real time, and the recognition / judgment results necessary for the running are obtained. Can also be displayed (S2307 in FIG. 25: travel status recognition / judgement).

Furthermore, as shown in FIG. 26, the data obtained in the post-processing can be transmitted / received to / from other vehicles and base stations (S2308 in FIG. 26: travel state recognition / judgment data transmission / reception).
In other words, in order to share the driving situation recognition / determination data as a result of recognition / judgment with the own vehicle with other vehicles, predetermined data can be transmitted to the multi-payment, and at the same time, other vehicles traveling around can It is possible to receive the traveling state recognition / determination data as a result of the determination and reflect it in the traveling of the host vehicle (see the third embodiment described later).
Furthermore, it is also possible to transmit the traveling state recognition / determination data of the own vehicle to a predetermined base station, receive the traveling state recognition / determination data sent from the base station, and reflect it in the traveling of the own vehicle (described later). See the third embodiment).

As described above, according to the navigation device of the present embodiment, a sufficient number of feature points are automatically detected from a plurality of frame images of a moving image captured by a camera mounted on a moving body such as a vehicle, and each frame is detected. By automatically tracking feature points between them, multiple feature points can be overlapped and camera vectors (camera position and rotation angle) and three-dimensional position coordinates of the feature points can be obtained with high accuracy.
Then, the three-dimensional coordinates of the obtained feature points are stored in advance in a recording medium, and the three-dimensional coordinates are compared with a camera image taken from a moving body that actually moves, or an image obtained from the camera From the above, it is possible to directly generate the three-dimensional coordinates of the camera position in real time and obtain highly accurate three-dimensional information indicating the current camera position, thereby being used as a navigation system for a mobile object.

Specifically, in the navigation device of the present invention, in order to obtain the current position coordinates of a moving body such as a vehicle in real time with higher accuracy than GPS, the image processing technique is used to obtain a plurality of features in the image. Pay attention and measure the three-dimensional coordinates of the feature points in advance with high accuracy. Then, a map (3D map) in which the feature points are described in three-dimensional coordinates is stored in a recording medium, and the recording medium is reproduced on the mobile body side, whereby the three-dimensional coordinates of the feature points can be read out. Further, the feature point in the video is extracted from the camera image obtained at the current location of the moving object, and the direction of the feature point and the direction of the feature point whose three-dimensional coordinates recorded in advance on the recording medium are known. In comparison, by obtaining the coordinates of points where the directions of a plurality of feature points coincide with each other, three-dimensional coordinates indicating the camera position, that is, three-dimensional coordinates indicating the current position of the moving body can be obtained.
As a result, the current position of a moving object such as a traveling vehicle is accurately indicated directly from a camera image or by a three-dimensional map generated and recorded in advance, which is impossible with a conventional GPS system. A highly accurate navigation system with an error range of about several centimeters can be realized.

[Second Embodiment]
Next, a second embodiment of the navigation device of the present invention will be described with reference to FIG.
FIG. 27 is a block diagram showing a schematic configuration of the navigation device 100 according to the second embodiment of the present invention. The navigation device 100 of the present embodiment includes an optional device 300 that can be selectively added.
As shown in the figure, in the present embodiment, as the optional device 300, a 3D map attribute adding device 310, a GPS device 320, a data updating device 330, and an optional display device 340 can be provided.

[3D map attribute addition device]
The 3D map attribute adding device 310 can add attribute information as additional information other than the three-dimensional coordinates of feature points as information recorded on the recording medium 120.
Here, the attribute of the feature point includes, for example, the name of the building to which the feature point belongs, the street name, the address, the history description, and the like. is there.
Specifically, the feature point attribute acquisition unit 311 acquires the feature point attribute.
The belonging object 3D shape coordinate acquisition unit 312 inputs the three-dimensional coordinates of the feature point to which the attribute is to be added from the feature point / camera vector calculation unit 115 of the feature point three-dimensional map generation device 110.
The attribute adding unit 313 adds attribute information corresponding to the input three-dimensional coordinates.
The attributed object 3D map generation unit 314 returns the 3D information to which the attribute information is added to the 3D map generation unit 119 of the feature point 3D map generation device 110.
Thereby, feature point attribute information is added as three-dimensional information recorded in the recording medium 120.

[GPS device]
The GPS device 320 outputs latitude / longitude altitude data obtained by GPS to the approximate current position specifying unit 132 of the point search navigation device 130, and specifies the approximate current position of the moving object in the approximate current position specifying unit 132.
Further, the GPS device 320 inputs three-dimensional data such as the camera position, direction, and posture indicating the current moving body situation obtained by the camera coordinate calculation unit 138 of the point search navigation device 130, and obtains the data obtained by GPS. It is corrected and used as an auxiliary device when a feature point cannot be obtained from the video.

The GPS is less accurate than the position information obtained by the navigation device of the present invention, but is suitable as information for specifying the approximate position. If there is approximate position information from the GPS, the point search navigation device 130 is a great clue to search for feature points around the current point.
Further, in such a measurement system mainly composed of GPS having poor accuracy, the high-accuracy position data according to the present invention is used as a correction signal to correct the data obtained by the GPS system. The accuracy equivalent to that of the present invention can be maintained for a certain period.
Further, in the navigation device of the present invention, it may be desirable to use the GPS system together at night or in a tunnel where it is difficult to obtain a camera image.

Therefore, in the present embodiment, when the GPS device 320 is combined as an optional device of the navigation device 100, measurement is performed at the measurable point by the navigation device 100, and image feature points cannot be sufficiently obtained. The GPS system can be operated with high accuracy by correcting and calibrating the GPS with the final data acquired by this apparatus. As a result, the advantages of the navigation device 100 and the GPS can be effectively utilized.
Specifically, the GPS data acquisition unit 321 acquires data obtained by GPS.

The device coordinate calculation unit 322 generates and outputs a coordinate signal to be input to the approximate current position designation unit 132 of the point search navigation device 130 based on the GPS data.
The GPS data correction unit 323 receives highly accurate position information obtained by the camera coordinate calculation 138 of the point search navigation device 130, detects a difference from the GPS measurement device, and generates a GPS correction calibration signal therefrom. To do. The GPS data correction unit 323 is provided with an output terminal for sending a highly accurate correction calibration signal generated to the GPS position measurement device.
The current location display unit 324 outputs and displays the GPS data corrected by the correction calibration signal as the current location display.

Thus, in this embodiment, by providing the GPS device 320 as an optional device, it normally operates as the point search navigation device 130, and it is difficult to continuously acquire video feature points such as nighttime. In this case, the GPS navigation function can be used together, and the point data obtained from the point search navigation device 130 acquired in pieces can be used as the correction signal to display the GPS data corrected with high accuracy.
In the present embodiment that includes the GPS device 320 as an option, data necessary for a navigator system using GPS can also be recorded on the recording medium 120.

[Data update device]
The data update device 330 is a device that can update the data in the recording medium 120 and adds the function of the feature point 3D map generation device 110 to the point search navigation device 130.
Specifically, the data update device 330 stores the camera video acquired in the point search navigation device 130 by the video recording unit 331.
The feature point data recalculation unit 332 tracks feature points of a recording medium over a plurality of frames and corresponding points with a small area image including the feature points in an image acquired by a camera, and a feature point three-dimensional map generation device In the same way as 110, the movement of the existing feature points is performed based on the correspondence between the feature points of the recording medium over a plurality of frames or the small region image including the feature points and the small region image in the video acquired by the camera. Etc. is required.
In addition, the new feature point acquisition unit 333 detects a new feature point from the video by the camera and adds it to the three-dimensional coordinates, thereby adding it as a feature point from the next time and adding it as a feature point from the next time.

The feature point data update unit 334 updates the data, and the update data recording unit 335 outputs the update data to the recording medium 120.
Thus, by providing the data update device 330, a function corresponding to the feature point three-dimensional map generation device 110 can be added to the point search navigation device 130 provided on the user side, while creating a map. It is possible to perform a search, and at the same time as a local point search, feature point detection and three-dimensional coordinate calculation are simultaneously performed and recorded, and data of the recording medium can be updated and used as data from the next time.

[Option display device]
The option display device 340 can display additional information other than the display content displayed on the point search navigation device 130. For example, even if it is not directly related to the recognition of the current location in the location search navigation device 130, an image or CG such as a traffic sign or a road display that assists the user's travel is displayed by the option display device 340 for better understanding. The navigation device is easy to operate and easy to operate.
Specifically, in the current location surrounding image display unit 341, display image data of the feature point 3D map reproduction unit 131 and the current location display unit 139 of the location search navigation apparatus 130 are input.
The display image attribute display unit 342 displays the attributes of the input display image data.
Further, the traffic sign recognition display unit 343 displays a traffic sign observed from the current location, an image such as a road display, and CG.

As described above, according to the navigation device according to the present embodiment, a sufficient number of feature points are automatically extracted from a plurality of frame images of a moving image captured in advance by a camera mounted on a vehicle or the like for generating a 3D map. By detecting and automatically tracking the feature points between the frames, it is possible to calculate the camera position and the rotation angle with high accuracy by performing overlap calculation on a large number of feature points.
Then, the obtained three-dimensional coordinates of the camera position are stored in a recording medium in advance (or the three-dimensional coordinates of the camera position are generated in real time), and the three-dimensional coordinates are actually moved from a navigation target vehicle or the like. By contrasting with the captured camera image, highly accurate three-dimensional information indicating the current camera position can be obtained, and as a result, it can be used as a navigation system for a mobile object.

[Third embodiment]
Next, a third embodiment of the navigation device of the present invention will be described with reference to FIGS.
FIG. 28 is a block diagram showing a schematic configuration of the navigation device according to the third embodiment of the present invention, and FIG. 29 is a schematic configuration of another embodiment of the navigation device according to the third embodiment of the present invention. FIG.
In the navigation apparatus shown in these drawings, the recording medium 120 and the point search navigation apparatus 130 are provided apart from each other, and predetermined three-dimensional information recorded in the recording medium 120 provided in the base station or other mobile object is It is transmitted to one or more other point search navigation devices 130 via a communication line.

Specifically, the navigation apparatus shown in FIG. 28 employs a satellite communication system. First, a recording medium 120 on which predetermined three-dimensional information is recorded is provided in a satellite apparatus 400 serving as a base station.
In the satellite device 400, the receiving unit 411 receives the update data from the data update device 332 via the data update reverse transmission device 350, and the data update unit 412 can update the data of the recording medium 120 at any time. Yes.
Then, the data in the recording medium 120 is transmitted to the point search navigation device by the transmission unit 413.

A receiving device 140 is provided on the spot search navigation device 130 side.
The receiving device 140 can receive the three-dimensional information data of the recording medium 120 transmitted from the satellite device 400 via the communication line by the receiving unit 141.
Furthermore, the navigation apparatus shown in FIG. 29 adopts an intercommunication system in addition to the satellite communication system shown in FIG. 28. Can be done.

According to such a navigation apparatus of the present embodiment, data of the recording medium 120 is received from the base station, update data and newly acquired data are transmitted to the base station, and data is directly exchanged between vehicles. Then, the data can be updated as needed and the data can be shared with other vehicles.
Thereby, compared with the case of only one vehicle, a wider and comprehensive navigation system can be realized.

The recording medium 120 is usually a DVD, a hard disk, or the like, and is loaded or installed in a device on the moving body side as a solid.
In the present embodiment, the data on the recording medium 120 is transmitted from the base station to a plurality of vehicles, so that the recording medium 120 can be handled as software itself and digital data itself. The reception of software has the same meaning as the loading of the recording medium 120 as a solid, and the usage range is expanded because it does not depend on the configuration or standard of the playback apparatus.
Further, in the present embodiment, the data newly acquired by the vehicle can be sent to the base station and directly exchanged between the vehicles. Further, the data can be easily updated by using the communication line, Data can be shared, and real-time data can be exchanged.
Thus, according to the present embodiment, a navigation device with more versatility can be provided.

[Fourth embodiment]
Furthermore, a fourth embodiment of the navigation device of the present invention will be described with reference to FIGS.
FIG. 30 is a block diagram showing a schematic configuration of the navigation device according to the third embodiment of the present invention.
As shown in the figure, the real-time navigation apparatus 200 of the present embodiment omits the recording medium 120 shown in the first embodiment, and the three-dimensional feature point in the first embodiment is placed on the side of the mobile object to be navigated. A feature point three-dimensional map generation / display device 210 is provided, in which the map generation device 110 has the function of the point search navigation device 130.

The feature point 3D map generation / display device 210 provided in the real-time navigation device 200 does not include the recording medium as described in the first embodiment, but is obtained by a camera provided in a moving body such as a vehicle to be navigated. On the basis of the actual video to be generated, the three-dimensional coordinates of the video feature points in the range observed from the moving body are generated in real time as they are, and the camera vector is generated.
Then, a 3D map is generated directly from the obtained three-dimensional coordinates of the feature points, and the three-dimensional distribution of the feature points and the position, speed, Among predetermined items including acceleration, viewpoint direction, three-axis rotation posture, three-axis rotation speed, and three-axis rotation acceleration, a plurality of items are output or displayed.

Specifically, the real-time kite navigation apparatus 200 of this embodiment includes a feature point three-dimensional map generation display apparatus 210 as shown in FIG.
The feature point 3D map generation / display apparatus 210 has substantially the same configuration as the feature point 3D map generation apparatus 110 of the first embodiment, and includes a camera video acquisition unit 211, a video recording unit 212, and a feature point extraction unit 213. , A feature point correspondence processing unit 214, a feature point / camera vector calculation unit 215, an error minimization unit 216, a shaking component detection unit 217, an absolute coordinate acquisition unit 218, and a 3D map generation display unit 219. Among these, the camera image acquisition unit 211, the video recording unit 212, the feature point extraction unit 213, the feature point correspondence processing unit 214, the feature point / camera vector calculation unit 215, the error minimization unit 216, the shake component detection unit 217, and the absolute coordinates The acquisition unit 218 includes a camera video acquisition unit 111, a video recording unit 112, a feature point extraction unit 113, a feature point correspondence processing unit 114, a feature point / camera vector calculation unit 115, an error minimization unit 116, The configuration is the same as that of the shake component detection unit 117 and the absolute coordinate acquisition unit 118.

In the feature point 3D map generation / display apparatus 210 of the present embodiment, the 3D map generation / display unit 219 is subjected to error minimization processing by the error minimization unit 216 and given absolute coordinates by the absolute coordinate acquisition unit 218. The three-dimensional shape of the image of a feature point or a small area including the feature point, its three-dimensional coordinates, and its distribution as a three-dimensional map together with the movement trajectory of the mobile object to be navigated and, if necessary, the planned movement path These are arranged and displayed directly together with objects including feature points.
32 and 33 show the three-dimensional coordinates of feature points generated and displayed by the real-time navigation method in the present embodiment and the current position of the moving object.
As shown in FIG. 32 (a), a large number of feature points around a traveling vehicle are extracted, and a three-dimensional map of roads and buildings on which the vehicle travels is generated from the feature points and moved into the map. The trajectory of the vehicle is shown. FIG. 32B shows the extracted feature points and the obtained trajectory of the vehicle in the actual camera video.
FIG. 33 is a planar development image of the video shown in FIG. 32 (b), in which the traveling locus of the vehicle, the current position, and the planned course are plotted in the video.

In this way, the real-time navigation device 200 of the present embodiment can search for the current location while making a 3D map directly while moving with a mobile object to be navigated, Real-time navigation that can perform 3D coordinate calculation and local point search at the same time, record, and update data on the recording medium is realized.
In the first embodiment described above, the feature point 3D map generation device 110 generates a 3D map by offline processing after acquiring an image, and then records the 3D map on the recording medium 120 and distributes it, etc. Thus, a method of confirming the current position by comparing the feature point recorded on the recording medium 120 with the current video is adopted.
In the present embodiment, a 3D map can be generated in real time while moving with a vehicle or the like, a current location can be searched, and the recording medium 120 can be omitted. Thereby, for example, even when traveling in an area not described in the feature point three-dimensional map recorded on the recording medium 120 or when the recording medium 120 is not provided, the feature point three-dimensional map is displayed in real time. You can display the current location while generating with. Of course, also in this embodiment, the recording medium 120 shown in the first embodiment can be used in combination.

Therefore, as a navigation apparatus according to the present invention, as shown in FIG. 31, a system including a recording medium 120 (navigation apparatus 100 shown in the figure) and a system omitting the recording medium 120 (navigation apparatus 200 shown in the figure). In the system including the recording medium 120, the feature point 3D map generation device 110 is provided separately from the point search navigation device 130, and the feature point 3D map generation device 110 and the point search navigation device are provided. The case where 130 is provided integrally can be implemented.
Furthermore, as shown in FIG. 31, the various optional devices 300 shown in the first embodiment can be provided, and the type, moving route, moving range, purpose of use, etc. of the moving body on which the navigation device of the present invention is mounted. Accordingly, the optional devices 300 can be selectively combined and employed.

Next, details of the real-time navigation device 200 of the present embodiment having the above-described real-time navigation function will be described with reference to FIGS. 34 to 36.
FIG. 34 is a block diagram illustrating a schematic configuration of an embodiment of the real-time navigation device, and describes a case where the real-time navigation device is mounted on a vehicle traveling on a road as a moving body.
In addition, the specific content of the processing operation | movement in each part of the real-time navigation apparatus shown below is the same as the content in corresponding 1st and 2nd embodiment.

[Basic type]
In the real-time navigation device 200 shown in the figure, an image acquisition unit 200-01 acquires a surrounding image by a camera mounted on the vehicle.
The temporary image recording unit 200-02 primarily records the surrounding image acquired by the image acquisition unit 200-01.
The feature point extraction unit 200-03 extracts feature points from the surrounding images recorded in the image temporary recording unit 200-02.
The feature point tracking unit 200-04 tracks feature points in adjacent images.

The feature point tracking coordinate table creation unit 200-05 records each coordinate of the feature point in the plurality of images tracked by the feature point tracking unit 200-04.
The vector calculation unit 200-06 selects some of the coordinates of the feature points in the image and obtains the camera vector and the feature point distribution by calculation.
The absolute coordinate conversion unit 200-07 gives absolute coordinates to the calculation result in the vector calculation unit 200-06.
The feature point distribution in-camera vector display unit 200-08 displays the calculation result given the absolute coordinates together with the three-dimensional distribution of the feature points.

[Absolute coordinate conversion by reference object]
In the map part 200-09, a planned traveling path of a vehicle constituting the moving body is described.
The reference object database unit 200-10 describes a reference object that is in a position that can be seen from the traveling path of the vehicle and whose coordinates and shape are known. Since the moving object is a vehicle as the reference object, for example, a traffic light or the like at each intersection of the traveling road is suitable. With the reference object database unit 200-10, if the approximate position is known, the specifications of the reference object (signals, etc.) can be known. Since the size is standardized, if the format is known, it can be used as a known reference object.
The reference object recognition unit 200-11 recognizes a reference object that is in the surrounding image acquired by the image acquisition unit 200-01 and whose shape and coordinates are known in the image.
The reference object position calculation unit 200-12 calculates the three-dimensional coordinates of the reference object from the position in the image of the reference object recognized by the reference object recognition unit 200-11.
The absolute coordinate conversion unit 200-13 compares the three-dimensional coordinates of the reference object obtained by the reference object position calculation unit 200-12 with the known data of the reference object, and converts the coordinates into absolute coordinates. .
The combination display unit 200-14 displays the converted absolute coordinates of the camera together with a map prepared in advance and displays the combined coordinates.

[Attitude control]
The camera posture signal acquisition unit 200-15 detects a three-axis posture signal of the camera from the camera vector obtained by the vector calculation unit 200-06.
The vehicle attitude control unit 200-16 performs vehicle attitude control based on the camera triaxial attitude signal detected by the camera attitude signal acquisition unit 200-15.
Since the rotation component of the camera can be extracted from the camera vector, the vehicle attitude can be measured. Then, a feedback signal can be generated from the vehicle attitude signal so that the vehicle attitude can be controlled so that the vehicle attitude maintains the target position.
The horizontal and vertical directions can be calibrated with a level or the like when the vehicle is stationary or at constant acceleration.

[Specify approximate position by GPS]
The GPS data acquisition unit 200-17 acquires position data by GPS.
The approximate position coordinate acquisition unit 200-18 specifies the approximate position and direction of the vehicle based on the GPS position data, and specifies the reference object in the surrounding image acquired by the image acquisition unit 200-01.
As shown in the first embodiment, the navigation system of the present invention is superior to the position accuracy of the existing GPS. Therefore, the position data can be narrowed down by using the position data by GPS for the approximate position acquisition. This is advantageous for calculation. In addition, it is possible to obtain latitude and longitude from position data by GPS and display the camera position in latitude and longitude. Furthermore, by capturing the GPS data, for example, when a feature point is not found, it is possible to travel by GPS navigation.

[Absolute measurement correction by parallel camera]
The image acquisition unit 200-01 described above can be added with a function of acquiring a parallel image by a camera in which a plurality of cameras that are arranged in parallel so that the fields of view overlap and whose positional relationship is fixed are stacked.
In the feature point tracking unit 200-04, a function of searching for the corresponding points of the feature points in the parallel images from the images from a plurality of cameras can be added.
Thereby, the vector calculation unit 200-06 can be added with a function for calculating the absolute length of the feature point and the camera vector from the coordinates of each corresponding point in the parallel image. In this case, since the absolute length can be acquired at all camera positions by the parallel camera, measurement with little error accumulation can be performed in long-distance measurement.

[Movement vector calculation]
The moving object feature point tracking unit 200-19 treats the feature point static coordinate system from which the feature points are removed as the feature point of the moving object.
The movement tracking coordinate table creation unit 200-20 generates a table of tracked feature points.
The moving body vector calculation unit 200-21 calculates a moving body vector for each moving body, converts the moving body vector into a stationary coordinate system, and synthesizes it with the previously obtained feature point of the stationary coordinate system. The feature point and the vector of the moving object are superimposed and displayed.
Details of this moving object vector extraction processing are shown in FIG.

In FIG. 35A, the camera vector determination unit S3401 based on the stationary coordinate system has already obtained the camera vector, that is, the camera position and direction three-dimensionally.
In the moving object feature point extraction unit S3402 other than the stationary coordinate system, the feature points other than the stationary coordinate system are selected and extracted from all the feature points, and it is determined that they include the feature points in the moving object. Tracking is performed in the feature point tracking unit S3403 that fixes the feature point and the camera is moved, and is stored as a table, and a vector of feature points of the moving object is obtained by calculation in the following process.

First, the feature points are moving because they are moving objects, but from the viewpoint of the camera, since it cannot be determined whether the coordinate system is a stationary coordinate system or a moving coordinate system, they are all treated as feature points of the stationary coordinate system.
In the feature point tracking unit S3403 in which the feature point is fixed and the camera is moved, the feature point is tracked, and the feature point tracking table is generated in the moving body feature point tracking table creation unit S3404. In this table, groups of different camera positions are generated for each moving object. Even if there is one camera, the moving object is observed as a group of feature points, and multiple camera positions are observed at different feature points corresponding to the moving object so that the group corresponds to one camera position. become.

In a plurality of camera position classification units S3405 viewed from the feature point coordinate system, several camera positions are obtained, and the moving object is classified for each camera position. In the feature point classification unit S3406, which is a group for each camera position, the feature points are classified for each moving object.
Since the camera position and the moving body are in relative motion, coordinate conversion is possible at any time. Therefore, by returning the camera to the original stationary coordinate system display in the coordinate system determining unit S3407 of each feature point group viewed from the stationary coordinate system, the feature point group and coordinate determining unit S3408 of each moving object The coordinates of feature points can also be converted to a stationary coordinate system. As a result, the moving object extraction unit S3409 can extract the moving object three-dimensionally into the stationary coordinate system.
It can also be calculated from the beginning based on the stationary coordinate system, in which case it is as shown in FIG.

According to the above-described real-time navigation device, the calculation is simplified in order to pursue real-time performance, and a recording medium such as a large hard disk is not used. By extracting feature points, extracting camera vectors and feature points, using only appropriate ones from the feature points for calculation, and displaying only others, the distribution map of sufficiently many feature points can be obtained. Generate a rough three-dimensional map.
And by displaying a camera vector in it, a driving | running | working position can be displayed in a road and the surrounding shape. In addition, the calculation speed can be increased by selecting a calculation feature point from among the feature points.

FIG. 36 shows an automatic take-off and landing device as an application example of the real-time navigation device. The figure shows an example in which the real-time navigation device according to the present embodiment is applied as an aircraft automatic take-off and landing device (or guidance device).
In the example shown in the figure, a wide-angle camera is attached at an arbitrary position on the ground side of the aircraft. The camera takes a picture of the ground side and captures the runway in part.
A sufficiently large number of feature points are automatically detected and tracked in the video imaged by the camera, and the three-dimensional position and attitude of the aircraft are calculated.
In addition, for the purpose of reconfirming the position and improving the position accuracy, a known object on the ground is obtained, compared with a known part stored in the database, recognized in the video, and the name and shape. Check the coordinates.

In this way, absolute coordinates are obtained, and the aircraft position and attitude are completely related in a three-dimensional manner with high accuracy as the positional relationship with the ground runway.
By setting the desired ideal course of the aircraft and detecting and controlling the deviation from the actual course of the aircraft acquired by this navigation device, it is possible to automatically obtain an approach path near the ideal course become. The same control can be performed during takeoff of the aircraft.
Application examples of the navigation device similar to the above include (1) automatic take-off and landing device, (2) spacecraft automatic docking device, (3) automatic stop position securing device for train vehicles, and (4) automatic vehicle parking. It can be applied to a device, (5) a ship automatic berthing device, and the like.

As mentioned above, although the preferable embodiment was shown and demonstrated about the navigation apparatus of this invention, the navigation apparatus which concerns on this invention is not limited only to embodiment mentioned above, A various change implementation is carried out in the scope of the present invention. It goes without saying that it is possible.
For example, since the navigation apparatus of the present invention has a three-dimensional map, the moving body to be applied is not limited to a vehicle traveling on the ground, but may be one that navigates a three-dimensional space. Because it can be used on airplanes, it is possible to navigate with high accuracy when landing. In addition, navigation on a cosmic scale is also possible using stars and constellations that can be seen from a spacecraft as feature points.

  As described above, the present invention can be used as a navigation device suitable for, for example, a car navigation device mounted on an automobile, a navigation device mounted on an airplane, a navigation device for automatic driving, a navigation device for a robot, and the like. .

Claims (14)

A recording medium in which video feature points in a range observed from a moving object to be navigationally recorded are recorded in three-dimensional coordinates;
3D coordinates that match the actual image by comparing the actual image obtained by the camera provided on the moving object to be navigated with the 3D coordinates of the image-like feature points obtained by reproducing the recording medium Find the upper point and direction, out of the predetermined items including the position, speed, acceleration, viewpoint direction, 3-axis rotation attitude, 3-axis rotation speed, 3-axis rotation acceleration of the camera provided on the moving body , A point search navigation device that outputs a plurality of items, or any combination thereof, and
A navigation device comprising: Information recorded on the recording medium is:
The types of video feature points in the range observed from the moving object and their three-dimensional coordinates,
A three-dimensional arrangement of the two-dimensional image of the small area including the video feature point and its three-dimensional coordinates
The shape of the object including the video feature point and its three-dimensional coordinates;
A peripheral image necessary for movement of a moving body other than the video feature point, a shape such as CG, and three-dimensional coordinates;
Including an image of a road on which a moving body moves, a vehicle traveling path, or a planned route, CG, its three-dimensional shape, and its three-dimensional coordinates,
The navigation device according to claim 1, wherein each of these pieces of information includes any one or a combination thereof, all of them, or their attribute information, and is recorded together with the three-dimensional map. The point search navigation device comprises:
A feature point 3D map reproduction unit for reproducing the recording medium;
An approximate current position designating unit that designates the approximate current position of the moving object and limits the search range at the initial setting;
From the three-dimensional map recorded on the recording medium, a plurality of feature points around the current location of the mobile object is read, and a current location feature point designation unit that designates the current location as a search target
A camera image acquisition unit for acquiring an image around the moving object from a camera provided in the moving object to be navigated;
A video temporary recording unit that records the video acquired by the camera video acquisition unit;
In-video feature point search unit for searching for feature point candidates to be the same as the search target in the video recorded in the video temporary recording unit;
The feature point candidates obtained by the in-video feature point search unit and the search target around the current location are compared and collated to obtain a correspondence relationship as the same object, and a predetermined number of correspondence points are determined from the candidates. An in-video feature point corresponding unit that receives the three-dimensional coordinates of the determined corresponding point from the recording medium;
Using the determined corresponding point and its three-dimensional coordinates, a camera coordinate calculation unit that determines three-dimensional data such as a camera position, a direction, and a posture indicating the current state of the moving object by calculation,
A current point display unit for displaying on the screen a combination of all or all of the three-dimensional data determined by the camera coordinate calculation unit alone or with information such as a map, video, and attribute recorded on the recording medium;
The navigation device according to claim 1, further comprising: Based on the actual image obtained by the camera provided on the moving body for generating the recording medium, the video feature points in the range observed from the moving body are recorded in three-dimensional coordinates and should be recorded on the recording medium. The navigation apparatus according to claim 1, further comprising a feature point three-dimensional map generation apparatus that generates information. The feature point three-dimensional map generation device includes:
A camera video acquisition unit for acquiring a surrounding image of the moving body from a camera provided in the moving body for generating the recording medium;
A video recording unit for recording the image acquired by the camera video acquisition unit;
A feature point extraction unit for automatically extracting a predetermined number of feature points from the image data recorded in the video storage unit;
About the feature points extracted by the feature point extraction unit, a feature point correspondence processing unit that automatically tracks within each frame image and obtains a correspondence relationship between the frame images;
A feature point / camera vector computing unit that obtains a three-dimensional position coordinate of the feature point for which the correspondence is obtained by the feature point correspondence processing unit, and obtains a camera vector corresponding to each frame image from the three-dimensional position coordinate;
The feature point three-dimensional coordinates and camera subjected to statistical processing so that the distribution of the three-dimensional position coordinates of each feature point and the camera vector obtained in the feature point / camera vector calculation unit is minimized and the error is minimized. An error minimizer that automatically determines the vector;
A mobile object to be used for navigation is a camera vector and a feature point subjected to error minimization processing by the error minimizing unit, a three-dimensional shape of an image of a small region including the feature point, its three-dimensional coordinates, and its distribution. A 3D map generation / recording unit that is arranged as a three-dimensional map with a passage of the object, and that is recorded on the recording medium together with an object including a feature point,
The navigation device according to claim 4. Based on the actual video obtained by the camera provided on the mobile object to be navigated, the video feature points in the range observed from the mobile object are generated in three-dimensional coordinates, and the camera vector is derived from the three-dimensional coordinates. Produces
While generating a three-dimensional map based on the generated three-dimensional coordinates, the three-dimensional distribution of feature points and the position, speed, acceleration, viewpoint direction, and triaxial rotation on the three-dimensional coordinates of the camera provided in the moving body A navigation device comprising: a feature point three-dimensional map generation / display device that outputs a plurality of items obtained by combining any of predetermined items including posture, three-axis rotation speed, and three-axis rotation acceleration. The feature point three-dimensional map generation and display device,
A camera video acquisition unit for acquiring a surrounding image of the mobile body from a camera provided in the mobile body;
A video recording unit for recording the image acquired by the camera video acquisition unit;
A feature point extraction unit for automatically extracting a predetermined number of feature points from the image data recorded in the video storage unit;
About the feature points extracted by the feature point extraction unit, a feature point correspondence processing unit that automatically tracks within each frame image and obtains a correspondence relationship between the frame images;
A feature point / camera vector computing unit that obtains a three-dimensional position coordinate of the feature point for which the correspondence is obtained by the feature point correspondence processing unit, and obtains a camera vector corresponding to each frame image from the three-dimensional position coordinate;
The feature point three-dimensional coordinates and camera subjected to statistical processing so that the distribution of the three-dimensional position coordinates of each feature point and the camera vector obtained in the feature point / camera vector calculation unit is minimized and the error is minimized. An error minimizer that automatically determines the vector;
A mobile object to be used for navigation is a camera vector and a feature point subjected to error minimization processing by the error minimizing unit, a three-dimensional shape of an image of a small region including the feature point, its three-dimensional coordinates, and its distribution. A 3D map generation and display unit arranged as a three-dimensional map together with a movement trajectory or a planned movement path if necessary, and displayed together with an object including a feature point,
The navigation device according to claim 6. The feature point / camera vector calculation unit is
The unit calculation for obtaining the three-dimensional coordinates of the desired feature point and the camera vector is repeated using any two frame images Fn and Fn + m (m = frame interval) used for the three-dimensional coordinate calculation of the camera vector and the feature point as unit images,
For a frame image between the two frame images Fn and Fn + m, the camera vector and the three-dimensional coordinates of the feature points are obtained by a simplified calculation.
The error minimizing unit includes:
As n advances continuously with the progress of the image, each camera vector obtained by calculating multiple times for the same feature point and the three-dimensional coordinate error of the feature point are adjusted and integrated so that the error is minimized, The navigation device according to claim 5 or 7, wherein final three-dimensional coordinates are determined. The feature point / camera vector calculation unit is
The navigation apparatus according to claim 8, wherein unit calculation is performed by setting the frame interval m so that m increases as the distance from the camera to the feature point increases according to the distance from the camera to the feature point. The feature point / camera vector calculation unit is
A feature point with a large error distribution of the obtained camera vector or feature point in three-dimensional coordinates is deleted, and if necessary, another feature point is recalculated to increase the accuracy of the three-dimensional coordinate calculation. 8. The navigation device according to 5 or 7. The recording medium and the point search navigation device are provided separately from each other,
3. The predetermined three-dimensional information recorded on a recording medium provided in a base station or another mobile body is transmitted to one or more point search navigation devices via a communication line. Navigation device. The navigation device according to claim 1 or 6, wherein the point search navigation device designates the approximate current position of the moving body by the approximate current position designation unit based on latitude and longitude altitude data obtained by GPS. The point search navigation device converts the three-dimensional data such as the camera position, direction, and posture obtained by the camera coordinate calculation unit indicating the current moving body status into a latitude and longitude altitude and corrects the GPS as a correction signal. 13. The navigation apparatus according to claim 12, wherein the navigation apparatus outputs an auxiliary signal for obtaining position data from GPS when a video feature point cannot be obtained. The navigation apparatus according to claim 1 or 6, wherein the mobile object to be navigated is an automobile, an aircraft, a ship, a person, a robot, a heavy machine, a spacecraft, a deep sea exploration ship, a machine having a moving part, or the like.

Images