JP2018081008A - Self position posture locating device using reference video map - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily, inexpensively, and highly precisely obtain self position posture location for locating a position and a posture of a moving vehicle or the like.SOLUTION: A self position posture locating device includes: a CV video map creating device 20 that performs a CV operation for obtaining a CV (camera vector) value indicative of three-dimensional coordinate values of a camera position and a posture of reference video, based on the reference video photographed by a CV video acquisition device 10, and generates a CV video map obtained by adding a CV value to the reference video; a CV video map database 30 for storing a CV video map; a CV video map/target image comparison device 50 for reading a CV video map corresponding to a target image photographed by a target moving body 40; and a self position posture locating device 60 for acquiring a CV value of a target image by transplanting a CV value added to the read CV video map in a feature point of the corresponding target image. The self position posture locating device 60 is configured to select a CV value added to a predetermined feature amount included in a CV video map as a CV value transplanted in the corresponding feature point of the target image.SELECTED DRAWING: Figure 14

Description

  The present invention is for, for example, automatic operation of various vehicles such as automobiles, mobile objects such as airplanes and ships, and automatic traveling of robots and the like for a mobile object such as a moving vehicle to determine its position and posture in real time. The present invention relates to a self-position posture locating device.

In general, in automatic operation of a moving body such as an automobile, an aircraft, and a ship, a self-position / posture positioning technique for a moving vehicle or the like to grasp and position its own position and posture is important.
Here, there are several methods for the self-position / posture determination of the moving body.
Specifically, as a self-position / posture locating apparatus for a moving vehicle or the like, in principle, six variables (position coordinates (X, Y, Z)) acquired by an expensive and highly accurate IMU / GYRO used for an aircraft or the like. And six degrees of freedom vector (variables) data of the rotation angles (Φx, Φy, Φz) of the respective coordinate axes. However, as a matter of fact, in order to achieve accuracy that can withstand automatic driving, the equipment and facilities are very expensive and are not practical.
In addition, GNSS (GPS) is a device that is generally spread as self-positioning, but since it is a three-variable only with coordinates, it cannot acquire six variables including posture.

In recent years, a self-localization method called a LIDAR method (Light Detection and Ranging, Laser Imaging Detection and Ranging) has become mainstream. This is a point cloud created by scanning a laser pulse, receiving scattered light from the three-dimensional space at the launch position, and measuring the distance from the time difference to create a point cloud and the three-dimensional space with the three-dimensional coordinates. It is a technology that is made up closely.
As a self-position / posture determination technique using such a point cloud, for example, in automatic driving of a vehicle or the like, the laser point cloud is made into a three-dimensional map and compared with laser scan data from an in-vehicle laser device, Is disclosed (Patent Document 1).

Recently, a technique has been proposed in which such a point cloud is processed to simultaneously perform self-position / posture determination and environmental map creation. This is a technology called SLAM (Simultaneous Localization And Mapping), which has become popular in recent years.
As a technology developed from this SLAM, there is V-SLAM (Visual SLAM) in which a point cloud is created in the same manner as a laser point cloud from an on-vehicle camera image, and the entire image is displayed as a three-dimensional point. In V-SLAM, an attempt has been made to determine the self-position / posture from a point cloud generated by directly processing an image from a vehicle-mounted camera.

JP 2014-086991 A

However, the conventional self-position / posture positioning method as described above has a problem that an enormous cost and an enormous amount of data are generated.
That is, if a 3D map is generated by the laser method as proposed in Patent Document 1, that is, the LIDAR method, it is very expensive, and the 3D map must be updated according to environmental changes. Because it does not become, it will be very expensive each time.
The biggest drawback of the LIDAR method is that it takes enormous effort and cost to create a three-dimensional point cloud and manage the data. Furthermore, renewal required similar efforts and costs, and was not practical.
Thus, in the conventional self-position / posture positioning method, a large amount of three-dimensional point data is managed, and the amount of data to be handled is enormous, which is not practical.

Furthermore, the conventional self-position / posture positioning technique has a problem in terms of accuracy.
That is, even if a three-dimensional map can be generated by the method proposed in Patent Document 1, the self-position must be obtained by automatic calculation with an expensive device or the like attached to the autonomous driving vehicle with reference to it. I must. Such a self-position calculation technique is still in the search stage, and the accuracy and stability are lacking for a high-priced apparatus for calculation.

In this way, with the conventionally proposed technology for self-position orientation, a huge amount of money is required for creating a three-dimensional map, and a device for obtaining the self-position is also expensive. There was also a problem that it would be a huge amount.
For this reason, in order to spread automatic driving, it was necessary to realize accurate self-position / posture determination more easily and inexpensively.
However, until now, there has been no technique or proposal that can effectively solve the problems of such a conventional self-position orientation determination.

  The inventor of the present application, as an invention that can solve the problem of such a self-position posture locating technique, has obtained a CV value indicating a camera position and a posture angle of a reference image based on a reference image taken in advance as a result of earnest research. A CV video map in which the six variables (position and orientation) are determined with high accuracy is generated, and the three-dimensional coordinates of the CV video map serving as the reference are generated in a target image taken from a vehicle or the like that is the target of self-position orientation. By transplanting / transferring, it was conceived that the three-dimensional position coordinates of a vehicle or the like can be obtained easily and at low cost with high speed and high accuracy.

In addition, by comparing the target image with the CV video map that is the reference video map, the image is used only if the high-accuracy CV values (six variables) necessary for the self-position / posture determination can be acquired in units of frames of the target image. In this case, the CV value cannot be obtained at a sample density equal to or higher than the frame unit of the image. This is an inevitable principle problem when using images.
Therefore, in order to solve this problem, the inventor of the present application is able to acquire temporally continuous CV values by interpolating between image frames using six variables that can be acquired by a mechanical sensor. did. Moreover, the present inventors have found that the mechanical sensor itself can be an inexpensive low-accuracy mechanical sensor.

  That is, the present invention has been proposed in order to solve the problems of the conventional techniques as described above, and is a vehicle that moves in automatic driving of various vehicles, aircraft, ships, etc., automatic driving of robots, etc. Self-position and orientation for locating its own position and orientation in real time can be obtained simply and at low cost, with high speed and high accuracy, using a CV video map as a reference video map. The purpose is to provide a position and orientation locator.

  In order to achieve the above object, a self-position / posture locating apparatus using a CV video map of the present invention as a reference video map is based on a reference video taken by a predetermined video acquisition means, and the camera position and orientation of the reference video CV video map creation means for generating a CV video map in which the CV value is added to the reference video by performing CV calculation for obtaining a CV (camera vector) value indicating a three-dimensional coordinate value of the CV video map, and storing the CV video map A CV video map database and a CV video map stored in the CV video map database are used as reference images, and a target image taken by a predetermined image acquisition unit provided in a target moving body is compared with the CV video map. The self-position / posture determination for acquiring the CV value of the target image by automatically associating the target image with a plurality of feature points indicating the same portion of the CV video map It is constituted comprising a stage, a.

According to the self-position / posture locating apparatus using the reference image map of the present invention, the CV calculation for obtaining the CV (camera vector) value indicating the three-dimensional coordinate values of the camera position and the posture is performed on the reference image prepared in advance. A CV video map in which the CV value is added to a reference video is generated, and based on the CV video map, three-dimensional coordinates are added to and transplanted to a target target image, thereby self-position / posture determination of the target image Can be performed at high speed and with high accuracy.
As a result, it is possible to obtain a self-position / posture determination that is necessary for automatic driving, automatic traveling, etc. of a vehicle, etc., in real time to determine the position and posture of the moving body in a simple, low-cost, high-speed and high-accuracy manner. It becomes possible.

1 is a block diagram showing a basic configuration of an embodiment of CV calculation means (CV video map creation device) for performing CV calculation of a reference video in a self-position / posture locating device using a reference video map of the present invention. FIG. It is the schematic which shows the means which image | photographs the all-around video image | video used by the CV calculating means shown in FIG. 1, and is a perspective view of the vehicle which mounts the all-around camera on the roof part. It is the schematic which shows the means which image | photographs the all-around video image | video used by the CV calculating means shown in FIG. 1, (a) is a front view of the vehicle which mounts an all-around camera in a roof part, (b) is a top view similarly. It is. It is explanatory drawing which shows the conversion image obtained from the image | video image | photographed with a omnidirectional camera, (a) is a virtual spherical surface to which a spherical image is affixed, (b) is an example of a spherical image affixed to a virtual spherical surface , (C) shows an image obtained by developing the spherical image shown in (b) on a plane according to the Mercator projection. It is explanatory drawing which shows the detection method of the specific camera vector in the CV calculating means based on one Embodiment of this invention. It is explanatory drawing which shows the specific detection method of a camera vector in the CV calculating means which concerns on one Embodiment of this invention. It is explanatory drawing which shows the specific detection method of a camera vector in the CV calculating means which concerns on one Embodiment of this invention. It is explanatory drawing which shows the designation | designated aspect of the desirable feature point in the detection method of the camera vector by the CV calculating means which concerns on one Embodiment of this invention. It is a graph which shows the example of the three-dimensional coordinate of the feature point obtained by the CV calculating means which concerns on one Embodiment of this invention, and a camera vector. It is a graph which shows the example of the three-dimensional coordinate of the feature point obtained by the CV calculating means which concerns on one Embodiment of this invention, and a camera vector. It is a graph which shows the example of the three-dimensional coordinate and camera vector of the feature point obtained by the CV data calculating means which concerns on one Embodiment of this invention. In the CV calculation means according to an embodiment of the present invention, a case is shown in which a plurality of feature points are set according to the distance of the feature points from the camera, tracked over adjacent frames, and a plurality of calculations are repeated. It is explanatory drawing. It is a figure at the time of displaying the locus | trajectory of the camera vector calculated | required by the CV data calculating means which concerns on one Embodiment of this invention in a video image | video. It is a block diagram which shows the basic composition of the self-position attitude | position determination apparatus using the reference | standard video map which concerns on one Embodiment of this invention. It is a block diagram which shows the detail of the transfer process operation | movement of CV value in the self-position attitude | position determination apparatus using the reference | standard video map shown in FIG. FIG. 16 is an explanatory diagram schematically showing a specific example of a CV value transfer processing operation in the self-position posture locating device using the reference video map shown in FIG. 15. Description schematically showing processing when the CV value of the target image obtained by the self-position / posture locating apparatus using the reference image map according to the embodiment of the present invention is improved with six variables obtained by the machine sensor. (A) is the whole of a plurality of frames constituting the target image, and (b) is an enlarged view of a part of the plurality of frames shown in (a). It is a functional block diagram which shows the system configuration | structure of the automatic driving | running | working system of the mobile body by the self-position posture orientation apparatus using the reference | standard video map which concerns on one Embodiment of this invention.

Hereinafter, a preferred embodiment of a self-position / posture locating apparatus using a reference image map according to the present invention will be described with reference to the drawings.
Here, the self-position / posture locating apparatus using the reference video map of the present invention described below is realized by processing, means, and functions executed by a computer in accordance with instructions of a program (software). The program sends commands to each component of the computer, and the following predetermined processing and functions, such as automatic extraction of feature points (reference points) and other feature points that serve as the reference in the video, and extracted criteria Automatic tracking of points, calculation of 3D coordinates of reference points, calculation of CV (camera vector) values, detection of reference points corresponding to reference images and target images, transplantation / integration of CV values between reference images and target images, machine Insertion between frames by 6 variables obtained by the sensor is performed. Thus, each process and means in the present invention are realized by specific means in which the program and the computer cooperate.

Note that all or part of the program is provided by, for example, a magnetic disk, optical disk, semiconductor memory, or any other computer-readable recording medium, and the program read from the recording medium is installed in the computer and executed. The
The program can also be loaded and executed directly on a computer through a communication line without using a recording medium.

[CV video map]
The self-position / posture locating device using the reference image map according to one embodiment of the present invention shown below is, for example, in automatic driving of various vehicles such as automobiles, moving bodies such as aircraft and ships, automatic driving of robots, etc. This is a means for a moving body such as a moving vehicle to determine its position and posture in real time.
Specifically, in the self-position / posture locating apparatus according to the present embodiment, a CV (camera vector) video map serving as a three-dimensional map is used to realize self-position / posture orientation.

In general, the current state of automatic traveling of a moving body, for example, automatic traveling of a vehicle and automatic traveling of a robot, is classified broadly and does not require a highly accurate three-dimensional (3D) map, and determines the surrounding environment by itself. There are two types of systems: an autonomous traveling system that travels in a moving manner, and a 3D map guiding system that requires a highly accurate 3D map for guiding a moving object. The present invention employs a method of using the latter three-dimensional map as a guide.
And in the present invention, in the automatic driving of various vehicles and aircraft, etc., in the automatic driving of the robot, etc., the moving vehicle, etc. refers to the image and video captured by itself and the already prepared three-dimensional map, The reference 3D map, the image (target image) taken from the camera attached to the moving object (automatically traveling vehicle, etc.), and 6 variables (3D position coordinates and rotation coordinates) acquired by the machine sensor A total of 6 variables) is acquired, and these are automatically compared and corrected to realize self-position / posture orientation in which the position and orientation of the subject are located in real time.

First, a CV image that is a three-dimensional map serving as a reference for self-position orientation determination will be described.
Prior to the target travel of the moving body, a shooting camera is installed in a vehicle for creating a reference CV video map, and a moving image or continuous still image is acquired, and feature points are extracted from the image. For example, the camera positions and orientations of all frames are calculated by mathematical calculation.
Specifically, the camera position and orientation are six variables, specifically, a vector of six degrees of freedom of camera position coordinates (X, Y, Z) and rotation angles (Φx, Φy, Φz) of the respective coordinate axes. A CV video can be generated by representing (camera vector: CV) one-to-one with each frame of the video (see FIGS. 1 to 13 described later).
What uses this CV video as a reference is a CV video map for automatic driving guidance.

Here, the six variables indicating the position and orientation of the target camera are six types of variables including coordinates [X, Y, Z] and orientation [Φx, Φy, Φz].
The above-described V-SLAM is also a technology born from a laser point cloud, so a three-dimensional point cloud is created and held as data. This seems to be similar to the present invention in that it uses an image, but there is an important difference between having or not having a huge 3D point cloud in the entire area of the image. This is very different from the present invention which does not have.
The present invention does not have 3D point cloud data directly, but aggregates all 3D information into the camera position and orientation, making the data extremely light and laborious, so that any point can be obtained at any time. 3D coordinates can be acquired. By doing so, the amount of data can be extremely reduced and the calculation process can be made more efficient.

That is, the self-position / posture locating apparatus according to the present embodiment acquires six variables indicating the position and posture of the target moving body. The acquisition of the six variables is to determine six variables [X, Y, Z] indicating the three-dimensional position coordinates and [Φx, Φy, Φz] indicating the posture as described above.
Specifically, first, CV values are acquired based on low-accuracy data that can be acquired from inexpensive mechanical sensors such as IMU / GYRO / GNSS (GPS) attached to a moving object for the purpose of automatic driving. I can keep it. Or it is also possible to acquire only the target variable of the CV value 6 variables according to the purpose.
Here, the reason why the mechanical sensor is made inexpensive and low-accuracy is that it is expensive and high-accuracy is not practical as in the above-described prior art.
CV values acquired from IMU / GYRO / GNSS (GPS), etc. are characterized by low accuracy but can be output continuously in time. In this respect, compared with CV values acquired in units of image frames It is an excellent advantage.

Then, the self-position / posture locating apparatus of the present embodiment three-dimensionally displays a video or continuous image (target image) from a camera attached to a moving object for the purpose of automatic traveling and a CV video map already prepared. It is a device for self-position / posture determination by comparing the two as a map, and this is called a CV video reference type self-position / posture determination device.
CV video maps can be given absolute coordinates, usually with high precision GNSS (GPS). The GNSS here is another GNSS different from the above-described low-accuracy GNSS, and an expensive and high-accuracy GNSS is assumed.
Furthermore, the CV value acquired from the IMU / GYRO / GPS is the CV value acquired in frame units (or an integer multiple thereof) (CV obtained by comparing the already prepared CV video map with the target image) The value) can be interpolated, for example, by proportionally allocating the CV values from the mechanical sensor to match the CV values at both ends of the discontinuity period, in order to fill the discontinuous portions in time. This is because the CV value obtained by the mechanical sensor is highly accurate in an extremely short time (about several seconds), but has a property that it is not practical because errors are accumulated in a long time.

In addition, such a self-position posture locating device according to this embodiment is based on a technology that three-dimensionally associates the camera position and posture of each of the new and old images by comparing the old image and the new image near the same location. Therefore, it can also be used as an update device for old and new images.
That is, by using the self-position / posture locating device according to the present embodiment, automatic traveling while updating an image is possible by moving / moving a moving body such as a vehicle.
Furthermore, an autonomous driving vehicle or the like may be run for image update.

[Characteristics of CV video map]
First, features of the CV video map according to the present invention will be described.
As will be described later (see FIGS. 1 to 13), any point in the CV video map is in a state where three-dimensional coordinates can be acquired at any time by performing corresponding point processing between adjacent frames. This method requires a single effort (CV calculation process) to three-dimensionalize an arbitrary point in the image. However, since the CV calculation process can be processed in milliseconds, the CV video is practically used. Means the same as having all 3D coordinates in the video. Therefore, the CV image can be used as a three-dimensional map serving as a reference for the self-position / posture locator.

Further, many feature points (several tens to hundreds of feature points) used in the CV calculation have three-dimensional coordinates in the middle of the calculation. When saved as data, these hundreds of feature points are used. And the 3D coordinates are discarded once. Even if discarded, if there are six variables of camera position and orientation, any point in the video can be three-dimensionalized at any time.
By the process of discarding the coordinate data of the three-dimensional feature points, the amount of data to be saved can be extremely reduced. In this respect, the CV image is extremely advantageous, which is greatly different from the conventional method (see Patent Document 1) in which the three-dimensional point group of the entire environment of the moving range of the moving object needs to be moved and processed in a state where all of them are stored. This is an excellent feature.

Further, for example, even if a part of the three-dimensional feature points (about several tens to several hundreds) obtained during the CV calculation is left without being discarded for some purpose later, Does not significantly increase the amount of data.
As described above, by using the CV video according to the present invention, it is possible for the first time to make a three-dimensional map of automatic driving a practical data amount.
Moreover, in such a CV video, an arbitrary point of an image in each frame does not yet have a three-dimensional coordinate as data, but an arbitrary point can be calculated by performing a few millisecond operations. There is an excellent feature that 3D coordinates can be acquired immediately and can be used as a 3D map.

As described above, by using the CV image according to the present invention as a three-dimensional map, not all the points of the target image obtained from the camera installed in the vehicle or the like that automatically travels, but only a few necessary features in the target image. By extracting only the points and comparing the two images, it is possible to determine the position and orientation of the target image.
As a result, unlike the prior art, the automatic movement of a moving object can be performed easily and quickly without causing a problem that the amount of data for creating a three-dimensional map and self-position / posture based on it is huge. It becomes possible to realize self-position orientation that is indispensable for driving.
Furthermore, in the present invention, since the image obtained from the camera mounted / installed on the mobile object becomes the target image as it is, the cost can be greatly reduced / reduced as compared with the prior art, and the cost can be reduced. It has an excellent feature that high-accuracy self-position orientation determination is possible.

[Definition of terms]
Next, definitions of terms and phrases used in the present specification and claims will be described.
・ CV video map:
A three-dimensional map serving as a reference standard in the self-position / posture locating device is a CV video generated by obtaining the camera position and posture with six variables from a video acquired in advance from a camera video or continuous images. It is used as an original map. This is called a CV video map.
Therefore, the CV video map can include a CV video, a 3D CG generated from the CV video, or generated by other methods, and various types of images generated by image processing from the CV video. It is possible to include a combination of dots, figures, etc. As a special case, even if only various three-dimensional shapes generated from a CV video are used, if they are originally CV video, they can be referred to as a CV video map. In particular, this can be referred to as a CV3D map.
That is, a CV video having the purpose of a 3D map can be called a CV video map.

Here, the CV video map according to the present invention does not have the three-dimensional coordinates of the entire space as a point group unlike the SLAM and V-Slam described above. Whether it is SLAM or V-Slam, it has a target 3D map and a point cloud (millions to hundreds of millions / km) that matches the accuracy required to perform the target self-orientation become.
On the other hand, the CV video according to the present invention basically has no point cloud, and can obtain the three-dimensional coordinates of a target point from an adjacent image by automatic calculation on the spot when necessary. It is. The three-dimensional coordinates of the entire space can be obtained by calculation from the camera position (six variables).

That is, the CV video map is basically based on having two variables of the two-dimensional video and the camera position without having the three-dimensional coordinates of the entire environment like the conventional SLAM and V-Slam. Necessary three-dimensional point coordinates can be obtained by calculation on the spot. The calculation speed per point is several milliseconds or less.
In general, since the coordinates of three-dimensional points are sufficient for self-position and orientation determination, four to ten points are sufficient. Therefore, by using the CV calculation according to the present invention, the necessary three-dimensional coordinates can be obtained at a sufficient speed. Coordinate values can be obtained. Since such a CV calculation method does not have a three-dimensional point group, the data is light and easy to handle, and data transmission can be realized in an extremely narrow band as compared with SLAM and V-Slam. become.

Furthermore, although the self-position / posture orientation of the target vehicle or the like obtained based on the CV video map is six variables, on the other hand, the six variables can also be acquired by IMU / GYRO / GNSS (GPS) of the mechanical sensor.
Therefore, even if the image (target image) obtained with the camera mounted on the vehicle is obtained by combining IMU / GYRO / GNSS (GPS), which is a mechanical sensor, this is a CV video map / CV3D map. can do.
The mechanical sensor (IMU / GYRO / GNSS) is sufficiently effective as a correction function for the target camera even if it is a low-priced IMU / GYRO / GNSS if it is extremely small time such as computation delay or time discontinuous period It is. The six variables acquired with the mechanical sensor are essentially equivalent to the six variables indicated by the CV value.
Therefore, a map incorporating IMU / GYRO / GNSS or the like can also be included in the CV video map.

CV value / CV video / CV image:
Six variables obtained by calculating the position and posture of the camera from six variables obtained by a mechanical sensor mounted on a moving object (for example, a vehicle or a robot) and a continuous still image or moving image obtained from the camera. Is called a camera vector (CV), its value is called a CV value, obtaining a CV value by calculation is called a CV calculation, and an image having such a CV value is called a CV video.
Note that the six variables obtained from the mechanical sensor integrated with the camera can also be the CV value according to the present invention.
A single image that is not continuous may be referred to as a CV image. Alternatively, when focusing on only one frame, the image of the one frame may be referred to as a CV image. That is, the CV image is a special state (single image) of the CV video.

・ CV machine map:
The CV3D map can be automatically read using a computer, not a human being. In this case, it may be called a CV machine map.
Further, such a CV machine map can be processed from a CV3D map so that it can be easily read by a computer, and this may also be called a CV machine map.
Further, the CV machine map stored in the database is referred to as CV machine map DB.
These are all CV video maps according to the present invention.
Note that the terms “CV video map / CV machine map” as described above are basically not clearly distinguished, and in the present specification, terms are used properly depending on the purpose.

・ Target moving body / target camera / target image:
A mobile object to be subjected to self-position and orientation by applying the present invention, that is, a mobile object to be subjected to automatic driving is called a target mobile object. For example, a vehicle or a robot that automatically travels.
A camera loaded on the target moving body is called a target camera, and an image acquired by the target camera is called a target image.

[CV calculation]
Next, details of the CV calculation in the CV video map used in the self-position / posture locating apparatus using the reference video map of the present invention as described above will be described with reference to FIGS.
The CV calculation means obtaining a CV value, and the obtained result is referred to as CV value or CV data. The notation CV is an abbreviation for camera vector, and the camera vector (CV) is a value indicating a three-dimensional position and a three-axis rotation posture of a camera such as a video camera that acquires an image for measurement or the like. .
The CV calculation acquires a moving image (video image), detects a feature point in the image, tracks it in a plurality of adjacent frames, and creates a triangle formed by the camera position and the tracking locus of the feature point in the image. The three-dimensional position of the camera and the three-axis rotational posture of the camera are obtained by generating a large number of images and analyzing the triangle.

In the CV calculation, it is an important characteristic that, in the process of obtaining the CV value, three-dimensional coordinates are simultaneously obtained for feature points (reference points) in the video at the same time.
Further, the CV value obtained by calculation from the moving image is obtained simultaneously with the three-dimensional camera position and the three-dimensional camera posture corresponding to each frame of the moving image. Moreover, in principle, the characteristic that a CV value is obtained corresponding to an image with a single camera is an excellent feature that can be realized only by CV calculation.
For example, in other measurement means (such as GPS and IMU), in order to simultaneously acquire each frame of a moving image and its three-dimensional camera position and three-dimensional camera posture, an image frame and a measurement sampling time are used. Since it has to be highly accurate and completely synchronized, it becomes a huge device, which is practically difficult to realize.

CV data obtained by calculation from a moving image is a relative value when not processed, but if it is a short section, three-dimensional position information and three-axis rotation angle information can be acquired with high accuracy.
In addition, since the CV data is acquired from the image, the acquired data is a relative value, but it can be realized by other methods that can measure the positional relationship with an arbitrary object in the image. With characteristics.
Further, since the CV value corresponding to the image can be obtained, the CV calculation capable of obtaining the camera position and its three-axis rotation posture directly from the image in the in-image measurement or survey is suitable for the in-image measurement or the in-image survey. .
The self-position / posture locating apparatus using the reference video map of the present invention performs coordinate integration processing (CV value transplantation) between the reference video and the target video (comparison video) based on the CV value data obtained by the CV calculation.・ Relocation).

[CV calculation means]
The CV calculation is performed by the CV calculation means 20 functioning as a CV video map creation device 20 (see FIG. 14) of the self-position / posture locating device using the reference video map of the present invention described later.
As shown in FIG. 1, the CV calculation means (CV video map creation device) 20 performs a predetermined CV calculation process on the video video input from the CV video acquisition device 10 composed of an in-vehicle video camera or the like. Specifically, the feature point extraction unit 21, the feature point correspondence processing unit 22, the camera vector calculation unit 23, the error minimization unit 24, the three-dimensional information tracking unit 25, and a high-precision camera vector And an arithmetic unit 26.

First, any video may be used for the CV calculation. However, in the video with a limited angle of view, the video is interrupted when the viewpoint direction is moved. ) Is desirable. Note that a moving image is similar to a continuous still image and can be handled in the same manner as a still image.
In general, a moving image recorded in advance is used as the video, but it is also possible to use a video captured in real time in accordance with the movement of a moving body such as an automobile.

Therefore, in this embodiment, as an image used for the CV calculation, an all-around image (see FIGS. 2 to 4) obtained by capturing the entire 360 ° circumference of a moving body such as a vehicle, or a wide-angle image close to the all-around image. And a CV video acquisition device 10 that generates and displays a composite image of a map and a video by flattening the entire circumference video in the viewpoint direction (see FIG. 1).
Here, the planar development of the all-around video is to express the all-around video as a normal image in perspective. Here, the term “perspective method” refers to the fact that the entire perimeter image itself is displayed in a different method from the perspective method, such as the Mercator projection or the spherical projection method (see FIG. 4). This is because it can be converted and displayed as a normal perspective image by unfolding and displaying.

In order to generate the all-round video in the CV video acquisition device 10, first, as shown in FIGS. 2 and 3, for the purpose of acquiring CV value data using the all-round video camera 11, With the all-around video camera 11 fixed to the moving body 11a, the periphery of the moving body is photographed with the movement of the moving body 11a.
Note that the mobile body 11a may be provided with a position measuring device or the like constituted by, for example, a GPS device alone or an IMU device to which absolute coordinates are acquired for the purpose of acquiring the position coordinates.
The all-round video camera 11 mounted on the moving body 11a may have any configuration as long as it captures and acquires a wide range of images. For example, a wide-angle lens, a camera with a fisheye lens, a moving camera, There are a fixed camera, a camera in which a plurality of cameras are fixed, a camera that can rotate around 360 degrees, and the like. In this embodiment, as shown in FIGS. 2 and 3, a plurality of cameras are integrally fixed to the vehicle, and an all-around video camera 11 that captures a wide range image as the moving body 11a moves is used. .

Then, according to the all-round video camera 11 as described above, as shown in FIG. 3, by being installed on the ceiling of the moving body 11a, images of the entire 360 ° circumference of the camera are simultaneously captured by a plurality of cameras. A wide range image can be acquired as moving image data by moving the mobile body 11a.
Here, the omnidirectional video camera 11 is a video camera that can directly acquire the omnidirectional video of the camera, but can be used as the omnidirectional video if more than half of the entire circumference of the camera can be acquired as video.
Even in the case of a normal camera with a limited angle of view, the accuracy of the CV calculation is reduced, but it can be handled as a part of the entire peripheral video.

Note that the wide-range video imaged by the all-around video camera 11 can be pasted as a single image on a virtual spherical surface that matches the angle of view at the time of imaging.
The spherical image data pasted on the virtual spherical surface is stored and output as spherical image (360 degree image) data pasted on the virtual spherical surface. The virtual spherical surface can be set to an arbitrary spherical shape centered on the camera unit that acquires a wide range image.
4A is an appearance image of a virtual spherical surface to which a spherical image is pasted, and FIG. 4B is an example of a spherical image pasted to the virtual spherical surface. FIG. 4C shows an example of an image obtained by developing the spherical image of FIG.

Then, the all-round video image generated / acquired as described above is input to the CV calculation means (CV image map creation device) 20 to obtain CV value data (see FIG. 1).
In the CV calculation means 20, first, the feature point extraction unit 21 has a sufficient number of feature points (reference points) from the moving image data that has been photographed and temporarily recorded by the all-around video camera 11 of the CV video acquisition device 10. ) Is automatically extracted.
The feature point correspondence processing unit 22 automatically obtains the correspondence relationship by automatically tracking the feature points automatically extracted in each frame image between the frames.
The camera vector calculation unit 23 automatically calculates a camera vector corresponding to each frame image from the three-dimensional position coordinates of the feature points for which the correspondence relationship has been determined.
The error minimizing unit 24 performs statistical processing so that the solution distribution of each camera vector is minimized by overlapping calculation of a plurality of camera positions, and automatically determines the camera position direction subjected to the error minimizing process. .

The three-dimensional information tracking unit 25 positions the camera vector obtained by the camera vector calculation unit 23 as an approximate camera vector, and based on the three-dimensional information sequentially obtained as part of the image in the subsequent process, a plurality of frame images 3D information is automatically tracked along an image of an adjacent frame. Here, three-dimensional information (three-dimensional shape) is mainly three-dimensional distribution information of feature points, that is, a collection of three-dimensional points, and this collection of three-dimensional points constitutes a three-dimensional shape. To do.
The high-precision camera vector calculation unit 26 generates and outputs a higher-precision camera vector than the camera vector obtained by the camera vector calculation unit 23 based on the tracking data obtained by the three-dimensional information tracking unit 25.
Then, the camera vector obtained as described above is input to the self-position / posture locating apparatus 10 using a reference video map, which will be described later, for the coordinate integration processing (transfer / integration of CV values) of the reference video and the target image. Will be used.

There are several methods for detecting a camera vector from feature points of a plurality of images (moving images or continuous still images). In the CV calculation means 20 of the present embodiment shown in FIG. By automatically extracting a number of feature points and automatically tracking them, a three-dimensional vector and a three-axis rotation vector of the camera are obtained by epipolar geometry.
By taking a sufficient number of feature points, camera vector information is duplicated, and an error can be minimized from the duplicated information to obtain a more accurate camera vector.

A camera vector is a vector of degrees of freedom possessed by a camera.
In general, a stationary three-dimensional object has six degrees of freedom of position coordinates (X, Y, Z) and rotation angles (Φx, Φy, Φz) of the respective coordinate axes.
Therefore, the camera vector is a vector of six degrees of freedom (6 variables) of the camera position coordinates (X, Y, Z) and the rotation angles (Φx, Φy, Φz) of the respective coordinate axes. When the camera moves, the direction of movement also enters the degree of freedom, but this can be derived by differentiation from the above six degrees of freedom (variables).
Thus, the detection of the camera vector in the present embodiment means that the camera takes six degrees of freedom values for each frame and determines six different degrees of freedom for each frame.

Hereinafter, a specific camera vector detection method in the CV calculation unit 20 will be described with reference to FIG.
First, the image data acquired by the all-round video camera 11 of the CV video acquisition device 10 described above is input to the feature point extraction unit 21 of the CV calculation unit 20 indirectly or directly, and the feature point extraction unit 21 A point or a small area image that should be a feature point is automatically extracted from a frame image that is appropriately sampled, and a feature point correspondence processing unit 22 automatically obtains a correspondence relationship between feature points among a plurality of frame images. .
Specifically, more than a sufficient number of feature points that are used as a reference for detecting a camera vector are obtained. An example of feature points between images and their corresponding relationships are shown in FIGS. In the figure, “+” is a feature point that is automatically extracted, and the correspondence is automatically tracked between a plurality of frame images (see correspondence points 1 to 4 shown in FIG. 7).
Here, for feature point extraction, as shown in FIG. 8, it is desirable to specify and extract a sufficiently large number of feature points in each image (see circles in FIG. 8). For example, about 100 feature points are extracted. Extract points.

Subsequently, the camera vector calculation unit 23 calculates the three-dimensional coordinates of the extracted feature points, and calculates the camera vector based on the three-dimensional coordinates. Specifically, the camera vector calculation unit 23 includes a sufficient number of feature positions that exist between successive frames, a position vector between moving cameras, a three-axis rotation vector of the camera, and each camera position and feature. Relative values of various three-dimensional vectors such as vectors connecting points are continuously calculated by calculation.
In this embodiment, for example, camera motion (camera position and camera rotation) is calculated by solving an epipolar equation from the epipolar geometry of a 360-degree all-round image.

Images 1 and 2 shown in FIG. 7 are images obtained by Mercator expansion of 360-degree all-round images. When latitude φ and light θ are assumed, points on image 1 are (θ1, φ1) and points on image 2 are ( θ2, φ2). The spatial coordinates of each camera are z1 = (cos φ1 cos θ1, cos φ1 sin θ1, sin φ1), z2 = (cos φ2 cos θ2, cos φ2 sin θ2, sin φ2). When the camera movement vector is t and the camera rotation matrix is R, z1 ^T [t] × Rz2 = 0 is the epipolar equation.
By providing a sufficient number of feature points, t and R can be calculated as a solution by the method of least squares by linear algebra calculation. This calculation is applied to a plurality of corresponding frames.

Here, it is preferable to use a 360-degree all-round image as an image used for the calculation of the camera vector.
The image used for the camera vector calculation may be any image in principle, but a wide-angle image such as a 360-degree all-round image shown in FIG. 7 can easily select many feature points. Therefore, in this embodiment, a 360-degree all-round image is used for the CV calculation, whereby the tracking distance of the feature points can be increased, and a sufficiently large number of feature points can be selected. It becomes possible to select a feature point convenient for each short distance. In addition, when correcting the rotation vector, the calculation process can be easily performed by adding the polar rotation conversion process. As a result, a calculation result with higher accuracy can be obtained.
In FIG. 7, in order to make it easy to understand the processing in the CV calculation means 20, a 360-degree all-round spherical image obtained by synthesizing images taken by one or a plurality of cameras is developed by a Mercator projection called map projection. However, in an actual CV calculation, it is not necessarily a developed image by the Mercator projection.

Next, the error minimizing unit 24 calculates a plurality of vectors based on each feature point according to a plurality of calculation equations based on the plurality of camera positions and the number of feature points corresponding to each frame, Statistical processing is performed so that the distribution of the position of each feature point and the camera position is minimized to obtain a final vector. For example, the optimal solution of the least square method is estimated by the Levenberg-Marquardt method for multiple frame camera positions, camera rotations, and multiple feature points, and errors are converged to determine the camera position, camera rotation matrix, and feature point coordinates. .
Further, feature points having a large error distribution are deleted, and recalculation is performed based on other feature points, thereby improving the accuracy of computation at each feature point and camera position.
In this way, the position of the feature point and the camera vector can be obtained with high accuracy.

9 to 11 show examples of three-dimensional coordinates of feature points and camera vectors obtained by CV calculation. 9 to 11 are explanatory diagrams showing a vector detection method by CV calculation according to this embodiment, and showing a relative positional relationship between a camera and an object obtained from a plurality of frame images acquired by a moving camera. FIG.
FIG. 9 shows the three-dimensional coordinates of the feature points 1 to 4 shown in the images 1 and 2 in FIG. 7 and the camera vector (X, Y, Z) that moves between the images 1 and 2.
10 and 11 show a sufficiently large number of feature points, the positions of the feature points obtained from the frame image, and the position of the moving camera. In the figure, a circle mark that continues in a straight line at the center of the graph is the camera position, and a circle mark located around the circle indicates the position and height of the feature point.

Here, the CV calculation in the CV calculation means 20 is performed in accordance with the distance from the camera to the feature point, as shown in FIG. 12, in order to obtain more accurate three-dimensional information of the feature point and the camera position. Set feature points and repeat multiple operations.
Specifically, the CV calculation means 20 automatically detects feature points that have image characteristics in the image, and obtains corresponding points of the feature points in each frame image. And the n + m-th two frame images Fn and Fn + m are used as unit calculations, and unit calculations with n and m appropriately set can be repeated.
m is the frame interval, and the feature points are classified into a plurality of stages according to the distance from the camera to the feature point in the image. The distance from the camera to the feature point is set so that m becomes larger. M is set to be smaller as the distance to is shorter. This is because the change in position between images is less as the distance from the camera to the feature point is longer.

Then, while sufficiently overlapping the classification of the feature points by the m value, a plurality of stages of m are set, and as n progresses continuously with the progress of the image, the calculation proceeds continuously. Then, the overlap calculation is performed a plurality of times for the same feature point in each step of n and m.
In this way, by performing unit calculation focusing on the frame images Fn and Fn + m, a precise camera vector is calculated over a long time between frames sampled every m frames (frames are dropped). However, in m frames (minimum unit frames) between the frame images Fn and Fn + m, a simple calculation that can be performed in a short time can be performed.

If there is no error in the precision camera vector calculation for every m frames, both ends of the camera vector of the m frames overlap with the Fn and Fn + m camera vectors that have been subjected to the high precision calculation. Accordingly, m minimum unit frames between Fn and Fn + m are obtained by a simple calculation, and both ends of the camera vector of the m minimum unit frames obtained by the simple calculation are Fn and Fn + m obtained by high precision calculation. The scale adjustment of m consecutive camera vectors can be made to match the camera vectors.
In this way, as n progresses continuously with the progress of the image, the scale adjustment is performed so that the error of each camera vector obtained by calculating a plurality of times for the same feature point is minimized, and integration is performed. A camera vector can be determined.
Accordingly, it is possible to speed up the arithmetic processing by combining simple arithmetic operations while obtaining a highly accurate camera vector having no error.

Here, there are various simple calculation methods depending on the accuracy. For example, when (1) many feature points of 100 or more are used in high-precision calculation, the minimum number of simple calculation is about ten. (2) Even if the number of the same feature points is the same as the feature points and camera positions, innumerable triangles are established there, and equations for that number are established. By reducing the number of equations, it can be simplified.
In this way, integration is performed by adjusting the scale so that the error of each feature point and camera position is minimized, distance calculation is performed, and feature points with large error distribution are deleted, and other features are added as necessary. By recalculating the points, the calculation accuracy at each feature point and camera position can be improved.

  In addition, by performing high-speed simple calculation in this way, it is possible to perform near real-time processing of camera vectors. High-speed calculation processing of the camera vector is performed with the minimum number of frames that can achieve the target accuracy and the minimum number of feature points that are automatically extracted, and the approximate value of the camera vector is obtained and displayed by high-speed calculation. As it accumulates, it is possible to increase the number of frames, increase the number of feature points, perform more accurate camera vector calculation, and replace the approximate value with a highly accurate camera vector value for display.

Furthermore, in this embodiment, it is possible to track three-dimensional information (three-dimensional shape) in order to obtain a more accurate camera vector.
Specifically, first, the three-dimensional information tracking unit 25 positions the camera vector obtained through the camera vector calculation unit 23 and the error minimization unit 24 as an approximate camera vector, and the image generated in the subsequent process. Based on the three-dimensional information (three-dimensional shape) obtained as a part, partial three-dimensional information included in a plurality of frame images is continuously tracked between adjacent frames to automatically track the three-dimensional shape.
Then, from the tracking result of the three-dimensional information obtained by the three-dimensional information tracking unit 25, a high-precision camera vector is obtained by the high-precision camera vector calculation unit 26.

In the feature point extraction unit 21 and the feature point correspondence processing unit 22 described above, feature points are automatically tracked in a plurality of inter-frame images, but the number of feature point tracking frames is limited due to disappearance of feature points. May come. In addition, since the image is two-dimensional and the shape changes during tracking, there is a certain limit in tracking accuracy.
Therefore, the camera vector obtained by the feature point tracking is regarded as an approximate value, and the three-dimensional information (three-dimensional shape) obtained in the subsequent process is traced on each frame image, and a high-precision camera vector is obtained from the trajectory. Can do.
The tracking of 3D shapes is easy to obtain matching and correlation accuracy, and the 3D shape does not change in size and size depending on the frame image, so it can be tracked over many frames. The accuracy of the camera vector calculation can be improved. This is possible because the approximate camera vector is already known by the camera vector calculation unit 23 and the three-dimensional shape is already known.

  When the camera vector is an approximate value, the error of 3D coordinates over a very large number of frames is small because there are few frames related to each frame by feature point tracking. The error of the three-dimensional shape when a part of the image is cut is relatively small, and the influence on the change and size of the shape is considerably small. For this reason, the comparison and tracking in the three-dimensional shape is extremely advantageous over the two-dimensional shape tracking. In tracking, when tracking with 2D shape, tracking changes in shape and size in multiple frames are unavoidable, so there are problems such as large errors and missing corresponding points. However, in tracking with a three-dimensional shape, there is very little change in shape, and in principle there is no change in size, so accurate tracking is possible.

Here, as the three-dimensional shape data to be tracked, there are, for example, a three-dimensional distribution shape of feature points, a polygon surface obtained from the three-dimensional distribution shape of feature points, and the like.
It is also possible to convert the obtained three-dimensional shape from a camera position into a two-dimensional image and track it as a two-dimensional image. Since the approximate value of the camera vector is known, projection conversion can be performed on a two-dimensional image from the camera viewpoint, and it is also possible to follow a change in the shape of the object due to movement of the camera viewpoint.

The camera vector obtained as described above can be displayed in an overlapping manner in the video image shot by the all-round video camera 11.
For example, as shown in FIG. 13, the image from the in-vehicle camera is developed in a plane, the corresponding points on the target plane in each frame image are automatically searched, and the corresponding points are combined to match the target plane. A combined image is generated and displayed in the same coordinate system.
Furthermore, the camera position and the camera direction can be detected one after another in the common coordinate system, and the position, direction, and locus can be plotted. The CV data indicates the three-dimensional position and the three-axis rotation, and the CV value can be observed simultaneously in each frame of the video image by displaying it over the video image. An example of an image displayed by superimposing CV data on a video image is shown in FIG.
If the camera position is correctly displayed in the video image, the position in the video image indicated by the CV value is the center of the image, and if the camera movement is close to a straight line, the CV values of all frames are displayed overlapping. Therefore, for example, as shown in FIG. 13, it is appropriate to display a position of 1 meter right below the camera position. Alternatively, it is more appropriate to display the CV value at the height of the road surface based on the distance to the road surface.

[Self-position orientation locator]
Next, the reference video map (CV video map) according to the present invention for performing the coordinate integration processing of the reference video (CV video) and the target image to be compared based on the CV value obtained as described above is used. An embodiment of the self-position / posture locating apparatus will be specifically described with reference to the drawings.
In the self-position / posture locator using the reference video map (CV video map) shown below, when the reference video / target image is expressed, the target image is not necessarily a new video in time and time, and the reference video Does not mean that is an old video.
For example, in the case of a self-position / posture locating device for the purpose of an image updating device, a video with a known CV value becomes a reference video, and the CV value is acquired (transplanted / integrated) based on the reference video. The video that becomes the target image (target video).

FIG. 14 is a block diagram showing a basic configuration of the self-position / posture locating apparatus 1 using a reference video map (CV video map) according to an embodiment of the present invention.
As shown in FIG. 14, the self-position / posture locating apparatus 1 using the reference video map (CV video map) according to the present embodiment uses the above-mentioned CV video map as a reference video, and a moving body such as a moving vehicle itself Is a device / means for locating the position and orientation of the mobile body in real time and realizing automatic operation of the moving body based on the self-position orientation.
Specifically, the self-position orientation determination apparatus 1 using the reference video map according to the present embodiment includes a CV video acquisition device 10, a CV video map creation device 20, and a CV video map database (CV machine map database) 30. A target moving body (automatic driving device) 40, a CV video map / target image comparing device 50, and a self-position / posture locating device 60.

The CV video acquisition device 10 is a means for capturing and acquiring a reference video for generating a reference video map for self-position orientation determination. As shown in FIGS. It is comprised by the moving bodies 11a, such as a traveling vehicle provided.
The moving body 11a travels within a certain range on a predetermined road or the like for the purpose of acquiring a reference video map, and thus the moving body 11a is moved by the all-round video camera 11 provided in the moving body 11a. Shoot and acquire video around a moving object as a reference video.
The reference video acquired by the CV video acquisition device 10 is input to the CV video map creation device 20, and a CV video map creation process based on the CV calculation described above is performed (see FIGS. 1 to 13).

The CV video map creation device 20 performs CV calculation for obtaining a CV (camera vector) value indicating the three-dimensional coordinate values of the camera position and orientation of the reference video based on the reference video taken by a predetermined video acquisition means. A means for generating a CV video map in which a CV value is added to a reference video, and constitutes a CV video map creating means according to claim 1 of the present application.
Specifically, the CV video map creation device 20 is configured by the CV calculation means shown in FIGS. The specific contents of the CV calculation by the CV video map creation device 20 are as described above (see FIGS. 1 to 13).

The CV video map database (CV machine map database) 30 is storage means for storing the CV video map generated by the CV video map creation device 20, and constitutes the CV video map database of claim 1 of the present application.
The CV video map stored in the CV video map database 30 is stored and held as 3D map data serving as a reference video for the self-position / posture locating process, read out by the self-position / posture locator 60, and stored in a predetermined manner. Comparison reference with the target image and coordinate integration are performed.

The target moving body (automated driving device) 40 is a moving object that is configured by a vehicle or the like that is an object of automatic driving and that is the target of the self-position / posture determination of this embodiment.
In the target moving body 40, as a means for capturing and acquiring an image / video that is a target image of self-position / posture determination, as with the CV video acquisition device 10 described above, for example, imaging such as a video camera or an in-vehicle camera is used. Means (target camera) are provided.
In addition, the target moving body 40 is provided with a description sensor such as IMU / GYRO / GNSS (GPS) as described above as means for acquiring position information of the moving body.

The target moving body 40 having a camera, a mechanical sensor, and the like travels along the movement of the moving body by a camera or the like provided in the moving body, for example, by traveling on a predetermined road or the like that is a target range of automatic driving. Thus, the video around the moving body is captured and acquired as a target image, and the three-dimensional position information of the target image is acquired and given by the mechanical sensor.
The target image acquired by the target moving body 40 and the 6-variable data obtained by the machine sensor are input to the self-position / posture locating device 60 via the CV video map / target image comparison device 50, and the reference video described above. The CV value that is a three-dimensional coordinate is transplanted, integrated, and corrected with respect to the target image by comparison and reference with the CV video map.

  The CV video map / target image comparison device 50 inputs a target image taken by a predetermined image acquisition means provided in the target moving body 40 and six variable data acquired by a mechanical sensor corresponding to the target image. Receive. Then, a CV video map serving as a reference image to be compared / referenced with the target image is read from the CV video map database 30, and the 6-variable data / CV video map obtained by the target image / machine sensor is self-position / posture locating device 60. Output to.

Here, as the CV video map serving as the reference video to be compared with the target image, for example, the CV video map corresponding to the target image is read based on the approximate position information attached to the CV video map. In this case, it is assumed that approximate position information is given to the target image by, for example, GPS.
As a result, the self-position / posture locating device 60 executes the self-position / posture locating process based on the respective data of the 6-variable / CV video map obtained by the target image / machine sensor.

The self-position / posture locating device 60 uses the CV video map stored in the CV video map database 30 as a reference image, and compares the target image taken by a predetermined image acquisition means provided in the target moving body 40 with the CV video map. The means for acquiring the CV value of the target image by automatically associating the target image with a plurality of feature points indicating the same portion of the CV video map. Means.
First, in the self-position / posture locating device 60, the target image obtained by the imaging means (target camera) of the target moving body 40 and the target movement obtained by the mechanical sensor are transmitted via the CV video map / target image comparing device 50. Six variables serving as position information of the body 40 are input.

The self-position / posture locator 60 reads a rough range to be compared / referenced with the target image from the CV video map stored in the CV video map database 30 as a reference image, and compares the CV video map / target image. It is read and input via the device 50.
The self-position / posture locating device 60 automatically selects a plurality of feature points indicating the same portion of the target image and the CV video map based on the input target image, the six variable data obtained by the machine sensor, and the CV video map. Thus, the coordinates are integrated by transplanting the CV value added to the CV video map to the feature point of the corresponding target image.
As a result, a CV value is given to the target image, and the target image is generated and held as a CV video (target CV value) having a highly accurate CV value.

Specifically, the CV value transplantation / integration processing between the CV video map serving as the reference video and the target image is performed as follows, for example.
First, the correspondence between common points in the target image and the reference video is initially set. Over each frame of the moving image starting from the initially set frame, a portion corresponding to a predetermined 3D reference point, 3D feature point, or 2D feature point in the reference image is included in the target image. Automatically corresponding to each other and searching for corresponding feature points or corresponding reference points over each frame.
Alternatively, a three-dimensional reference point, a three-dimensional feature point, or a two-dimensional feature point in the target image is searched for in the reference image and tracked over each frame of the reference image.
Further, in the case where the target image is a plurality of images or video images over a plurality of frames, the corresponding reference points associated by the search process are tracked over each frame in which the target image proceeds. Similarly, when the reference image is a plurality of images or moving images over a plurality of frames, the corresponding reference point associated by the search process is traced over each frame in which the reference image proceeds.
Then, the three-dimensional coordinates of the three-dimensional reference point of the reference image are transplanted to the corresponding reference point of the target image based on the correspondence result of the corresponding reference point (same as the corresponding feature point).

Furthermore, the CV value of the target image can be obtained from the three-dimensional coordinates of the transplanted corresponding reference point by the above-described CV calculation. When the target image extends over a plurality of frames, the CV value can be obtained over each frame by tracking.
In other words, the correspondence between the reference image and the target image means that the three-dimensional coordinates of the reference point of the reference image are transplanted to the target image. If there are four or more points, the camera position of the target image The posture can be obtained by CV calculation, and the CV value of the target image can be acquired.
In addition, the reference image to which the CV value is added in this way and the target image for which the CV value is obtained are coordinate-integrated in the same coordinate system.

As described above, the self-position / posture locating device 60 compares the CV video map with the target image, but conversely, with the known three-dimensional points automatically acquired from the target image as a reference, Corresponding points can be obtained to obtain the CV value of the target image.
This corresponds to a case where three-dimensional coordinates are acquired only on the target image side by automatic recognition. That is, instead of measuring the 3D coordinates of the target image on the spot, the 3D shape and 3D coordinates of the target object are announced, for example, as in a real world 3D marker (see FIG. 15) described later. In such a case, by acquiring the target object, it can be used when the three-dimensional coordinates of the published object can be acquired from the outside.

In addition, even when such a target object moves with the passage of time or the like, it is possible to acquire three-dimensional coordinates from the target image side.
There are few objects that are completely fixed on the earth at all times. For example, the object may move even in an earthquake, and a sign may be tilted by wind and snow. Therefore, it is necessary to always obtain the latest information, and for that purpose, it is useful to obtain the three-dimensional coordinates from the target image side.
In the self-localization device 60 according to the present embodiment, the combination of the CV video map and the target image is used to convert the three-dimensional feature points from the CV video map to the target image, or simultaneously from the target image to the CV video map. By performing the original coordinate transfer, the CV value of the target image can be acquired.

Furthermore, as described above, the CV calculation according to the present invention is obtained by calculating a moving image or a continuous still image, but it is also possible to perform CV calculation including both a CV video map and a target image. is there.
In this way, the calculation is performed with only the known CV value fixed, and the unknown CV value on the target image side is obtained, so that the CV value can be obtained with higher accuracy than the simple calculation using the corresponding point processing. Can be obtained.

  Specifically, in the self-localization device 60, in the combination of the CV video map and the target image, both of the two-dimensional feature points (2D) and the three-dimensional feature points (3D) are mixed. The calculation is performed, and at that time, the CV calculation can be performed by using all the feature points of the feature point (3D) with the known three-dimensional coordinates while fixing the three-dimensional coordinates. As a result, all feature points in which the CV video map and the target image are mixed simultaneously have three-dimensional coordinates, and then the CV value acquired by the target image is treated as a new CV value, so that the mixed CV value is obtained. The CV value of the target image can be separated from the target image, and the self-position / posture can be determined.

Details of the self-position / posture locating process in the self-position / posture locating apparatus 60 as described above will be described later with reference to FIGS.
Then, the self-position / posture locating device 60 as described above selects seven types of feature amounts described later as predetermined feature amounts included in the CV video map as feature points indicating the same portion of the target image and the CV video map. (See FIGS. 15 and 16).
The seven types of feature amounts will be described later with reference to FIGS.

Further, the self-position / posture locating device 60 uses the 6-variable data indicating the self-position and posture of the target moving body 40 acquired by the mechanical sensor provided in the target moving body 40 based on the CV value of the target image. By correcting, six variables indicating the self-position and posture of the target moving body 40 that are temporally continuous are acquired.
As a result, the CV value of the target image can be corrected and supplemented by the six variable data obtained by the machine sensor (see FIG. 17).
The CV value correction / complement processing by the mechanical sensor will be described later with reference to FIG.

Then, by performing the self-position / posture determination based on the CV video map as described above, automatic driving control of a moving body such as a vehicle can be performed based on the generated / output self-position / posture determination result.
For example, as indicated by a broken line in FIG. 14, the target moving body 40 such as a vehicle to be automatically driven is controlled and driven by automatic driving means.
Specifically, the target moving body 40 includes an output signal of the vehicle surrounding state determination device 70 configured by various sensors and the like, and a vehicle control signal generation device 80 that controls the operation of the vehicle such as running / stopping / rotation based on the output signal. The automatic operation is performed by the output signal.

High-accuracy three-dimensional position information based on the CV value is obtained by inputting the self-position / posture orientation information generated by the above-described self-position / posture orientation device 60 to the automatic driving means of the target moving body 40. Thus, accurate position information of the target moving body 40 can be obtained at high speed and at low cost.
Details of the automatic driving control of the vehicle or the like using the self-position / posture locating device 60 of the present invention as described above will be described later with reference to FIG.

[Self-position orientation determination processing]
Next, details of the self-position / posture locating process in the self-position / posture locator 60 described above will be described with reference to FIGS.
Here, the self-position / posture determination is to determine a CV value of a target moving body (vehicle, robot, aircraft, moving general object, etc.). By determining the CV value, the position and orientation of the moving object (target moving body) in the CV video can be uniquely determined.
The obtained CV value is originally a relative value of a relative coordinate, but as a CV video map, by giving a real scale (absolute coordinate) thereto, the CV value has a real scale and is converted into an absolute coordinate. It will be.

This technique is indispensable for automatic driving and robot traveling. For this purpose, the position and orientation (six variables) of the target moving body must be obtained by real-time processing.
However, if the purpose is to update the CV video map, real-time processing is not necessary, and a part or all of the previous CV video map is updated by post-processing to determine the new position of the acquired self-position / posture. be able to. Therefore, the self-position / posture locating device of the present invention can also be used as a CV video map updating device.

That is, by performing the self-position / posture determination according to the present embodiment, the purpose of the self-position / posture determination is achieved even if the target image for which the CV value has been acquired is subsequently discarded. However, the data of the target image from which the CV value is acquired in this way can be used as a part of a new CV video map without being discarded.
For this purpose, the target image from which the CV value is acquired can be taken into the CV video map database and stored as overlapping data in the CV video map, or can be updated by replacing part or all of the CV video map.
In this way, the target image and the six variables obtained from the target image are stored together with the image without being discarded, so that it can be used not only as a self-position / posture locating device but also as a CV video map updating device. it can. Furthermore, it is also possible to operate an automatic driving vehicle or an automatic traveling robot for automatic updating only for the purpose of updating the CV video map.

As described above, according to the CV calculation technique which is the premise of the present invention, a moving image is acquired from a camera installed on a moving object, a feature point is extracted, and it is tracked between frames and used. Thus, the CV value can be obtained by calculation (see FIGS. 1 to 13).
In the present invention, a CV video map whose CV value is already known is used as a three-dimensional map, and the CV value of the video or image of the target moving body is not directly calculated, but before the target moving body moves, Prepare a nearby CV video map including the existing point in advance, and determine the correspondence from the relationship between the camera video or camera image loaded on the target mobile object and the CV video map already prepared. By obtaining, the CV value of the target moving body is obtained.
The CV value thus obtained may be referred to as a target CV value.

In the present embodiment, the device for obtaining the self position and the posture (six variables) is characterized by the combination of the CV video map and the target image.
In particular, in order to obtain the relationship between the CV video map and the target mobile object with respect to the types of image feature points and the like necessary for self-position orientation, at least one of the following seven feature amounts Is used.
Hereinafter, this will be referred to as “seven types of feature amounts”.

[7 types of features]
Hereinafter, the seven types of feature amounts in the present embodiment will be described in detail with reference to FIGS.
FIG. 15 is a block diagram showing details of the CV value transfer processing operation in the self-position / posture locating apparatus 60 shown in FIG.
FIG. 16 is an explanatory diagram schematically showing a specific example of the CV value transfer processing operation shown in FIG.
Note that the feature quantities (feature points) shown below are not necessarily points with no area, but are actually small surfaces with very small areas, surfaces with features, and characteristic attributes. It may be an area you have.
For this reason, in this specification, the feature points including the feature points will be referred to as “feature quantities”.

[1. Feature point (2D)]
As described in the above-described CV calculation processing (see FIGS. 1 to 13), the feature points in the CV video serving as the reference video can be automatically extracted by the image processing technique. A feature point in the CV video is a feature point (2D) 60a (see FIGS. 15 and 16A).
The feature points in this video can be defined as points, but in reality, they are often small area images that can be regarded as points in terms of coordinates. The feature point is not limited to two-dimensional or three-dimensional, and refers to a feature point in a CV video map, or a feature point indicating the same point in an image acquired by a camera installed on a target mobile object. is there.
In particular, the feature point (2D) is defined as a two-dimensional quantity in a video or an image.
Also, the feature point (2D) is tracked across adjacent frames of the video, and becomes a three-dimensional feature point (3D) by performing CV calculation.

[2. Feature points (3D)]
Since the above-described feature points in the CV video are three-dimensionalized in the process of generating the CV video, they can be handled as three-dimensional feature points. This three-dimensional feature point is a feature point (3D) 60b (see FIGS. 15 and 16B).
As shown in FIG. 16B, the three-dimensional feature points are obtained by transferring the three-dimensional coordinates to the feature points indicating the same point in the image (target image) acquired by the camera installed on the corresponding target moving body. Can do. As a result, three-dimensional coordinates are given to a part of the image acquired in the corresponding target image.
If the three-dimensional coordinates can be transferred to any four or more points in the image by the same method, the camera position and orientation of the image can be obtained.

[3. Designated feature point (2D)]
The above-described feature point (2D) 60a is a feature point automatically generated by an image processing technique. When the person designates the feature point in the CV video or the target image, the above feature point (2D) 60a. It can be handled exactly the same as 60a.
This is the designated feature point (2D) 60c (see FIGS. 15 and 16A).

[4. Designated feature point (3D)]
Similarly, when a human specifies a feature point in an image, acquires its three-dimensional coordinates in a CV image, and obtains the three-dimensional coordinates, it can be handled in the same manner as the above-described feature point (3D). .
This is the designated feature point (3D) 60d (see FIGS. 15 and 16B).

[5. Recognized objects such as signs (3D)]
A sign or the like can be handled as the two types of designated feature points 60c and 60d described above.
Specifically, an object with three-dimensional coordinates such as a sign, a signboard, or a feature acquired by the recognition technique is particularly designated as a recognition object (3D) 60e such as a sign (see FIG. 15).
Since the recognition object (3D) such as a sign already has a known three-dimensional coordinate, if the area is large, the target CV value can be transplanted with a single object. It becomes possible.

[6. Real world 3D marker]
For example, a real-world marker is installed on the road or its surroundings to acquire the three-dimensional coordinates for the purpose of self-position / posture, not the target (3D) that existed from the beginning, such as the above sign. Alternatively, as known, the corresponding point is obtained in the target image, and the three-dimensional coordinates are transplanted, and the CV value can be obtained therefrom by calculation.
This is the real world three-dimensional marker 60f (see FIGS. 15 and 16D).
Here, the real-world three-dimensional marker is different from the following three-dimensional marker (3D marker) in the image, and for the purpose of self-localization, for example, for automatic driving of a vehicle or automatic driving of a robot. As a 3D marker actively installed in the real world as well as in the CV map.

[7. 3D marker in image]
The real-world three-dimensional marker 60f is a case where a marker is placed in the real world, but more easily, a characteristic place is selected in the CV video map, and this is used as the in-image three-dimensional marker 60g. (See FIGS. 15 and 16 (c)).
Based on the three-dimensional marker 60g in the image, the CV value can be acquired by searching for the corresponding portion in the target image.

Here, the in-image three-dimensional marker (3D marker) is extracted as a mark in advance in the CV video map for the purpose of locating the own position and orientation in order to efficiently operate the automatic position and orientation locator. This is a 3D marker having 3D coordinates installed in a CV video map.
Specifically, the 3D marker can use an object that is not originally installed for automatic traveling. For example, it can be used as a 3D marker by acquiring shapes and coordinates in advance, such as road markings, corners of buildings, and windows.
In this respect, it can be distinguished from the above-described three-dimensional marker actually installed in the real world.

As described above, in the present embodiment, the CV value of the target image can be obtained by calculation by combining one or several of the seven types of feature amounts (feature points) as described above.
At that time, at least one kind of feature quantity to be selected is automatically extracted, and the feature quantity to be selected has already been automatically acquired part or all of the position and the three-dimensional coordinates of the three-dimensional shape. It shall be.
For example, in the case of automatic driving of a vehicle, such an object that is automatically extracted includes a sign having a three-dimensional coordinate, a road marking, a part of a building, a power pole, a curb, and the like existing around a road. It is. In the case of automatic running of the robot, the corner of the wall within the active range, the shape of the room that does not move, the fixed object, the characteristic pattern of the floor, etc. In such a recognition object, the three-dimensional coordinates have already been obtained by automatic calculation.

[Acquire CV value by transferring 3D feature points to the target image]
The self-position orientation in the present embodiment is obtained by adding a three-dimensional feature point from a reference CV video to a target target image by using a part or a combination of the seven types of feature amounts (feature points) as described above. By transferring, the CV value of the target image is acquired.
That is, the feature amounts of both the reference video and the target image are coordinate-integrated, and the CV value of the target image is acquired, so that the self-position / posture is determined.
This is the CV transfer 61 by the corresponding point calculation shown in FIG.

As described above, the CV value of the target image can be obtained by transferring and transplanting the three-dimensional coordinates to the feature points in the target image from the CV video map in which the CV value prepared in advance is known. It is CV value acquisition. However, if there are known three-dimensional points or surfaces in the target image, using them will contribute to lowering the calculation cost for acquiring the CV value of the target image.
The feature point transfer / transplant processing operation for acquiring the CV value of the target image will be specifically described below.

[CV transfer by corresponding point calculation]
As described above, obtaining the CV value (six variables) of the target image to be processed is the essence of the self-position / posture determination according to the present invention.
The method of extracting and tracking the feature point (2D) and obtaining the CV value by the CV calculation is as described above (see FIGS. 1 to 13).
Therefore, a method for performing self-position / posture determination by acquiring a CV value of a target image based on a reference CV video will be described below.

[1. Computation method 1]
Using the above-described CV calculation (FIGS. 1 to 13), the feature point (2D) in the target image can be independently CV-calculated and then coordinate-integrated with the CV value in the reference CV video. . Here, in order to integrate coordinates, it is necessary to acquire many common feature points in both images.
As a result, the CV values in both images can be displayed in the same coordinate system, and the acquisition of the CV values in the target image in the same coordinate system is completed. That is, the self-position / posture determination has been completed.
Here, the both images mean both the reference CV video and the target image.

[2. Computation method 2]
The CV image of the reference CV image map is returned to the state before the CV calculation, and the frames and feature points of both the CV image map and the target image are mixed without using the three-dimensional coordinates of the feature points used in the calculation. In this state, CV calculation is performed.
As a result, the CV values of both the CV video map and the target image are obtained. Here, when the mixed CV value is obtained, the three-dimensional coordinates of only the feature points on the CV video map side are given as known. The CV value on the target image side is automatically obtained with high accuracy.

[3. Computation method 3]
The feature point (3D) is calculated as a known coordinate by mixing the feature point (2D) and the feature point (3D) in the images of both the CV video map and the target image, as well as the points and planes whose 3D coordinates are known. And obtaining the CV value in the target image that has been unknown, thereby completing the CV value acquisition of the target image. That is, the self-position orientation is completed.
That is, while the two-dimensional feature points and the three-dimensional feature points of both the CV video map and the target image are mixed, CV calculation is performed by combining both of them, and the three-dimensional coordinates of the feature points whose known three-dimensional coordinates are fixed. The CV value of the target image can be acquired by performing the CV calculation using all the feature points.

[4.3D feature point coordinate transfer method]
By automatically corresponding a plurality of three-dimensional known points in the reference CV video to corresponding points (2D) in the target image, the three-dimensional coordinates are transferred to corresponding points in the target image.
Then, a CV value in the target image is geometrically calculated from a plurality of four or more three-dimensional feature points in the target image whose three-dimensional coordinates are known.
Thereby, the CV value acquisition of the target image is completed, and the self-position / posture determination is completed.
Here, the feature plane can be considered to be composed of a plurality of feature points, and therefore can be regarded as a correspondence of a plurality of feature points. Since all of these are handled as three-dimensional feature points, this is the method that requires the least calculation cost, that is, the CV value is obtained at high speed.

[5. Inverse 3D feature point coordinate transfer method]
This method corresponds to a case where the relationship between the reference CV video and the target image is partially or limitedly reversed in principle. However, the reference image is a CV video map.
Specifically, for example, as shown in FIG. 16 (d), a case where a known real world marker of three-dimensional coordinates is installed in the real world corresponds. In this case, the three-dimensional coordinates of the marker are acquired directly without using the reference CV video.
However, since the other feature points are acquired from the CV video, the calculation content is that the CV video map is involved as the reference image in the final CV value acquisition.

In addition, it is preferable to use a real world marker as described above from the viewpoint of safety and the like.
In the self-position / posture locating apparatus according to the present invention, for the convenience of using the camera while the vehicle or the like is traveling, it may be difficult to travel the vehicle or the like at night or in fog. Even in such a case, if a real-world 3D marker is used, it can be safely and inexpensively installed on the road or in the vicinity of the road, so the real-world 3D marker method is very safe in terms of safety. Promising.
Of course, even when the 3D marker in the target image is used, self-position / posture determination is sufficiently possible, but it is preferable to use a real-world 3D marker defined by law for administrative considerations or the like.

[6. CV value acquisition method by mechanical sensor]
It is also possible to directly acquire the CV value by a mechanical sensor provided in the target moving body 40 (see FIG. 14), for example, IMU / GYRO / GNSS.
However, the mechanical sensors that are currently popular have low accuracy, and it is difficult to obtain the accuracy that can withstand practical use by itself. On the other hand, the greatest advantage of mechanical sensors is that real-time output can be obtained.
Therefore, in this embodiment, it is possible to adopt a technique for obtaining a real-time approximate CV value by using relatively low price IMU / GYRO / GNSS or the like. Even if it is an approximate value, it is possible to obtain a relative value with little error in a very short time. Therefore, in order to know the variation between frames of the time discontinuous CV value by the relative value, or know the real-time value. Because it becomes effective.
Details of correction and complementary CV value acquisition by the mechanical sensor will be described later with reference to FIG. 17 described later.

[CV integration calculation]
As described above, the CV value of the target image can be acquired and transplanted based on the reference CV video, but the target is obtained by the same method as that for acquiring and generating the CV video map (see FIGS. 1 to 13). The CV value in the target image can be acquired by directly acquiring the CV value in the image, integrating the coordinates, and performing alignment.
This is the CV integration calculation 62 shown in FIG.
In addition, such CV calculation and CV value acquisition of the target image alone are not used alone, but are used in combination with other methods described above depending on the situation and the like.

[Real-time correction]
In principle, if a mechanical sensor such as IMU / GYRO is attached to the camera, CV values, that is, six variables can be acquired. However, in order to directly acquire the CV video, a very expensive (high accuracy) IMU / GYRO or the like is required, and in reality, the use of a mechanical sensor alone is not practical.
On the other hand, mechanical sensors such as IMU / GYRO have an excellent feature that a real-time output can be obtained.
Therefore, in the present embodiment, the feature of the mechanical sensor is effectively used and used as real-time correction for correcting time delay and time discontinuity associated with calculation of CV image and CV value acquisition.

In the CV calculation according to the present invention, the calculation of the feature amount related to the image processing, in principle, has a slight time delay due to the image processing time.
Therefore, using a mechanical sensor such as IMU / GYRO, it is possible to correct in real time a change in CV in which a slight time delay has occurred.
Specifically, the image processing time required for CV calculation and CV value acquisition is about several milliseconds to several seconds. That is, only this time is supplemented by six variables obtained from the mechanical sensor. With such a time, even with an inexpensive (low accuracy) mechanical sensor such as IMU / GYRO, time discontinuity correction and real-time correction can be performed.
Furthermore, in order to convert the CV value into absolute coordinates, the acquired relative coordinates may be converted into absolute coordinates by GCP installed in the environment or GNSS (GPS) rigidly coupled to the camera that acquires the target image. it can.

[High accuracy of target CV value by mechanical sensor]
Hereinafter, with reference to FIG. 17, the high accuracy (correction / complementation) of the CV value using the mechanical sensor will be specifically described.
FIG. 17 schematically shows processing when the CV value of the target image obtained by the self-position / posture locating apparatus using the reference video map according to the present embodiment is improved with six variables obtained by the mechanical sensor. It is explanatory drawing, (a) shows the whole some frame which comprises a target image, (b) expands and shows a part of some frame shown to (a).

Here, in FIG. 17, the CV value of the target image is plotted on the vertical axis, but displaying all six variables is cumbersome and hinders understanding, so only one variable of the target CV value is plotted on the vertical axis [01. ]. In addition, the horizontal axis [08] indicates the passage of time, and one segment of the time axis corresponds to the frame interval of the target camera.
The CV value [02, 06, 10] obtained by comparing the CV video map with the target image is indicated by ●.
One of the six variables acquired by the mechanical sensor is indicated by a dotted line 1 [05].
As shown in FIG. 17B, since the mechanical sensor has a real-time output, the target CV value acquired from the target image has a time delay Δt [04].
Further, due to insufficient accuracy of the mechanical sensor, an error Δd [03] occurs in the variable value itself.

When such an output is obtained, the delay time of the CV value acquired from the target image is known, the time axis is shifted, the real time axis is returned, and the output from the machine sensor is superimposed.
Then, the CV value obtained from the target image is set as a true value, and the CV value at the same time of the mechanical sensor is corrected so as to match the values at both ends of the frame. In this case, when it is not matched only by parallel movement, it is matched by proportional distribution.
By doing in this way, it was able to complement between the frames with the CV value from the mechanical sensor. Thus, acquiring the real-time CV value at the current time is meaningful as a self-position / posture locating device.

By the way, in order to record and save a high-precision real-time CV value, it is as described above. In the case of self-position / posture determination for automatic driving that requires only the CV value at the current time, one frame is used. There is a case where it is necessary to correct beyond this, and it becomes as follows. That is, the CV value acquired from the target image always has a delay of Δt due to the calculation processing time. At this time, Δt may exceed one frame period. That period will be supplemented with mechanical sensors. At this time, the machine sensor data [05] from the last frame [10] to the current time [12], not at both ends of the frame, is extended from the final CV value [10], and the target CV value at the current time [ 11].
In FIG. 17, the calculation delay time is illustrated as being within one frame, but the meaning does not change even if it exceeds one frame. The length of Δt [04] only extends further to the left of the drawing.
As a result, the CV value of the target camera is finally secured in real time.

As described above, in the present embodiment, when the self-position / posture of a target moving body (target moving body) is determined by a relatively low-precision and inexpensive mechanical sensor such as IMU / GYRO that does not satisfy the required accuracy, The self-position and posture (6 variables) of the moving body (target moving body) are obtained by the mechanical sensor with an error included.
Then, in order to correct the accumulated error, 6 variables obtained by the mechanical sensor with a more accurate CV value obtained by comparing the CV video map and the camera image loaded on the target moving body are intermittently obtained. 6 variables of continuous self-position / posture with no time delay can be acquired.

In this way, the six variables of raw data obtained from the mechanical sensor are continuous in time and have real-time characteristics, but the mechanical sensor is considerably less accurate than the CV value of the CV video map. is there.
On the other hand, the CV value based on the CV video map has advantages and disadvantages, such as a time delay due to computation, a temporally fragmented value, and lack of continuity.
Therefore, by combining the two, the advantages of both can be extracted.

By combining the CV video map and the machine sensor in this way, not only can the effect of correcting the delay error due to the arithmetic processing be expected, but it also has real-time properties.
That is, a delay occurs because the calculation processing time for obtaining the CV value (6 variables) of the target image is finite. This is a self-positioning and posture error. In order to correct the progress of the CV value that occurs within the delay time, the result is obtained by interpolating with the machine sensor the minimum time period from the most recent end point where the CV value of the target image is blank to the current time. Real-time performance can be improved.
In the first place, in order to achieve accuracy with only mechanical sensors, expensive equipment and devices are necessary, and there are other problems such as difficulty in calibration.

Therefore, by using the CV calculation technique according to the present invention, the CV value of the target camera with high accuracy can be obtained, and correction / complementation using the mechanical sensor and the CV video map can be performed.
That is, it is possible to interpolate six variables of the position and orientation of real-time output by a mechanical sensor such as IMU / GYRO that has low accuracy but no delay. The real-time output of the mechanical sensor is low in accuracy, but utilizes the characteristic that six variables with few errors can be acquired for an extremely short time.
In this way, the CV value is basically obtained by CV calculation, but only the latest ultra-short time is complemented by the mechanical sensor and is corrected in real time.

[Automatic driving system]
Next, automatic operation of the target moving body by the self-position / posture locating apparatus using the reference image map according to the present embodiment as described above will be described with reference to FIG.
FIG. 18 is a functional block diagram showing a system configuration of the automatic driving system 100 for a moving body by the self-position / posture locating apparatus using the reference video map according to the present embodiment.
In the system configuration of the automatic driving system 100 shown in FIG. 18, a portion not directly related to the above-described self-position / posture determination device (see FIG. 14) is indicated by a dotted line, and a portion directly related to the self-position / posture determination device is indicated by a solid line. ing.

In the automatic driving system 100, the target image acquisition / target camera unit 101 that acquires the target image installed in the target moving body (in this case, the autonomous driving vehicle) is compared with the CV video map unit 102 that is created in advance. The feature point comparison unit 105 takes the corresponding points of both images. Thereby, first, a target CV value having an operation delay is acquired.
Next, even if it is the low accuracy acquired from the mechanical sensor / 6 variable acquisition part 106, the CV value of a real-time output is output.
Next, the CV value output from the feature point comparison unit 105 is corrected by the self-position / posture determination unit 109 with the CV value without delay output from the machine sensor / 6 variable acquisition unit 106, and the final object without delay is corrected. A CV value is generated and output.

The self-localization device as described above is a main device for automatic operation. However, in order to perform automatic operation, the 3D space / environment attribute identification unit 103 can be used outdoors, indoors, intersections, on roads, The general attributes of the environment, such as outside the road and tunnel, are grasped.
Further, the target object recognition unit 104 recognizes a target part in the vicinity of the traveling road, and acquires its three-dimensional coordinates.
If the obstacle 3D recognition unit 107 determines that the object near the travel path is an obstacle, a signal is sent to the driving parameter instruction unit 111.

Thus, a translation vehicle, an oncoming vehicle, a parked vehicle, a person, other moving bodies, etc. around the self-driving vehicle are recognized. For the translation vehicle and the oncoming vehicle, the size and its six variables are the parked vehicle. As for the size and its three-dimensional coordinates, and for humans and other moving objects, information such as the size and direction of movement is acquired, and a signal is sent to the operation parameter instruction unit 111.
The driving parameter instruction unit 111 is under the control of the driving condition setting unit 110. Conditions necessary for driving are set. Finally, the vehicle is automatically controlled by the automatic guide traveling unit 112 of the vehicle. Is executed.

As described above, the automatic driving system 100 according to the present embodiment can perform automatic driving and automatic driving of a moving vehicle and the like accurately and safely using the above-described self-positioning device 1 using the reference video map. .
When the above self-position / posture locating device fails for some reason, the vehicle must be guided safely and stopped safely. In the case of a highway, it may be impossible to stop immediately.
Therefore, it is usually configured to automatically guide and stop the vehicle with only the information acquired by itself without receiving any external signal in an emergency, while performing automatic operation with the self-position and posture locating device activated. It is necessary to prepare.

Since automatic detection devices and systems always have an obstacle detection device, the device on which the obstacle detection device is installed is used as an auxiliary device, and automatic operation is performed while always operating this device and the auxiliary device. It is desirable to do.
Similarly, in the robot, there is a need for an auxiliary device that automatically moves to a safe place and stops even if the self-position / posture device breaks down.
Thus, in the present embodiment, in addition to the self-position / posture locating device, an autonomous traveling system that is loaded on the target vehicle and independent from the device is loaded, and in an emergency, by other cameras or other sensors, A self-position / posture locating device with a safety device that can safely guide and stop a target vehicle can be provided.

[Example]
Next, a more specific example of the self-position / posture locating apparatus using the reference video map according to the present embodiment will be described.
In the following, an example of a standard self-position posture locating device in automatic operation will be shown.
In the following embodiments, description will be made with reference to FIGS. 1 and 6 as appropriate.

A CV video map database 30 is created by the CV video map creation device 20 by performing CV calculation from a video obtained by photographing the environment obtained by the CV video acquisition device 10 in advance.
Next, from the CV video database 30, only the map in the range necessary for automatic driving is cut out, and the CV video map / target image comparison device 50 acquires the CV value of the target camera.
On the other hand, at the same time, the apparatus is integrated with the target moving body 40 with the camera, and first, six variables are directly acquired by a mechanical sensor. As a result, the CV value acquired from the image from the camera and the CV value acquired from the mechanical sensor, that is, the six variables of both are acquired.

Of these two types of data, six variables obtained directly by the mechanical sensor are temporally continuous (see FIG. 17 [05]), but the accuracy is low. On the other hand, although the image is fragmentary data [see FIG. 17 [06]], the accuracy is high.
Therefore, the self-position / posture locating device 60 corrects the CV value of the mechanical sensor by calibrating at a time (integer multiple of the frame) that matches the CV value data of the target camera among the continuous CV values of the mechanical sensor. (FIG. 17 [04/03]), and the intermediate part is interpolated with the data of the mechanical sensor to obtain a CV value with improved accuracy as a whole.

The signal acquired by the self-position / posture locating device 60 is sent to the vehicle surrounding state determination device 70, positioned in the CV video map, and simultaneously positioned with surrounding objects, pedestrians, obstacles, and the like.
These objects are three-dimensionally and their positional relationships are clarified, and the vehicle control signal generator 80 generates a vehicle control signal, and the vehicle (target moving body 40) is automatically controlled. Automatic operation is realized.

Note that obtaining the CV value of the target image is the self-position / posture orientation itself, but it is not always necessary to obtain all the frames of the target image by feature point tracking.
In general, since it is not possible to obtain a highly accurate value (CV value) unless a certain amount of time is taken for the calculation, it is desirable to take the calculation time as much as possible, but the frame falls accordingly.
Therefore, in this embodiment, CV acquisition is performed with a skipped frame on the target image side, and the skipped frame is not calculated, but is filled with six variables obtained from the simultaneously acquired mechanical sensor.

Specifically, it can be performed as follows.
[Example 1]
When the target image is 6 fps (6 frames per second), the CV calculation takes 1 second, that is, only the 1/6 frame is calculated.
As a result, high-precision calculation can be performed for only one frame, and six variables acquired from the mechanical sensor can be interpolated for the remaining five frames.

[Example 2]
When the target image is 0.5 fps (1 frame per 2 seconds) as shown in FIG. 17, it takes 0.3 seconds for the CV calculation, and therefore a delay (delay) of 0.3 seconds occurs. To do.
In this case, it is possible to output a real-time CV value with a mechanical sensor supplementing 0.3 seconds (see FIG. 17).

Of course, in this case, it is a condition that the error due to the mechanical sensor is less than or equal to the error of the CV calculation. This condition is extremely reasonable.
If the target CV calculation takes 1 second, this means that the 6 variables obtained from the machine sensor are calibrated once a second.
The condition that the accumulated error generated in one second of the mechanical sensor is lower than the error due to the CV calculation means that even an inexpensive IMU or GYRO can sufficiently cope with it. This is therefore a very realistic method.
Since the target CV value obtained from the target image and the mechanical sensor have errors, the number of camera frames is determined in consideration of the correction period, the number of camera frames, the performance of the mechanical sensor, and the like.

As described above, according to the self-position / posture locating apparatus 1 using the reference image map of the present embodiment, as a CV image map serving as a reference image, an image from a moving camera or a continuous still image (two-dimensional coordinates). And a total of 6 variables of 3D coordinates (X, Y, Z) and rotation amounts (Φx, Φy, Φz) indicating the position and orientation of the camera are given to all frames as CV values. It is possible to obtain a state in which the three-dimensional coordinates of an arbitrary place can be acquired.
Therefore, the position of a 3D point at any point can be obtained at any time by performing several millisecond arithmetic processing when necessary, without having to maintain 3D points (point clouds) for the entire environment where automatic driving is performed. Information can be acquired and generated, and data representing a three-dimensional space can be greatly compressed.
As a result, information indicating the position and posture of the moving body itself can be obtained easily, quickly and with high accuracy in automatic driving of a vehicle or aircraft, automatic driving of a robot moving in a three-dimensional space, etc. Become.

As described above, the SLAM and V-SLAM used for conventional self-position / posture determination have three-dimensional points in the entire area from the beginning, and also transfer data in order to exchange enormous point cloud data. As a result, enormous calculation costs and expenses are incurred for recording, and it is not realistic to automatically drive a wide area and to determine the self-position posture.
In SLAM and V-SLAM, the point cloud necessary for automatic operation is generally several hundred million points / km.
Even in the self-position / posture locating apparatus of the present embodiment, data of some characteristic three-dimensional points are actually held, but even if it has a three-dimensional point of several hundred points / km. The number of point clouds by the conventional LIDAR method or V-SLAM is so small that it cannot be compared, and the amount of data that increases due to only a few hundred points is negligible.

What actually has data is not the three-dimensional coordinates of the space, but only the six variables of the camera position. In spite of this, it is an excellent feature of the present invention that these 6 variables can be used as a 3D map in a state in which the three-dimensional coordinates of all points in the image can be obtained by a simple calculation.
A CV video map is generated by generating a reference image or a continuous reference still image (two-dimensional image) by a reference camera moving in an environment for the purpose of automatically running in advance and a reference CV value of the environment. By associating the two-dimensional video and six variables with each frame, the three-dimensional information of the environment can be collected and held.
Moreover, the CV video map does not have an enormous three-dimensional point group, and the three-dimensional coordinates of an arbitrary point in the CV video map can be obtained by calculation when necessary. The feature is that data can be kept and managed with a large weight reduction.

As described above, the CV video map used as the reference video in the present invention is held in a state where the three-dimensional coordinates in the environment can be acquired anytime and anywhere by taking one effort, and thus the data is extremely light. Thus, it can sufficiently withstand communication, and automatic driving can be made realistic.
Then, using such a CV video map as a reference image, this is compared with the target image, a plurality of feature points indicating the same location of both are automatically associated, and the CV value (6 variables) of the target image can be calculated. By acquiring, the target image self-position / posture can be acquired quickly.

Therefore, in the self-position / posture locating apparatus 1 using the reference video map according to the present embodiment, the following excellent effects can be realized by comparing the CV video with the target image and combining with the mechanical sensor. it can.
That is, by combining the CV video map and the target image, firstly, the data to be handled becomes light and the calculation becomes efficient.
Secondly, correction by a mechanical sensor can be added to the CV value of the moving image acquired from the camera and the target image acquired from the camera. These device configurations are simple, robust, easy to handle, low-cost, and high-performance self-positioning posture locating devices.
Thirdly, by incorporating the output from the GNSS as a mechanical sensor as one feature point or directly as a known CV value into the CV calculation, it is possible to easily increase the accuracy of the target CV value.

Further, time discontinuity correction by IMU / GYRO can be performed.
Such CV video accuracy can be improved by post-processing and post-shooting, and post-processing and design that match the accuracy is possible. If higher accuracy is required, it can be improved at any time. It is.
On the other hand, in the conventional self-position / posture locating method as disclosed in Patent Document 1 described above, it is extremely difficult to improve the accuracy of a 3D map using a laser point group, and the acquired data is changed. It is extremely difficult.

As described above, the self-position / posture determination apparatus using the reference video map of the present invention has been described with reference to the preferred embodiment. However, the self-position / posture determination apparatus using the reference video map according to the present invention is the same as the above-described embodiment. Needless to say, the present invention is not limited thereto, and various modifications can be made within the scope of the present invention.
For example, in the above-described embodiment, the application of the self-position / posture locating device using the reference image map of the present invention has been described assuming automatic driving of a moving body such as a vehicle, but the self-position according to the present invention is described. The posture locating device can be applied to any device or means that requires self-position posture locating, and needless to say, its use and usage are not particularly limited.

In the above-described embodiment, when automatic driving or the like is performed, a CV image as a reference image in which a CV value is obtained with high accuracy in advance as a reference three-dimensional map (with CV values in all frames). It has been explained that the position of the host vehicle can be acquired with high accuracy by transplanting and integrating CV values according to the present invention using a real-time video captured from a vehicle-mounted camera as a target image.
However, the same applies to a reference image based on a three-dimensional map generated by actual measurement using a mechanical sensor, a surveying device, or the like.
Therefore, the self-position / posture locating apparatus according to the present invention can use measured value data such as a mechanical sensor together as three-dimensional coordinate data.

In addition, according to the present invention, the position accuracy can be expected to be one hundred times that of real-time GPS. Therefore, the GPS can also be used as the approximate position setting means in the present invention.
In addition, as described above, even when updating a three-dimensional map, the self-position / posture locating apparatus using the reference video map of the present invention is effectively used, and of course when updating a map from video. Can be used.
Furthermore, according to the present invention, high-accuracy position coordinates can be obtained without using GPS, so that it can be expected to be used for high-precision navigation technology.

  The present invention can be suitably used for, for example, automatic driving of various vehicles such as automobiles, moving bodies such as airplanes and ships, automatic traveling of robots, and the like.

DESCRIPTION OF SYMBOLS 10 CV image acquisition device 20 CV image map creation device 30 CV image map database 40 Target moving body 50 CV image map / target image comparison device 60 Self-position / posture locating device

Claims (11)

Based on a reference image captured by a predetermined image acquisition means, CV calculation is performed to obtain a CV (camera vector) value indicating a three-dimensional coordinate value of the camera position and orientation of the reference image, and the CV value is applied to the reference image. A CV video map creating means for generating a CV video map to which
A CV video map database for storing the CV video map;
The CV video map stored in the CV video map database is used as a reference image, and the target image captured by a predetermined image acquisition means provided in the target moving body is compared with the CV video map, and the target image and the CV Self-position / posture locating means for automatically acquiring a plurality of feature points indicating the same part of the video map to obtain a CV value of the target image;
A self-position / posture locating device using a reference image map. The CV video map creating means includes:
The reference, which is the CV (camera vector) value when the reference image is acquired without holding the three-dimensional information of the three-dimensional space imprinted on the reference image as the three-dimensional coordinate data of the space. Generating 6-variable data indicating the position and orientation of the image acquisition means that has captured the image;
The CV video map database is
3D coordinates of an arbitrary point in the reference image can be obtained as needed from the CV value. The CV in which the reference image and the CV value are held corresponding to each other with reduced weight of 3D data. The self-position / posture locating apparatus using the reference video map according to claim 1, wherein the video map is stored. The self-position posture locating means is
The 6-variable data indicating the self-position and posture of the target moving body acquired by the mechanical sensor provided in the target moving body is treated as data without delay, and the calculation delay time of the CV value of the target image is converted. 3. The reference according to claim 1, wherein a real-time CV value or six variables indicating the self-position and posture of the target moving body that are temporally continuous are acquired by correcting to a real-time value. Self-position and posture locator using video map. The self-position posture locating means is
As a feature point indicating the same location of the target image and the CV video map, a predetermined feature amount included in the CV video map is selected,
The three-dimensional recognition object such as a sign having an automatically extracted three-dimensional coordinate included in the CV video map is selected as the predetermined feature amount. Self-position / posture locator using the reference video map described. The self-position posture locating means is
As a feature point indicating the same location of the target image and the CV video map, a predetermined feature amount included in the CV video map is selected,
The reference video map according to any one of claims 1 to 4, wherein a three-dimensional marker in the image having a known three-dimensional coordinate included in the CV video map is selected as the predetermined feature amount. Self-position posture locator using The self-position posture locating means is
As a feature point indicating the same location of the target image and the CV video map, a predetermined feature amount included in the CV video map is selected,
The reference video map according to any one of claims 1 to 5, wherein a real-world three-dimensional marker having a known three-dimensional coordinate included in the CV video map is selected as the predetermined feature amount. Self-position posture locator using The self-position posture locating means is
7. The target image from which the CV value is acquired is taken into the CV video map database, and a part or all of the CV video map stored in the CV video map database is updated. A self-position / posture locating device using the reference video map according to any one of the preceding claims. The self-position posture locating means is
By combining the CV video map and the target image and transferring the three-dimensional coordinates of the three-dimensional feature points from the CV video map to the target image or from the target image to the CV video map, The CV value of an image is acquired. The self-position / posture locating apparatus using the reference video map according to claim 1. The self-position posture locating means is
The CV video map and the target image are combined, and the CV calculation is performed by integrating the two-dimensional feature points and the three-dimensional feature points included in the CV video map and the target image. The CV value of the target image is acquired by performing CV calculation for all feature points included in the CV video map and the target image, with the three-dimensional coordinates fixed.
The CV value of the target image is separated from all CV values included in the CV video map and the target image. The self using the reference video map according to any one of claims 1 to 8, Position and orientation locator. The target moving body consists of an autonomous traveling system and a target vehicle equipped with a camera or a sensor,
The reference image according to any one of claims 1 to 9, wherein the target vehicle is guided and stopped based on the target image to which the CV generated by the self-position / posture locating unit is added. Self-position posture locator using map. The CV video map creating means is
A feature point extraction unit for automatically extracting a predetermined number of feature points from the image data of the moving image;
About the extracted feature points, a feature point correspondence processing unit that automatically tracks within each frame image of the moving image and obtains a correspondence relationship between the frame images;
A camera vector calculation unit that obtains the three-dimensional position coordinates of the feature point for which the correspondence relationship is obtained, and obtains a camera vector composed of the three-dimensional position coordinates and the three-dimensional rotation coordinates of the camera corresponding to each frame image from the three-dimensional position coordinates. The self-position posture locating device using the reference video map according to any one of claims 1 to 10.