WO2018142533A1

WO2018142533A1 - Position/orientation estimating device and position/orientation estimating method

Info

Publication number: WO2018142533A1
Application number: PCT/JP2017/003757
Authority: WO
Inventors: 隆博加島
Original assignee: 三菱電機株式会社
Priority date: 2017-02-02
Filing date: 2017-02-02
Publication date: 2018-08-09
Also published as: JPWO2018142533A1; JP6479296B2; TW201830336A

Abstract

A position/orientation estimating device (1): retrieves, from a kilometer post database (4), the positional coordinates of a reference point of a kilometer post corresponding to kilometers recognized from a photographed image; calculates the azimuth of the kilometer post on the basis of the retrieved positional coordinates of the reference point; specifies a kilometer post in the photographed image on the basis of the calculated azimuth of the kilometer post and the photograph azimuth of a photographing device (2); and estimates the position and the orientation of the photographing device (2) on the basis of the correspondence between the position coordinates of the reference point of the identified kilometer post in a world coordinate system and the positon coordinates thereof in the image coordinate system.

Description

Position / orientation estimation apparatus and position / orientation estimation method

The present invention relates to a position / orientation estimation apparatus and a position / orientation estimation method for estimating the position and orientation of an estimation object.

A technique for estimating the position and orientation of an estimation object using position information of a distance marker that has been image-recognized from a captured image is known. For example, Patent Literature 1 describes a vehicle position detection device that corrects a vehicle position using position information of a kilometer post (distance mark). This vehicle position detection device specifies the position information of the kilometer post corresponding to the feature information recognized from the photographed image among the pre-stored position information and feature information of the kilometer post, and uses this position information to determine the vehicle position. Is corrected.

JP 2009-250718 A

However, in the vehicle position detection device described in Patent Document 1, when a plurality of kiloposts having the same feature information are present in different places, only the feature information recognized by the image from the photographed image is used to capture the kilopost taken by the camera. It cannot be accurately identified. For this reason, there was a possibility of being corrected to an incorrect position.

For example, let us consider a case where a kilometer post is installed every 100 meters starting from the start point of the down line and the end point of the up line on the road where the lanes are separated into the up line and the down line.
The kilometer post indicates the number of kilometers that is the distance from the starting point set on the road to the kilometer post, so the same number of kilometers is present on the upstream kilometer post and the downstream kilopost on the corresponding position. Marked. In the above vehicle position detection device, the number of kilometers is the feature information. Therefore, the position information of the kilometer post on the upstream line should be specified, but the position information of the kilometer post on the downstream line in the corresponding position is erroneously specified. There is a possibility that.

This invention solves the said subject, and it aims at obtaining the position and orientation estimation apparatus and position and orientation estimation method which can improve the estimation precision of the position and orientation of an estimation target object.

A position and orientation estimation apparatus according to the present invention includes an object search unit, an orientation calculation unit, an object identification unit, and a position and orientation estimation unit.
The object search unit converts the reference point set for the recognition object in the three-dimensional space and the label information recognized from the photographed image from the database in which the coordinate information indicated by the recognition object is associated. Search for the corresponding recognition object.
The azimuth calculating unit calculates the azimuth of the recognition target based on the position coordinates of the reference point set for the recognition target searched by the target search unit.
The object specifying unit is based on the shooting direction of the estimated object obtained by shooting the shot image and the direction of the recognition target calculated by the direction calculation unit, from among the recognition targets searched by the target search unit. A recognition object in the captured image is specified.
The position / orientation estimation unit, based on the correspondence between the position coordinates in the three-dimensional space of the reference point set for the recognition object specified by the object specifying unit and the position coordinates in the captured image, Estimate posture.

According to the present invention, it is possible to accurately specify the recognition object in the photographed image from the recognition objects that indicate the same marking information based on the shooting direction of the estimation object and the direction of the recognition object. The position and orientation of the estimation object can be accurately estimated based on the correspondence relationship between the position coordinates of the reference point of the accurately identified recognition object in the three-dimensional space and the position coordinates in the captured image.

It is a block diagram which shows the function structure of the position and orientation estimation apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the example of a distance marker database. FIG. 3A is a block diagram showing a hardware configuration for realizing the function of the position / orientation estimation apparatus according to Embodiment 1. FIG. 3B is a block diagram illustrating a hardware configuration that executes software that implements the functions of the position and orientation estimation apparatus according to Embodiment 1. 3 is a flowchart showing an operation of the position and orientation estimation apparatus according to the first embodiment. It is a figure which shows the outline | summary of the process which recognizes a distance marker from a picked-up image. It is a figure which shows the outline | summary of the process which calculates the azimuth | direction of a distance marker. It is a figure which shows the outline | summary of the specific process of a distance marker. It is a figure which shows the example of installation of a distance marker.

Hereinafter, in order to describe the present invention in more detail, modes for carrying out the present invention will be described with reference to the accompanying drawings.
Embodiment 1 FIG.
FIG. 1 is a block diagram showing a functional configuration of a position / orientation estimation apparatus 1 according to Embodiment 1 of the present invention. As shown in FIG. 1, the position / orientation estimation device 1 is connected to each of the imaging device 2, the sensor device 3, and the distance marker database 4, and estimates the position and orientation of the imaging device 2.
The imaging device 2 is an estimation target whose position and orientation are estimated by the position / orientation estimation device 1, and is installed at a predetermined location of the vehicle to image the periphery of the vehicle.
In the following description, “position” indicates the position of the estimation object in the three-dimensional space, and “posture” is the tilt angle or rotation angle of the estimation object in the three-dimensional space. The imaging device 2 is assumed to be an in-vehicle camera that images the front of the vehicle.

The sensor device 3 is a sensor mounted on the vehicle, and is a three-axis acceleration sensor and a three-axis geomagnetic sensor. The acceleration sensor detects acceleration along the x-axis, acceleration along the y-axis, and acceleration along the z-axis in a three-dimensional space. The geomagnetic sensor detects geomagnetism along each of the x-axis, y-axis, and z-axis.
In the vehicle, the positional relationship between the imaging device 2 and the sensor device 3 is fixed.

The distance marker database 4 is a database in which the position coordinates of the reference point set in the distance marker in the three-dimensional space are associated with the number of kilometers of the distance marker.
The distance marker is a recognition object whose number of kilometers is recognized from the photographed image photographed by the photographing device 2. The number of kilometers is the marking information marked on the distance marker that is the recognition object, and indicates the distance from the starting point set on the road to the distance marker.

FIG. 2 is a diagram showing an example of the distance marker database 4. In FIG. 2, the distance marker is a rectangular sign board, and the reference points are the four corner points of the surface on which the kilometer is marked in the distance marker. The distance marker registered in the distance marker database 4 shown in FIG. 2 is a distance marker set on a road in which lanes are separated into an up line and a down line. In the distance marker database 4, the distance markers indicating the same number of kilometers are the distance marker on the up line side and the distance marker on the down line side at the position corresponding to the distance marker.

The reference points registered in the distance marker database 4 are points in a three-dimensional space representing the real space. The three-dimensional space defines, for example, an x-axis (for example, positive in the east direction) in the east-west direction and a y-axis (for example, positive in the north direction) in the north-south direction from a specific point on the earth as the origin. It is represented by a three-dimensional coordinate system that defines the z-axis (for example, the height direction of the distance marker is positive) in the elevation direction.
The position coordinates of the four corner points are distances along the x-axis, y-axis, and z-axis from the origin to the four corner points, expressed in meters.
Hereinafter, the three-dimensional coordinate system representing the real space is referred to as a “world coordinate system”.

The position / orientation estimation apparatus 1 includes a distance marker recognition unit 5, an orientation estimation unit 6, a distance marker search unit 7, an orientation calculation unit 8, a distance marker identification unit 9, and a position / orientation estimation unit 10.
The distance marker recognition unit 5 is a component corresponding to the object recognition unit, and from the captured image captured by the imaging device 2, the number of kilometers marked on the distance marker and the reference point set for the distance marker above. Recognize position coordinates in a two-dimensional coordinate system of a captured image.
For example, the distance marker recognition unit 5 performs pattern recognition processing or character recognition processing on a captured image that is a two-dimensional photographed image, thereby recognizing the number of kilometers of the distance marker captured in the captured image and the position coordinates of the reference point. To do.
Hereinafter, the two-dimensional coordinate system of the captured image is referred to as an “image coordinate system”.

The orientation estimation unit 6 estimates the imaging orientation of the imaging device 2 based on the detection information of the sensor device 3. The shooting direction of the shooting device 2 is a direction in which the viewpoint of the shooting device 2 is located. This azimuth may be an angle indicating the shooting azimuth of the image taking device 2. For example, the north may be represented by 0 degrees, the east by 90 degrees, the south by 180 degrees, and the west by 270 degrees. It may be expressed as

The distance marker search unit 7 is a component corresponding to the object search unit, and searches the distance marker database 4 for a distance marker corresponding to the number of kilometers recognized by the distance marker recognition unit 5 from the photographed image. In the distance marker database 4 shown in FIG. 2, the distance marker on the up line side and the distance marker on the down line side corresponding to the distance marker have the same number of kilometers. Two distance markers corresponding to the number are searched. For example, if the kilometer recognized by the distance marker recognition unit 5 is 123.4, two distance markers corresponding to the kilometer are retrieved, and the position coordinates of the reference point set for each of these distance markers are calculated. Is output to the unit 8.

The azimuth calculating unit 8 calculates the normal direction of the surface on which the number of kilometers is indicated as the azimuth of the distance marker based on the position coordinates of the reference point set in the distance marker searched by the distance marker searching unit 7.
The azimuth of the distance marker may be an angle indicating the azimuth of the distance marker with respect to the imaging device 2. For example, as with the azimuth estimation unit 6, 0 degrees north, 90 degrees east, 180 degrees south, and 270 west The angle may be expressed in degrees or the angle may be expressed in radians.

The distance marker specifying unit 9 is a component corresponding to the object specifying unit, and based on the shooting direction of the imaging device 2 and the direction of the distance marker, from the distance markers searched by the distance marker searching unit 7, The distance marker in the captured image is specified. Here, the shooting direction of the shooting apparatus 2 is the direction estimated by the direction estimation unit 6, and the direction of the distance marker is the direction calculated by the direction calculation unit 8.

The position / orientation estimation unit 10 determines the position of the imaging device 2 based on the correspondence between the position coordinates in the three-dimensional space of the reference point set in the distance marker specified by the distance marker specifying unit 9 and the position coordinates in the captured image. And estimate posture.
For example, the position / orientation estimation unit 10 generates a PnP (Perspective n-Points) problem using the internal parameters of the imaging apparatus 2 based on the correspondence between the position coordinates of the reference point in the world coordinate system and the position coordinates in the image coordinate system. Solve. Thereby, the position and orientation of the photographing apparatus 2 are estimated. The internal parameters (intrinsic parameters) are information including the focal length and principal point of the photographing lens provided in the photographing apparatus 2.

The distance marker database 4 may be constructed on a storage area of an external storage device provided separately from the position and orientation estimation device 1. In this case, the distance marker search unit 7 searches the distance marker by accessing the distance marker database 4 in the external storage device via a communication line such as the Internet or an intranet.
The distance marker recognizing unit 5 may be a constituent element included in the photographing apparatus 2. In this case, the distance marker recognizing unit 5 is removed from the position / orientation estimation device 1 and the label information recognized from the captured image is output from the imaging device 2 to the distance marker searching unit 7 and is recognized from the captured image. The position coordinates of the points are output from the imaging device 2 to the position / orientation estimation unit 10.
The direction estimation unit 6 may be a component included in the sensor device 3. In this case, the azimuth estimation unit 6 is removed from the position / orientation estimation device 1, and information indicating the photographing orientation of the photographing device 2 is output from the sensor device 3 to the distance indicator specifying unit 9.

FIG. 3A is a block diagram showing a hardware configuration for realizing the function of the position / orientation estimation apparatus 1. The imaging device 100, the sensor device 101, the storage device 102, and the processing circuit 103 are connected to each other by a bus. FIG. 3B is a block diagram illustrating a hardware configuration that executes software that implements the functions of the position and orientation estimation apparatus 1. The imaging device 100, the sensor device 101, the storage device 102, the CPU (Central Processing Unit) 104, and the memory 105 are connected to each other by a bus.

3A and 3B, the image capturing device 100 is an image capturing device 2 that captures the periphery of the vehicle on which the position and orientation estimation device 1 is mounted, and the sensor device 101 is a sensor that constitutes the sensor device 3. The storage device 102 stores a distance marker database 4. The storage device 102 is realized by, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an HDD (Hard Disk Drive), or the like, and may be a storage device that combines these. Further, a part or all of the storage area of the storage device 102 may be provided in the external device. In this case, the position / posture estimation apparatus 1 communicates with the external apparatus via a communication line such as the Internet or an intranet, and the distance marker search process is executed.

In the position / orientation estimation apparatus 1, the functions of the distance marker recognition unit 5, the orientation estimation unit 6, the distance marker search unit 7, the orientation calculation unit 8, the distance marker identification unit 9, and the position / orientation estimation unit 10 are realized by a processing circuit. The That is, the position / orientation estimation apparatus 1 includes a processing circuit for executing these functions. The processing circuit may be dedicated hardware or a CPU that executes a program stored in a memory.

When the processing circuit is the dedicated hardware processing circuit 103 illustrated in FIG. 3A, the processing circuit 103 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), or the like. ), FPGA (Field-Programmable Gate Array), or a combination thereof.
In the position / orientation estimation apparatus 1, each function of the distance marker recognition unit 5, the direction estimation unit 6, the distance marker search unit 7, the direction calculation unit 8, the distance target specification unit 9, and the position / orientation estimation unit 10 is realized by a processing circuit. Alternatively, each function may be realized by a single processing circuit.

When the processing circuit is the CPU 104 shown in FIG. 3B, the functions of the distance marker recognition unit 5, the azimuth estimation unit 6, the distance marker search unit 7, the azimuth calculation unit 8, the distance marker identification unit 9, and the position and orientation estimation unit 10 are as follows: It is realized by software, firmware, or a combination of software and firmware. Software and firmware are described as programs and stored in the memory 105.

The CPU 104 realizes each function by reading and executing a program stored in the memory 105. That is, the position / orientation estimation apparatus 1 includes a memory 105 for storing a program that, when executed by the CPU 104, results in the processing from step ST1 to step ST9 shown in FIG.
These programs cause the computer to execute the procedures or methods of the distance marker recognition unit 5, the direction estimation unit 6, the distance marker search unit 7, the direction calculation unit 8, the distance marker identification unit 9, and the position and orientation estimation unit 10. .

The memory is, for example, non-volatile or volatile semiconductor memory such as RAM, ROM, flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically Programmable EPROM), magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disk) and the like are applicable.

In addition, a part of the functions of the distance marker recognition unit 5, the direction estimation unit 6, the distance marker search unit 7, the direction calculation unit 8, the distance marker identification unit 9, and the position and orientation estimation unit 10 are realized by dedicated hardware. A part may be realized by software or firmware.
For example, the distance marker recognition unit 5 and the azimuth estimation unit 6 realize their functions by a dedicated hardware processing circuit, and include a distance marker search unit 7, an azimuth calculation unit 8, a distance marker identification unit 9, and a position and orientation estimation unit 10. As for, the function is realized by the CPU 104 executing the program stored in the memory 105.
As described above, the processing circuit can realize the above-described functions by hardware, software, firmware, or a combination thereof.

Next, the operation will be described.
FIG. 4 is a flowchart showing the operation of the position / orientation estimation apparatus 1 and shows a series of processes from obtaining a captured image in front of the vehicle until the position and orientation of the imaging apparatus 2 are estimated.
First, when the distance marker recognition unit 5 inputs a captured image in front of the vehicle from the imaging device 2 (step ST1), the distance marker recognizes the distance marker from the captured image (step ST2).
FIG. 5 is a diagram showing an outline of processing for recognizing the distance marker 401 from the captured image 2A. In FIG. 5, the pixel in the upper left corner of the captured image 2A is the origin (0, 0), the axis extending rightward from the origin is defined as the x axis, and the axis extending downward from the origin is defined as the y axis. Each point on the captured image 2A is defined in units of pixels (pixels) and represents a position in the image coordinate system.
However, the definition of the origin and the coordinate axes described above is merely an example, and the present invention is not limited to this.

When the distance marker recognition unit 5 recognizes the distance marker 401 in the captured image 2A by performing image processing on the captured image 2A (step ST3; YES), the distance marker recognition unit 5 recognizes the number of kilometers displayed on the distance marker 401. To the distance marker search unit 7. In FIG. 5, the number information “123.4” is output to the distance marker search unit 7.
The distance marker recognition unit 5 recognizes the position coordinates of the four corner points of the distance marker 401 in the image coordinate system and outputs them to the position and orientation estimation unit 10. The four corner points of the distance marker 401 are points on the surface where the number of kilometers in the distance marker 401 is marked, and are

points

402a, 402b, 402c, and 402d shown in FIG.
On the other hand, when the distance marker is not shown in the photographed image, the distance marker recognition unit 5 cannot recognize the distance marker from the photographed image (step ST3; NO), and returns to the process of step ST1.

The distance marker search unit 7 searches the distance marker database 4 for a distance marker corresponding to the number of kilometers recognized from the captured image 2A by the distance marker recognition unit 5 (step ST4).
For example, when the distance marker search unit 7 searches the distance marker database 4 shown in FIG. 2 based on “123.4”, which is the number of kilometers input from the distance marker recognition unit 5, the number of kilometers “123.4”. Information on the two distance markers corresponding to is read out and output to the azimuth calculation unit 8. The above-mentioned distance marker information is position coordinates in the world coordinate system of each point at the four corners of the distance marker.

Next, the azimuth calculating unit 8 calculates the azimuth of the distance marker with respect to the photographing apparatus 2 based on the position coordinates in the world coordinate system of the four corners of the distance marker input from the distance marker searching unit 7 (step ST5). ). As described above, when two distance markers corresponding to the number of kilometers “123.4” recognized from the photographed image 2A are retrieved from the distance marker database 4, the azimuth calculation unit 8 determines each of these distance markers. Calculate the bearing. The azimuth calculation unit 8 outputs the calculated azimuth for each distance marker and the position coordinates in the world coordinate system of the four corners (reference points) of the distance marker for which the azimuth has been calculated to the distance marker identification unit 9.

FIG. 6 is a diagram showing an outline of processing for calculating the azimuth of the distance marker 601 and shows a state where the distance marker 601 is viewed from the sky. In FIG. 6, the positive direction of the x axis is the east direction, and the positive direction of the y axis is the north direction. A point 602 is a point at the upper left corner of the distance marker 601, and a point 603 is a point at the upper right corner of the distance marker 601.

Points

602 and 603 are the reference points described above. An arrow 604 indicates the installation direction of the distance marker 601, that is, the normal direction of the surface on which the kilometer is marked.

The angle 605 is an angle indicating the azimuth of the distance marker 601 calculated by the azimuth calculation unit 8. The azimuth calculation unit 8 calculates the angle 605 using the position coordinates of the upper left corner point 602, the position coordinates of the upper right corner point 603, and the inverse trigonometric function of the distance marker 601.
The azimuth calculation unit 8 calculates the azimuth for all the distance markers searched by the distance marker search unit 7 and outputs the calculated azimuth to the distance marker identification unit 9.
In the above description, the azimuth of the distance marker 601 is calculated using the position coordinates of the upper left corner point 602 and the upper right corner point 603 of the distance marker 601, but instead of the lower left corner of the distance marker 601. The azimuth of the distance marker 601 may be calculated using the position coordinates of the point and the point at the lower right corner. Alternatively, the azimuth of the distance marker may be calculated using a combination of an upper left corner point, an upper right corner point, a lower left corner point, and a lower right corner point.

Further, the bearing calculation unit 8 calculates the bearing of the distance marker based on the outer product of the vector from the upper left corner point to the lower left corner point and the vector from the upper left corner point to the upper right corner point. May be. For example, the azimuth calculation unit 8 calculates a vector perpendicular to the surface on which the number of kilometers is marked in the distance marker based on the value of the outer product, and uses the calculated vector and the inverse trigonometric function to calculate the distance marker. Calculate the bearing.

Next, when the orientation estimation unit 6 acquires the sensor value from the sensor device 3 (step ST6), the orientation estimation unit 6 estimates the imaging orientation of the imaging device 2 based on the acquired sensor value (step ST7).
For example, the azimuth estimation unit 6 calculates the posture of the photographing device 2 with respect to the ground based on the gravitational acceleration detected by the acceleration sensor, and based on the calculated posture and the value of the geomagnetism detected by the geomagnetic sensor. The shooting direction of 2 is estimated.
When the photographing device 2 is in a posture perpendicular to the ground (the photographing direction is horizontal to the ground) or when the posture of the photographing device 2 is fixed to the vehicle and is known, the azimuth estimation unit 6 includes a geomagnetic sensor. The photographing direction of the photographing apparatus 2 may be estimated using only the sensor information.

Subsequently, the distance marker specifying unit 9 searches for the distance marker searched by the distance marker searching unit 7 based on the azimuth of the distance marker calculated by the azimuth calculating unit 8 and the shooting azimuth of the imaging device 2 estimated by the azimuth estimating unit 6. A distance marker in the captured image is specified from among the images (step ST8).
FIG. 7 is a diagram showing an outline of the distance marker specifying process. In FIG. 7, a shooting direction 701 is a shooting direction of the shooting apparatus 2 estimated by the direction estimation unit 6 and has an azimuth angle of 60 degrees. The azimuth | direction 702 and the azimuth | direction 703 shown with a broken line are the azimuth | directions of the said two distance markers calculated by the azimuth | direction calculation part 8. FIG. The azimuth angle of the azimuth 702 is 50 degrees, and the azimuth angle of the azimuth 703 is 260 degrees.

The distance marker specifying unit 9 inverts the azimuth angle of the imaging azimuth 701 by 180 degrees in order to facilitate comparison with the azimuth of the distance marker. In FIG. 7, an azimuth 704 is an azimuth obtained by inverting the azimuth angle of the shooting azimuth 701 by 180 degrees, and the azimuth angle is 240 degrees.
Next, the distance marker specifying unit 9 compares the azimuth 704 obtained by inverting the imaging azimuth 701 by 180 degrees with the

azimuths

702 and 703 of the two distance markers.

For example, the distance marker specifying unit 9 calculates the absolute value 705 of the difference between the azimuth angle of the azimuth 704 and the azimuth angle of the azimuth 702 and calculates the absolute value 706 of the difference between the azimuth angle of the azimuth 704 and the azimuth angle of the azimuth 703. The absolute value 705 and the absolute value 706 are compared in magnitude. The distance marker specifying unit 9 specifies the distance marker having the smallest absolute value of the azimuth angle difference as the distance marker captured by the imaging device 2, that is, the distance marker in the captured image. Since the absolute value 705 is 170 degrees and the absolute value 706 is 20 degrees, the distance marker of the azimuth 703 is specified as the distance marker in the captured image.
The distance marker specifying unit 9 outputs the position coordinates in the world coordinate system of each point (reference point) at the four corners of the specified distance marker to the position and orientation estimating unit 10.

In a road where the lanes are separated into an up line and a down line, if distance signs are installed every 100 meters starting from the start point of the down line and the end point of the up line, the distance sign is set to the road. The number of kilometers that is the distance from the starting point to the mileage mark is displayed.
In this case, as shown in FIG. 8, the same kilometer is marked on the up-distance side distance indicator 800a and the down-line side distance indicator 800b corresponding to the up-distance side distance indicator 800a. The same number of kilometers is indicated on the distance marker 801b on the down line side at the position corresponding to.

In the conventional technique, in order to identify the distance marker being photographed based on the number of kilometers recognized from the photographed image, for example, the distance marker 800a on the up-line side should be identified, and it is at a position corresponding to this. There is a possibility that the distance marker 800b on the down line side is specified by mistake.
On the other hand, the distance marker specifying unit 9 searches for the distance searched by the distance marker search unit 7 based on the direction of the distance marker calculated by the direction calculation unit 8 and the shooting direction of the shooting device 2 estimated by the direction estimation unit 6. A distance marker in the captured image is specified from the markers. Thereby, it is possible to accurately identify the distance marker 800a on the upline side that is actually reflected in the captured image.

The position / orientation estimation unit 10 associates the position coordinates in the world coordinate system of each point of the four corners of the distance marker specified by the distance marker specifying unit 9 with the position coordinates in the image coordinate system of each point of the four corners of the distance marker. Based on the relationship, the position and orientation of the imaging device 2 are estimated (step ST9).
Since the distance marker specified by the distance marker specifying unit 9 is a distance marker captured by the imaging device 2, the position / orientation estimation unit 10 determines the position coordinates in the image coordinate system of each point at the four corners of the distance marker. It can be acquired from the mark recognition unit 5.
The position / orientation estimation unit 10 solves the PnP problem using the internal parameters of the imaging apparatus 2 based on the correspondence between the position coordinates in the world coordinate system and the position coordinates in the image coordinate system of the four corners of the distance marker. By solving, the position and orientation including the rotation component of the imaging device 2 are estimated.

When the positional relationship between the vehicle on which the position / orientation estimation device 1 is mounted and the imaging device 2 is known, the position / orientation estimation unit 10 is based on the estimated position and orientation of the imaging device 2 and the positional relationship. Can be accurately estimated. Since the vehicle position can be estimated accurately, the navigation accuracy can be improved.

Among the processes shown in FIG. 4, there is no dependency between the process from step ST1 to step ST5 and the process from step ST6 to step ST7, so the former process and the latter process are performed in parallel. May be executed.

So far, the case where the recognition target object is a distance marker and the labeling information is the number of kilometers has been described, but Embodiment 1 is not limited to this.
For example, the recognition target object may be equipment installed along a road, and the marking information may be code information such as numbers, character strings, images, and two-dimensional barcodes marked on the equipment. . That is, the recognition target object and the labeling information may be anything that can be recognized from the captured image.

Although the case where the reference point of the distance marker is each of the four corners has been shown, the present invention is not limited to this. For example, the reference point may be a point on the edge of the distance marker or a point on the marking surface in kilometer.
Furthermore, although the case where the position / orientation estimation apparatus 1 is mounted on a vehicle is shown, the present invention is not limited to this. For example, the position / orientation estimation apparatus 1 may be mounted on a moving body such as a railway or an aircraft, or the position / orientation estimation apparatus 1 may be realized by a mobile terminal such as a smartphone or a tablet PC and carried by a person.

As described above, the position / orientation estimation apparatus 1 according to the first embodiment searches the distance marker database 4 for the position coordinates of the reference point of the distance marker corresponding to the number of kilometers recognized from the captured image, and the retrieved distance marker. The azimuth of the distance marker is calculated based on the position coordinates of the reference point, the distance marker in the photographed image is identified based on the calculated azimuth of the distance marker and the photographing azimuth of the photographing device 2, and the identified distance marker The position and orientation of the photographing apparatus 2 are estimated based on the correspondence between the position coordinates of the reference point in the world coordinate system and the position coordinates in the image coordinate system.
Thus, based on the shooting direction of the shooting apparatus 2 and the direction of the distance marker, the distance marker in the captured image can be accurately specified from the distance markers indicating the same number of kilometers.
It is possible to accurately estimate the position and orientation of the photographing apparatus 2 based on the position coordinates of the reference point of the distance marker that is accurately specified.
Since the position of the photographed distance marker can also be specified accurately, the position / orientation estimation apparatus 1 can be applied to the record of road maintenance inspection work.
Further, since the position and orientation of the position / orientation estimation apparatus 1 can be accurately estimated based on the position and orientation of the photographing apparatus 2, the augmented reality application that requires the accurate position and orientation of the object is also applicable. Can be applied.

In the position / orientation estimation apparatus 1 according to the first embodiment, the orientation estimation unit 6 estimates the orientation of the imaging device 2 with respect to the ground based on the detection information of the acceleration sensor, and the imaging orientation of the imaging device 2 based on the estimated orientation. Is estimated. By comprising in this way, the imaging | photography azimuth | direction of the imaging device 2 attached with the free attitude | position with respect to the ground can be estimated.

In the present invention, any component of the embodiment can be modified or any component of the embodiment can be omitted within the scope of the invention.

The position / orientation estimation apparatus according to the present invention can improve the estimation accuracy of the position and orientation of the estimation object, and is suitable, for example, for an in-vehicle navigation apparatus.

1 position and orientation estimation device, 2,100 photographing device, 2A photographed image, 3,101 sensor device, 4 distance marker database, 5 distance marker recognition unit, 6 bearing estimation unit, 7 distance marker search unit, 8 bearing calculation unit, 9 Distance marker specifying unit, 10 position and orientation estimation unit, 102 storage device, 103 processing circuit, 104 CPU, 105 memory, 401, 601 distance marker, 604 arrow, 605 angle, 701 shooting orientation, 702 to 704 orientation, 705, 706 absolute Value, 800a, 800b, 801a, 801b distance markers.

Claims

The recognition target object corresponding to the labeling information recognized from the photographed image from the database in which the position coordinates of the reference point set in the recognition target object in the three-dimensional space and the labeling information indicated by the recognition target object are associated with each other. An object search part for searching for
An azimuth calculating unit that calculates the azimuth of the recognition object based on the position coordinates of the reference point set in the recognition object searched by the object searching unit;
Based on the shooting direction of the estimated object obtained by shooting the captured image and the direction of the recognition object calculated by the direction calculation unit, from among the recognition objects searched by the object search unit, An object specifying unit for specifying the recognition object in the captured image;
Based on the correspondence between the position coordinates in the three-dimensional space of the reference point set for the recognition object specified by the object specifying unit and the position coordinates in the captured image, the position and orientation of the estimation object A position and orientation estimation apparatus comprising: a position and orientation estimation unit that estimates
The position / orientation estimation apparatus according to claim 1, further comprising an object recognition unit that recognizes labeling information and a position coordinate of the reference point in the captured image from the captured image captured by the estimated object. .
The position / orientation estimation apparatus according to claim 1, further comprising an azimuth estimation unit configured to estimate a shooting azimuth of the estimation target object.
The azimuth estimating unit estimates a posture of the estimation object with respect to the ground based on detection information of an acceleration sensor, and estimates a shooting azimuth of the estimation object based on the estimated posture. The position and orientation estimation apparatus described.
The position / orientation estimation apparatus according to claim 1, wherein the reference points are a plurality of points on a surface on which labeling information is marked in the recognition target object.
A step of recognizing the labeling information indicated by the recognition target object and the position coordinates of the reference point set in the recognition target object in the captured image from the captured image captured by the estimation target object;
The object search unit is configured to perform the object recognition from the database in which the position coordinates in the three-dimensional space of the reference point set to the recognition object and the labeling information indicated by the recognition object are associated with each other by the object recognition unit. Searching the recognition object corresponding to the labeling information recognized from the captured image;
An azimuth calculating unit calculating the azimuth of the recognition object based on the position coordinates of the reference point set in the recognition object searched by the object searching unit;
An azimuth estimating unit estimating a shooting azimuth of the estimation object;
The object specifying unit is searched by the object searching unit based on the shooting direction of the estimated object estimated by the direction estimating unit and the direction of the recognition object calculated by the direction calculating unit. Identifying the recognition object in the captured image from the recognition object;
The position and orientation estimation unit is configured to perform the estimation based on a correspondence relationship between a position coordinate in the three-dimensional space of the reference point set in the recognition target specified by the target specifying unit and a position coordinate in the captured image. A position and orientation estimation method comprising: estimating a position and orientation of an object.