JP6727032B2 - Mobile terminal, self-position estimation system using the same, server and self-position estimation method - Google Patents

Description

The present invention relates to a mobile terminal for estimating the self-position of a mobile terminal outdoors, a self-position estimation system using the same, a server, and a self-position estimation method in a system for performing maintenance such as maintenance and inspection of roads. is there.

There are times when you want to record your own position when you find an abnormal point in the inspection work of a large infrastructure such as a road. A GPS (Global Positioning System) can be used for estimating the self-position, but there is an error of about 10 meters at the maximum, and the self-position accuracy necessary for business cannot be secured.
Therefore, there is disclosed a method of obtaining a highly accurate shooting position by previously having a 3D (three-dimensional) model of an artificial structure and by associating an object in the current camera image with the 3D model. (For example, Patent Document 1)

Japanese Unexamined Patent Publication No. 2012-79129 (pages 7 to 11, FIG. 2)

Since the conventional self-position estimation system is configured as described above, in a place where structures of the same shape are lined up, such as on a housing complex or an expressway, the position cannot be identified only by the shape of the structures, and the self-position cannot be determined. There was a problem that it could not be estimated.

The present invention has been made to solve the above-described problems, and a mobile terminal capable of highly accurately estimating the self-position even in a place where there is no structure capable of uniquely identifying the position and self-position estimation using the mobile terminal An object is to obtain a system, a server, and a self-position estimation method.

In a mobile terminal according to the present invention, an image capturing unit that captures an image with a camera, an identification information extracting unit that extracts identification information in an image captured by the image capturing unit, and identification information including pre-created position information The identification information acquisition unit that acquires the identification information from the server having the identification information, the identification information extracted by the identification information extraction unit, and the identification information acquired from the server by the identification information acquisition unit are compared, and the identification information is associated with both identification information. An information comparison unit and a self-position estimation unit that estimates the self-position based on the identification information associated by the identification information comparison unit, the identification information having a character string and a size of the character string, and the self-position When estimating the self-position , the estimation unit calculates the distance from the character string to the self-position based on the size of the character string of the associated identification information, and uses the plurality of character strings to determine the direction of the self-position. It is to be calculated .

According to the present invention, an image capturing unit that captures an image with a camera, an identification information extracting unit that extracts identification information in an image captured by the image capturing unit, and a server that has identification information including pre-created position information From the identification information acquisition unit that acquires the identification information, the identification information extracted by the identification information extraction unit, and the identification information acquired from the server by the identification information acquisition unit, and the identification information comparison unit that associates both identification information , And a self-position estimating unit that estimates the self-position based on the identification information associated by the identification information comparing unit, the identification information having a character string and a size of the character string, and the self-position estimating unit , In estimating the self-position, the distance from the character string to the self-position is calculated based on the size of the character string of the associated identification information, and the direction of the self-position is calculated using a plurality of character strings . The self-position can be estimated with high accuracy even in a place where structures having the same shape are lined up.

1 is a block diagram showing a self-position estimation system according to Embodiment 1 of the present invention. It is a figure which shows the hardware constitutions of the mobile terminal of the self-position estimation system by Embodiment 1 of this invention. It is a figure which shows the hardware constitutions of the server of the self-position estimation system by Embodiment 1 of this invention. It is a figure which shows the data structure of identification information DB (database) of the self-position estimation system by Embodiment 1 of this invention. It is a figure which shows the display screen of the self-position estimation system by Embodiment 1 of this invention. 5 is a flowchart showing a flow until the identification information DB in the server of the self-position estimation system according to the first embodiment of the present invention is updated. 5 is a flowchart showing a flow of storing identification information around a designated position in the self-position estimation system according to the first embodiment of the present invention in an identification information DB in a mobile terminal. 3 is a flowchart showing a flow of self-position estimation of the self-position estimation system according to the first embodiment of the present invention. It is a block diagram which shows the self-position estimation system by Embodiment 2 of this invention. 7 is a flowchart showing a flow of self-position estimation of the self-position estimation system according to the second embodiment of the present invention. It is a block diagram which shows the self-position estimation system by Embodiment 3 of this invention. 9 is a flowchart showing a flow of self-position estimation of the self-position estimation system according to the third embodiment of the present invention. It is a block diagram which shows the self-position estimation system by Embodiment 4 of this invention. 13 is a flowchart showing a flow of self-position estimation of the self-position estimation system according to the fourth embodiment of the present invention. It is a block diagram which shows the self-position estimation system by Embodiment 5 of this invention. 9 is a flowchart showing a flow of self-position estimation of a self-position estimation system according to a fifth embodiment of the present invention.

Embodiment 1.
Embodiment 1 of the present invention will be described below with reference to the drawings.
1 is a block diagram showing a self-position estimation system according to a first embodiment of the present invention.
In FIG. 1, the self-position estimation system has a mobile terminal 100 and a server 200 that are communicable via a communication line.
The mobile terminal 100 is configured as follows.
The camera 110 photographs the surrounding area. The image capturing unit 120 captures a peripheral image through the camera 110. The screen display unit 130 (display unit) displays a camera image, map information, etc. on the screen of the display device.
The in-image identification information extraction unit 140 (identification information extraction unit) extracts identification information such as a character string included in the image. The identification information acquisition unit 150 acquires the identification information of the designated position from the server 200. The identification information DB 160 holds the identification information acquired by the identification information acquisition unit 150.
The identification information comparison unit 170 compares the identification information extracted from the image with the identification information stored in the identification information DB 160, and associates the identification information with each other. The self-position estimation unit 180 estimates the self-position from the identification information associated by the identification information comparison unit 170. The GPS receiver 190 receives position information from GPS satellites.

The server 200 is configured as follows.
The 3D model generator 210 generates a 3D model from 3D point cloud data measured by MMS (mobile mapping system) or the like. The 3D model DB 220 holds the 3D model generated by the 3D model generation unit 210.
The 3D model identification information extraction unit 230 extracts identification information such as a character string from the 3D model. The identification information DB 240 (identification information database) holds the identification information extracted by the 3D model identification information extraction unit 230. The identification information transmitting unit 250 transmits the identification information in response to the request from the mobile terminal 100.

FIG. 2 is a diagram showing a hardware configuration of a mobile terminal of the self-position estimation system according to the first embodiment of the present invention.
In FIG. 2, main hardware of the mobile terminal 100 is shown. Software stored in a ROM (Read Only Memory) 12 causes a CPU (Central Processing Unit) 11 to perform processing while temporarily storing data in a RAM (Random Access Memory) 13, and to display a camera image and processing on a display device 15. Display the result. The identification information DB 160 is stored in the flash memory 14.
The software stored in the ROM 12 includes an image capturing unit 120, an in-image identification information extraction unit 140, a screen display unit 130, an identification information acquisition unit 150, an identification information comparison unit 170, a self-position estimation unit 180, and a GPS reception unit 190. ..

FIG. 3 is a diagram showing a hardware configuration of a server of the self-position estimation system according to the first embodiment of the present invention.
FIG. 3 shows the main hardware of the server 200. With the software stored in the ROM 22, the CPU 21 performs processing while temporarily storing data in the RAM 23. The 3D model DB 220 and the identification information DB 240 are stored in the HDD (hard disk drive) 24.
The software stored in the ROM 22 includes a 3D model generation unit 210, a 3D model identification information extraction unit 230, and an identification information transmission unit 250.

FIG. 4 is a diagram showing a data structure of the identification information DB of the self-position estimation system according to the first embodiment of the present invention.
In FIG. 4, each data of the identification information DB 240 has identification information and position information. The identification information has a character string and its size, and the position information is the latitude and longitude of a signboard having the character string.

FIG. 5 is a diagram showing a display screen of the self-position estimation system according to the first embodiment of the present invention.
In FIG. 5, a screen 1301 displayed by the screen display unit 130 includes an area 1302 for displaying a camera image, an area 1303 for displaying the self position of the mobile terminal 100 on a map, and an area for displaying the latitude and longitude of the self position. 1304 and.
The character string of the identification information in the camera image in the area 1302, which is associated with the character string of the identification information held in the identification information DB 160, is highlighted.

Next, the operation will be described with reference to FIGS. 6, 7, and 8.
FIG. 6 shows a flow until the identification information DB 240 in the server 200 is updated based on 3D point cloud data measured by MMS or the like.
FIG. 7 shows a flow until the identification information around the position designated by the user is stored in the identification information DB 160 in the mobile terminal 100.
FIG. 8 shows a flow when performing self-position estimation on the spot.

First, the flow until the identification information DB 240 in the server 200 is updated based on 3D point cloud data measured by MMS or the like will be described with reference to FIG.
The 3D model generation unit 210 of the server 200 receives the 3D point cloud data measured by MMS or the like and generates a 3D model (step ST1001).
The 3D model generation unit 210 stores the generated 3D model in the 3D model DB 220 (step ST1002).
The 3D model identification information extraction unit 230 extracts identification information such as character strings from signboards and signs in the 3D model stored in the 3D model DB 220, distance markers on the side of the road, and the like, as shown in FIG. And its size and position information such as a signboard having a character string are stored in the identification information DB 240 (step ST1003).

Next, a flow (third step) until the identification information around the position designated by the user is stored in the identification information DB 160 in the mobile terminal 100 will be described with reference to FIG. 7.
The identification information acquisition unit 150 of the mobile terminal 100 requests the identification information transmission unit 250 of the server 200 for the identification information near the designated position (step ST2001). This designated position is designated based on the position information received by the GPS receiving unit 190.
The identification information transmission unit 250 acquires the identification information near the designated position from the identification information DB 240 and transmits it to the identification information acquisition unit 150 of the mobile terminal 100 (step ST2002).
Identification information acquisition section 150 stores the received identification information in identification information DB 160 (step ST2003).

Next, the flow when the mobile terminal 100 performs self-position estimation on the spot will be described with reference to FIG.
The image capturing unit 120 of the mobile terminal 100 captures an image of the current location through the camera 110 (step ST3001, first step).
The in-image identification information extraction unit 140 extracts identification information such as a character string drawn on a signboard, a sign, a distance sign, or the like, that is, a character string and its size from the image (step ST3002, second step).
The identification information comparison unit 170 compares the identification information extracted by the in-image identification information extraction unit 140 with the identification information in the identification information DB 160, and identifies the corresponding identification information in the identification information DB 160 (step ST3004, fourth). Step). If the character string can be specified, the corresponding character string in the area 1302 for displaying the camera image on the screen 1301 is highlighted.

The self-position estimation unit 180 estimates the self-position from the position information of the identification information of the identified identification information DB 160 (step ST3005, fifth step).
At this time, the distance to the character string is calculated based on the size of the character string in the identification information extracted by the in-image identification information extraction unit 140. Moreover, when a plurality of character strings are specified, the direction of the self-position is also calculated. From these, the self-position is estimated.
That is, the character string is used to identify the identification information, the size of the character string is used to calculate the distance, and the direction of the self-position is calculated for the plurality of character strings.
Then, the estimated self-position is displayed on the map in the area 1303 of the screen 1301 and the accurate latitude and longitude are displayed in the area 1304.

According to the first embodiment, by comparing identification information such as a character string drawn on a signboard, a sign, a distance marker, etc. extracted from 3D point cloud data acquired by MMS, etc., with the identification information in the camera image, Since the self-position is estimated, there is an effect that the self-position can be estimated with high accuracy even in a place where structures having the same shape are lined up.

Embodiment 2.
FIG. 9 is a block diagram showing a self-position estimation system according to the second embodiment of the present invention.
9, 100 to 250 are the same as those in FIG. In FIG. 9, the mobile terminal 100 is provided with a self-position estimation complementing unit 181 that estimates a self-position by detecting a manhole.

In the second embodiment, as shown in FIG. 9, a self-position estimation complementing unit 181 that estimates a self-position by detecting a manhole is added to the first embodiment.

The operation of the second embodiment will be described below with reference to FIG.
FIG. 10 shows the flowchart of FIG. 8 according to the first embodiment to which a supplementary process of self-position estimation by a manhole is added.
Steps ST3001, 3002, 3004, and 3005 in FIG. 10 are the same as the processes in FIG. In FIG. 10, after the processing of step ST3002, a determination processing (step ST3003) of whether or not the identification information can be extracted is added, and when it cannot be extracted, the processing of steps ST3101 and ST3102 is added.

In FIG. 10, after the process of step ST3002, it is determined whether or not the identification information can be extracted (step ST3003), and if it cannot be extracted, the self-position estimation complementing unit 181 detects a manhole from the captured image (step). ST3101).
Next, the self-position estimation complementing unit 181 estimates the self-position from the detected position of the manhole and the distance to the manhole (step ST3102). The self-position estimation complementing unit 181 displays the estimated self-position on the map in the area 1303 of the screen 1301 and also displays the accurate latitude and longitude in the area 1304.
The position of the manhole shall be clear.

According to the second embodiment, by detecting a manhole whose position can be identified, there is an effect that the self-position estimation can be continued even when there is a place where the identification information is small and it is difficult to estimate the self-position.

Embodiment 3.
11 is a block diagram showing a self-position estimation system according to a third embodiment of the present invention.
11, 100 to 250 are the same as those in FIG. In FIG. 11, the mobile terminal 100 is provided with a self-position estimation complementing unit 182 that estimates the amount of movement of the mobile terminal 100 by motion stereo.

In the third embodiment, as shown in FIG. 11, a self-position estimation complementing unit 182 for estimating the moving direction and the moving amount of the mobile terminal 100 by motion stereo is added to the first embodiment.

The operation of the third embodiment will be described below with reference to FIG.
FIG. 12 is a flowchart obtained by adding the process of estimating the movement amount from the change amount of continuous camera images to the flowchart of FIG. 8 of the first embodiment.
Steps ST3001 to ST3005 in FIG. 12 are the same as the processes in FIG. In FIG. 12, the processes of steps ST3201 to ST3203 are added as the processes when the identification information cannot be extracted in step ST3003.

In FIG. 12, after the process of step ST3102, it is determined whether or not the identification information can be extracted (step ST3003), and if the identification information cannot be extracted, the self-position estimation complementing unit 182 responds from two consecutive captured images. The feature points to be extracted are extracted (step ST3201).
Next, the self-position estimation complementing unit 182 calculates the moving direction and the moving amount of the mobile terminal 100 from the extracted change amount (difference) of the feature points (step ST3202).
Next, the self-position estimation complementing unit 182 estimates the self-position from the movement direction and the movement amount calculated in step ST3202, based on the self-position before the movement (step ST3203). The self-position estimation complementing unit 182 displays the estimated self-position on the map in the area 1303 of the screen 1301 and also displays the accurate latitude and longitude in the area 1304.

According to the third embodiment, the movement direction and the movement amount are calculated from the variation amount of the feature points in the camera image to estimate the self-position, so that there is little identification information around and it is difficult to estimate the self-position from the identification information. Even if there are many different places, self-position estimation can be continued.

Fourth Embodiment
FIG. 13 is a block diagram showing a self-position estimation system according to Embodiment 4 of the present invention.
In FIG. 13, 100 to 250 are the same as those in FIG. In FIG. 13, the mobile terminal 100 calculates the moving direction and the moving amount of the mobile terminal 100 based on the acceleration measured by the acceleration sensor 111 and the angular velocity measured by the gyro sensor 112, and estimates the self-position. An estimation complementing unit 183 is provided.

The fourth embodiment is different from the first embodiment in that, as shown in FIG. 13, a self-position estimation complement for estimating a moving direction and a moving amount from a measurement value of a sensor mounted on the mobile terminal 100 and estimating a self-position. The part 183 is added.

The operation of the fourth embodiment will be described below with reference to FIG.
FIG. 14 is a flowchart of FIG. 8 of the first embodiment, in which the moving direction and the moving amount of the mobile terminal 100 are calculated based on the acceleration measured by the acceleration sensor 111 and the angular velocity measured by the gyro sensor 112. This is the addition of processing for estimating the position.
Steps ST3001 to ST3005 of FIG. 14 are the same as the processes of FIG. In FIG. 14, the processes of step ST3301 and step ST3302 are added as the process when the identification information cannot be extracted in step ST3003.

In FIG. 14, subsequent to the process of step ST3102, it is determined whether or not the identification information can be extracted (step ST3003). Then, the angular velocity is measured to calculate the moving direction and the moving amount of the mobile terminal 100 (step ST3301).
Next, the self-position estimation complementing unit 183 estimates the self-position from the calculated movement direction and movement amount based on the self-position before the movement (step ST3302). The self-position estimation complementing unit 183 displays the estimated self-position on the map in the area 1303 of the screen 1301 and also displays the accurate latitude and longitude in the area 1304.

According to the fourth embodiment, the movement direction and the movement amount are calculated from the measurement values of the sensors mounted on the mobile terminal, and the self-position is estimated, so that there is little identification information in the surroundings and the self-position is estimated from the identification information. Even if there is a difficult place to continue, the self-position estimation can be continued.

Embodiment 5.
FIG. 15 is a block diagram showing a self-position estimation system according to the fifth embodiment of the present invention.
15, 100 to 140 and 200 to 240 are the same as those in FIG. In FIG. 15, the mobile terminal 100 is provided with a self-position estimation requesting unit 184 that requests the server 200 to perform self-position estimation. Further, in the server 200, the identification information extracted from the image is compared with the identification information held in the identification information DB 240 (identification information database), and the identification information is compared with the identification information comparison unit 260 that associates the identification information. A terminal position estimation unit 270 that estimates the position of the mobile terminal 100 from the information is provided.

The fifth embodiment differs from the first embodiment in that the identification information comparing unit and the self-position estimating unit are moved from the mobile terminal 100 to the server 200, as shown in FIG.

The operation of the fifth embodiment will be described below with reference to FIG.
16 is different from the flowchart of FIG. 8 of the first embodiment in that the identification information comparison processing after extracting the identification information of the current location from the camera image and the self-position estimation processing are performed by the server 200. It was done.
Steps ST3001 and ST3002 in FIG. 16 are the same as the processes in FIG. In FIG. 16, the processes of steps ST3401 to ST3405 are added as processes subsequent to step ST3002. Of these, the processes of steps ST3402 to ST3405 are processes of the server 200.

In FIG. 16, self-position estimation requesting section 184 of mobile terminal 100 requests self-position estimation to terminal position estimating section 270 of server 200 (step ST3401). At this time, the self-position estimation requesting unit 184 sends the identification information extracted by the in-image identification information extracting unit 140 to the terminal position estimating unit 270 of the server 200.
The terminal position estimation unit 270 of the server 200 requests the identification information comparison unit 260 to compare the identification information (step ST3402).
The identification information comparison unit 260 compares the identification information (designated identification information) sent from the mobile terminal 100 with the identification information in the identification information DB 240, and the identification information DB 240 corresponding to the identification information sent from the mobile terminal 100. The identification information of is identified (step ST3403).

Terminal position estimation section 270 estimates the position of mobile terminal 100 from the identified identification information and the location information in identification information DB 240 (step ST3404).
Next, terminal position estimating section 270 transmits the estimation result to self-position estimation requesting section 184 (step ST3405). The transmitted estimation result includes screen display information to be displayed on the screen display unit 130. The self-position estimation requesting unit 184 displays the estimated self-position on the map in the area 1303 of the screen 1301 and also displays the accurate latitude and longitude in the area 1304.

According to the fifth embodiment, by performing the comparison process of the identification information on the server, it is not necessary to transmit the identification information from the identification information DB of the server to the mobile terminal in advance, and thus the working time can be reduced. There is.
In addition, by performing the identification information comparison processing on the server having high computing power, there is an effect that the self-position estimation time can be shortened.

It should be noted that, in the present invention, the respective embodiments can be freely combined, or the respective embodiments can be appropriately modified or omitted within the scope of the invention.

11 CPU, 12 ROM, 13 RAM, 14 flash memory, 15 display device,
21 CPU, 22 ROM, 23 RAM, 24 HDD,
100 mobile terminal, 110 camera, 111 acceleration sensor,
112 gyro sensor, 120 image capturing unit, 130 screen display unit,
140 identification information extraction unit in image, 150 identification information acquisition unit, 160 identification information DB,
170 identification information comparing unit, 180 self position estimating unit, 181 self position estimating complementing unit,
182 self-position estimation complementing unit, 183 self-position estimation complementing unit,
184 self-position estimation requesting unit, 190 GPS receiving unit, 200 server,
210 3D model generation unit, 220 3D model DB,
230 3D model identification information extraction unit, 240 identification information DB, 250 identification information transmission unit, 260 identification information comparison unit, 270 terminal position estimation unit

Claims (10)

An image capturing unit that captures images with a camera,
An identification information extraction unit that extracts identification information in the image captured by the image capturing unit,
An identification information acquisition unit that acquires the identification information from a server that has identification information including position information created in advance,
The identification information extracted by the identification information extraction unit and the identification information acquired from the server by the identification information acquisition unit are compared, and an identification information comparison unit that associates both identification information with each other,
And a self-position estimation unit that estimates the self-position based on the identification information associated by this identification information comparison unit,
The identification information has a character string and the size of the character string,
In estimating the self-position , the self -position estimating unit calculates the distance from the character string to the self-position based on the size of the character string of the associated identification information, and uses a plurality of character strings. A mobile terminal that calculates the direction of its own position . A self-position estimation complementing unit that complements the estimation by the self-position estimating unit by recognizing a manhole on the road from the image captured by the image capturing unit and estimating the self-position by the position of the manhole is provided. The mobile terminal according to claim 1, wherein the mobile terminal is a mobile terminal. By calculating the moving direction and the moving amount of the mobile terminal from the difference between the image captured immediately before by the image capturing unit and the current image, and estimating the self-position based on the moving direction and the moving amount, the self-position is calculated. The mobile terminal according to claim 1, further comprising a self-position estimation complementing unit that complements the estimation by the estimating unit. The movement direction and the movement amount of the mobile terminal are estimated from the measurement values of the sensors mounted on the mobile terminal, and the self-position is estimated by the movement direction and the movement amount to complement the estimation by the self-position estimation unit. The mobile terminal according to claim 1, further comprising: a self-position estimation complementing unit. The image capturing unit displays an image captured by the image capturing unit, and includes a display unit that highlights the identification information on the displayed image when the identification information comparing unit associates the identification information with each other. The mobile terminal according to claim 1, wherein the mobile terminal is a mobile terminal. The mobile terminal according to claim 5, wherein the display unit displays the self-position estimated by the self-position estimation unit in a screen area different from the display area of the image. The mobile terminal according to any one of claims 1 to 6,
And a self-position estimation system comprising a server having an identification information database storing identification information including previously created position information. The server is
A 3D model generation unit that generates a 3D model from the separately measured 3D point cloud data;
And a 3D model identification information extraction unit that extracts identification information including position information from the 3D model generated by the 3D model generation unit,
The self-position estimation system according to claim 7, wherein the identification information database stores the identification information extracted by the 3D model identification information extraction unit. A 3D model generation unit that generates a 3D model from separately measured 3D point cloud data,
A 3D model identification information extraction unit that extracts identification information including position information from the 3D model generated by the 3D model generation unit,
An identification information database storing the identification information extracted by the 3D model identification information extraction unit,
An identification information comparison unit that compares the identification information transmitted from the mobile terminal with the identification information stored in the identification information database, and associates the two pieces of identification information with each other.
And a terminal position estimation unit that estimates the position of the mobile terminal based on the identification information associated by the identification information comparison unit,
The identification information has a character string and the size of the character string,
In estimating the position of the mobile terminal , the terminal position estimation unit calculates the distance from the character string to the mobile terminal based on the size of the character string of the associated identification information, and calculates a plurality of characters. A server, characterized in that the direction of the mobile terminal is calculated using a sequence . The first step in which the image capturing section captures an image with the camera,
The identification information extraction unit, a second step of extracting the identification information in the image captured by the first step,
A third step of acquiring the identification information by the identification information acquisition unit from the server having the identification information including the position information created in advance,
The identification information comparing unit compares the identification information extracted in the second step with the identification information acquired from the server in the third step, and a fourth step of associating both identification informations,
And the self-position estimation unit includes a fifth step of estimating the self-position based on the identification information associated by the fourth step,
The identification information has a character string and the size of the character string,
The fifth step, in estimating the self position , based on the size of the character string of the associated identification information, calculates the distance from the character string to the self position, using a plurality of character strings A method for estimating a self-position , comprising calculating the direction of the self-position.

Images