JP2017211284A - Mobile terminal, self position estimation system using the same, server, and self position estimation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a self position estimation system capable of estimating a self position with high accuracy even in a place where there is no structure capable of specifying a position uniquely.SOLUTION: A self position estimation system comprises a mobile terminal 100 and a server 200. The mobile terminal 100 causes: an image capture unit 120 to capture an image through a camera 110; an in-image identification information extraction unit 140 to extract identification information in the image; an identification information acquisition unit 150 to acquire identification information from the server 200 having the identification information containing pre-created position information; an identification information comparison unit 170 to compare the acquired identification information with the identification information extracted by the in-image identification information extraction unit 140 to associate both of the identification information; and a self position estimation unit 180 to estimate a self position based on the associated identification information.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a mobile terminal that estimates the self-position of a mobile terminal outdoors, a self-position estimation system using the same, a server, and a self-position estimation method in a system that performs maintenance management such as road maintenance and inspection. is there.

In some cases, you may want to record your location when you find an unusual part in an inspection of a large infrastructure such as a road. GPS (Global Positioning System) can be used to estimate the self-position, but there is an error of about 10 meters at the maximum, and the self-position accuracy required for business cannot be secured.
For this reason, there is disclosed a technique for obtaining a highly accurate shooting position by holding a 3D (three-dimensional) model of an artificial structure in advance and taking correspondence between an object in the current camera image and the 3D model. (For example, Patent Document 1)

JP 2012-79129 A (pages 7 to 11 and FIG. 2)

  Since the conventional self-position estimation system is configured as described above, in a place where structures of the same shape such as housing estates and highways are lined up, the position cannot be identified only by the shape of the structure, and the self-position There was a problem that it could not be estimated.

  The present invention has been made to solve the above-described problems, and a mobile terminal capable of estimating a self-position with high accuracy even in a place where there is no structure capable of uniquely identifying the position, and self-position estimation using the mobile terminal It is an object to obtain a system, a server, and a self-position estimation method.

  In the mobile terminal according to the present invention, an image capturing unit that captures an image with a camera, an identification information extracting unit that extracts identification information in an image captured by the image capturing unit, and identification information including position information created in advance The identification information acquisition unit that acquires identification information from the server having the identification information, the identification information extracted by the identification information extraction unit, and the identification information acquired from the server by the identification information acquisition unit, and the identification information associated with each other An information comparison unit and a self-position estimation unit that estimates the self-position based on the identification information associated by the identification information comparison unit are provided.

  According to the present invention, an image capturing unit that captures an image with a camera, an identification information extracting unit that extracts identification information in an image captured by the image capturing unit, and a server having identification information including position information created in advance. Identification information acquisition unit that acquires identification information, identification information extracted by the identification information extraction unit, and identification information comparison unit that compares the identification information acquired from the server by the identification information acquisition unit and associates both identification information And a self-position estimation unit that estimates the self-position based on the identification information associated with the identification information comparison unit, so that the self-position can be estimated with high accuracy even in a place where structures of the same shape are arranged. be able to.

It is a block diagram which shows the self-position estimation system by Embodiment 1 of this invention. It is a figure which shows the hardware constitutions of the mobile terminal of the self-position estimation system by Embodiment 1 of this invention. It is a figure which shows the hardware constitutions of the server of the self-position estimation system by Embodiment 1 of this invention. It is a figure which shows the data structure of identification information DB (database) of the self-position estimation system by Embodiment 1 of this invention. It is a figure which shows the display screen of the self-position estimation system by Embodiment 1 of this invention. It is a flowchart which shows the flow until it updates the identification information DB in the server of the self-position estimation system by Embodiment 1 of this invention. It is a flowchart which shows the flow until it stores in the identification information DB in a mobile terminal the identification information around the designated position of the self-position estimation system by Embodiment 1 of this invention. It is a flowchart which shows the flow of the self-position estimation of the self-position estimation system by Embodiment 1 of this invention. It is a block diagram which shows the self-position estimation system by Embodiment 2 of this invention. It is a flowchart which shows the flow of the self-position estimation of the self-position estimation system by Embodiment 2 of this invention. It is a block diagram which shows the self-position estimation system by Embodiment 3 of this invention. It is a flowchart which shows the flow of the self-position estimation of the self-position estimation system by Embodiment 3 of this invention. It is a block diagram which shows the self-position estimation system by Embodiment 4 of this invention. It is a flowchart which shows the flow of the self-position estimation of the self-position estimation system by Embodiment 4 of this invention. It is a block diagram which shows the self-position estimation system by Embodiment 5 of this invention. It is a flowchart which shows the flow of the self-position estimation of the self-position estimation system by Embodiment 5 of this invention.

Embodiment 1 FIG.
Embodiment 1 of the present invention will be described below with reference to the drawings.
FIG. 1 is a block diagram showing a self-position estimation system according to Embodiment 1 of the present invention.
In FIG. 1, the self-position estimation system includes a mobile terminal 100 and a server 200 that can communicate with each other via a communication line.
The mobile terminal 100 is configured as follows.
The camera 110 captures the periphery. The image capturing unit 120 captures surrounding images through the camera 110. The screen display unit 130 (display unit) displays camera images, map information, and the like on a display device.
The in-image identification information extraction unit 140 (identification information extraction unit) extracts identification information such as a character string included in the image. The identification information acquisition unit 150 acquires the identification information of the specified position from the server 200. The identification information DB 160 holds the identification information acquired by the identification information acquisition unit 150.
The identification information comparison unit 170 compares the identification information extracted from the image with the identification information held in the identification information DB 160 and associates the identification information with each other. The self-position estimation unit 180 estimates the self-position from the identification information associated with the identification information comparison unit 170. The GPS receiving unit 190 receives position information from GPS satellites.

The server 200 is configured as follows.
The 3D model generation unit 210 generates a 3D model from 3D point cloud data measured by MMS (mobile mapping system) or the like. The 3D model DB 220 holds the 3D model generated by the 3D model generation unit 210.
The 3D model identification information extraction unit 230 extracts identification information such as a character string from the 3D model. The identification information DB 240 (identification information database) holds the identification information extracted by the 3D model identification information extraction unit 230. The identification information transmission unit 250 transmits identification information in response to a request from the mobile terminal 100.

FIG. 2 is a diagram showing a hardware configuration of the mobile terminal of the self-position estimation system according to Embodiment 1 of the present invention.
In FIG. 2, the main hardware of the mobile terminal 100 is shown. With software stored in a ROM (Read Only Memory) 12, a CPU (Central Processing Unit) 11 performs processing while temporarily storing data in a RAM (Random Access Memory) 13. Display the results. The identification information DB 160 is stored in the flash memory 14.
The software stored in the ROM 12 includes an image capturing unit 120, an in-image identification information extraction unit 140, a screen display unit 130, an identification information acquisition unit 150, an identification information comparison unit 170, a self-position estimation unit 180, and a GPS reception unit 190. .

FIG. 3 is a diagram showing a hardware configuration of the server of the self-position estimation system according to Embodiment 1 of the present invention.
In FIG. 3, the main hardware of the server 200 is shown. The CPU 21 performs processing while temporarily storing data in the RAM 23 by software stored in the ROM 22. The 3D model DB 220 and the identification information DB 240 are stored in an HDD (Hard Disk Drive) 24.
The software stored in the ROM 22 includes a 3D model generation unit 210, a 3D model identification information extraction unit 230, and an identification information transmission unit 250.

FIG. 4 is a diagram showing a data structure of the identification information DB of the self-position estimation system according to Embodiment 1 of the present invention.
In FIG. 4, each piece of data in the identification information DB 240 has identification information and position information. The identification information has a character string and its size, and the position information is the latitude and longitude of a signboard with the character string.

FIG. 5 is a diagram showing a display screen of the self-position estimation system according to Embodiment 1 of the present invention.
In FIG. 5, a screen 1301 displayed by the screen display unit 130 includes an area 1302 for displaying a camera image, an area 1303 for displaying the self-position of the mobile terminal 100 on a map, and an area for displaying the latitude and longitude of the self-position. 1304.
A character string associated with the character string of the identification information held in the identification information DB 160 in the character string of the identification information in the camera image in the area 1302 is highlighted.

Next, the operation will be described with reference to FIG. 6, FIG. 7, and FIG.
FIG. 6 shows a flow until the identification information DB 240 in the server 200 is updated based on 3D point cloud data measured by MMS or the like.
FIG. 7 shows a flow until the identification information around the position designated by the user is stored in the identification information DB 160 in the mobile terminal 100.
FIG. 8 shows a flow when self-position estimation is performed on site.

First, the flow until the identification information DB 240 in the server 200 is updated based on 3D point cloud data measured by MMS or the like will be described with reference to FIG.
The 3D model generation unit 210 of the server 200 receives 3D point cloud data measured by MMS or the like, and generates a 3D model (step ST1001).
The 3D model generation unit 210 stores the generated 3D model in the 3D model DB 220 (step ST1002).
The 3D model identification information extraction unit 230 extracts identification information such as a character string from a signboard, a sign, a roadside distance marker, etc. in the 3D model stored in the 3D model DB 220. As shown in FIG. Then, the size and position information such as a signboard having a character string are stored in the identification information DB 240 (step ST1003).

Next, a flow (third step) until the identification information around the position designated by the user is stored in the identification information DB 160 in the mobile terminal 100 will be described with reference to FIG.
Identification information acquisition section 150 of mobile terminal 100 requests identification information near the designated position from identification information transmission section 250 of server 200 (step ST2001). This designated position is designated based on the position information received by the GPS receiving unit 190.
Identification information transmission section 250 acquires identification information in the vicinity of the designated position from identification information DB 240 and transmits it to identification information acquisition section 150 of mobile terminal 100 (step ST2002).
The identification information acquisition unit 150 stores the received identification information in the identification information DB 160 (step ST2003).

Next, the flow when the mobile terminal 100 performs self-location estimation at the site will be described with reference to FIG.
The image capturing unit 120 of the mobile terminal 100 captures an image of the current location through the camera 110 (step ST3001, first step).
The in-image identification information extraction unit 140 extracts identification information such as a character string drawn on a signboard, a sign, a distance marker, or the like, that is, a character string and its size from the image (step ST3002, second step).
The identification information comparison unit 170 compares the identification information extracted by the in-image identification information extraction unit 140 with the identification information in the identification information DB 160, and identifies the identification information in the corresponding identification information DB 160 (step ST3004, fourth). Step). If it can be identified, the corresponding character string in the area 1302 for displaying the camera image on the screen 1301 is highlighted.

The self-position estimation unit 180 estimates the self-position from the position information of the identification information in the identified identification information DB 160 (step ST3005, fifth step).
At this time, the distance to the character string is calculated based on the size of the character string in the identification information extracted by the in-image identification information extraction unit 140. When a plurality of character strings are specified, the direction of the self position is also calculated. From these, the self-position is estimated.
In other words, the character string is used to specify the identification information, the size of the character string is used to calculate the distance, and the direction of the self position is also calculated using a plurality of character strings.
Then, the estimated self-position is displayed on the map of the area 1303 of the screen 1301 and the exact latitude and longitude are displayed in the area 1304.

  According to Embodiment 1, by comparing identification information such as a character string drawn on a signboard, a sign, a distance marker, etc. extracted from 3D point cloud data acquired by MMS or the like with identification information in a camera image, Since the self-position is estimated, there is an effect that the self-position can be estimated with high accuracy even in a place where structures having the same shape are arranged.

Embodiment 2. FIG.
FIG. 9 is a block diagram showing a self-position estimation system according to Embodiment 2 of the present invention.
In FIG. 9, reference numerals 100 to 250 are the same as those in FIG. In FIG. 9, the mobile terminal 100 is provided with a self-position estimation complementing unit 181 that estimates the self-position by detecting a manhole.

In the second embodiment, as shown in FIG. 9, a self-position estimation complementing unit 181 that estimates a self-position by adding a manhole is added to the first embodiment.

Hereinafter, the operation of the second embodiment will be described with reference to FIG.
FIG. 10 is obtained by adding a self-position estimation complementary process by manholes to the flowchart of FIG. 8 of the first embodiment.
Steps ST3001, 3002, 3004, and 3005 in FIG. 10 are the same processes as those in FIG. In FIG. 10, after the process of step ST3002, a process for determining whether or not the identification information has been extracted (step ST3003) is added. When the identification information cannot be extracted, the processes of step ST3101 and step ST3102 are added.

In FIG. 10, after the process of step ST3002, it is determined whether or not the identification information has been extracted (step ST3003). If the identification information has not been extracted, the self-position estimation complementing unit 181 detects a manhole from the captured image (step ST3003). ST3101).
Next, the self-position estimation complementing unit 181 estimates the self-position from the detected position of the manhole and the distance from the manhole (step ST3102). The self-position estimation complementing unit 181 displays the estimated self-position on the map of the area 1303 on the screen 1301 and displays the accurate latitude and longitude in the area 1304.
It is assumed that the position of the manhole is clear.

  According to the second embodiment, by detecting a manhole that can identify the position, there is an effect that the self-position estimation can be continued even when the location where the identification information is small and the self-position estimation is difficult continues.

Embodiment 3 FIG.
FIG. 11 is a block diagram showing a self-position estimation system according to Embodiment 3 of the present invention.
In FIG. 11, reference numerals 100 to 250 are the same as those in FIG. In FIG. 11, the mobile terminal 100 is provided with a self-position estimation complementing unit 182 that estimates the amount of movement of the mobile terminal 100 using motion stereo.

  In the third embodiment, as shown in FIG. 11, a self-position estimation complementing unit 182 that estimates the moving direction and the moving amount of the mobile terminal 100 by motion stereo is added to the first embodiment.

Hereinafter, the operation of the third embodiment will be described with reference to FIG.
FIG. 12 is obtained by adding a process of estimating a movement amount from a change amount of continuous camera images to the flowchart of FIG. 8 of the first embodiment.
Steps ST3001 to ST3005 in FIG. 12 are the same processes as those in FIG. In FIG. 12, the process of step ST3201-step ST3203 is added as a process when identification information cannot be extracted by step ST3003.

In FIG. 12, it is determined whether or not the identification information has been extracted following the processing in step ST3102, and if the identification information cannot be extracted, self-position estimation complementing section 182 responds from two consecutive captured images. Feature points to be extracted are extracted (step ST3201).
Next, self-position estimation complementing section 182 calculates the movement direction and movement amount of mobile terminal 100 from the extracted change amount (difference) of feature points (step ST3202).
Next, self-position estimation complementing section 182 estimates the self-position from the movement direction and movement amount calculated in step ST3202 based on the self-position before movement (step ST3203). The self-position estimation complementing unit 182 displays the estimated self-position on the map of the area 1303 on the screen 1301 and also displays accurate latitude and longitude in the area 1304.

  According to the third embodiment, it is difficult to estimate the self position from the identification information because there is little identification information around by calculating the movement direction and the movement amount from the change amount of the feature point in the camera image and estimating the self position. Even if a difficult place continues, the self-position estimation can be continued.

Embodiment 4 FIG.
FIG. 13 is a block diagram showing a self-position estimation system according to Embodiment 4 of the present invention.
In FIG. 13, reference numerals 100 to 250 are the same as those in FIG. In FIG. 13, the mobile terminal 100 calculates the moving direction and the moving amount of the mobile terminal 100 based on the acceleration measured from the acceleration sensor 111 and the angular velocity measured from the gyro sensor 112, and estimates the self position. An estimation complementing unit 183 is provided.

  As shown in FIG. 13, the fourth embodiment estimates the moving direction and the moving amount from the measured values of the sensors mounted on the mobile terminal 100, as shown in FIG. A part 183 is added.

Hereinafter, the operation of the fourth embodiment will be described with reference to FIG.
FIG. 14 is a flow chart of FIG. 8 according to the first embodiment, which calculates the moving direction and moving amount of the mobile terminal 100 based on the acceleration measured from the acceleration sensor 111 and the angular velocity measured from the gyro sensor 112. A process for estimating the position is added.
Steps ST3001 to ST3005 in FIG. 14 are the same processes as those in FIG. In FIG. 14, steps ST3301 and ST3302 are added as processing when identification information cannot be extracted in step ST3003.

In FIG. 14, it is determined whether or not the identification information has been extracted following the process of step ST3102 (step ST3003). If the identification information cannot be extracted, the self-position estimation complementing unit 183 receives the acceleration and gyro sensor 112 from the acceleration sensor 111. Is measured, and the moving direction and moving amount of the mobile terminal 100 are calculated (step ST3301).
Next, self-position estimation complementing section 183 estimates the self-position from the calculated movement direction and movement amount based on the self-position before movement (step ST3302). The self-position estimation complementing unit 183 displays the estimated self-position on the map of the area 1303 on the screen 1301 and also displays accurate latitude and longitude in the area 1304.

  According to the fourth embodiment, the direction of movement and the amount of movement are calculated from the measured values of the sensors mounted on the mobile terminal, and the self-position is estimated, so that there is less identification information around and the self-position is estimated from the identification information. Even when difficult locations continue, self-position estimation can be continued.

Embodiment 5. FIG.
FIG. 15 is a block diagram showing a self-position estimation system according to Embodiment 5 of the present invention.
In FIG. 15, 100 to 140 and 200 to 240 are the same as those in FIG. In FIG. 15, the mobile terminal 100 is provided with a self-position estimation requesting unit 184 that requests the server 200 for self-position estimation. Further, the server 200 compares the identification information extracted from the image with the identification information held in the identification information DB 240 (identification information database), and associates the identification information with the identification information comparison unit 260 that associates the identification information. A terminal position estimation unit 270 that estimates the position of the mobile terminal 100 from the information is provided.

  In the fifth embodiment, as shown in FIG. 15, the identification information comparison unit and the self-position estimation unit are moved from the mobile terminal 100 to the server 200 as compared with the first embodiment.

Hereinafter, the operation of the fifth embodiment will be described with reference to FIG.
FIG. 16 differs from the flowchart of FIG. 8 of the first embodiment in that the identification information comparison process after extracting the identification information of the current location from the camera image and the self-position estimation process are executed by the server 200. It is a thing.
Step ST3001 and step ST3002 in FIG. 16 are the same processes as those in FIG. In FIG. 16, steps ST3401 to ST3405 are added as the next processing after step ST3002. Among these, the processing of step ST3402 to step ST3405 is processing of the server 200.

In FIG. 16, self-position estimation requesting section 184 of mobile terminal 100 requests self-position estimation from terminal position estimating section 270 of server 200 (step ST3401). At this time, the self-position estimation request unit 184 sends the identification information extracted by the in-image identification information extraction unit 140 to the terminal position estimation unit 270 of the server 200.
Terminal position estimation section 270 of server 200 requests identification information comparison section 260 to compare identification information (step ST3402).
The identification information comparison unit 260 compares the identification information (designated identification information) sent from the mobile terminal 100 with the identification information in the identification information DB 240, and the identification information DB 240 corresponding to the identification information sent from the mobile terminal 100. Is identified (step ST3403).

Terminal position estimation section 270 estimates the position of mobile terminal 100 from the identified identification information and the position information in identification information DB 240 (step ST3404).
Next, terminal position estimation section 270 transmits the estimation result to self position estimation request section 184 (step ST3405). The estimation result to be transmitted includes screen display information to be displayed on the screen display unit 130. The self-position estimation requesting unit 184 displays the estimated self-position on the map of the area 1303 on the screen 1301 and also displays accurate latitude and longitude in the area 1304.

According to the fifth embodiment, by performing identification information comparison processing on the server, it becomes unnecessary to transmit the identification information from the server identification information DB to the mobile terminal in advance, thereby reducing the work time. There is.
Moreover, there is an effect that the self-position estimation time can be shortened by performing the identification information comparison process with a server having a high calculation capability.

  It should be noted that the present invention can be freely combined with each other within the scope of the invention, and each embodiment can be appropriately modified or omitted.

11 CPU, 12 ROM, 13 RAM, 14 Flash memory, 15 Display device,
21 CPU, 22 ROM, 23 RAM, 24 HDD,
100 mobile terminals, 110 cameras, 111 acceleration sensors,
112 gyro sensor, 120 image capturing unit, 130 screen display unit,
140 in-image identification information extraction unit, 150 identification information acquisition unit, 160 identification information DB,
170 identification information comparison unit, 180 self-position estimation unit, 181 self-position estimation complement unit,
182 Self-position estimation complement part, 183 Self-position estimation complement part,
184 Self-position estimation requesting unit, 190 GPS receiving unit, 200 server,
210 3D model generator, 220 3D model DB,
230 3D model identification information extraction unit, 240 identification information DB, 250 identification information transmission unit, 260 identification information comparison unit, 270 terminal position estimation unit

Claims (11)

An image capture unit that captures images with the camera,
An identification information extraction unit for extracting identification information in an image photographed by the image photographing unit;
An identification information acquisition unit for acquiring the identification information from a server having identification information including position information created in advance;
An identification information comparison unit that compares the identification information extracted by the identification information extraction unit with the identification information acquired from the server by the identification information acquisition unit, and associates both identification information;
And a mobile terminal comprising a self-position estimating unit that estimates the self-position based on the identification information associated with the identification information comparing unit.   The mobile terminal according to claim 1, wherein the identification information is a character string.   A self-position estimation complementing unit for recognizing a manhole on the road from the image photographed by the image photographing unit and estimating the self-position based on the position of the manhole to complement the estimation by the self-position estimation unit The mobile terminal according to claim 1, wherein the mobile terminal is a mobile terminal.   By calculating the moving direction and moving amount of the mobile terminal from the difference between the image captured immediately before by the image capturing unit and the current image, and estimating the self position based on the moving direction and moving amount, the self position The mobile terminal according to claim 1, further comprising a self-position estimation complementing unit that complements estimation by the estimation unit.   The estimation by the self-position estimation unit is complemented by estimating the movement direction and movement amount of the mobile terminal from the measured values of the sensors mounted on the mobile terminal, and estimating the self-position based on the movement direction and movement amount. The mobile terminal according to claim 1, further comprising a self-position estimation complementing unit.   A display unit that displays an image captured by the image capturing unit and highlights the identification information on the displayed image when the identification information comparison unit associates the identification information with each other. The mobile terminal according to any one of claims 1 to 5, wherein the mobile terminal is configured as described above.   The mobile terminal according to claim 6, wherein the display unit displays the self-position estimated by the self-position estimation unit in a screen area different from the display area of the image. The mobile terminal according to any one of claims 1 to 7,
And a self-position estimation system comprising a server having an identification information database storing identification information including position information created in advance. The server
A 3D model generation unit that generates a 3D model from separately measured 3D point cloud data;
A 3D model identification information extraction unit that extracts identification information including position information from the 3D model generated by the 3D model generation unit;
9. The self-position estimation system according to claim 8, wherein the identification information database stores identification information extracted by the 3D model identification information extraction unit. A 3D model generation unit that generates a 3D model from separately measured 3D point cloud data;
A 3D model identification information extraction unit that extracts identification information including position information from the 3D model generated by the 3D model generation unit;
An identification information database storing identification information extracted by the 3D model identification information extraction unit;
An identification information comparison unit that compares the identification information transmitted from the mobile terminal with the identification information stored in the identification information database, and associates both identification information;
And a server comprising a terminal position estimation unit that estimates the position of the mobile terminal based on the identification information associated by the identification information comparison unit. A first step in which the image capturing unit captures an image with the camera;
A second step in which the identification information extraction unit extracts the identification information in the image photographed in the first step;
A third step of acquiring the identification information by an identification information acquisition unit from a server having identification information including position information created in advance;
A fourth step in which the identification information comparison unit compares the identification information extracted in the second step with the identification information acquired from the server in the third step, and associates the identification information with each other;
And a self-position estimating method including a fifth step of estimating the self-position based on the identification information associated in the fourth step.

Images