WO2019234936A1 - Mobile terminal, camera position estimation system, camera position estimation method, and signboard - Google Patents

Abstract

The present invention makes it possible to accurately estimate, with a simple configuration, the current position and orientation of a user even in a location where radio waves from a satellite, etc., do not reach. This mobile terminal 100 comprises a camera 121 and a processing unit for processing an image acquired by the camera 121. The processing unit comprises an image acquisition unit 171 for using the camera 121 to acquire a sign image that is an image including two sign points that are at different positions removed from the mobile terminal 100 and have known position information, and a calculation unit 172 for analyzing the acquired sign image, calculating the planar orientation and interior angles of a first triangle formed by the two sign points and a terminal position that is the position of the mobile terminal 100, and using the planar orientation and interior angles to calculate the terminal position.

Description

Portable terminal, camera position estimation system, camera position estimation method, and sign board

The present invention relates to position and orientation information estimation technology, and more particularly, to position and orientation information estimation technology in places where radio waves from artificial satellites and the like do not reach.

There is a technology to acquire (estimate) current position information and direction information even in places where radio waves from artificial satellites do not reach. For example, in Patent Document 1, “a plurality of position information display devices for installing latitude / longitude information in a plurality of buildings, and a position information reading device for reading latitude / longitude information described in the position information display device; The current position information is obtained by reading the latitude / longitude information from the position information display device closest to the current position using the position information reading device.The direction information display device installed in the vicinity where the latitude / longitude information is installed , And obtains azimuth information by reading the azimuth information from the azimuth information display device closest to the current position (summary excerpt).

Japanese Unexamined Patent Publication No. 2000-205888

What is obtained by the technique disclosed in Patent Document 1 is only position information of a position information display board installed in advance, and position information of a place without a position information display board cannot be obtained. That is, the current position information of the user cannot be obtained. For example, in a large-scale underground shopping center or shopping center, an error is likely to occur when searching for a destination such as a store by grasping the current position and direction.

The present invention has been made in view of the above circumstances, and provides a technique for accurately estimating a user's current position and direction with a simple configuration even in a place where radio waves from artificial satellites or the like do not reach. Objective.

The present invention is a portable terminal comprising a camera and a processing unit that processes an image acquired by the camera, wherein the processing unit is provided with two marker points at different positions away from the portable terminal by the camera. An image acquisition unit that acquires a sign image that is an image including a sign point whose position information is known, and the acquired sign image is analyzed, and a terminal position that is the position of the mobile terminal and the two sign points A calculation unit that calculates a plane orientation of the first triangle and an interior angle of the first triangle formed by calculating the terminal position using the plane orientation and the interior angle. To do.

In addition, the present invention is a camera position estimation system that includes a camera and a processing device that processes an image acquired by the camera, and estimates the camera position that is the position of the camera by analyzing the image. The camera acquires a sign image that is an image including two sign points at different positions away from the camera and includes a sign point whose position information is known, and the processing device acquires the sign image acquired by the camera. Analyzing the sign image, calculating the plane orientation of the first triangle formed by the camera position and the two sign points and the interior angle of the first triangle, and using the plane orientation and the interior angle The camera position is calculated.

Furthermore, the present invention provides a camera position estimation method for estimating a camera position that is a position of the camera by analyzing the image in a system including a camera and a processing device that processes an image acquired by the camera. The camera acquires a marker image that is an image including two marker points at different positions away from the camera and includes a marker point whose position information is known, and the marker image acquired by the camera is obtained. Analyzing, calculating the plane orientation of the first triangle formed by the camera position and the two marker points and the interior angle of the first triangle, and using the plane orientation and the interior angle, the camera position Is calculated.

Even in places where radio waves from artificial satellites do not reach, the current position and direction of the user can be estimated accurately with a simple configuration. Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

It is explanatory drawing for demonstrating the outline | summary of the process of 1st embodiment. It is explanatory drawing for demonstrating an example of the marker board of 1st embodiment. (A) is a hardware block diagram of the portable terminal of 1st embodiment, (b) is a functional block diagram of the terminal position estimation part of the portable terminal of 1st embodiment. (A) is explanatory drawing for demonstrating the mode at the time of terminal position estimation of 1st embodiment, (b) is explanatory drawing for demonstrating the position calculation method by a general triangulation. . (A)-(d) is explanatory drawing for demonstrating the calculation method of the angle which the portable terminal and signboard surface of 1st embodiment each comprise. It is explanatory drawing for demonstrating the terminal position estimation method in the position of 1st embodiment. It is a flowchart of the terminal position estimation process of 1st embodiment. (A) And (b) is explanatory drawing for demonstrating the modification of the marker board of 1st embodiment. (A) And (b) is explanatory drawing for demonstrating the terminal position estimation method of the modification of 1st embodiment. It is explanatory drawing for demonstrating the terminal position estimation method of the modification of 1st embodiment. (A) And (b) is explanatory drawing for demonstrating the service and example of a display screen which are provided by the modification of 1st embodiment. It is explanatory drawing for demonstrating an example of the sign board of 2nd embodiment. (A)-(c) is explanatory drawing for demonstrating the terminal position estimation method of 2nd embodiment. (A) And (b) is explanatory drawing for demonstrating the terminal position estimation method of 2nd embodiment. (A)-(c) is explanatory drawing for demonstrating the terminal position estimation method of 2nd embodiment. (A) And (b) is explanatory drawing for demonstrating the terminal position estimation method of 2nd embodiment. (A) And (b) is explanatory drawing in the case of displaying the three-dimensional position of a target point. It is explanatory drawing for demonstrating an example of the marker point of 3rd embodiment. (A)-(c) is explanatory drawing for demonstrating the terminal position estimation method of 3rd embodiment. (A) And (b) is explanatory drawing for demonstrating the terminal position estimation method of 3rd embodiment. (A)-(c) is explanatory drawing for demonstrating the terminal position estimation method of the modification of 3rd embodiment. (A)-(d) is explanatory drawing for demonstrating an example of the positional relationship of the marker point of the modification of 3rd embodiment, and an additional marker point, respectively. (A) And (b) is explanatory drawing for demonstrating the terminal position estimation method of the modification of 3rd embodiment. (A) And (b) is explanatory drawing for demonstrating the terminal position estimation method of the modification of 3rd embodiment. It is explanatory drawing for demonstrating the terminal position estimation method of the modification of 3rd embodiment. (A)-(c) is explanatory drawing for demonstrating the terminal position estimation method of the modification of 3rd embodiment. It is explanatory drawing for demonstrating the terminal position estimation method of the modification of 3rd embodiment. (A) And (b) is explanatory drawing for demonstrating the terminal position estimation method of the modification of 3rd embodiment. (A) And (b) is explanatory drawing for demonstrating the terminal position estimation method of the modification of 3rd embodiment. (A) And (b) is explanatory drawing for demonstrating the estimated position data of the application example of 3rd embodiment. It is a flowchart of the terminal position estimation process of the application example of 3rd embodiment. It is explanatory drawing for demonstrating the estimated position log | history of the application example of 3rd embodiment. It is a flowchart of the terminal position estimation process of the application example of 3rd embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. Hereinafter, in this specification, those having the same function are denoted by the same reference numerals unless otherwise specified, and repeated description is omitted.

<< First Embodiment >>
In the present embodiment, a landmark whose position information is known and is located away from the camera is photographed by the camera, and the position of the camera is calculated (estimated) by image processing.

Hereinafter, in the present embodiment, a landmark whose position information is known is a sign board attached or installed in advance on the surface of an object such as a pillar or a wall. It is assumed that position information of the sign board and direction information of the sign board surface are registered in the sign board. In the present embodiment, the sign plate has a shape in which an angle formed by a line segment connecting the camera position and the sign plate center and the sign plate surface can be grasped.

In this embodiment, two sign boards are photographed, the position information of each sign board, the angle formed by the line connecting each sign board surface, the camera position and the sign board center, and the orientation information of each sign board surface Then, the position information of the camera position is calculated.

FIG. 1 is a diagram for explaining an outline of processing according to the present embodiment. As shown in the figure, the owner 910 holds a portable information processing apparatus (hereinafter referred to as a portable terminal) 100 having a photographing function (camera) and photographs the sign board 200.

Note that the mobile terminal 100 can be connected to the server 960 via a preset access point (AP) 970 and a network 940 or via a base station 950 of a mobile phone company.

[Signboard]
Details of the sign board 200 will be described with reference to FIG. As shown in this figure, the sign board 200 of this embodiment has, for example, a square shape with one side LS.

Further, as shown in FIG. 2, the sign board 200 is capable of photographing position information and azimuth information (hereinafter collectively referred to as position azimuth information), and analyzing the photographed image. It has a position / orientation information display area 201 displayed in a readable manner. Alternatively, the position / orientation information may be described in a QR code (registered trademark). In the example of this figure, the position / orientation information is described on the sign surface of the sign plate 200.

The position information is, for example, the latitude and longitude of the center point of the sign board 200. The azimuth information is indicated by, for example, an angle formed by a horizontal arrow 202 (hereinafter referred to as a directional arrow 202) on the sign surface of the sign board 200 and a predetermined reference direction. Here, it is assumed that the angle formed by the direction of the azimuth arrow 202 and the north direction is registered.

The owner 910 can visually check the description in the position / orientation information display area 201. The sign board 200 may be photographed by the camera of the portable terminal 100 and acquired by image processing.

Note that the position / orientation information may not be clearly indicated on the sign board 200. For example, the URL of the information acquisition destination may be described in a QR code (registered trademark) or the like, and the position and orientation information of the sign board 200 may be acquired from a server or the like by photographing with a camera.

[Configuration of portable device]
Next, the configuration of the mobile terminal 100 will be described. The mobile terminal 100 is an information processing apparatus having a shooting function and an information processing function. For example, a mobile phone, a smartphone, a tablet terminal, a wearable terminal such as a watch or a head-mounted display, a feature phone, or other portable digital device. Further, an autonomous device such as a drone may be used.

FIG. 3A is a hardware configuration diagram of the mobile terminal 100 of the present embodiment. As shown in the figure, the mobile terminal 100 includes a CPU (Central Processing Unit) 101, a storage device 110, a photographing device 120, a user interface (I / F) 130, a sensor device 140, and a communication device 150. And an expansion I / F 160. Each device is connected to the CPU 101 via the bus 102.

The CPU 101 is a microprocessor unit that controls the entire mobile terminal 100. A bus 102 is a data communication path for performing data transmission / reception between the CPU 101 and each device of the portable terminal 100.

The storage device 110 includes a ROM (Read Only Memory) 111, a RAM (Random Access Memory) 112, and a storage 113.

The ROM 111 is a memory in which a basic operation program such as an operating system and other operation programs are stored. As the ROM 111, for example, a rewritable ROM such as an EEPROM (Electrically Erasable and Programmable Read Only Memory) or a flash ROM is used.

The storage 113 stores the operation program and operation setting value of the mobile terminal 100 and various programs and various data necessary for realizing each function of the present embodiment.

The storage 113 holds information stored in the portable terminal 100 even when external power is not supplied. For example, a device such as a flash ROM, SSD (Solid State Drive), or HDD (Hard Disk Drive) is used as the storage 113.

RAM 112 is a work area for executing basic operation programs and other operation programs.

The ROM 111 and the RAM 112 may be integrated with the CPU 101. Further, the ROM 111 may not use an independent configuration as shown in FIG. 3A but may use a partial storage area in the storage 113. That is, all or part of the functions of the ROM 111 may be replaced by a partial area of the storage 113.

It should be noted that each operation program stored in the ROM 111 or the storage 113 can be updated and function expanded by, for example, a download process from each distribution server on the network.

The photographing apparatus 120 includes a camera 121, an image processor 122, and an image memory 123.

The camera 121 acquires surroundings and objects as image data by converting light input from the lens into an electrical signal using an image sensor such as a CCD (Charge-Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) sensor. To do.

The image processor 122 performs format conversion, menu and other OSD (On-Screen Display) signal superimposing processing on the image data acquired by the camera 121 as necessary. The image processor 122 includes a video RAM (not shown), and drives a display 131 (to be described later) based on image data input to the video RAM. The image processor 122 includes, for example, a codec unit that compresses / decompresses a captured image or video, an image quality improvement processing unit that improves image quality of an image or video, an angle correction or rotation correction from a captured image, a QR code, or the like The function of an image processing unit or the like for recognizing information and correcting the information is realized.

The image memory 123 temporarily stores image data acquired by the camera 121 or image data processed by the image processor 122.

The user I / F 130 includes a display 131, for example. The display 131 is a display device such as a liquid crystal panel, for example, and displays a processing result by each unit of the mobile terminal 100 as a display unit. In addition, a touch panel function may be provided in which operation devices functioning as reception units that receive input of operation instructions are stacked.

Note that the function of the operation device may be realized by a keyboard, a mouse, or the like connected via an expansion I / F 160 described later.

The sensor device 140 is a sensor group for detecting the state of the mobile terminal 100. In the present embodiment, for example, a GPS (Global Positioning System) receiver 141, a three-axis gyro sensor 142, and a three-axis acceleration sensor 143 are provided. In addition, a ranging sensor, a geomagnetic sensor, an illuminance sensor, a proximity sensor, a biological information sensor, an atmospheric pressure sensor, and the like may be provided.

These sensors detect the position, tilt, direction, movement, etc. of the mobile terminal 100. If the current position is a position where GPS radio waves can be acquired, the position information is acquired by the GPS receiver 141.

The communication device 150 performs communication between the mobile terminal 100 and an external device. In the present embodiment, for example, a LAN (Local Area Network) communication device 151, a telephone network communication device 152, and a short-range communication device 153 are provided.

The LAN communication device 151 is connected to the network via an access point (AP) device by wireless connection such as Wi-Fi (registered trademark), and transmits / receives data to / from other devices on the network.

The telephone network communicator 152 performs a call and data transmission / reception by wireless communication with a base station of the mobile telephone communication network.

The short-range communication device 153 transmits and receives data to and from other devices in the vicinity of the mobile terminal 100 by a wired connection means such as USB (Universal Serial Bus). In addition, data is transmitted and received by wireless communication with other devices including a short-range communication device. The short-range communication device 153 is, for example, an I / F of short-range wireless communication (NFC (Near Field Communication)), and is an extremely short distance of about several centimeters to about 10 centimeters, and between devices equipped with NFC chips. Two-way communication may be realized. For example, it corresponds to a service using a non-contact IC chip such as electronic money mounted on the mobile terminal 100. Further, the short-range communication device 153 may transmit and receive data by wireless communication with other devices including the wireless communication device. For example, Bluetooth (registered trademark) or the like realizes simple exchange of information using radio waves between information devices with a distance of several meters to several tens of meters.

The LAN communication device 151, the telephone network communication device 152, and the short-range communication device 153 each include a coding circuit, a decoding circuit, an antenna, and the like. The communication device 150 may further include a communication device that realizes infrared communication and other communication devices.

The extension I / F 160 is an interface group for extending the functions of the mobile terminal 100. In the present embodiment, a video / audio I / F, an operation device I / F, a memory I / F, and the like are provided. The video / audio I / F performs input of video signals / audio signals from external video / audio output devices, output of video signals / audio signals to external video / audio input devices, and the like. External operation devices such as a keyboard are connected via an operation device I / F. The memory I / F transmits and receives data by connecting a memory card and other memory media.

Note that the configuration example of the mobile terminal 100 illustrated in FIG. 3A focuses on the configuration essential to the present embodiment. The mobile terminal 100 may be further added with a configuration (not shown) such as a digital broadcast receiving function and an electronic money settlement function in addition to these configurations.

Next, functions of the mobile terminal 100 will be described. Here, the configuration related to the terminal position estimation function (terminal position estimation unit) for calculating the current position information of the mobile terminal 100 according to the present embodiment will be mainly described. As illustrated in FIG. 3B, the mobile terminal 100 according to the present embodiment includes an image acquisition unit 171, a calculation unit 172, a display control unit 173, and a data storage unit 174 as the terminal position estimation unit 170. Prepare.

The image acquisition unit 171 controls the photographing device 120 to acquire an image. In the present embodiment, the acquired image is further subjected to processing such as image compression / decompression processing and image improvement. In this embodiment, an image of the sign board 200 is acquired. More specifically, two different sign boards 200 are acquired.

The calculation unit 172 analyzes the image acquired by the image acquisition unit 171 and calculates the current position of the mobile terminal 100.

For example, the calculation unit 172 analyzes the contents of the sign board 200 using various correction functions and character recognition techniques, specifies the direction and angle of the sign board 200 from the photographed shape of the sign board 200, and converts it into data. Or

The display control unit 173 controls display on the display 131.

The data storage unit 174 stores data necessary for processing, a processing halfway and processing result, and generated data.

In addition, in the sign board 200, when the position / orientation information is registered with a QR code, a QR code analysis unit 175 may be further provided. The QR code analysis unit 175 analyzes the content of the QR code photographed by the photographing device 120. The analysis result may be displayed on the display 131 by the display control unit 173.

In addition, each part of the terminal location estimation part 170 except the data storage part 174 is implement | achieved when CPU101 expand | deploys and executes the program stored in the storage 113 or ROM111 on RAM112. However, not all of the software may be software, and for example, a part of the software may be hardware for speeding up. The data storage unit 174 is constructed in the storage device 110.

[Terminal location estimation]
Next, terminal position estimation processing by the terminal position estimation unit 170 of this embodiment will be described. Here, a method in which the information of the two different signboards 200 acquired by the image acquisition unit 171 is analyzed and the calculation unit 172 calculates the current position of the owner 910 (terminal position estimation) is shown in FIG. ) To FIG. In the present embodiment, the current position is calculated as the position of the mobile terminal 100 held by the owner 910. Hereinafter, in the present embodiment, the calculated current position is referred to as a terminal position.

Here, as shown in FIG. 4A, the case where the owner 910 uses the camera 121 of the portable terminal 100 held by the owner 910 to photograph the sign plates 211 and 212 will be described as an example. The portable terminal 100 analyzes the photographing result and calculates the terminal position.

Hereinafter, the center position of the sign board 211 is point A, the center position of the sign board 212 is point B, and the terminal position is point C. In the present embodiment, it is assumed that the points A, B, and C are on the same horizontal plane. In addition, when it is not on the same horizontal plane, an error occurs in the estimated value of the terminal position. However, this method can be used as long as the error is allowed. Hereinafter, this plane is referred to as x, y plane, the coordinates of point A are (xa, ya), the coordinates of point B are (xb, yb), and the coordinates of point C are (xc, yc). In the present embodiment, by photographing the sign board 211 and the sign board 212, an angle α1 formed by the point C and the sign surface of the sign board 211 and an angle β1 formed by the point C and the sign surface of the sign board 212 are obtained. Measurement is performed to obtain the coordinates of the point C. Note that the coordinates of the points A and B are actually represented by latitude and longitude, but are converted into the coordinates on the x and y planes described above for calculation.

Here, the terminal position C is calculated by triangulation. Triangulation is a survey method using trigonometry and geometry that calculates the position of a point to be measured by measuring the angle from a known point at both ends of a certain baseline to the point to be measured.

算出 Briefly explain the method for calculating the position of the point you want to measure by triangulation. Here, the position calculation method by triangulation will be briefly described with reference to FIG. As shown in this figure, in triangulation, a line segment CA and a line segment AB are formed in a triangle formed by a point C whose position is to be obtained and two points (points A and B) whose positions are known. The position of the point C is calculated by measuring the angle and the angles α and β formed by the line segment BC and the line segment AB.

In FIG. 4B, the coordinate system in which the direction connecting point A and point B is the x-axis direction is considered. Here, the length of the line segment AB is L, the distance between the point C and the line segment AB is d, the angle formed by the line segment CA and the line segment AB, and the both end angles which are the angles formed by the line segment BC and the line segment AB, Let α and β be respectively.

If the coordinates of the point A are A (xa, ya), the coordinates of the point B are B (xa + L, ya). And the coordinate (xc, yc) of the point C is represented by the following formula | equation.
xc = xa + d / tanα (1)
yc = ya−d (2)
Here, d and L are calculated by the following equations.
d = L / (1 / tan α + 1 / tan β) (3)
= L · sin α · sin β / sin (α + β) (4)

By substituting this d into the coordinates of the point C, the coordinates xc and yc of the point C are expressed using L, α and β, respectively.

In the present embodiment, as described above, the image acquisition unit 171 acquires the images (label images) 221 and 222 of the two different sign plates 211 and 212, respectively.

The calculation unit 172 analyzes the sign images 221 and 222 acquired by the image acquisition unit 171 and calculates the position information and both end angle information of the sign plates 211 and 212. Note that both end angle information is the end points of the side connecting the sign plates 211 and 212 of the triangle having the center point of the sign plate 211, the center point of the sign plate 212 and the terminal position as vertices, that is, the above-described angles α and β. Is an angle corresponding to.

Hereinafter, in the present embodiment, on the horizontal plane including the sign board 200 and the mobile terminal 100, the intersection line of the sign face of the sign board 200 and the horizontal plane and the line connecting the mobile terminal 100 and the center point of each sign face are The angle formed is simply referred to as the angle formed between the sign surface and the mobile terminal 100.

The position information of the sign plates 211 and 212 is obtained by reading the position and orientation information from the sign images 221 and 222.

For example, when the position / orientation information is described in letters and numbers, it is obtained by encoding with an OCR (Optical Character Recognition) function. Further, when the position / orientation information is described in a QR code or the like, the position / orientation information is acquired by reading the QR code.

The angles α1 and β1 are calculated using the sizes of the sign images 221 and 222 and the direction information of the position and direction information. Specifically, the ratio between the length of the vertical side and the length of the horizontal side of the sign image 221 and the azimuth information are used.

Here, as shown in FIG. 5A, the sign board 200 is a square having a side length of LS0. As shown in FIG. 5B, on the horizontal plane including the sign board 200 and the portable terminal 100, when the angle formed by the sign face of the sign board 200 and the portable terminal 100 is α1, FIG. As shown, the ratio between the vertical side LS1 and the horizontal side LS2 of the sign image 221 captured by the mobile terminal 100 is sin α1. When the angle α1 formed by the sign surface and the mobile terminal 100 is 90 °, sin (90 °) = 1 as shown in FIG.

For example, as shown in FIG. 6, the angle between the mobile terminal 100 and the sign surface of the sign board 211 is α1, and the angle between the mobile terminal 100 and the sign face of the sign plate 212 is β1.

Here, α2 is registered as the direction of the arrow indicating the azimuth on the sign plate 211 (the angle formed by the arrow on the sign surface to the north), and β2 is registered as the direction of the arrow indicating the azimuth of the sign plate 212. It shall be.

Furthermore, an angle between the direction from the sign plate 211 toward the sign plate 212 and the east-west direction is γ2. Γ2 is calculated from the latitude and longitude information of the sign board 211 and the sign board 212. That is, it is calculated by the following formula.
L = √ {(xa−xb) ² + (ya−yb) ² } (5)
tan (γ2) = (yb−ya) / (xb−xa) (6)

In this case, as shown in FIG. 6, there is the following relationship among α, α1, α2, and γ2.
α1 + α + (90 degrees−α2) + γ2 = 180 degrees (7)
Therefore,
α = 90 degrees -α1 + α2-γ2 (8)
Further, there is the following relationship among β, β1, β2, and γ2.
β1 + β + (90 degrees−γ2−β2) = 180 degrees (9)
Therefore,
β = 90 degrees−β1 + β2 + γ2 (10)

As described above, the position information (xa, ya) and the direction information α2 of the point A, the position information (xb, yb) and the direction information β2 of the point B, the angle α1 formed by the sign surface of the sign plate 211 and the portable terminal 100. If the angle β1 formed by the sign surface of the sign plate 212 and the portable terminal 100 is known, the distance L between the point A and the point B, the angle α formed by the line segment CA and the line segment AB, the line segment CB, and the line The angle β formed by the minute AB can be calculated. Therefore, position information (xc, yc) of the point C can be obtained.

Next, the flow of terminal location estimation processing by the image acquisition unit 171 and the calculation unit 172 of this embodiment will be described. FIG. 7 is a process flow of the terminal position estimation process of the present embodiment. This process is started by an instruction from the owner 910. For example, when the mobile terminal 100 includes a plurality of modes, the mobile terminal 100 may be started when an instruction to shift to a mode for executing the terminal position estimation process of the present embodiment is received.

Note that the owner 910 confirms that two different marker plates 211 and 212 are in a position where they can be photographed, and starts this processing.

The image acquisition unit 171 acquires the sign image 221 of the first sign plate 211 according to the instruction from the owner 910 (step S1101).

The calculation unit 172 analyzes the sign image 221 and obtains the position and orientation information of the first sign plate 211 (step S1102). Here, an area corresponding to the position / orientation information display area 201 on the sign image 221 is subjected to image processing, and the position information of the center position of the first sign board 211 and the direction of the direction arrow 202 are acquired.

Further, the calculation unit 172 analyzes the sign image 221 and calculates an angle α1 formed by the sign surface and the mobile terminal 100 (step S1103).

Next, the image acquisition unit 171 acquires the sign image 222 of the second sign plate 212 in accordance with the instruction from the owner 910 (step S1104).

The calculation unit 172 analyzes the sign image 222 and acquires the position / orientation information of the second sign plate 212 (step S1105). Here, the area corresponding to the position / orientation information display area 201 on the sign image 222 is subjected to image processing, and the position information of the center position of the second sign plate 212 and the direction of the direction arrow 202 are acquired.

Further, the calculation unit 172 analyzes the sign image 222 and calculates an angle β1 formed by the sign surface and the mobile terminal 100 (step S1106).

The calculation unit 172 calculates the terminal position, which is the current position of the mobile terminal 100, using the above method, using the position and orientation information and angle of the first sign board and the position and orientation information and angle of the second sign board. (Step S1107).

The calculation unit 172 stores the calculation result in the data storage unit 174, and the display control unit 173 displays the calculation result on the display 131 (step S1108) and ends the process.

As described above, in the present embodiment, a sign image that is an image including two sign points that are separated from the mobile terminal 100 and have different positions and including a sign point with known position information is acquired, and the obtained sign Analyzing the image, calculating the plane orientation of the first triangle formed by the terminal position C, which is the position of the mobile terminal 100, and the two sign points, and the interior angle of the first triangle, the plane orientation and the interior angle The terminal position C is calculated using At this time, in this embodiment, two different sign boards 200 in which the terminal position C and the center point are on the same horizontal plane are photographed as the sign images. And let the center point of each marker board 200 be two marker points. The sign board 200 has a position and orientation information display area 201 that can be read by analyzing the sign image of the position information of the center point and the direction of the sign face with respect to the reference direction, respectively. Prepare for. In addition, a line segment connecting the camera 121 and the center point due to the distortion of the sign image has a shape capable of measuring the angle formed with the normal line of the sign surface. In this embodiment, a calculation part calculates the both-ends angle | corner of the line segment which connects two marker points among the interior angles of a 1st triangle from a marker image. Then, the terminal position C is calculated by triangulation.

As described above, according to the present embodiment, the portable terminal 100 can calculate its own position only by photographing the two sign boards 200. Therefore, according to the present embodiment, an accurate position can be acquired even when the GPS function cannot be used, such as in an underground shopping center or a large-scale shopping center.

Further, a so-called QR code that can access information such as advertisements may be described on the sign board 200 in addition to the position information. In this case, the mobile terminal 100 can also read this QR code when acquiring an image to estimate position information. Thereby, information that can be acquired based on the QR code can also be displayed.

The capacity of a general QR code is a maximum of 7,089 characters in numbers and a maximum of 1,817 characters in kanji / kana, providing a sufficient amount of information. Therefore, with this configuration, the terminal position X is estimated, and at the same time, the name and position (longitude / latitude / height) of the store in the underground shopping mall or large-scale shopping center are displayed on the detailed map. Information on products, events, etc. can be provided. Further, nearby sign board position information may be described.

<Variation 1 of the first embodiment>
In the above embodiment, the case where the sign plate 200 is a square has been described as an example. However, the reason why the sign plate 200 is a square is to simplify the calculation of the angle between the terminal position and the sign surface.

However, the sign board 200 has a line connecting the position of the mobile terminal 100 including the camera 121 that has photographed the sign board 200 and the center of the sign board 200 due to distortion of the photographed image of the sign board 200 on the face of the sign board 200. It is only necessary to have an angle measurement configuration capable of measuring an angle formed with a normal line of a certain marking surface.

For example, the shape of the sign plate 200 may be any shape that can calculate the angle between the camera position and the sign surface using the photographed sign image. For example, it may be a rectangle with a known ratio of the long side to the short side, a circle, or an ellipse with a known ratio of the major axis to the minor axis.

For example, FIG. 8A shows an example where the sign plate 200a is circular. For example, in the case of a circular shape, let f be the diameter of the circle. When the sign board 200 is viewed obliquely on the same horizontal plane, the height (vertical) length is f, but the horizontal (horizontal) length is shorter than f. This aspect ratio varies depending on the angle formed by the sign board 200 and the portable terminal 100.

For example, as shown in FIG. 5B, when the sign board 200 is photographed from the direction of the angle α1, the following formula is established.
sin α1 = f1 / f (11)
Note that f1 is the horizontal length when the vertical length is f.

Therefore, in this case, the calculation unit 172 analyzes the acquired sign image and calculates the angle α1 according to the above formula.

Further, the shape of the sign plate 200 itself is not limited. For example, as shown in FIG. 8B, reference points (points where marks indicated by crosses are present) are provided at four vertices of a virtual rectangle. 213 may be indicated. In this case, regarding the four reference points 213, the ratio between the length in the vertical direction and the length in the horizontal direction between the reference points 213 is assumed to be known.

Also, the shape of the sign plate 200 itself may not have the above characteristics. For example, what is necessary is just to provide the mark of the shape which has the said characteristic on the marking surface of the marking board 200. FIG. The mark is, for example, a rectangular frame or a circular frame.

<Modification 2 of the first embodiment>
Further, when calculating the terminal position, a focal length calculation function provided in the distance measuring sensor or the camera 121 may be used. The calculation method in this case will be described with reference to FIGS. In addition, when using the focal distance calculation function with which the camera 121 is provided, it is limited to the range in which the camera 121 can calculate a focal distance.

First, the calculation principle will be explained. Here, as in FIG. 4B, in the triangle ABC having three points, A, B, and C as vertices, the positions of the points A and B, the length (distance) LB of the side AC, The description will be made assuming that the position of the point C is calculated when the length (distance) LA of the side BC is known. In the present embodiment, the distance LB and the distance LA are acquired by the distance measuring sensor or the focal length calculation function of the camera 121.

Here, of the internal angles of the triangle ABC, the angle formed by the side CA and the side AB is α, and the angle formed by the side CB and the side AB is β. In this case, the coordinates (xc, yc) of the point C are represented by the following expressions.
xc = xa + LBcos α (12)
yc = √ {(LB ² − (LBcos α) ² )} (13)
Here, cos α can be calculated from the cosine theorem as follows.
cos α = (LB ² + L ² −LA ² ) / 2 · LB · L (14)

Here, FIG. 9B shows a case where the surfaces of the two sign plates 211 and 212 are not in a straight line. In this case, from the image acquired by the camera 121 at the point C, the angles α1 and β1 with the sign surface are calculated by the method of the above embodiment. Then, the internal angles α and β are calculated by the above formulas (8) and (10). Thereby, the position information of the point C is calculated.

When using the function of calculating the focal length of the distance measuring sensor or the camera 121, one terminal board 200 may be photographed to calculate the terminal position. A calculation method in this case will be described with reference to FIG.

As in the above embodiment, the angle between the sign surface and the portable terminal 100 obtained from the captured image is α1, and the direction is α2. Here, as shown in FIG. 10, assuming that the angle α3 formed by the line connecting the mobile terminal 100 and the center point of the sign surface and the east-west direction, the coordinates (xc, yc) of the point C are expressed by the following equations. The
xc = xa + LBcos α3 (15)
yc = ya−LBsin α3 (16)
In addition, between α1, α2 and α3,
α1 + α3 + (90 degrees−α2) = 180 degrees (17)
Therefore,
α3 = 90 degrees−α1 + α2 (18)

<Modification 3 of the first embodiment>
In addition, in the above description, the sign board 200 is described as a plate-like thing created individually. However, the sign board 200 is not limited to this, and may be printed on a part of an advertisement in a store, for example. Thereby, it is possible to renew the sign board 200 every time the advertisement is changed and to insert new information into the QR code.

Also, the position where the sign board 200 is installed does not always have to be the same place, and the place may be moved. Each time it moves, position and orientation information after movement is given.

In the above description, it is assumed that each sign board 200 is not known in advance. However, the present invention is not limited to this. For example, an entrance of an underground shopping center or a large-scale shopping center, each staircase, an elevator, The place where each sign board 200 is present may be displayed in the vicinity of the escalator or in a busy place. Alternatively, the information may be registered on the website. Alternatively, it may be installed in places where customers are relatively easy to see and check, such as the entrance of each store and each product corner.

<Modification 4 of the first embodiment>
In the above embodiment, the calculated terminal position is displayed on the display 131, but the calculation result processing by the calculation unit 172 is not limited to this. For example, you may output to the other application which requires the present position of the portable terminal 100. FIG.

For example, in a large-scale shopping center or home center, it is an application that guides a customer to a desired store location.

Application is stored in the ROM 111 or the storage 113 of the mobile terminal 100, for example. This is realized by loading the CPU 101 into the RAM and executing it in accordance with an instruction from the user. Hereinafter, a function executed by this application is referred to as a navigation unit (navigation unit) 176.

This method will be described with reference to FIGS. 11 (a) and 11 (b). It is assumed that the store information of each store is registered in advance and stored in the server 960 or the like. The store information includes a store name, a store description, and store location information. The position information is registered in latitude and longitude.

The navigation unit 176 accesses the server 960 in which the store information is registered via the network 940, and acquires the store information. Then, it is displayed on the display 131 in a predetermined display form.

Then, when the user selects a desired store, navigation information from the current location to the location of the store is calculated and displayed. Note that the terminal position calculated by the above method is used as the current position.

For example, in the example of FIG. 11A, the navigation unit 176 causes the display 131 to display a store on the map according to its position information. Further, the current position of the mobile terminal 100 is also displayed. For example, the mobile terminal 100 stores the map information in advance in the data storage unit 174 or acquires the map information from the server 960.

11A, the current position of the owner 910 of the mobile terminal 100 is the east corner of the area A21. Further, it is assumed that the store selected by the owner is in the area A13. The navigation unit 176 displays the route on this map.

At this time, as shown in FIG. 11 (b), navigation information from the current position to the desired store may be displayed on the display 131 in text. In this example, for example, information such as “the target store is located one block north and two blocks east” may be displayed.

In addition, according to this modification, it can be seen which direction the mobile terminal 100 is facing. Using the current location and orientation information of the mobile terminal 100 and the location information of the target store, the direction in which the owner 910 is facing at that time, such as “go forward”, “return backward”, etc. A navigation instruction based on a reference may be given.

Furthermore, each store may be provided with detailed product information, and when entering the store, the location of the product may be displayed. The merchandise information includes a merchandise name, a merchandise description, and an arrangement position of the merchandise. Then, a list of product names may be displayed, and when the user selects a desired product, navigation to the product may be performed similarly in the store.

The navigation unit 176 acquires display information from the server 960 via the access point 970 and the network 940 via the communication device 150, for example.

<Modification 5 of the first embodiment>
Note that the position information that is the basis for calculating the position information of the mobile terminal 100 is not limited to latitude and longitude. For example, the coordinate value of an original coordinate system such as each shopping center may be used. In this case, the position information of the product is also registered with coordinate values in the same coordinate system.

Further, the QR code for each store may be shown on the information center of the shopping center, and the mobile terminal 100 may read the QR code to perform navigation to the store.

In addition, the QR code of the handling product may be shown at the entrance of each store and the navigation may be performed in the same manner.

* While moving to the target location, the position of the owner 910 changes every moment. However, it is troublesome to always measure the above-mentioned position. Therefore, regarding the movement between them, for example, the movement distance and direction may be detected by the triaxial acceleration sensor 143, the triaxial gyro sensor 142, and the clock function, and the corrected value may be indicated in the map information.

Further, when it can be determined that the position estimation error by the internal sensor has increased, the position of the nearby sign board 200 is indicated to prompt the owner 910 to perform position measurement, or the portable terminal 100 automatically performs position measurement. It may be. For example, when the movement distance from the previous position measurement point has increased, or when it appears that it is moving on a location that is not a passage on the map, it is determined that the position estimation error by the internal sensor has increased.

<< Second Embodiment >>
Next, a second embodiment of the present invention will be described. In the first embodiment, the length of the line segment connecting the center points of the two sign boards 200 is determined on the sign board 200 from the ratio of the distortion of the sign shape between the horizontal plane direction and the direction orthogonal thereto in the sign image. The terminal position is calculated by calculating both end angles.

In this embodiment, a circle is drawn on the sign board 200, and information necessary for terminal position estimation is acquired from the degree of distortion of the circular shape on the captured image. In the present embodiment, unlike the first embodiment, the sign board 200 and the mobile terminal 100 may not be on the same horizontal plane. Moreover, the number of the sign board 200 which acquires an image is sufficient.

Also, in the present embodiment and the third embodiment, unlike the first embodiment, the position information is a three-dimensional amount including the height. Then, consider two line segments that are not on the same straight line with known positions, and at least two triangles formed from each line segment and terminal position, and calculate each angle and length of each side of those triangles. By doing so, the terminal position is estimated three-dimensionally. Since the two triangles share the terminal position, the plane orientation of the triangle is determined by calculating the respective triangle shapes, and the terminal position can be calculated.

Hereinafter, the present embodiment will be described focusing on the configuration different from the first embodiment.

[Signboard]
Prior to the description of the terminal position calculation method, first, the sign board 230 of the present embodiment will be described. As shown in FIG. 12, the sign plate 230 of this embodiment includes a sign circle 231 having a circular shape, an azimuth line 232, and a position / direction information display area 233.

In the position / orientation information display area 233, information (position / orientation information) necessary for photographing the sign board 230 and calculating terminal position information is recorded. For example, the position information of the center of the marker circle 231, the diameter of the marker circle 231, the azimuth line direction, and the surface normal direction are described.

For example, latitude, longitude, and height are registered as center position information. The azimuth line direction is a three-dimensional direction of the azimuth line 232, and is indicated by a unit vector component, for example. The surface normal direction is the three-dimensional direction of the normal of the sign surface of the sign plate 230, and is indicated by, for example, a component of the surface normal vector.

Note that, for example, an arrow, a triangle, or the like may be attached to the azimuth line 232 so that the orientation thereof can be easily grasped.

[Mobile devices]
The hardware configuration and functional blocks of the mobile terminal 100 of the present embodiment have basically the same configuration as that of the first embodiment. However, the calculation method by the calculation unit 172 is different.

[Position calculation method]
Next, a method in which the calculation unit 172 captures the sign board 230 and calculates the current position of the mobile terminal 100 will be described.

Hereinafter, in the present embodiment, the distance in the real space between the point I and the point J is denoted as _LIJ . Further, an angle formed by the line segment IJ and the line segment JK at points I, J, and K that are not in a straight line is referred to as an angle IJK. Further, this angle IJK is also referred to as an expected angle at which the line segment IK is expected from the point J or an expected angle at which the interval between the point J and the point IK is expected. For the two vectors V _A and V _B , the vector product is represented as [V _A , V _B ] and the scalar product is represented as (V _A , V _B ).

The point used for calculating the terminal position on the sign circle 231 of the sign board 230 is called a sign point. The coordinates of the sign point can be calculated from the position / orientation information described on the sign plate 230. In addition, the prospective angle to expect between the two marker points from the terminal position is geometrical based on the focal length of the camera 121 and the length on the image sensor between the marker points obtained from the image with the two marker points in the field of view. Can be calculated. The focal length is specified from the positional relationship between the lens of the camera 121 and the image sensor. Then, in the triangle formed by the two sign points and the terminal position, the unknown side length and the interior angle value are measured, and the expected angle for estimating the distance between the two sign points from the sign point coordinate value and the terminal position is measured. The terminal position is calculated from the value.

Furthermore, if the distance measuring sensor can be used in consideration of measurement errors, the output of the distance measuring sensor may be used.

FIG. 13A shows a state where the owner 910 located away from the place where the single sign board 230 is installed, such as a pillar, is shooting using the mobile terminal 100.

Suppose that the position (terminal position) of the mobile terminal 100 (owner 910) is X (point X). In addition, the owner 910 photographs a single sign board 230 from an oblique position (a direction different from the normal direction of the sign face) with respect to the face of the sign plate 230. In this case, on the display 131 of the mobile terminal 100, the sign circle 231 of the sign plate 230 is displayed in an elliptical shape as an image of the sign circle 231 (mark circle image 231i) shown in FIG.

In addition, since it does not limit that the portable terminal 100 and the sign board 230 exist on the same horizontal surface here, the portable terminal 100 and the sign board 230 can have an angle of a height direction. Accordingly, the calculation is performed on the assumption that the major axis 234i of the ellipse of the sign circle image 231i may be different from the azimuth line (azimuth line image 232i) on the image.

FIG. 13 (c) shows only the sign circle image 231i displayed on the display 131 of FIG. 13 (b). It should be noted that in the following explanatory diagrams, the images other than the images corresponding to the marker circle 231 and the azimuth line 232 are omitted in order to simplify the drawing.

Suppose that the center point of the sign circle image 231i is Oi. Further, the point Ai and the point Bi are the intersections of the bearing image 232i and the marked circle image 231i, respectively, and the points Ci and Di are the intersections of the major axis 234i and the ellipse of the marked circle image 231i, respectively, Ei , Fi are the intersections of the minor axis and the ellipse of the sign circle image 231i, respectively.

Note that the calculation unit 172 obtains each point Ai to point Fi on the image by analyzing the photographed sign circle image 231i, and calculates the length of the major axis and the minor axis on the image.

Next, FIG. 14A is a diagram in which the image is expanded in the short axis direction on the image and converted so as to overlap the marker circle 231 using the ratio of the long axis length to the short axis length on the image. ). FIG. 14A shows points Ai, Bi, Ci, Di, Ei, and Fi that have been converted to the actual marker circle 231 and azimuth line 232 by image expansion, respectively, A, B, Indicated as C, D, E, F. Here, the angle AOC is denoted as η _AC .

Here, the length of each line segment AB, CD, EF is known from the diameter of the mark circle 231, and since the line segment AB is the azimuth line 232, its three-dimensional direction is known. Further, the surface normal direction of the sign plate 230 is also known. Using these, the three-dimensional direction of the line segment CD, the three-dimensional direction of the short axis EF, and the spatial coordinates of the axis end points (C, D, E, F) used as the marker points are obtained. In addition, the center point O of the marker circle 231 is also used as the marker point, but the spatial coordinates of O are known from the position and orientation information.

Consider two or more triangles to calculate the terminal position X. First, as shown in FIG. 14B, a triangle XOC having the terminal position X, the point C, and the point O as vertices is considered (first triangle; may be a triangle XOD). Since the line segment CD (first diameter) is in the major axis direction on the image in the sign circle image 231i, the terminal position X exists at any position 236 orthogonal to the line segment CD.

That is, the line segment CD is orthogonal to a straight line (XO) connecting the point X as the terminal position and the center point O of the marker circle 231. Therefore, the triangle XOC is a right triangle as shown in FIG. The distance L _OC between the point O and the point C is the radius R (1/2 of the diameter) of the marker circle 231. The diameter is acquired by analyzing a region corresponding to the position / orientation information display region 233 on the captured image. Therefore, the distance L _OX between the point X and the point O can be calculated by the following equation. Note that the angle CXO is θ _C. θ _C is measured from an image captured by the camera 121.
L _OX = R · cot θ _C (19)

Next, in order to calculate the angle formed by the triangle XOC with the sign surface, a triangle (second triangle) formed by two sign points that are not on the same straight line as the line segment CD and the terminal position X is considered. Here, the line segment EF is adopted as the second diameter, and the triangle XOE and the triangle XOF shown in FIG. 15B are considered.

As shown in FIG. 15B, the angle EXO is θ _E , the angle FXO is θ _F , the angle EOX is λ _E , the angle FOX is λ _F , the angle OEX is ξ _E , and the angle OFX is ξ _F.
The following equation holds according to the sine theorem.
L _OX / sinξ _E = L _OE / sin θ _E (20)
L _OX / sinξ _F = L _OF / sin θ _F (21)

Here, L _OE = L _OF , but here, the description will be made as it is without using the relationship L _OE = L _OF . In order to simplify the notation, κ _E and κ _F represented by the following equations are used.
κ _E = L _OE / sin θ _E (22)
κ _F = L _OF / sin θ _F (23)

L _OE and L _OF are the radius R of the marker circle 231 and are known. Further, the above θ _E and θ _F can be calculated geometrically from the focal length of the camera 121 and the length on the image sensor corresponding to EF. The focal length is specified from the positional relationship between the lens in the camera 121 and the image sensor. Therefore, κ _E and κ _F can be calculated using these.

When κ _E and κ _F are used, the distance L _OX between the terminal position X and the sign plate 230 is expressed by the following equation.
_{L OX = κ E sinξ E =} κ F sinξ F ··· (24)

Here, since it is desired to obtain λ _E , when the above relational expression is rewritten using λ _E , it is expressed by the following expression.
ξ _E = 180 degrees -λ _E -θ _E (25)
ξ _F = λ _E −θ _F (26)
Therefore, the distance L _OX is expressed by the following formula.
L _OX = κ _E sin (λ _E + θ _E ) = κ _F sin (λ _E −θ _F ) (27)

When the above equation is expanded and arranged, it is expressed by the following equation according to the addition theorem.
_{sinλ E (κ F cosθ F -κ} E cosθ E) = cosλ E (κ E sinθ E + κ F sinθ F) ··· (28)

Here, Sinramuda _{E, and, (κ E sinθ E + κ} F sinθ F) Since is positive, the sign of the the _{cosλ E (κ F cosθ F -κ} E cosθ E) coincide.
Taking the square of both sides of the above equation, it is expressed by the following equation.
cos ² λ _E {(κ _E sin θ _E + κ _F sin θ _F ) ² + (κ _F cos θ _F −κ _E cos θ _E ) ² } = (κ _F cos θ _F −κ _E cos θ _E ) ² (29)
Here, the relationship of sin ² λ _E + cos ² λ _E = 1 was used.

When attention is paid to the sign relationship between cos λ _E and (κ _F cos θ _F −κ _E cos θ _E ), cos λ _E is calculated as follows.
_{cosλ E = (κ F cosθ F} -κ E cosθ E) / {(κ E sinθ E + κ F sinθ F) 2 + (κ F cosθ F -κ E cosθ E) 2} 1/2 ··· (30)
Similarly, sin λ _E is calculated as follows.
sin λ _E = (κ _E sin θ _E + κ _F sin θ _F ) / {(κ _E sin θ _E + κ _F sin θ _F ) ² + (κ _F cos θ _F −κ _E cos θ _E ) ² } ^1/2 (31)

When the notation of κ _E and κ _F is returned, the following is obtained.
cosλ _E = (L _OF cot θ _F −L _OE cot θ _E ) / {(L _OE + L _OF ) ² + (L _OF cot θ _F −L _OE cot θ _E ) ² } ^1/2 (32)
sinλ _E = (L _OE + L _OF ) / {(L _OE + L _OF ) ² + (L _OF cot θ _F −L _OE cot θ _E ) ² } ^1/2 (33)

Therefore, L _OX is represented by the following formula.
L _OX = κ _E sinξ _E
= (L _OE / sin θ _E ) sin (λ _E + θ _E )
_{= (L OE / sinθ E)} (sinλ E cosθ E + cosλ E sinθ E)
_{= L OE L OF (cotθ E} + cotθ F) / {(L OE + L OF) 2 + (L OF cotθ F -L OE cotθ E) 2} 1/2 ··· (34)
Since it is not used in the following calculation, details are not shown, but L _EX and L _FX of the remaining sides can also be obtained from the sine theorem. That is, all the corners and sides of the triangle formed by E, O, F, and X can be calculated by the above-described method.

Here, when the above equation is rewritten using the relationship L _OE = L _OF = R, it is as follows.
cos λ _E = (cot θ _F −cot θ _E ) / {4+ (cot θ _F −cot θ _E ) ² } ^1/2 (35)
sinλ _E = 2 / {4+ (cot θ _F −cot θ _E ) ² } ^1/2 (36)
Therefore, L _OX is obtained by the following equation.
L _OX = R (cot θ _E + cot θ _F ) / {4+ (cot θ _F −cot θ _E ) ² } ^1/2 (37)

As described above, as shown in FIG. 15B, from the measurement of two points on the same straight line where the distance between the points is known and the two expected angles between the three points from the terminal position X, the terminal position X and It is possible to calculate an unknown corner and a side length of the triangle formed from the three points. That is, as shown in FIG. 15C, the terminal position X is specified as the position where the angle EOX is λ _E among the positions 236 at the distance L _OX from the line segment CD.

As described above, the distance L _OX can be calculated as L _OX = Rcot θ _C using the point C and the point D corresponding to the long axis of the sign circle image 231i as shown in the above equation (19). Either method may be used to calculate the distance L _OX , or the average value of both may be used.

Next, the terminal position X is calculated. Here, the central angle AOC of the arc AC shown in FIG. 16A is positive in the counterclockwise direction from point A to point C with respect to η _AC .

FIG. 16B shows the direction of each unit vector to be described later. In FIG. 16B, the unit vector in the direction from the point O to the point A in the real space is V _A , the surface normal vector (unit vector) of the sign board 230 in the real space is V _S, and V _A the O center to rotate 90 degrees in the sign surface in the clockwise vector and V _T.

Since V _T is obtained as a vector product of V _A and V _S , it can be calculated by the following equation using the position and direction information.
V _T = [V _A , V _S ] (38)
Here, the angle formed by the vector V _T and the line segment OE is equal to η _AC .

A unit vector in the direction from the point O toward the point X is defined as V _X. V _X is orthogonal to the long axis CD. For this reason, the projection of V _X onto the marking surface is on the line segment EF. Further, since the angle between _{V X} and the line segment OE is lambda _{E, V X} is calculated by the following equation.
_{V X = cosλ E cosη AC V} T + cosλ E sinη AC V A + sinλ E V S ··· (39)

The position coordinates of the real space of the center point O of the label plate 230 P _O, when the position coordinates in the real space of the terminal position X and P _X, P _X is calculated by the following equation.
P _X = P _O + L _OX V _X (40)

As described above, in this embodiment, the position information of the center point and the orientation information of the surface are known, and the sign circle 231 described on the sign plate 230 and the string passing through the center of the sign circle 231 And a base line connecting the two points and a predetermined diameter at which the predetermined diameter and circumference on the mark circle 231 intersect by analyzing the captured image. And the two end angles connecting the portable terminal 100 and the predetermined two points, respectively, to calculate the terminal position. And the sign board 230 has the structure which can calculate the direction of the surface containing the positional information of said 2 points | pieces, a terminal position, and said 2 points | pieces by analyzing a picked-up image.

Thus, according to this embodiment, the terminal position can be calculated by photographing the sign board 230 as in the first embodiment. Therefore, according to the present embodiment, as in the first embodiment, even when the GPS function cannot be used, the current position and orientation of the owner 910 of the mobile terminal 100 can be calculated with a simple configuration and with high accuracy. .

Furthermore, according to the present embodiment, the terminal position X is obtained as a three-dimensional position including the height. In addition, the direction from the terminal position X to the sign board 230 is obtained as a direction vector in real space. Therefore, it can be seen in which direction the casing of the mobile terminal 100 is oriented in the real space. Thereby, since precise calibration including the three-dimensional direction of the internal coordinate system of the portable terminal 100 can be performed, position guidance including the height is also possible. It is also possible to display an image indicating the three-dimensional position of the target point by superimposing it on the actual image.

For example, at the place of the shelf where the product is stored, the product position may be superimposed on the image of the shelf and displayed as an AR (Augmented Reality) image. An example is shown in FIG. On the display 131 of the mobile terminal 100, a live-action image 301 of a product shelf photographed by the mobile terminal 100 is displayed, and an AR image 300 indicating the location of the target product is superimposed and displayed. Detailed location information including the height of the product may be acquired from a store via the Internet. Moreover, you may obtain the thing image | photographed beforehand from the store as the real image | video 301 of a goods shelf. In this case, the part in the store that the owner 910 is viewing may be estimated from the direction in which the mobile terminal 100 is facing and the video may be displayed.

Further, the navigation screen displays the position of the owner 910 on the plan view as shown in FIG. 11A until it arrives in the vicinity of the target product, and when the navigation screen arrives in the vicinity of the target product. Thus, the mode may be automatically switched to the mode for displaying the actual image of the product shelf, and the product shelf and the target product image may be displayed. At this time, if the target product image is not within the field of view of the camera 121 of the mobile terminal 100, an instruction as to which direction the mobile terminal 100 should be directed may be displayed (FIG. 17). (B)). Whether or not it is near the target product may be determined based on whether or not the condition that the product is visible and the owner 910 is within a predetermined distance from the product is satisfied.

Furthermore, the position guidance target is not limited to the product, but may indicate a route such as a door on the way to reach the target location.

In particular, according to the present embodiment, the current position and direction can be calculated only by photographing one sign board 230. Even if the portable terminal 100 is not on the same horizontal plane as the sign board 230, the terminal position can be calculated. Compared to the first embodiment, the terminal position can be calculated with a simpler configuration. In addition, since the degree of freedom of installation of the sign board 230 is increased and, for example, a ceiling or a floor may be used, the owner 910 of the mobile terminal 100 has a wider range for seeing the sign board 230 and the possibility of calculating the terminal position is increased. .

<< Third Embodiment >>
Next, a third embodiment of the present invention will be described. In the present embodiment, the position of the mobile terminal 100 (terminal position) is calculated using a plurality of marker points, each of which has a known three-dimensional position, without using the shape of the marker.

Hereinafter, the configuration of the mobile terminal 100 of the present embodiment is basically the same as that of the first embodiment. Hereinafter, a description will be given focusing on the configuration different from the first embodiment. The notation of distance, angle, etc. is the same as in the second embodiment.

As described above, in this embodiment, in order to calculate the terminal position, the shape of the sign is not used, but the three-dimensional position information of the sign point is used. Here, the terminal position is calculated using four marker points. In addition, the four marker points are on the same plane. First, an example in which four mark points are written on one mark plate will be described.

[Marking point]
An example of the sign point of this embodiment is shown in FIG. As shown in the figure, in this embodiment, four marker points 241 are arranged on one marker plate 240. In the present embodiment, it is assumed that four points are arranged at each vertex of a virtual square on the sign board 240. That is, it is assumed that three or more points are not arranged on the same straight line. Hereinafter, each marker point 241 is referred to as point A, point B, point C, and point D, respectively.

Also, when there are marker points 241 on the four vertices of the quadrangle, the intersection point 242 of the diagonal line is set as a point O. The point O may be displayed on the sign board 240 or may be processed as internal data for image processing.

The sign board 240 of this embodiment includes a position information display area 243. In the position information display area 243, the position coordinates of each marker point 241 are described. The position coordinates are, for example, the latitude / longitude height of each marker point 241. Note that the position coordinates of the intersection 242 (point O) can be calculated from the position coordinates of each marker point 241, but may be described in the position information display area 243.

In the present embodiment, the image acquisition unit 171 acquires four marker points 241 and position information of each marker point 241 as an image.

And the calculation part 172 analyzes the image which the image acquisition part 171 acquired, and calculates the positional information on the terminal position X of the portable terminal 100 as a present position.

Hereinafter, an example of a calculation method of the current position by the calculation unit 172 of the present embodiment will be described. Here, as shown in FIG. 18, the four marker points 241 are in a positional relationship where a line segment connecting point A and point C and a line segment connecting point B and point D intersect at point O. Shall.

In order to estimate the terminal position X, here, as shown in FIG. 19A and FIG. 19B, as a line segment composed of the sign points 241, a group of AO and OC, a group of BO and OD, think of. Lines do not appear on a straight line between groups. The corners and side lengths of the triangle formed by these line segments and the terminal position X are calculated. Here, the distances L _OA , L _OC , L _OB , and L _OD between the marker points 241 including the point O can be calculated using the position information. In addition, the angle AXO (θ _A ), the angle CXO (θ _C ), the angle BXO (θ _B ), and the angle DXO (θ _D ) are as described with reference to FIG. From the focal length and the corresponding length on the image sensor, it can be calculated geometrically using the captured image.

Hereinafter, as shown in FIGS. 19A and 19B, the angle XAO is ξ _A , the angle XCO is ξ _C , the angle XOA is λ _A , the angle XOC is λ _C , the angle XBO is ξ _B , and the angle XDO Is represented as ξ _D , the angle XOB as λ _B , and the angle XOD as λ _D.

Λ _A , λ _B , and L _OX can be calculated by the following equations in the same calculation as that of the modification of the second embodiment described with reference to FIG.
_{cosλ A = (L OC cotθ C} -L OA cotθ A) / {(L OA + L OC) 2 + (L OC cotθ C -L OA cotθ A) 2} 1/2 ··· (41)
_{sinλ A = (L OA + L} OC) / {(L OA + L OC) 2 + (L OC cotθ C -L OA cotθ A) 2} 1/2 ··· (42)
cos λ _B = (L _OD cot θ _D −L _OB cot θ _B ) / {(L _OB + L _OD ) ² + (L _OD cot θ _D −L _OB cot θ _B ) ² } ^1/2 (43)
sinλ _B = (L _OB + L _OD ) / {(L _OB + L _OD ) ² + (L _OD cot θ _D −L _OB cot θ _B ) ² } ^1/2 (44)
L _OX = L _OA L _OC (cot θ _A + cot θ _C ) / {(L _OA + L _OC ) ² + (L _OC cot θ _C −L _OA cot θ _A ) ² } ^1/2 (45)
L _OX = L _OB L _OD (cot θ _B + cot θ _D ) / {(L _OB + L _OD ) ² + (L _OD cot θ _D −L _OB cot θ _B ) ² } ^1/2 (46)

As shown in FIG. 19C, when λ _A , λ _B , and L _OX are determined, the terminal position X is uniquely determined.

In this embodiment, the distance L _OX can be calculated using only the triangle AXC or the triangle BXD as described above. As in the modification of the second embodiment, either value may be used. For example, the calculation unit 172 calculates an expected angle (θ _A + θ _C ) from which the line segment AC is expected and an expected angle (θ _B + θ _D ) from which the line segment BD is expected from the terminal position X, and compares both. The distance L _OX may be obtained using the larger one. Or you may use the average value of both.

Next, the terminal position X is obtained. In the present embodiment, as shown in FIG. 20 (a), a unit vector in the direction from the intersection point O to the point A and V _A, the unit vector in the direction from the intersection point O to the point B and V _B. Both can be calculated from the position coordinates of the sign point. A unit vector perpendicular to the sign plane from the intersection O and facing the sign side is V _S, and an angle BOA measured counterclockwise from the point B to the point A is η _BA . V _S is obtained by the following equation.
V _S = [V _B , V _A ] / sin η _BA (47)

Furthermore, the unit vector from the intersection O to point X and _{V X,} further, as shown in FIG. 20 (b), _{V X} forms a label surface of the label plate 240 and the angle a _{(V X} and _{V S} angle Is assumed to be λ _X (≦ 90 degrees). Here, when the vector triple product formula is applied, the relationship among V _X , V _B , and V _A is expressed by the following equation.
[V _X , [V _B , V _A ]] = (V _X , V _A ) V _B- (V _X , V _B ) _VA (48)
Take the square (as the inner product) of both sides of the above equation. At this time, V _B and V _A can be expressed using V _S from the above equation (47). That is, the relationship between V _X , V _B , and _VA is expressed by the following equation.
[V _X , [V _B , V _A ]] = [V _X , V _S ] sin η _BA (49)
Taking the square of both sides of the above formula (as the inner product), the left side and the right side are respectively expressed by the following formulas.
(Left side) ² = sin ² η _BA ([V _X , V _s ], [V _X , V _s ])
= Sin ² η _BA sin ² (90 degrees -λ _X )
= Sin ² η _BA cos ² λ _X (50)
(Right _{^{side) 2 = ((cosλ A V}} B -cosλ B V A), (cosλ A V B -cosλ B V A))
^{_{= Cos 2 λ A + cos 2}} λ B -2cosλ A cosλ B cosη BA ··· (51)
Therefore, from the above equation, cos λ _X can be calculated by the following equation.
^{_{cosλ X = (cos 2 λ A}} + cos 2 λ B -2cosλ A cosλ B cosη BA) 1/2 / | sinη BA | ··· (52)

Here, in the case of cos λ _X = 0, λ _X = 90 degrees, that is, V _X = V _S.

Next, consider the case of cos λ _X ≠ 0 (λ _X <90 degrees).

First, as shown in FIG. 20 (b), the intersection point of the perpendicular line from the terminal position X to the sign surface and the sign surface is H, and the unit vector in the direction from the point O to the point H is V _H. Here, when the unit vector in the direction of the next vector product [V _X , V _S ] is V _T , V _T is expressed by the following equation.
V _T = [V _X , V _S ] / cos λ _X
= [V _X , [V _B , V _A ]] / (cos λ _X sin η _BA )
= {(V _X , V _A ) V _B − (V _X , V _B ) V _A } / (cos λ _X sin η _BA )
= (Cos λ _A V _B -cos λ _B V _A ) / (cos λ _X sin η _BA ) (53)
V _T is parallel to the marking surface, V _T and V _H are orthogonal, and when V _T is rotated 90 degrees counterclockwise, it becomes equal to V _H. Therefore, _VH is obtained as follows.
V _H = [V _S , V _T ]
= [V _S , (cos λ _A V _B −cos λ _B V _A )] / (cos λ _X sin η _BA )
= [[V _B , V _A ], (cos λ _A V _B −cos λ _B V _A )] / (cos λ _X sin ² η _BA )
= {Cos λ _A [[V _B , V _A ], V _B] −cos λ _B [[V _B , V _A ], V _A ]} / (cos λ _X sin ² η _BA )
_{= {Cosλ A (V A -cosη} BA V B) -cosλ B (cosη BA V A -V B)} / (cosλ X sin 2 η BA)
_{= {(Cosλ A -cosλ B cosη} BA) V A + (cosλ B -cosλ A cosη BA) V B} / (cosλ X sin 2 η BA) ··· (54)

Thus, _(obtained from the label information) the position coordinates of the intersection point O P _O, the position coordinates of the terminal position X and P _X, P _X is determined as follows.
_{P X = P O + L OX} sinλ X V S + L OX cosλ X V H
_{= P O + (L OX /} sin 2 η BA) × {sgn (sinη BA) (sin2η BA -cos 2 λ A -cos 2 λ B + 2cosλ A cosλ B cosη BA) 1/2 [V B, V A] _{+ (cosλ A -cosλ B cosη BA} ) V A + (cosλ B -cosλ A cosη BA) V B} ··· (55)
Here, sgn (sin η _BA ) is as follows.
sgn (sin η _BA ) = + 1 if sin η _BA > 0 (56)
sgn (sin η _BA ) = − 1 if sin η _BA <0 (57)

When cosλ _X = 0, that is, when V _X is orthogonal to the label surface, P _X = P _O + L _OX V _S. Since cos λ _A = cos λ _B = 0, the above equation is as follows.
P _X = P _O + L _OX [V _B , V _A ] / sin η _BA = P _O + L _OX V _S (58)
Note that the above equation includes the case of cos λ _X = 0.

As described above, in the present embodiment, the four marker points 241 on the same plane are photographed, and the terminal position of the mobile terminal 100 is calculated by image analysis. For example, by image analysis, a line segment connecting two marker points (point A, point C) among the four marker points 241 and a line connecting the terminal position (point X) and each of the two marker points A and C. The angle formed by the minute (angle XAC, angle XCA) and the direction of the plane including the points X, A, and C are calculated. The portable terminal 100 of this embodiment estimates a terminal position using these.

That is, according to the present embodiment, the terminal position can be calculated by photographing four different marker points. Therefore, according to the present embodiment, the current position and orientation of the owner 910 of the portable terminal 100 can be calculated with a simple configuration with high accuracy even when the GPS function cannot be used, as in the above embodiments.

In particular, according to the present embodiment, the current position and direction can be calculated only by photographing one sign board 240 having four sign points 241. At this time, the terminal position X is obtained as a three-dimensional position including the height. Also, the direction from the terminal position X to the sign board 240 is obtained as a direction vector in real space. Therefore, it can be seen in which direction the casing of the mobile terminal 100 is oriented in the real space. Thereby, position guidance including the height is also possible.

That is, according to the present embodiment, the current position can be calculated even when the mobile terminal 100 is not on the same horizontal plane as the sign board 240. For this reason, the degree of freedom with which the sign plate 240 is attached increases, and the possibility of calculation for the owner 910 of the mobile terminal 100 also increases.

<Variation 1 of the third embodiment>
In the present embodiment, the position information may not be arranged on the sign board 240. It suffices if it is on the same virtual plane and is photographed at the same time as the four marker points 241 and described at a position where the information can be acquired. When each sign point 241 is arranged at a distant position, the position coordinates of the point may be described for each sign point.

In the following modified examples including this modified example, all the sign points 241 may not fit in one image. In this case, the expected angle between two terminal points is measured from the terminal position using the output of the three-axis gyro sensor 142.

For example, it is assumed that the sign point A is photographed with the image a and the sign point B is photographed with the image b. In each image, from the terminal position, the prospective angles to be expected between the image center and the sign point A in the image and between the image center and the sign point B in the image are the focal length of the camera 121, It can be calculated geometrically from the length on the image sensor between the marker points. Further, the spatial orientation in the internal coordinate system in the center direction of each image is calculated from the output of the three-axis gyro sensor 142. Accordingly, the orientation of the sign points A and B in the spatial orientation of the internal coordinate system can be calculated, and the prospective angle for predicting the distance between the two sign points A and B from the terminal position can be calculated. Here, in calculating the prospective angle, the difference in the central direction between the image a and the image b only needs to be known relatively, so the orientation of the internal coordinate system calculated from the output of the three-axis gyro sensor 142 is It may be deviated from the actual orientation of the space.

Also in the present embodiment, the position information acquisition destination such as URL is described on the sign board 240 as the position information, and the position coordinates of each sign point 241 are acquired from the management server via this QR code. May be.

Also, the position coordinates may not be the latitude and longitude height. The position coordinate based on the coordinate system peculiar to the building and the region where the sign board 240 is installed may be used. For example, when the sign board 240 is arranged in a shopping center, it may be associated with a floor map of the shopping center.

<Modification 2 of the third embodiment>
Further, each marker point 241 may not be a square vertex. It is sufficient that at least one point on the sign plate 240 is not on a straight line connecting the other two or more sign points 241.

Among these, the calculation method of the current position by the calculation unit 172 in the case of the sign board 240 having three sign points 241 on one straight line and one sign point 241 not on the straight line is shown in FIG. This will be described with reference to FIG.

In this modified example, as shown in FIG. 21A, the point A, the point O, and the point C are on a straight line, and only the point B is not on the line segment AC. The coordinates of each marker point 241 are described in the position information display area 243. The configuration other than the arrangement of the marker points 241 is the same as that of the present embodiment.

FIG. 21B is a triangle AXC connecting the terminal position X, the point A, and the point C. As in the above embodiment, the angle AXO is represented as θ _A , the angle CXO is represented as θ _C , the angle XAO is represented as ξ _A , the angle XCO is represented as ξ _C , the angle XOA is represented as λ _A , and the angle XOC is represented as λ _C. The triangle AXC has the same configuration as the triangle AXC in FIG. Therefore, L _OX and λ _E are obtained by the same calculation as described above.

FIG. 21C shows a triangle BXO connecting the terminal position X, the point O, and the point B. As in the above embodiment, the angle BXO is represented as θ _B , the angle XBO is represented as ξ _B , and the angle XOB is represented as λ _B.

The value of L _OB is calculated from the position information. Further, θ _B is obtained by measurement with the camera 121. L _OX is obtained by calculation from the properties of the triangle AXC. Therefore, ξ _B and λ _B are obtained by the following sine theorem.
L _OB / sin θ _B = L _OX / sinξ _B (59)

Thereafter, the terminal position X is obtained by the same calculation as described with reference to FIG.

As described above, according to the present modification, if position information (positional coordinates) of four marker points 241 that are on the same plane and at least one point is not on the same straight line is given, The terminal position X of the mobile terminal 100 can be calculated by measuring the expected angle from which the terminal point X is expected to be between each marker point. In addition, the same effects as in the above embodiment can be obtained.

<Modification 3 of the third embodiment>
(When the four sign points are not on the same plane)
Further, the four marker points 241 may not be on the same plane. As long as all the four sign points 241 are not arranged on a straight line, the terminal position X can be calculated. Hereinafter, a calculation method of the terminal position X by the calculation unit 172 in this case will be described.

First, divide the four sign points A, B, C, and D into two pairs of sign points, and consider two straight lines that pass through each of the two mark points. Here, it is classified into a set of point A and point B and a set of point C and point D. In the classification, as shown in FIG. 22A, the line AB passing through the points A and B and the line CD passing through the points C and D are selected so as not to be parallel.

Note that FIG. 22A is a view of each sign point 241 viewed from the direction of the terminal position X. FIGS. 22B, 22C, and 22D are diagrams in which each marker point 241 is viewed from the normal direction of the plane determined by the terminal position X and the line segment CD.

Two points on the straight line (referred to as apparent intersections) in the same direction (or opposite directions) when viewed from the mobile terminal 100 are also referred to as new additional sign points O and U (hereinafter simply referred to as points O and U). .) The “opposite direction” is a case where the mobile terminal 100 is sandwiched between two straight lines as shown in FIG. Point O is a point on line AB, and point U is a point on line CD. Here, it is assumed that the point O is closer to the terminal than the point U. As shown in FIG. 22D, the point O and the point U are not necessarily between the point A and the point B and between the point C and the point D.

Here, a plane determined by the terminal position X and the straight line AB is considered. On this plane, there are also point O and point U.

An example of the relationship between the marking points A, B, U and the terminal position X of the mobile terminal 100 on this plane is shown in FIG. The intersection of the line segment AB and the line segment UX is O. Angle of each vertex, as shown, respectively, the angular UAO eta _A, the angular UBO eta _B, the angular AUO zeta _A, the angular OUB zeta _B, the angular XAO xi] _A, the angular XBO xi] _B, the angular AOX Is represented as λ _A , the angle BOX as λ _B , the angle AXO as θ _A , and the angle BXO as θ _B.

From the sine theorem regarding the triangle XOA and the triangle XOB, the following equation holds.
L _OX = L _OA sin ξ _A / sin θ _A = L _OB sin ξ _B / sin θ _B (60)
Further, the following equation holds from the sine theorem regarding the triangle UOA and the triangle UOB.
_{L OU = L OA sinη A /} sinζ A = L OB sinη B / sinζ B ··· (61)
From the above two equations, the following relational expression holds.
_{sinξ A sinζ A / sinθ A sinη} A = sinξ B sinζ B / sinθ B sinη B ··· (62)
When this relational expression is transformed, the following expression is obtained.
cot λ _A (cot η _A + cot η _B −cot θ _A −cot θ _B ) = cot θ _B cot η _B −cot θ _A cot η _A (63)

When the position of the sign point U is determined, η _A and η _B are determined, and λ _A is obtained from the above equation. Therefore, the inner angle and the side length of each triangle shown in FIG. 23A are obtained, and the orientation of the surface of the triangle XAB on which the terminal position X is placed is also determined, so that the terminal position X is obtained.

However, there is an exceptional case where λ _A is not determined. This is a case where each interior angle satisfies the relationship represented by the following expression.
cot η _A + cot η _B −cot θ _A −cot θ _B = 0 (64)
At this time, the following relationship also holds.
cotθ _B cotη _B- cotθ _A cotη _A = 0 (65)
When the above two equations are combined and transformed, the following equation is obtained.
cot η _A = cot θ _B (66)
cot η _B = cot θ _A (67)
That is, since η _A , η _B , θ _A , and θ _B are the interior angles of the triangle, the following relationship is established.
η _A = θ _B (68)
η _B = θ _A (69)

Considering the circumference angle theorem, it means that point A, point B, point U, and point X are on the same circumference. On the other hand, the above equation is established at an arbitrary position on the circumference of the point X. Therefore, λ _A is not determined. In this case, the mobile terminal 100 is moved to another location and measurement is performed again. That is, the calculation unit 172 outputs a message that prompts movement and re-shooting.

) Except in exceptional cases, if the sign point U is determined, the terminal position X is obtained. The sign point U is on the straight line CD. Therefore, when the terminal position X is calculated using each point on the straight line CD as the marker point U, the candidate point of the terminal position X is placed on a predetermined curve (first curve) in the space.

As shown in FIG. 23B, the same calculation is performed using the marker points C, D, O, and U. As a result, λ _C is obtained by the following equation.
cot λ _C (cot η _C + cot η _D + cot θ _C + cot θ _D ) = cot θ _D cot η _D −cot θ _C cot η _C (70)

Similarly, if the terminal position X is calculated using each point on the straight line AB as a point O, a curve (second curve) different from the first curve in which the candidate point of the terminal position X is placed in the space is determined. The terminal position X is an intersection of the first curve and the second curve.

As shown in FIG. 24A, in the triangle ABU, a perpendicular line is drawn from the point U to the side AB, and the perpendicular leg is defined as W. When cot η _A and cot η _B are represented by the distance between points, the formula (63) is expressed as follows.
cotλ _A (L _AB −L _UW (cot θ _A + cot θ _B )) = L _BW cot θ _B −L _AW cot θ _A (71)

Next, as shown in FIG. 24B, in the triangle CDO, when a perpendicular line is dropped from the point O to the side CD and the foot of the perpendicular line is Q, the expression (70) is expressed as follows.
cotλ _C (L _CD + L _OQ (cot θ _C + cot θ _D )) = L _DQ cot θ _D −L _CQ cot θ _C (72)

For example, as shown in FIG. 18, when the sign points A, B, C, and D are on the same plane, the points U, O, W, and Q are the same in the equations (71) and (72). Which is consistent with the notations of equations (41) to (44).

Here, the length (distance) of the line segment AO, the line segment OX, the line segment CU, and the line segment UX is calculated as follows.
L _AO = L _AW -L _UW cotλ _A (73)
L _OX = (L _AW −L _UW cot λ _A ) (cot θ _A + cot λ _A ) sin λ _A (74)
L _CU = L _CQ + L _OQ cotλ _C (75)
L _UX = (L _CQ + L _OQ cot λ _C ) (cot θ _C + cot λ _C ) sin λ _C (76)

When the point U is determined, the point W is determined, and the coordinates of the terminal position X are determined by using the equations (71), (73), and (74). On the other hand, when the point O is determined, the point Q is determined, and the coordinates of the terminal position X are determined using the equations (72), (75), and (76). The point U and the point O are searched so that the terminal position X calculated in both becomes the same coordinate.

As described above, given four marker points 241 whose coordinate values in the external coordinate system are known, which are not on the same plane and at least one point is not on the same straight line, A prospective angle that is expected from the terminal position X between the sign point 241 and the intersection point is measured, a straight line connecting the sign points 241 of different directions, an angle of a straight line connecting the sign point 241 and the terminal position X, and the sign point 241 The distance from the terminal position X to the terminal position X can be obtained, and the coordinate value of the terminal position X in the external coordinate system can be obtained. The intersection point is one of apparent intersection points (point O, point U) viewed from the terminal position X of two straight lines connecting the sign points 241. Further, since the direction in the external coordinate system from the terminal position X toward the two or more sign points 241 is known, the orientation of the internal coordinate system of the mobile terminal 100 in the external coordinate system can also be grasped.

The measurement of the prospective angle for predicting the distance between the two sign points 241 from the terminal position X can be performed by analyzing the photo when the two sign points 241 fit within one picture. Even if there is no sensor (hereinafter referred to as an orientation sensor) for estimating the orientation (the orientation of the terminal in space) of the mobile terminal 100 such as the three-axis gyro sensor 142, the estimation of the terminal position X and the orientation of the terminal (Direction in the external coordinate system) can be estimated. Therefore, it is desirable that the camera 121 can capture all solid angles.

When the azimuth sensor is mounted on the terminal, a plurality of photographs are taken while changing the attitude of the mobile terminal 100, and 2 from the terminal position X from the azimuth of each marker point viewed from the internal coordinate system of the mobile terminal 100. A prospective angle for expecting between two marker points 241 can be measured. In this case, since only the relative relationship in the direction of each marker point 241 in the internal coordinate system is used, the relationship between the orientations of the internal coordinate system and the external coordinate system can be accurately obtained. It can also be used for calibration.

<Modification 4 of the third embodiment>
In addition, if the output of the three-axis gyro sensor 142 that is an azimuth sensor is also taken into consideration, the marker points 241 are not limited to four points. As long as all the points are not on a straight line, it may be three or more. If there are three or more marker coordinates of the marker point 241, it is possible to calculate the orientation of the surface including the marker point 241. As will be described later, two or more points may be used as long as the orientation of the internal coordinate system is correct.

(If 3 sign points are given)
Hereinafter, a method of calculating the terminal position X by the calculation unit 172 when three marker points 241 are given will be described. In this modification, the output of the three-axis gyro sensor 142 that is an orientation sensor of the mobile terminal 100 is used. Hereinafter, this method will be described.

First, three mark points 241 and a projection plane 410 and a projection plane 420 that are not parallel to each other are set. The projecting surface 410 and the projecting surface 420 need only be non-parallel and do not need to be orthogonal. FIG. 25 shows an example of three marker points A, B, and C, a projection plane 410, and a projection plane 420.

The coordinates of each marker point 241 are projected onto the projection plane 410 and the projection plane 420, respectively. Here, projection points of each point onto the projection plane 410 are points A ′, B ′, and C ′, respectively. Further, the projection points on the projection plane 420 are points A ″, B ″, and C ″, respectively. The terminal position X is also projected onto each projection plane, and the projection points are X ′ and X ″, respectively.

From the expected angles (angles) at which the sign points are estimated from the terminal position X in the real space, the expected angles (angles) at which the projected points A ′, B ′, C ′ are estimated from the projected points X ′ are calculated. Since the projection direction is known in the internal coordinate system, the angle in the projection plane 410 can also be calculated.

FIGS. 26 (a) to 26 (c) show diagrams of the marker points projected on the projection plane 410 and the terminal position X. FIG. Since the projection surface 420 is basically the same, a case where the projection surface 420 is projected onto the projection surface 410 will be described as an example.

Since the positional relationships are the same as those in FIGS. 23 (a), 23 (b), and 21 (b), respectively, the projection is similarly performed from the measured value of the expected angle from the terminal position X to the interval between the marker points 241. The terminal position X ′ in the plane 410 is obtained.

As shown in FIG. 27, let W ′ be the leg of a perpendicular line drawn from the sign point C ′ to the straight line A′B ′. When the sign point C ′ is located farther than the terminal position X ′ across the straight line A′B ′ (FIG. 26A), L _{C′W ′} > 0, η _{A ′} > 0, η _{Let B ′} > 0. When the sign point C ′ is located closer to the terminal position X ′ across the straight line A′B ′ (FIG. _26B ), L _{C′W ′} <0, η _{A ′} <0, η _{B ′.} <0. When the sign point C ′ is on the straight line A′B ′ (FIG. _26C ), L _{C′W ′} = 0. Thereby, the expression can be unified as follows.
_{cotλ A '(L A'B' -L} C'W '(cotθ A' + cotθ B ')) = L B'W' cotθ B '-L A'W' cotθ A '··· (77)
L _{A′O ′} = L _A′W′ −L _{C′W ′ cot} λ _{A ′} (78)
L _{O′X ′} = (L _A′W′− L _{C′W ′} cot λ _{A ′} ) (cot θ _{A ′} + cot λ _{A ′} ) sin λ _{A ′} (79)
In the case of FIG. 26C, the point W ′ and the point O ′ coincide.

However, the terminal position X ′ cannot be obtained when the mark points A′B′C ′ and the terminal position X ′ are aligned on the same plane or on the same circumference in the projection plane 410. In this case, the projection plane is reset or moved and remeasured.

As a result, the terminal positions X ′ and X ″ projected onto the projection plane 410 and the projection plane 420 are obtained. The terminal position in the real space is obtained as the intersection of the straight lines extending in the projection direction from the terminal positions X ′ and X ″. X can be determined.

Furthermore, since the direction in which the mobile terminal 100 faces can be estimated from the measurement, the direction of the internal coordinate system can be corrected. At that time, the terminal position X may be obtained again by the same procedure using the corrected internal coordinate system. This procedure can be repeated until the terminal position X and the direction of the internal coordinate system have the required accuracy.

(When two sign points are given)
In addition, when the azimuth by the internal coordinate system of the mobile terminal 100 is obtained from the output of the three-axis gyro sensor 142 that is the azimuth sensor, the terminal position X can be calculated from only the information of the two marker points 241.

For example, as shown in FIG. 28 (a), when two sign points 241 (point A and point B) are given, the terminal position X follows a circumference 244 passing through points A and B along a straight line AB. Is on a curved surface rotated around the axis. Note that the angle AXB (θ _AB ) can be calculated from the acquired image.

If the direction (V _A ^(o) , V _B ^(o) ) from the terminal position X to each marker point 241 is known, the terminal position X ^(o) on the curved surface can be specified.

When only two sign points 241 are given, the direction of the internal coordinate system of the portable terminal 100 cannot be corrected only by measuring the sign point direction (θ _AB ). However, in another way, if the estimated value X of the terminal position X ^(d) is obtained, and the estimated value X ^(d), a new weighted average value of the specified terminal position X ^(o) by the above technique mobile The terminal position X of the terminal 100 is set, and the direction of the internal coordinate system is corrected to match the direction (V _A ) of the marker point calculated from the terminal position X.

(When one sign point is given)
Here, a method of correcting the direction of the internal coordinate system when one mark point 241 is given will be described. In this case, as shown in FIG. 28 (b), it is only known that the terminal position X ^(o) is on the straight line L passing through the landmark A. V _A ^(o) is the direction of the sign point 241 viewed from the terminal position X of the mobile terminal 100 measured in the internal coordinate system.

Let X ^(o) be a leg of a perpendicular line drawn down to a straight line L from the terminal position X ^(d) estimated by another method. Further, the weighted average value of the terminal position X ^(d) and the terminal position X ^(o) is set as a new terminal position X, and the internal coordinate system is matched with the direction (V _A ) of the sign point calculated from the terminal position. Correct the direction.

<Modification 5 of the third embodiment>
(When the marker points are measured at multiple points)
Further, in the above embodiment, the owner 910 images a plurality of sign points 241 at one place and calculates the terminal position X. However, it is not limited to this. The owner 910 may calculate the terminal position X using a plurality of landmarks 241 acquired at different positions.

For example, the owner 910 of the mobile terminal 100 may not be able to photograph a plurality of sign points 241 necessary for terminal position estimation at one location. This modification assumes such a case.

In this modified example, as shown in FIG. 29 (a), the owner 910 of the mobile terminal 100 moves when the plurality of sign points 241 cannot be photographed at one place, and moves as many times as necessary to estimate the terminal position. The sign points 241 (in FIG. 29A, points 241a, 241b, 241c, and 241d) are photographed. At this time, the movement amount is calculated by an internal function of the mobile terminal 100.

The calculation unit 172 of the present modification corrects the position information of each marker point 241 using the movement amount (three-dimensional vector amount) so that the marker point 241 is photographed at the same location. That is, the position information of the sign points 241 acquired at different positions is corrected to position information based on the current position. Then, the terminal position X is calculated using the position information of the sign point 241 after correction.

Hereinafter, a method for estimating the terminal position X using the sign points 241 acquired at different locations will be described with reference to FIG.

Point S, the representation of the internal coordinate system of the portable terminal 100 and Q _S. It is assumed that the direction of the marker point A is measured at the point S and the direction of the marker point B is measured at the point T. A movement vector _DST from the point S to the point T as viewed is expressed by the following equation.
D _ST = Q _T -Q _S (80)

By the amount to cancel out the movement amount indicated by the motion vector D _ST, and change the landmark A (coordinates P _A) (moved). The direction when the sign point A ′ (coordinate value P _{A ′} ) after the change is viewed from the point T is equal to the direction of the mark point A when viewed from the point S. Therefore, the following equation holds.
P _{A ′} = P _A + D _ST (81)
V _{TA ′} = V _SA (82)
Here, the symbol V _IJ is a unit vector from the point I to the point J in the internal coordinate system.

In this way, a point A ′ obtained by virtually moving the marker point 241 (point A) measured outside the point T is set as a new marker point 241. The movement amount is an amount that cancels the movement amount between the measurement points. Then, it is assumed that a new sign point 241 (point A ′) is measured at the point T, and the terminal position X is estimated by the above method using another sign point 241 (point B) measured at the point T.

In the estimation of the terminal position X, the direction and the prospective angle between the marker points 241 are obtained as described above. Also in this modification, the direction and the expected angle between the marker points 241 can be measured as the marker angle and the expected angle in the internal coordinate system of the mobile terminal 100.

Also, the movement vector between measurement points is measured by an internal sensor. For example, the movement vector is obtained as a two-time integral value of the triaxial acceleration sensor 143.

<Application example 1>
In addition, as an application of this modification, the owner 910 of the mobile terminal 100 acquires a sign point necessary for estimating the terminal position while moving, and each time a new sign point is acquired, A method for updating the current position using the marked points will be described.

In this application example, each time the current terminal position X is estimated, the calculation unit 172 manages the estimated terminal position X as estimated position data in association with the estimated date and time. The estimated position data is stored in the data storage unit 174, for example.

An example of the estimated position data 180 is shown in FIG. As shown in this figure, the estimated position data 180 is a point No. that is identification information that uniquely identifies each estimated position. The date and time (date / time) 182 at which the marker point used to estimate the position is measured, the estimated terminal position (point coordinates) 183, the terminal speed 184 at the time of measurement, and the time The coordinates (marking coordinates) 185 of the sign point 241 photographed in step 185 and the direction (marking direction) 186 from the portable terminal 100 of the sign point 241 are registered.

Next, the flow of terminal position estimation processing for calculating the current position of the mobile terminal 100 using the technique of this application example will be described. FIG. 31 is a processing flow of terminal position estimation processing of this application example. Hereinafter, in this example, the current position is calculated using the four marker points 241. In addition, first, position coordinates (label coordinates) of at least four sign points 241 are acquired, and terminal positions (terminal coordinates) are calculated.

In addition, it is assumed that the terminal position first calculated by the calculation unit 172 is registered in the estimated position data 180 in association with the calculation time. At this time, the position information (marking coordinates) of each marking point 241 used for the calculation and the marking direction in the internal coordinate system of the portable terminal 100 are also registered.

When the owner 910 holds the mobile terminal 100, moves a predetermined distance, and finds a new sign point 241, the owner 910 performs a position estimation process via the user I / F 130 of the mobile terminal 100. At the same time, the camera 121 is directed to a new sign point 241.

Upon receiving an instruction from the owner 910, the terminal position estimation unit 170 causes the image acquisition unit 171 to acquire an image of a new sign point 241 (step S3101).

When the image acquisition unit 171 acquires an image of a new sign point 241, the calculation unit 172 analyzes the image, acquires position information (marking coordinates) of the new sign point 241, and creates a new one in the internal coordinate system. The direction (marking direction) of the sign point 241 is acquired (step S3102).

The calculating unit 172 further refers to the estimated position data 180, extracts the date and time of the immediately preceding record, and calculates the average speed of the mobile terminal 100 from the date and time to the current time as the speed (step S3103). Note that the velocity is calculated and recorded at predetermined time intervals from the output of the triaxial acceleration sensor 143.

The calculation unit 172 uses the acquired sign coordinates, sign direction, and calculated speed as new point data to create a new point number. Are registered in the estimated position data 180 (step S3104).

Next, the calculation unit 172 calculates the terminal position X using the latest four sign points 241. At this time, in this example, the mobile terminal 100 is acquired in a different location except for the new sign point 241. Therefore, first, the movement vector is calculated by the above method. Then, by converting the label direction by the moving vector, the marker coordinates assumed to be measured from the current position are obtained (step S3105).

Then, the estimated position data 180 is updated with the obtained sign direction and sign coordinates (step S3106). For example, in the example of FIG. The current position is calculated using P _B1 , P _C1 , P _D1 and P _E2 acquired in step 1. Therefore, P _B1 , P _C1 , and P _D1 are converted into coordinate values based on the current position X ₂ according to the movement amount, and P _B2 , P _C2 , and P _D2 are obtained. At this time, the sign direction is changed from V _1B , V _1C , and V _1D because both the terminal position and the sign coordinates are changed.

The calculation unit 172 calculates the estimated position coordinates of the current position X ₂ using the latest four mark points after the conversion (label coordinates P _B2 , P _C2 , P _D2 , P _E2 ) (step S3107). Then, the calculation result is stored in the estimated position data 180 (step S3108), and the process ends.

After estimation of the current position _{X 2,} an estimated position data 180, shown in FIG. 30 (b). Rewrite the record used to calculate the new estimated position. Information before rewriting may be left as a history.

Next, when the owner 910 moves, shoots a new sign point 241 (or a plurality of sign points), and gives an instruction to estimate the current position, the calculation unit 172 of the mobile terminal 100 sets the sign point 241 to a point. No. 3 records are created and registered. Then, the number of records just before supplementing to make the four marker points (in the example of FIG. _30B , the marker coordinates are selected from the marker point records P _C2 , P _D2 , and P _E2 ) are converted by the amount of movement. . The coordinate position after conversion, direction, using the position orientation information of the new landmarks 241, calculates the terminal position X ₃ of the imaging locations.

<Application example 2>
Furthermore, in this modification, each time the owner 910 captures an image, the owner 910 determines the presence / absence of the marker point 241 in the captured image. Although the terminal position X is calculated, the number of marker points used for calculating the terminal position X is not limited to four. Furthermore, even if it is the same mark point, when it measures from a different terminal position, you may handle as different mark point data. Hereinafter, the process in this case will be described.

In this case, for example, the data of the estimated position and the marker point 241 are stored in the data storage unit 174 in the acquired state. An example of the estimated position history 190 in this case is shown in FIG. As shown in this figure, the estimated position history 190 holds data of items substantially similar to the estimated position data 180.

That is, data No. which is identification information for uniquely specifying each measurement data. Corresponding to 191, the date and time (date / time) 192 at which the landmark used to estimate the position was measured, the estimated terminal position (point coordinates) 193, the terminal speed 194 at the estimated time, and the time The coordinates (marking coordinates) 195 of the sign point 241 photographed in step 195 and the direction (marking direction) 196 from the portable terminal 100 of the sign point 241 are registered.

Next, the flow of the terminal position estimation process using the estimated position history 190 using the marker points measured in the past will be described. FIG. 33 is a processing flow of terminal position estimation processing of this example. Here, the image acquisition unit 171 starts processing upon acquisition of a new image. Note that when a moving image is being shot, the processing may be started at regular intervals.

First, the calculation unit 172 analyzes the acquired image and determines whether or not the sign point 241 has been detected (step S4101). If the moving speed is equal to or lower than a certain value, a plurality of pieces of image data taken continuously may be used assuming that the moving speed is still. Whether or not it has been detected is determined, for example, based on whether or not predetermined position information has been extracted by image analysis.

If it could not be detected (step S4101; No), the process is terminated as it is.

On the other hand, when one or more landmarks 241 are detected (step S4101; Yes), first, the calculation unit 172 estimates the terminal position X ₀ ^(d) based on the internal sensor (step S4102).

Here, at the current position, the point No. The integrated value of the acceleration of the mobile terminal 100 is added to the coordinates (point coordinates) 193 of the terminal position X1 of ₁ and the speed 194 of the mobile terminal 100. The integral value of acceleration is obtained by subtracting the gravitational acceleration from the acceleration measurement value by the triaxial acceleration sensor 143 that is an internal sensor. Specifically, the terminal position coordinates X ₀ ^(d) and the terminal speed S ₀ ^(d) are estimated by the following equations.

Here, a (u) is an acceleration measured by a triaxial acceleration sensor, and g ₁ is a gravitational acceleration plus a correction acceleration, both of which are based on the internal coordinate system. . Also, t is a point No. It represents the elapsed time from the time when 1 measurement was performed.

Next, the calculation unit 172 specifies the number M (M is an integer equal to or greater than 1) of the detected sign points 241 (step S4103). Specifically, the calculation unit 172 analyzes the image acquired at the terminal position X ₀ ^(d) and detects the number of marker points in the image. The calculation unit 172 obtains an estimated value X ₀ ^(o) of a new terminal position by combining the detected marker point 241 and information of the marker point 241 measured in the past.

Next, when calculating the estimated value X ₀ ^(o) of the terminal position, the calculation unit 172 determines whether to use the number N of sign points to be used (N is an integer equal to or greater than 1) including a new sign point. (Step S4104).

Specifically, it is determined as follows.
(Case 1): In a set of four marker points 241 selected in order from the newest one including the marker points 241 newly acquired from the current image, the distance from the current point of each marker point 241 is obtained. Then, the maximum value of the obtained distance is compared with d _max4 and a predetermined value d ₄ . If d _max4 value _{d 4} is less than uses its four landmarks 241.
(Case 2): When the case 1 is not satisfied, the maximum value d _{max3 of the} distance in the set of three mark points 241 selected in the new order is _compared with a predetermined value d ₃ . If d _max3 is smaller than the value d ₃ , the three landmarks 241 are used.
(Case 3): When Case 2 is not satisfied, the maximum value d _{max2 of the} distances of the two marker points selected in the new order is compared with a predetermined value d ₂ . If d _max2 is smaller than the value d ₂ , the two measurement points are used.
(Case 4): When Case 3 is not satisfied, only the newest sign point 241 is used.
As the distance limit value, for example, the following relationship is set. However, it is not limited to this.
d ₄ <d ₃ <d ₂ (85)

When the number N of marker points to be used is determined, an estimated value X ₀ ^(o) of the terminal position is obtained by a method according to the number N of marker points described above (step S4105).

The coordinate of the newly measured marker point 241 is P _n , and the new marker point direction vector based on the internal coordinate system at this time is V _n ^(o) (a plurality of marker points 241 measured at the new terminal position are plural. In some cases, the direction of each marker point 241 is also included). In the above method, when the terminal position cannot be calculated, the terminal position is moved and a new measurement is performed. Measurement points may be used instead.

Next, the weighting coefficients for the case n in the case of the K _n, depending on each case, to calculate a new terminal position X ₀ by the following equation (step S4106).
X ₀ = (1−K _n ) X ₀ ^(d) + K _n X ₀ ^(o) (86)
Here, each weighting coefficient shall have the following relations, for example.
1 ≧ K ₄ > K ₃ > K ₂ > K ₁ > 0 (87)
However, the relationship between the weighting factors is not limited to this.

Next, the direction of the internal coordinate system is updated (step S4107). Here, the direction vector V _n of the new sign point 241 viewed from the terminal position is calculated from the new terminal position coordinate X ₀ with the external coordinate value and the new sign point coordinate P _n with the same external coordinate value. . Then, from the difference between the direction vector V _n ^(o) and V _{n of} each marker point 241 measured with the current internal coordinate system reference, the internal vector rotated so that the direction vector measured with the internal coordinate system becomes equal to T _n. Set the coordinate system as the new target internal coordinate system.

At this time, if there are a plurality of new sign points 241, the calculated value and the measured value of the direction vector V _n may not be completely matched at all the new sign points 241. In this case, for example, a new target internal coordinate system is set so that the sum of direction errors is minimized. Although the target internal coordinate system may be adopted as a new internal coordinate system, the rotation angle for obtaining the target internal coordinate system may be reduced by using a weighting coefficient.

In order to convert from the current internal coordinate system to the target internal coordinate system, for example, rotation is performed with three orthogonal axes, but each rotation angle may be multiplied by a weight coefficient K ′ _{n with} respect to case n. Here, it is assumed that the weighting coefficient has the following relationship, for example.
1 ≧ K ′ ₄ > K ′ ₃ > K ′ ₂ > K ′ ₁ > 0 (88)
However, the relationship between the weighting factors is not limited to this.

Finally, the parameter for position estimation by the internal sensor is updated (step S4108). In this case, a new estimate of the velocity at a point X ₀ is obtained by the following equation.
S ₀ = S ₀ ^(d) − (X ₀ ^(d) −X ₀ ) / Δt (89)
Here, Δt is No. This is the elapsed time from the measurement time at point 1 to the time when a new measurement was performed.

Further, it can be presumed that the cause of the error in the position estimation obtained from the integration of acceleration is the first cause in the measured value of acceleration. Therefore, the acceleration measurement value is corrected by correcting the gravitational acceleration. First, the correction is performed by the following equation before the correction of the internal coordinate system.
g ₀ = g ₁ − (S ₀ −S ₀ ^(d) ) / Δt (90)
Here, g ₀ is the corrected gravitational acceleration. Furthermore, changing the g ₀ to representation in the new internal coordinate system.

Since the value has been updated, the estimated position history 190 is rewritten. Specifically, it can be rewritten according to the following procedure.

First, the past data is No. The data of (k) is No. Copy to (k + N). Next, no. From (1) to No. The date and time measured as “date / time” as data up to (N), X ₀ as “point coordinates”, S ₀ as “speed”, P _n as “sign coordinates”, and V _n as “sign direction” Write. Further, the corrected g ₀ is set as the value of g ₁ used in the equation (83) in the next step, and the process is terminated.

In the following, the above process is repeated each time an image is acquired using the mobile terminal 100.

As described above, while moving, a new marker point is measured, the position of the terminal in the external coordinate system is sequentially estimated, and correction can be performed so that the direction of the internal coordinate system matches the direction of the external coordinate system.

<Modification 6 of the third embodiment>
In the present embodiment, the terminal position is estimated in advance as a sign point 241 attached to a sign board or the like. However, the present invention is not limited to this. For example, a characteristic point of a building such as a representative point of a landmark or a corner of a ceiling may be used. In this case, the calculation unit 172 may acquire the position coordinates of these representative points or feature points from the server 960 and perform position calculation.

<Modification 7 of the third embodiment>
In the present embodiment, the position coordinates of the sign point 241 are acquired from the position information arranged near the sign point 241 or the QR code, but the present invention is not limited to this. For example, it is good also as a structure which can transmit position information (coordinate value) from the marker point 241. FIG.

In this case, for example, the sign point 241 may emit a signal having good straightness such as light. Note that the electromagnetic wave transmitted from the marker point 241 may be a radio wave as long as it has a rectilinearity and a receiving device capable of measuring the direction. The sign point 241 continues to send coordinate information of the sign point 241 as a beacon signal. Thereby, a compact sign point 241 can be configured. Furthermore, the identification signal of the sign point 241 may be placed on the beacon signal. In that case, the position information of the sign point 241 may be acquired from the server 960 using the identification signal.

Also, a marker point 241 that outputs a beacon signal may be mounted on the moving body. In this case, the moving body on which the marker point 241 is mounted or the marker point 241 itself has a configuration capable of calculating its own position from signals from other marker points 241. As a result, the sign point 241 can automatically set its own position and can cope with movement.

Further, the mobile terminal 100 itself may become a beacon point 241 by a beacon. When the terminals are close to each other, it is possible to accurately grasp the relative position between the terminals. For example, it is effective for collision avoidance when drones fly densely.

<Modification>
In each of the above-described embodiments and modifications, the current position is estimated (calculated) in the mobile terminal 100. However, it is not limited to this. An image may be acquired by the mobile terminal 100 and transmitted to the server 960 via the access point 970 and the network 940, and the server 960 may calculate the current position of the mobile terminal 100 that is the transmission source. In this case, the calculation result is returned to the mobile terminal 100 that is the transmission source.

It should be noted that the present invention is not limited to the above-described embodiments and modifications, and includes various modifications. For example, the above-described embodiments and modification examples have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Moreover, it is possible to replace a part of the configuration of a certain embodiment or modification with the configuration of another embodiment or modification. Moreover, it is also possible to add the structure of another embodiment or modification to the structure of a certain embodiment or modification. Furthermore, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment or modification.

In addition, each of the above-described configurations, functions, processing units, processing means, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by the processor. Information such as programs, tables, and files that realize each function can be stored in a memory unit, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD. .

Also, the control lines and information lines indicate what is considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. In practice, it may be considered that almost all the components are connected to each other.

100: mobile terminal, 101: CPU, 102: bus, 110: storage device, 111: ROM, 112: RAM, 113: storage, 120: photographing device, 121: camera, 122: image processor, 123: image memory, 130 : User I / F, 131: Display, 140: Sensor device, 141: GPS receiver, 142: 3-axis gyro sensor, 143: 3-axis acceleration sensor, 150: Communication device, 151: LAN communication device, 152: Telephone network Communication device 153: Short-range communication device 160: Extended I / F 170: Terminal position estimation unit 171: Image acquisition unit 172: Calculation unit 173: Display control unit 174: Data storage unit 175: QR Code analysis unit, 176: navigation unit, 180: estimated position data, 184: speed, 190: estimated position history, 194: speed,
200: sign board, 200a: sign board, 201: position and orientation information display area, 202: bearing arrow, 211: sign board, 212: sign board, 213: reference point, 221: sign image, 222: sign image, 230: Marking plate, 231: Marking circle, 231i: Marking circle image, 232: Direction line, 232i: Direction line image, 233: Position / direction information display area, 234i: Long axis, 240: Marking plate, 241: Marking point, 242: Intersection, 243: position information display area, 244: circumference,
410: Projection plane, 420: Projection plane, 910: Owner, 940: Network, 950: Base station, 960: Server, 970: Access point

Claims (14)

1. A portable terminal comprising a camera and a processing unit that processes an image acquired by the camera,
   The processor is
   An image acquisition unit that acquires a marker image that is an image including two marker points at different positions away from the mobile terminal and including a marker point whose position information is known;
   Analyzing the acquired sign image, calculating the plane orientation of the first triangle formed by the terminal position that is the position of the mobile terminal and the two sign points and the interior angle of the first triangle, A mobile terminal comprising: a calculation unit that calculates the terminal position using a plane orientation and the interior angle.
2. The mobile terminal according to claim 1,
   A mobile terminal, further comprising: a display control unit configured to display the calculated terminal position on a display of the mobile terminal.
3. The mobile terminal according to claim 1,
   A navigation unit for performing navigation;
   The calculation unit outputs the calculated terminal position to the navigation unit.
4. The mobile terminal according to claim 1,
   The image acquisition unit acquires the sign image by photographing two different sign boards with the terminal position and the center point on the same horizontal plane with a camera,
   The two mark points are the center points of the two mark plates,
   Each of the two sign boards has a position and orientation information display area that can be read by analyzing the sign image, and the position information of the center point and the orientation of the sign face that is the face of the sign board with respect to a reference direction. The line segment connecting the camera and the center point from the distortion of the marker image has a shape capable of measuring the angle formed with the normal line of the marker surface, while preparing for the marker surface,
   The calculation unit acquires, from the marker image, both end angles of a line segment connecting the two marker points among the inner angles of the first triangle.
5. The mobile terminal according to claim 1,
   The two marking points are intersections of a first diameter, which is a predetermined diameter of a circle drawn on a marking surface which is a surface of a predetermined marking plate, and the circle,
   The sign board includes, on the sign surface, the circle, an azimuth line that is a predetermined diameter of the circle, and a position / azimuth information display area,
   In the position / orientation information display area, the position information of the center point of the circle, the diameter of the circle, and the orientation of the bearing line in real space are readable by analyzing the marker image. And
   The image acquisition unit obtains the sign image by photographing the sign plate with the camera,
   The calculation unit includes one of intersections of a second diameter orthogonal to the first diameter and the circle, a center point of the circle, and an interior angle of a second triangle formed by the terminal position, A portable terminal characterized by further calculating a distance between a terminal position and a center point of the circle, and calculating the plane orientation.
6. The mobile terminal according to claim 1,
   The image acquisition unit acquires an image further including an additional sign point whose position information is known, which is different from the two sign points,
   The two sign points and the additional sign point are not in a straight line,
   The calculation unit includes a predetermined point on a line segment connecting the two mark points, the additional mark point, an interior angle of a second triangle formed by the terminal position, and the terminal position and the predetermined point. A mobile terminal characterized by further calculating a distance and calculating the plane orientation.
7. The mobile terminal according to claim 1,
   The portable terminal characterized in that the two mark points are representative points of landmarks whose position information is known in advance.
8. The mobile terminal according to claim 1,
   It further includes a display that serves as both a reception unit and a display unit that receive setting of the target point from the user,
   Based on the calculated terminal position and the orientation of the mobile terminal, an image indicating a three-dimensional position of the target point is displayed superimposed on an actual image acquired by the camera.
9. A camera position estimation system that includes a camera and a processing device that processes an image acquired by the camera, and estimates the camera position that is the position of the camera by analyzing the image,
   The camera acquires a sign image that is an image including a sign point having two known sign points at different positions away from the camera and having known position information;
   The processing device analyzes the sign image acquired by the camera, and calculates a plane orientation of a first triangle formed by the camera position and the two sign points and an interior angle of the first triangle. The camera position is calculated using the plane orientation and the internal angle.
10. In a system comprising a camera and a processing device that processes an image acquired by the camera, a camera position estimation method for estimating a camera position that is a position of the camera by analyzing the image,
    With the camera, a sign image that is an image including two sign points at different positions away from the camera and including a sign point with known position information,
    Analyzing the sign image acquired by the camera, calculating the plane orientation of the first triangle formed by the camera position and the two sign points, and the interior angle of the first triangle, A camera position estimation method, wherein the camera position is calculated using the interior angle.
11. A sign board attached to the surface of an object,
    A position and direction information display area for displaying the position information and the direction information in a manner capable of capturing the position information and the direction information on the sign surface which is the surface of the sign plate and analyzing the captured image;
    A mark capable of measuring an angle formed by a line connecting the position of the camera and the center point of the marker plate and a normal of the marker surface from distortion of a captured image obtained by photographing the marker plate with a camera. The sign board characterized by comprising.
12. The sign board according to claim 11,
    The mark is a rectangular frame or a circular frame described on the mark surface.
13. The sign board according to claim 11,
    The mark is a point indicating the positions of four vertices of a virtual rectangle described on the sign surface.
14. The sign board according to claim 11,
    The mark includes a mark circle described on the mark surface and an azimuth line having a predetermined diameter of the mark circle.
    

Images