JP2013042246A - Portable terminal and location and azimuth estimation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect the location and azimuth of a portable terminal with high accuracy in a portable terminal and a location and azimuth estimation program by using a relatively inexpensive sensor incorporated in the portable terminal.SOLUTION: In a portable terminal, an inertia navigation method for performing short-range wireless communication with an external device having own location information stored therein is used. The portable terminal is configured to include: a detection unit which, upon detecting the acceleration of the portable terminal, outputs the acceleration information; a photographing unit which photographs a picture image; a communication unit which communicates with an external device; and a control unit which, when the communication unit becomes ready to communicate with the external device, sets the location information received from the external device to the initial location of the portable terminal, as well as predicts photographing timing on the basis of an AC component of the acceleration information to control the photographing unit so that the image of the external device will be photographed with the photographing timing, and sets a yaw angle estimated from the photographed image of the external device to an initial azimuth of the portable terminal.

Description

  The present invention relates to a portable terminal and a position and orientation estimation program.

  Some portable terminals such as a smart phone have a function of estimating the position and orientation (or posture) of the portable terminal. Various methods have been proposed for estimating the position and orientation of the mobile terminal. The self-location and direction estimated by the mobile terminal are used to provide various services represented by navigation to the user of the mobile terminal.

  As an example of the service, there is a service that provides an intuitive interface with respect to a device arranged in the same space as the mobile terminal by estimating the position and orientation of the mobile terminal. For example, in a home application, if the mobile terminal is located near the TV and has an orientation facing the TV, the TV program guide is displayed on the mobile terminal, making it easy and intuitive to operate the TV. Can provide an interface capable of In addition, in the application example in the office, when the portable terminal is located in the vicinity of the printer and is stationary for a certain period of time in the direction facing the printer, the printer can easily print the content displayed on the portable terminal. In addition, an interface capable of intuitive operation can be provided.

  In a portable terminal equipped with an acceleration sensor, a gyro sensor, a geomagnetic sensor, and the like, the position and orientation of the portable terminal can be estimated within a predetermined time using inertial navigation. However, when using inertial navigation, it is necessary to determine in advance the initial position and initial orientation (or initial orientation) of the mobile terminal. Conventionally, in inertial navigation devices for airplanes and ships, initial values of state quantities such as posture in a stationary state of the inertial navigation device using a relatively high-performance inertial sensor by an initial setting called initial alignment. To decide. However, since such an initial setting must use a relatively high-performance inertial sensor, the performance of a MEMS (Micro Electro Mechanical Systems) sensor mounted on a general mobile terminal is The initial setting cannot be supported.

  On the other hand, there is also a method for estimating the position and orientation of a mobile terminal using information detected by a geomagnetic sensor, a GPS (Global Positioning System) sensor, or the like outdoors. However, in a room such as a factory or office, the geomagnetism is disturbed and the accuracy of information detected by the magnetic sensor may be reduced, and detection by the GPS sensor may not be performed. It is difficult to apply to estimation.

  For example, as in Patent Document 1, a method of detecting the attitude of a mobile terminal by using a dedicated tag that is optically read has also been proposed. However, it is necessary to move the portable terminal to a position where the dedicated tag can be optically read and to keep the portable terminal stationary until the reading is completed. Not suitable for office applications.

  As described above, the conventional technique uses a relatively inexpensive sensor mounted on the mobile terminal, and the initial position and the initial orientation (or so that the position and orientation detection accuracy is not greatly affected by the motion state of the mobile terminal. , Initial posture) is difficult to determine.

US Pat. No. 7,287,706 Special table 2010-520688 JP 2006-194725 A JP 2007-18451 A JP 2008-33526 A JP 2009-287940 A JP 2010-271167 A

  In the prior art, there has been a problem that it is difficult to detect the position and orientation of the mobile terminal with high accuracy using a relatively inexpensive sensor mounted on the mobile terminal.

  Therefore, the present invention provides a mobile terminal and a position and orientation estimation program capable of detecting the position and orientation of the mobile terminal with high accuracy using a relatively inexpensive sensor mounted on the mobile terminal. For the purpose.

  According to an aspect of the present invention, there is provided a mobile terminal that uses inertial navigation that performs short-range wireless communication with an external device that stores self-location information, and detects acceleration of the mobile terminal and outputs acceleration information. When the communication unit becomes communicable with the external device, the position information received from the external device is stored in the initial position of the mobile terminal. And setting the position, predicting the shooting timing based on the AC component of the acceleration information, controlling the shooting unit to take the image of the external device at the shooting timing, and estimated from the shot image of the external device There is provided a portable terminal comprising a control unit that sets a yaw angle to an initial orientation of the portable terminal.

  According to an aspect of the present invention, there is provided a program for causing a computer of a mobile terminal using inertial navigation that performs short-range wireless communication with an external device that stores self-position information to perform position and orientation estimation processing. A procedure for determining whether or not the communication unit can communicate with the external device, and if it is determined that the communication unit can communicate with the external device, position information received from the external device is transmitted to the mobile terminal. A procedure for setting the initial position, a procedure for predicting a shooting timing based on an AC component of acceleration information output from the acceleration sensor of the portable terminal, and the portable terminal so that an image of the external device is shot at the shooting timing. And a procedure for setting a yaw angle estimated from a captured image of the external device as an initial orientation of the portable terminal. A program characterized by causing executed computer is provided.

  According to the disclosed mobile terminal and the position and orientation estimation program, it is possible to detect the position and orientation of the mobile terminal with high accuracy by using a relatively inexpensive sensor mounted on the mobile terminal.

It is a block diagram which shows an example of the hardware constitutions of the portable terminal in one Example of this invention. It is a figure explaining the freedom degree of the position and azimuth | direction of a portable terminal. It is a perspective view explaining the position acquisition process of a NFC tag. It is a figure which shows an example of the coordinate system showing the presumed position of a portable terminal. It is a figure explaining the estimation process of the initial azimuth | direction of a portable terminal. It is a flowchart explaining the estimation process of the initial azimuth | direction of a portable terminal. It is a flowchart explaining the estimation of the movement state of the portable terminal linked with the NFC tag. It is a flowchart explaining the state estimation process of a portable terminal. It is a figure which shows SMA in case of triaxial acceleration and N = 1. It is a figure explaining an imaging timing determination process. It is a flowchart explaining an imaging timing determination process. It is a flowchart explaining the determination process of a still area. It is a figure explaining an estimation process of a roll angle and a pitch angle. It is a figure explaining the feature extraction of a tag image, and the direction calculation of a portable terminal. It is a figure explaining an example of direction calculation of a portable terminal.

  In the disclosed portable terminal and the position and orientation estimation program, the acceleration information of the portable terminal that performs short-range wireless communication with the external device that stores the self-position information is detected by the first detection unit, and the orientation information of the portable terminal is the second. It is detected by the detector. When the mobile terminal moves to a position where short-distance wireless communication with an external device is possible, the control unit of the mobile terminal sets the position information received from the external device as the initial position of the mobile terminal, and based on the AC component of the acceleration information The shooting timing of the external device by the shooting unit is predicted. The control unit controls the photographing unit to photograph the image of the external device at the predicted photographing timing, and sets the yaw angle estimated from the photographed image of the external device as the initial orientation of the mobile terminal.

  The control unit may set the yaw angle estimated from the captured image of the external device and the roll angle and pitch angle estimated from the DC component of the acceleration information and the orientation information as the initial orientation of the mobile terminal.

  Hereinafter, embodiments of the disclosed portable terminal and position and orientation estimation program will be described with reference to the drawings.

  FIG. 1 is a block diagram illustrating an example of a hardware configuration of a mobile terminal according to an embodiment of the present invention. In FIG. 1, the mobile terminal 500 includes a CPU 501, a storage unit 502, an input unit 503 including a numeric keypad, a display unit 504, a communication unit 505, and a sensor unit 506. In this example, the storage unit 502, the input unit 503, the display unit 504, the communication unit 505, and the sensor unit 506 are connected to the CPU 501 via the bus 507, but the storage unit 502, the input unit without using the bus 507 503, the display unit 504, the communication unit 505, and the sensor unit 506 may be directly connected to the CPU 501. When signals are directly transmitted to and received from the CPU 501 without going through the bus 507, the problem of traffic on the bus 507 can be reduced. A smart phone having a hardware configuration as shown in FIG. 1 is well known.

  At least a part of the input unit 503 and the display unit 504 may be integrally provided by a touch panel or the like. The sensor unit 506 includes, for example, an acceleration sensor 506a, a gyro sensor (or gyroscope) 506b, and a camera 506c. The sensor unit 506 may further include, for example, a geomagnetic sensor, a GPS sensor, and the like.

  The CPU 501 controls the entire mobile terminal 500 and can implement various functions of the mobile terminal 500 by executing various programs including a position and orientation estimation program. The storage unit 502 stores various data including programs executed by the CPU 501 and intermediate data such as calculation processing executed by the CPU 501 such as map data. The storage unit 502 can be formed by, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), an HDD (Hard Disk Drive), or the like. The input unit 502 is operated by the user when inputting commands and data to the mobile terminal 500. A display unit 503 displays input information from the input unit 502, a message to the user, various operation menus, a map, and the like. The communication unit 505 performs wireless communication with another device (not shown).

  A program (position and direction estimation program) for causing the portable terminal 500 to have at least a position and orientation estimation function causes the CPU 501 to operate as the portable terminal 500 having a position and orientation estimation function using inertial navigation. The program may be stored in a computer-readable storage medium. The computer-readable storage medium is limited to, for example, a semiconductor storage device such as an IC (Integrated Circuit) card memory, a magnetic disk such as a floppy (registered trademark) disk, a magneto-optical disk, and a portable recording medium such as a CD-ROM. It includes various recording media that are accessible by the CPU 500. Further, a computer-readable storage medium may be formed by the storage unit 502, for example.

  Needless to say, the CPU 501 may further realize at least a part of various functions such as a call function and a photographing function of the mobile terminal 500 by executing various programs in addition to the position and orientation estimation function.

  A smart phone is an example of the mobile terminal 500. The portable terminal 500 is not particularly limited as long as it is an electronic device having a wireless communication function, an acceleration detection function, an orientation (or orientation) detection function, and a photographing function. The wireless communication function can be realized by the communication unit 505 including, for example, a wireless transmitter and a wireless receiver, and at least performs two-way wireless communication with the NFC tag 10 that is well-known in near field communication (NFC) described later. Includes functions to perform. When it is not necessary to write information to the NFC tag 10, the wireless communication function can be realized by the communication unit 505 including the tag reader 505-1 that reads at least information stored in the NFC tag 10.

  The acceleration detection function can be realized by, for example, the acceleration sensor (or acceleration detection unit) 506a of the sensor unit 506. The direction detection function can be realized by, for example, the gyro sensor (or posture detection unit) 506b of the sensor unit 506. The imaging function can be realized by a camera (imaging unit or imaging unit) 506c such as a digital camera or a digital video camera of the sensor unit 506, for example.

  FIG. 2 is a diagram illustrating the position and orientation freedom of the mobile terminal. FIG. 2 shows a case where six degrees of freedom in space (three degrees of freedom for position and three degrees of freedom for direction (or posture)) are detected. In FIG. 2, X, Y, and Z indicate three axes of the XYZ coordinate system in which the position of the mobile terminal 500 is represented, and the orientation of the mobile terminal 500 is a rotation angle Roll such as an arrow about the X, Y, and Z axes. , Pitch, Yaw.

  Next, the operation of the portable terminal 500 will be described with reference to FIG. FIG. 3 is a perspective view for explaining the NFC tag position acquisition process.

  First, the NFC tag 10 is installed at an arbitrary position in an environment where the mobile terminal 500 is used, for example, at home or office. The NFC tag 10 is an example of an external device having a function of performing short-range wireless communication (NFC: Near Field Communication) with the mobile terminal 500. For example, the NFC tag 10 is disposed on the reference surface 11 at a height position where a user holding the mobile terminal 500 can easily hold the mobile terminal 500 against the NFC tag 10. The reference plane 11 is a horizontal plane that is provided outside the mobile terminal 500 and is substantially perpendicular to the direction of gravity, and in this example is the upper surface of a table or the like.

  NFC is an international standard for wireless communication. For example, 13.56 MHz radio waves are used for communication between the portable terminal 500 and the NFC tag 10, and bidirectional communication of, for example, 100 kbps to 400 kbps is possible at a short distance of about 10 cm. As shown in FIG. 3, if the mobile terminal 500 is at a short distance where communication with the NFC tag 100 is possible, the communication unit 505 can write information to the NFC tag 10 or read information from the NFC tag 10.

  The communication unit 505 (for example, tag reader 505-1) of the mobile terminal 500 reads the tag position information stored in the NFC tag 10 when the mobile terminal 500 comes close to the NFC tag 10. The fact that the communication unit 505 of the mobile terminal 500 can read the information of the tag position of the NFC tag 10 by NFC means that the mobile terminal 500 is close to the NFC tag 10, so the read tag position is the estimated position of the mobile terminal 500. To do. FIG. 4 is a diagram illustrating an example of a coordinate system representing the estimated position of the mobile terminal. FIG. 4 shows an XYZ coordinate system representing the estimated position of the mobile terminal 500.

  The azimuth (or posture) of the mobile terminal 500 represented by a roll angle, a pitch angle, and a yaw angle is estimated by executing the following processing.

  When the communication unit 505 detects the NFC tag 10 because it can communicate with the NFC tag 10, it starts buffering processing of the acceleration detected by the acceleration sensor 506 a and the direction detected by the gyro sensor 506 b, and also detects the detected acceleration. Based on this, the determination process of the movement section of the mobile terminal 500 is started in real time. Further, when the removal from the stationary state of the mobile terminal 500 (or the transition to the motion state) is detected by a method described later, the shooting time (or shooting timing) of the NFC tag 10 based on the prediction in the initial speed and the initial orientation. Calculate It is assumed that the initial position and initial orientation of the mobile terminal 500 are set by touching the NFC tag 10 with the mobile terminal 500. For this reason, the communication unit 505 detects the NFC tag 10, the integral value (ie, velocity) of the linear acceleration component of the acceleration detected by the acceleration sensor 506a is zero (0), and the angular velocity detected by the gyro sensor 506b is zero. If it is (0), it can be seen that the mobile terminal 500 is stationary at a position close to the NFC tag 10, so that the contact state between the mobile terminal 500 and the NFC tag 10 can be estimated. Further, if the communication unit 505 cannot detect the NFC tag 10, that is, if it cannot communicate with the NFC tag 10, it is possible to estimate the non-contact state of the mobile terminal 500, the NFC tag 10, and the NFC tag 10. The camera 506c captures an image of the NFC tag 10 at the calculated capturing time. When the photographing of the NFC tag 10 by the camera 506c is completed, the contact state or non-contact state between the mobile terminal 500 and the NFC tag 10 is estimated, and a stationary section for estimating the Roll angle and the Pitch angle is obtained again. The roll angle and pitch angle are estimated by calculating the gravitational vector in the obtained stationary section. Further, the Yaw angle of the portable terminal 500 is obtained by extracting the characteristics of the image of the photographed NFC tag 10. The position and orientation are updated again up to the current time with the position and orientation of the mobile terminal 500 estimated in the stationary section, and the position and orientation are continuously estimated.

  Accordingly, the position and orientation of the mobile terminal 500 can be estimated simply by touching the NFC tag 10 placed on the horizontal plane (hereinafter referred to as touch) with the mobile terminal 500. When the NFC tag 10 is touched with the mobile terminal 500, one surface (for example, the bottom surface) of the mobile terminal 500 is brought into contact with the surface of the NFC tag 10. At this time, the position and orientation of the mobile terminal 500 can be accurately estimated without being affected by the presence or absence of a touch impact. In order to estimate the position and orientation of the mobile terminal 500, an operation that burdens the user such as keeping the mobile terminal 500 stationary for a relatively long time is unnecessary. For example, by calculating the average of a plurality of estimated positions and the average of a plurality of estimated orientations by touching a plurality of NFC tags 10, it is generated by inertial navigation using an inexpensive inertial sensor (acceleration sensor 506a, gyro sensor 506b). The estimation error may be reduced. If the existing sensors 506a, 506b, and 506c installed in the mobile terminal 500 are used to estimate the position and orientation of the mobile terminal 500, the hardware configuration of the mobile terminal 500 is changed or the mobile terminal 500 is used. There is no need to install a large number of dedicated devices in the environment.

  As described above, the position and orientation of the mobile terminal 500 are estimated by linking the acceleration sensor 506a, the gyro sensor 506b, and the camera 506c mounted on the mobile terminal 500. When estimating the position of the portable terminal 500, the distance that the portable terminal 500 can communicate with the NFC tag 10 is as short as 1 cm to 10 cm, for example, and therefore the position (X, Y, Z) of the NFC tag 10 is the position of the portable terminal 500. Estimate as Further, when estimating the azimuth (or posture) of the mobile terminal 500, the mobile terminal 500 has moved to a position where short-range wireless communication with the NFC tag 10 is possible, that is, between the mobile terminal 500 and the NFC tag 10. By using the start of short-range wireless communication as a trigger, and obtaining a stationary section in which the NFC tag 10 is touched on the portable terminal 500 based on the acceleration detected by the acceleration sensor 506a and the angular velocity detected by the gyro sensor 506b. Specify a roll angle and pitch angle of 500. Furthermore, the motion of the mobile terminal 500 is predicted, and an image of the NFC tag 10 is captured by the camera 506c and analyzed, so that the Yaw angle of the mobile terminal 500 is estimated.

  Next, the estimation of the initial position and initial orientation (or initial posture) of the mobile terminal 500 will be described in more detail.

  The initial position of the portable terminal 500 can be estimated (or set) by the following procedures ST1 to ST4. First, in step ST1, the position (x, y, z) of the NFC tag 10 observed in advance in the absolute coordinate system is measured. In step ST2, the measured position of the NFC tag 10 is written to the NFC tag 10 by a known method as unique information (that is, position information) of the NFC tag 10. For example, the position information of the NFC tag 10 is written into the NFC tag 10 using the tag writer 505-2 of the communication unit 505 or the tag writer of the dedicated device for initial setting. In procedure ST3, when the NFC tag 10 is touched with the portable terminal 500, the position information of the NFC tag 10 is read by a known method using the tag reader 505-1 of the communication unit 505. In step ST4, since the tag reader 505-1 reads the position information of the NFC tag 10 by short-range wireless communication, the position of the NFC tag 10 indicated by the read position information is set as the estimated position of the mobile terminal 500 in the absolute coordinate system. .

  On the other hand, the initial azimuth (or initial posture) of the mobile terminal 500 is obtained by grouping the Roll angle and Pitch angle that can be estimated with the triaxial orthogonal acceleration data and the Yaw angle that cannot be estimated with the triaxial orthogonal acceleration data. Estimate in different ways. FIG. 5 is a diagram illustrating an estimation process of the initial orientation of the mobile terminal, and FIG. 6 is a flowchart illustrating the estimation process of the initial orientation of the mobile terminal.

  FIG. 5 illustrates a data buffering period for buffering detection data of the acceleration sensor 506a and the gyro sensor 506b, an image capturing period for capturing an image of the NFC tag 10 by predicting the movement of the mobile terminal 500, and a sensor section (or capturing). (Timing) specification and the direction estimation period based on the captured image, and the current position update period in the estimated direction are indicated by white arrows. The approaching section is a section in which the mobile terminal 500 approaches the NFC tag 10, the stationary section is a section in which the mobile terminal 500 is in a stationary state by touching the NFC tag 10, and the separation section is in a state in which the portable terminal 500 is detached from the stationary state (that is, , Move) and transition to the motion state. The portable terminal 500 can communicate with the NFC tag 10 from time t1 (that is, the NFC tag 10 can be detected), touch the NFC tag 10 to be in a stationary state at time t2, and can communicate with the NFC tag 10 from time t3. Leave the state (that is, the NFC tag 10 cannot be detected).

  In FIG. 6, when the mobile terminal 500 detects the NFC tag 10 because the mobile terminal 500 can communicate with the NFC tag 10, buffering of the detection data 502 </ b> A and 502 </ b> B of the acceleration sensor 506 a and the gyro sensor 506 b to the storage unit 502 is performed. Starts and starts discriminating the movement section in real time based on the acceleration. For example, when the communication unit 505 of the mobile terminal 500 starts communication with the NFC tag 10, it can be detected that the mobile terminal 500 can communicate with the NFC tag 10. In step S <b> 2, when detecting the removal of the mobile terminal 500 from the stationary state, the imaging time of the NFC tag 10 by the camera 506 c is calculated based on the initial addition and the prediction in the initial orientation, and stored in the storage unit 502. In step S3, the camera 506c captures the NFC tag 10 at the calculated capturing time, and the captured image data 502C is stored in the storage unit 502.

  In step S4, when photographing is completed, the contact state or non-contact state between the mobile terminal 500 and the NFC tag 10 is confirmed based on the detection data 502A and 502B, and then the stationary section is identified and stored in the storage unit 502. Store. In parallel with step S4, step S5 extracts the feature of the photographed image from the photographed image data 502C, specifies the Yaw angle of the portable terminal 500 from the extracted feature, and stores it in the storage unit 502. After step S <b> 4, step S <b> 6 specifies the roll angle and pitch angle of the portable terminal 500 based on the acceleration detection data 502 </ b> A in the stationary section and stores it in the storage unit 502. After step S5 or S6, step S7 updates again the position and direction up to the current time with the position and direction estimated in the stationary section and stores them in the storage unit 502.

  In order to accurately estimate the Roll angle and Pitch angle from the three-axis orthogonal acceleration data, for the reason described later, in order to perform the process of FIG. The stationary state is specified by estimating the motion state such as first. In addition, in order to determine the timing for capturing an image of the NFC tag 10 in order to specify the Yaw angle, the capturing timing based on the motion state of the mobile terminal 500 is predicted. As a result, it is possible to specify the roll angle and the pitch angle of the mobile terminal 500 based on the acceleration, and the Yaw angle of the mobile terminal 500 based on the image capturing.

  Next, estimation of the motion state of the mobile terminal 500 in conjunction with the NFC tag 10 will be described. The estimation of the motion state of the portable terminal 500 is performed until the image capturing of the NFC tag 10 is completed after the NFC tag 10 is detected. FIG. 7 is a flowchart illustrating the estimation of the motion state of the mobile terminal in conjunction with the NFC tag.

  In FIG. 7, the timing of state estimation of the mobile terminal 500 can be determined as follows. When the mobile terminal 500 approaches a certain distance (for example, 1 cm to 10 cm) from the NFC tag 10, the communication unit 505 can communicate with the NFC tag 10, and thus an NFC tag detection event occurs in step S <b> 11. When this NFC tag detection event occurs, the state estimation process of the portable terminal 500 is activated. Thus, since the state estimation process is activated only when an NFC tag detection event occurs, the power consumption of the mobile terminal 500 can be suppressed.

  The filtering process for the state estimation process can be performed as follows. In order to estimate the state of the mobile terminal 500, step S12 includes gravity data (component) based on the gravity vector and triaxial acceleration (Linear Acceleration) data (component) based on the motion of the mobile terminal 500 from the acceleration data detected by the acceleration sensor 506a. Isolate. The gravity data is obtained as the DC component 502A-2 of the acceleration data by passing the acceleration data through a low-pass filter (LPF). On the other hand, the triaxial acceleration data is obtained as an AC component 502A-1 of acceleration data by passing the acceleration data through a high-pass filter (HPF). Since LPF and HPF are well known, the description thereof is omitted. By using the separated triaxial acceleration data, it is possible to estimate the motion state or the stationary state of the mobile terminal 500.

  The buffering process can be performed as follows. In order to estimate the state of the portable terminal 500, step S12 buffers the filtered AC component 502A1-1 and acceleration DC component 502A-2 in the storage unit 502, and also detects the gyro data detected by the gyro sensor 506b. 502B is buffered in the storage unit 502. In step S12, the tag position information received from the NFC tag 10 detected in step S11 is buffered in the storage unit 502 as the initial position information of the portable terminal 500.

  The stationary state and the exercise state (or detachment from the stationary state) of the mobile terminal 500 can be estimated as follows. Since the linear acceleration data excluding the gravity data indicates the motion state of the mobile terminal 500, the motion state of the mobile terminal 500 can be determined in real time. Therefore, in step S13, the stationary state and the motion state of the mobile terminal 500 are estimated from the AC component 502A-1.

  FIG. 8 is a flowchart for explaining the state estimation process of the mobile terminal. The state estimation process in FIG. 8 corresponds to the processes in steps S12 and S13. In FIG. 8, step S21 acquires acceleration data detected by the acceleration sensor 506a. In step S22, the acquired acceleration data is separated into an acceleration DC component 502A-2 that is gravity data and an acceleration AC component 502A-1 that is triaxial acceleration data. In step S23, an SMA (Signal Magnitude Area) of the AC component 502A-1 of acceleration is calculated. The SMA is a feature amount indicating the magnitude (energy) of the momentum within a certain time T, and can be calculated from the following equation.

  If the SMA level is equal to or greater than the certain time TM and the SMA level is equal to or less than the motion and stillness threshold indicated by the thick broken line in FIG. On the other hand, when the level of the SMA exceeds the motion and stillness threshold, it is possible to detect the transition of the mobile terminal 500 from the stationary state to the motion state and determine that the mobile terminal 500 is in the motion state. Step S24 classifies whether portable terminal 500 is in a stationary state or an exercise state based on the level of SMA.

  Returning to the description of FIG. 7, the shooting timing based on the estimation of the motion state of the mobile terminal 500 can be determined as follows. In order to estimate the Yaw angle of the mobile terminal 500 based on image processing to be described later, when the mobile terminal 500 leaves the NFC tag 10 after touching the NFC tag 10, the mobile terminal 500 is placed at an appropriate height position from the NFC tag 10. In step S14, an appropriate shooting timing is determined based on prediction (that is, calculation) so that the NFC tag 10 can be shot in a state of being (or distance). If the shooting timing is too early or too late, there is a possibility that the characteristics of the NFC tag 10 necessary for image processing cannot be extracted (or captured) from the image data of the shot NFC tag 10. Therefore, the acceleration AC component 502A-1 buffered in step S12 and detected in step S13 so that the mobile terminal 500 can photograph the NFC tag 10 at an appropriate height position (distance from the NFC tag 10). The imaging timing determination process is performed using the transition from the stationary state to the motion state (or the departure from the stationary state).

  FIG. 10 is a diagram illustrating the photographing timing determination process. In FIG. 10, S indicates the minimum photographing distance required for photographing the NFC tag 10 by the camera 506c, and is assumed to be designated in advance, for example. As shown in FIG. 10, when the mobile terminal 500 is detected to be stationary or detached from the stationary state in step S13, the photographing timing is determined in step S14, and then the NFC tag 10 is photographed by the camera 506c in step S15. Is done. Details of step S15 will be described later.

  Next, determination of the stationary section of the mobile terminal 500 will be described. In FIG. 7, step S16 determines a stationary section of the mobile terminal 500 for estimating the roll angle and the pitch angle of the mobile terminal 500 based on the DC component 502A-2 of acceleration, as will be described later. Although it is possible to detect the stationary state of the mobile terminal 500 based on the SMA of the AC component 502A-1 of acceleration, the rotation of the mobile terminal 500 that cannot be detected only by the AC component 502A-1 of acceleration is also taken into consideration. Also good. Therefore, in this example, the SMA of the gyro data 502-B detected by the gyro sensor 506b is also evaluated at the same time as the SMA of the AC component 502A-1 of the acceleration, and the common section on the time axis of the stationary section obtained from each SMA. (That is, the product set) is determined as a stationary interval for estimation.

  FIG. 12 is a flowchart for explaining still area determination processing. In FIG. 12, in step S <b> 41, buffered gyro data 502 -B and acceleration AC component 502 </ b> A- 1 are acquired from the storage unit 502. In step S42, SMAs of the acquired gyro data 502-B and the AC component 502A-1 of the acquired acceleration are calculated. A step S43 specifies a section in which each calculated SMA is equal to or less than a specified SMA threshold. In step S44, a common section (that is, a product set) of the two sections identified in step S43 is identified and determined as a still section for estimating the Roll angle and the Pitch angle.

  Next, a process for estimating the Roll angle and the Pitch angle using the three-axis orthogonal acceleration data in the determined still section will be described. In FIG. 7, step S17 is a gravitational vector or gravity data (that is, a DC component of acceleration data) with the three-axis orthogonal acceleration data (that is, the AC component 502A-1 of acceleration) of the stationary section of the mobile terminal 500 determined in step S16. 502A-2) is used to estimate the roll angle and pitch angle of the portable terminal 500.

  FIG. 13 is a diagram for explaining the roll angle and pitch angle estimation processing. As shown in FIG. 13, first, the Z axis of the absolute coordinate system (X, Y, Z) is set in the direction of gravity. Rotate around the Y axis of the absolute coordinate system by an angle θ (X, Y, Z → X ', Y', Z '), and then rotate around the X' axis by an angle φ (X ', Y', Z '→ X ", Y", Z "). Also, the coordinate system (X", Y ", Z") is the body coordinate system (local coordinate system) that is actually placed on the mobile device 500. By calculating the rotation angles φ and θ based on the observed triaxial acceleration data, the roll angle φ and the pitch angle θ of the mobile terminal 500 can be estimated. The above two rotation transformation matrix is as follows.

  Next, a process for estimating the Yaw angle based on the image data of the NFC tag 10 photographed by the camera 506c will be described. It is difficult to estimate the Yaw angle of the portable terminal 500 based on the acceleration data detected by the acceleration sensor 506a. Therefore, in FIG. 7, step S15 estimates the Yaw angle based on the image data of the NFC tag 10 taken by the camera 506c. Specifically, first, the orientation data of the NFC tag 10 is registered in advance in the NFC tag 10 or, for example, registered in a database (DB: Data-Base) in the storage unit 502. Further, the NFC tag 10 is photographed at the photographing timing determined in step S14 by the camera 506c of the portable terminal 500. Furthermore, the orientation of the portable terminal 500 is calculated based on the relative orientation relationship between the camera 506c and the NFC tag 10 and the registered absolute orientation of the NFC tag 10 by extracting the characteristics of the image data of the photographed NFC tag 10.

  FIG. 14 is a diagram for explaining the feature extraction of the tag image and the orientation calculation of the mobile terminal. As shown in FIG. 14, in step S101, features of the NFC tag 10 are extracted by well-known template matching using the captured raw image of the NFC tag 10 and the template image. In this example, since the corner (that is, tag corner) of the NFC tag 10 is extracted as a feature of the NFC tag 10, the template image is a corner image (or corner template) of the NFC tag 10. The template can be prepared in advance and can be registered in the NFC tag 10. The template can also be registered in a remote server or a DB in the storage unit 502 of the portable terminal 500 so that it can be accessed with a tag ID corresponding to the NFC tag 10.

  In step S102, the orientation of the NFC tag 10 and the NFC tag 10 (that is, the relative orientation with respect to the camera 506c) is determined based on the geometric feature of the extracted feature of the NFC tag 10 (the tag corner indicated by a circle in FIG. 14). In step S103, the orientation of the camera 506c (that is, the portable terminal 500) is calculated based on the orientation of the NFC tag 10 determined in step S102 and the absolute orientation of the NFC tag 10 registered in the NFC tag 10. Thereby, as shown in FIG. 14, the coordinate system of the portable terminal 500 under the absolute coordinate system O is obtained with respect to the absolute coordinate system O, the tag coordinate system T, and the difference d extracted from the image by template matching.

  By the way, after detecting the NFC tag 10, the mobile terminal 500 estimates the Yaw angle by the image processing as described above by continuously capturing images with the camera 506 c instead of determining the capturing timing based on the prediction. You may do it. In this case, it is possible to select a plurality of images from which many features can be extracted from continuously shot images, and it is possible to improve the estimation accuracy of the Yaw angle.

  Also, the Roll angle and Pitch angle estimated based on the acceleration data of the stationary section and the Yaw angle estimated based on the image captured at the predicted capturing timing are acquired at different timings, so they are used as the initial direction of inertial navigation When doing so, the Roll angle and Pitch angle may be updated again. In this case, since the gyro data 502B until the image of the NFC tag 10 is captured is buffered, the Yaw angle estimation is completed after the mobile terminal 500 detects the movement away from the NFC tag 10. The roll angle and the pitch angle may be corrected with the gyro data 502B. The Roll angle and Pitch angle corrected in this way and the Yaw estimated from the captured image may be set as the initial orientation.

  According to the above embodiment, since the initial position and the initial direction of the mobile terminal 500 can be accurately set with a relatively simple operation, the position and the direction of the mobile terminal 500 are then subjected to inertial navigation based on the initial position and the initial direction. It becomes possible to detect with high accuracy by a known method to be used.

  As an example of a service provided after setting the initial position and initial orientation of the mobile terminal 500, the position and orientation of the mobile terminal 500 with respect to the initial position and initial orientation can be estimated by a well-known method. A service that provides an intuitive interface for devices arranged in a space. For example, in an application example at home, if the mobile terminal 500 is located near a television (not shown) and has an orientation facing the television, the television program guide is displayed on the display unit 504 of the mobile terminal 500. Therefore, it is possible to provide an interface that allows simple and intuitive operation of the television. Also, in an application example in an office, when the mobile terminal 500 is located near a printer (not shown) and is stationary for a certain time in an orientation facing the printer, the content displayed on the display unit 504 of the mobile terminal 500 It is possible to provide an interface that allows simple and intuitive operation of the printer such as printing with a printer.

  In the above example, the mobile terminal can communicate with the NFC tag, but the device with which the mobile terminal communicates is not particularly limited as long as the device can perform short-range wireless communication (NFC) and can store its own position.

The following additional notes are further disclosed with respect to the embodiment including the above examples.
(Appendix 1)
A mobile terminal using inertial navigation for short-range wireless communication with an external device storing self-location information,
A first detector that detects acceleration of the mobile terminal and outputs acceleration information;
A shooting section for shooting images;
A communication unit that communicates with the external device;
When the communication unit can communicate with the external device, the position information received from the external device is set to the initial position of the mobile terminal, and the imaging timing is predicted based on the AC component of the acceleration information, The image capturing unit is controlled to capture an image of a device at the image capturing timing, and a control unit is provided that sets a yaw angle estimated from an image captured by the external device as an initial orientation of the mobile terminal. Mobile device.
(Appendix 2)
A second detector for detecting orientation information of the mobile terminal;
The control unit sets the yaw angle estimated from the captured image of the external device, the roll angle and the pitch angle estimated from the DC component of the acceleration information and the orientation information as the initial orientation of the mobile terminal, The mobile terminal according to appendix 1.
(Appendix 3)
When the communication unit is able to communicate with the external device, the control unit estimates a stationary section where the mobile terminal contacts the external device based on an AC component of the acceleration information and the orientation information, and the stationary section The mobile terminal according to appendix 2, wherein the roll angle and the pitch angle are estimated.
(Appendix 4)
A storage unit;
The mobile terminal according to appendix 2 or 3, wherein the control unit filters the acceleration information into the AC component that is a linear acceleration component and the DC component that is a gravity component, and stores the filtered information in the storage unit. .
(Appendix 5)
The control unit performs the estimation of the stationary section and the estimation of the roll angle and the pitch angle in the stationary section, triggered by the start of communication of the communication unit with the external device. The terminal device of any one of thru | or 4.
(Appendix 6)
A program for causing a computer of a mobile terminal using inertial navigation to perform short-range wireless communication with an external device storing self-position information to perform position and orientation estimation processing,
A procedure for determining whether the communication unit of the mobile terminal can communicate with the external device;
When it is determined that the communication unit can communicate with the external device, a procedure for setting position information received from the external device as an initial position of the mobile terminal;
A procedure for predicting shooting timing based on an AC component of acceleration information output by the acceleration sensor of the mobile terminal;
A procedure for controlling the photographing unit of the portable terminal so as to photograph the image of the external device at the photographing timing;
A program for causing the computer to execute a procedure for setting a yaw angle estimated from a captured image of the external device as an initial orientation of the mobile terminal.
(Appendix 7)
The procedure for setting the initial azimuth includes a yaw angle estimated from a captured image of the external device, a DC component of the acceleration information, and a roll angle and a pitch angle estimated from the azimuth information output by the gyro sensor of the mobile terminal. The program according to appendix 6, wherein the program is set to the initial orientation of the mobile terminal.
(Appendix 8)
When it is determined that the communication unit can communicate with the external device, a procedure for estimating a stationary section where the mobile terminal contacts the external device based on the AC component of the acceleration information and the orientation information;
The program according to claim 7, further causing the computer to execute a procedure for estimating the roll angle and the pitch angle in the stationary section.
(Appendix 9)
The computer further executes a procedure for filtering the acceleration information into the AC component that is a linear acceleration component and the DC component that is a gravity component and storing the filtered information in the storage unit of the mobile terminal. Or the program of 8.
(Appendix 10)
The procedure for setting the initial position determines that the communication unit can communicate with the external device upon detecting the start of communication between the communication unit and the external device. The program according to any one of the above.

  As described above, the disclosed mobile terminal and the position and orientation estimation program have been described by way of examples. However, the present invention is not limited to the above examples, and various modifications and improvements can be made within the scope of the present invention. Needless to say.

10 NFC tag 500 Mobile terminal 501 CPU
502 storage unit 505 communication unit 506 sensor unit 506a acceleration sensor 506b gyro sensor 506c camera

Claims (5)

A mobile terminal using inertial navigation for short-range wireless communication with an external device storing self-location information,
A first detector that detects acceleration of the mobile terminal and outputs acceleration information;
A shooting section for shooting images;
A communication unit that communicates with the external device;
When the communication unit can communicate with the external device, the position information received from the external device is set to the initial position of the mobile terminal, and the imaging timing is predicted based on the AC component of the acceleration information, The image capturing unit is controlled to capture an image of a device at the capturing timing, and a control unit is provided that sets a yaw angle estimated from a captured image of the external device as an initial orientation of the mobile terminal. Mobile device. A second detector for detecting orientation information of the mobile terminal;
The control unit sets the yaw angle estimated from the captured image of the external device, the roll angle and the pitch angle estimated from the DC component of the acceleration information and the orientation information as the initial orientation of the mobile terminal, The mobile terminal according to claim 1.   When the communication unit is able to communicate with the external device, the control unit estimates a stationary section where the mobile terminal contacts the external device based on an AC component of the acceleration information and the orientation information, and the stationary section The mobile terminal according to claim 2, wherein the roll angle and the pitch angle are estimated. A storage unit;
The mobile unit according to claim 2 or 3, wherein the control unit filters the acceleration information into the AC component that is a linear acceleration component and the DC component that is a gravity component and stores the filtered information in the storage unit. Terminal. A program for causing a computer of a mobile terminal using inertial navigation to perform short-range wireless communication with an external device storing self-position information to perform position and orientation estimation processing,
A procedure for determining whether the communication unit of the mobile terminal can communicate with the external device;
When it is determined that the communication unit can communicate with the external device, a procedure for setting position information received from the external device as an initial position of the mobile terminal;
A procedure for predicting shooting timing based on an AC component of acceleration information output by the acceleration sensor of the mobile terminal;
A procedure for controlling the photographing unit of the portable terminal so as to photograph the image of the external device at the photographing timing;
A program for causing the computer to execute a procedure for setting a yaw angle estimated from a captured image of the external device as an initial orientation of the mobile terminal.