JP2020113159A - Information terminal device and program - Google Patents

Abstract

To provide an information terminal device which can suppress delay and estimate a posture.SOLUTION: The information terminal device includes: an imaging unit 1 for imaging and obtaining a taken image; a first estimation unit 3 for estimating attitude information of the imaging unit at a past time from the taken image; a sensor unit 2 for continuously acquiring sensor values on which the attitude of the imaging unit is to be reflected; a second estimation unit 4 for estimating attitude change information to a future time from the sensor values acquired continuously; and a synthesis unit 5 for obtaining synthesis attitude information formed by synthesizing the attitude information and the attitude change information.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information terminal device and a program capable of estimating a posture while suppressing a delay.

Techniques for estimating the orientation of a camera that captures an image include, for example, techniques disclosed in Patent Documents 1 and 2 and Non-Patent Document 1. Patent Document 1 proposes a method of estimating a posture change between images by extracting a feature descriptor from an image and obtaining a correspondence relationship between the descriptors in a plurality of images. Reliability is enhanced by using sensor information for normalization of feature descriptors. Patent Document 2 proposes a method of estimating an initial posture with a sensor and correcting it with an image. It is argued that the sensor reduces the solution space of pose estimation and enhances the convergence of pose optimization by images. In Non-Patent Document 1, the posture is estimated from an image and a sensor by using deep learning.

JP 2014-241155 JP Special table 2015-532077 bulletin

Ronald Clark, Sen Wang, Hongkai Wen, Andrew Markham, Niki Trigoni, ``VINet: Visual-Inertial Odometry as a Sequence-to-Sequence Learning Problem,'' Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pp. 3995 --4001, 2017. Kato, H., Billinghurst, M. ``Marker Tracking and HMD Calibration for a video-based Augmented Reality Conferencing System,'' In Proc. of the 2nd Int. Workshop on Augmented Reality, 1999. D.G.Lowe, ``Distinctive image features from scale-invariant key points,'' Proc. of Int. Journal of Computer Vision, 60(2) pp.91-110, 2004.

However, the above conventional techniques have the following problems. That is, since a processing load is required to estimate a posture from an image having a large amount of information, it takes time from image capturing to posture calculation. Therefore, the posture may change when the posture calculation result is obtained. There is. In particular, if you want to realize an augmented reality (AR) display application by using this attitude calculation result when the real world scenery can be viewed without delay on an optical see-through type head mounted display (HMD), etc. There is a problem that the reflection of the estimation result will be delayed.

As a result of this delay, when the camera is moving with respect to the background or the object, the display may be unnatural from the standpoint of the user who experiences the AR display using such conventional posture estimation. was there. In other words, the AR display is performed based on the posture at the past time that is different from the posture at the current time with respect to the visible background or object at the current time, so that the AR display deviates from the original position to be superimposed. There is a risk that superimposition will be performed at the position where it is displayed, resulting in an unnatural display. For example, when the movement is fast, even if the posture at the past time with a delay of only 0.1 seconds to 0.2 seconds from the current time is used, the display may be unnatural.

In view of the problems of the above-mentioned conventional techniques, it is an object of the present invention to provide an information terminal device and a program capable of estimating a posture while suppressing a delay.

In order to achieve the above-mentioned object, the present invention is an information terminal device, and an image capturing unit that captures an image to obtain a captured image, and first estimation that estimates posture information of the image capturing unit at a past time from the captured image. Unit, a sensor unit that continuously acquires a sensor value that reflects the attitude of the imaging unit, and a second estimation unit that estimates attitude change information toward a future time from the continuously acquired sensor value. A combining unit that obtains combined posture information that combines the posture information and the posture change information. Further, it is a program for causing a computer to function as the information terminal device.

According to the present invention, the posture information in the past is estimated with high accuracy from the captured image, the posture change information toward the future time is estimated based on the sensor value, and the posture information and the posture change information are calculated. By obtaining the synthesized combined posture information, when the current time reaches the future time, delay can be suppressed by this combined posture information and a highly accurate posture estimation result can be obtained.

It is a functional block diagram of the information terminal device which concerns on one Embodiment. It is a figure which shows the example of the hardware constitutions of the general computer apparatus which can implement|achieve an information terminal device. It is a figure which shows the schematic example of the time transition of operation|movement as a whole information terminal device by each part of an information terminal device operating in cooperation. 6 is a flowchart of the operation of the imaging unit according to the embodiment. It is a flow chart of operation of the first estimating part concerning one embodiment. 6 is a flowchart of the operation of the sensor unit according to the embodiment. It is a flow chart of operation of the second estimating part concerning one embodiment. It is a flow chart of operation of a synthetic part and a presentation part concerning one embodiment.

FIG. 1 is a functional configuration diagram of an information terminal device according to an embodiment. As illustrated, the information terminal device 10 includes an imaging unit 1, a sensor unit 2, a first estimating unit 3, a second estimating unit 4, a synthesizing unit 5, and a presenting unit 6. The hardware configuration of the information terminal device 10 can be realized by a general computer device implemented as a smartphone, a tablet terminal, a desktop or laptop computer, an HMD, or the like.

FIG. 2 is a diagram showing an example of a hardware configuration of a general computer device 20 capable of realizing the information terminal device 10. As shown in FIG. 2, the computer device 20 includes a CPU (central processing unit) 101 that executes a predetermined instruction, and a dedicated processor 102 that executes a part or all of the execution instructions of the CPU 101 instead of the CPU 101 or in cooperation with the CPU 101. (GPU (graphics processing unit), deep learning dedicated processor, etc.), RAM103 as a main storage device that provides a work area for the CPU101 and dedicated processor 102, ROM104 as an auxiliary storage device, camera 201, a normal see-through type display 202, a see-through type display 203 for realizing an optical see-through type HMD, a sensor 204 and a communication interface 205, and a bus B for exchanging data among them. It should be noted that the display 202 and the see-through display 203 may be provided with only one of them depending on the embodiment.

Each unit of the information terminal device 10, which will be described in detail later, can be realized by the CPU 101 and/or the dedicated processor 102 that reads a predetermined program corresponding to the function of each unit from the ROM 104 and executes the program. Here, when the imaging-related processing is performed, the camera 201 further operates in an interlocking manner, and when the display-related processing is performed, the display 202 or the see-through display 203 further operates in an interlocking manner. When various sensor values are acquired, the sensor 204 (one or more types) further operates in conjunction, and when communication-related processing related to data transmission/reception via a network is performed, the communication interface 205 further interlocks. And work.

The information terminal device 10 may be realized by one computer device 20 such as a smartphone terminal, or may be realized by two or more computer devices 20. For example, a first computer device 20 that is an HMD terminal including at least a camera 201 and a see-through display 203, and a second computer device 20 that is a smartphone terminal that provides AR superimposed content to be displayed on this HMD terminal. The information terminal device 10 may be a system that enables mutual communication by short-range wireless communication via the communication interface 205 of both terminals. In addition to this or instead of this, in the third computer device 20 which is a server terminal on the network, a part of the information terminal device 10 (when realizing AR display at the site where the user exists as a use, The functions of the imaging unit 1, the sensor unit 2, and the presentation unit 6) may be realized.

Hereinafter, each unit of the information terminal device 10 will be described. FIG. 3 is a diagram showing a schematic example of a time transition of the operation of the information terminal device 10 as a whole by the respective parts of the information terminal device 10 operating in cooperation with each other. In FIG. 3, the time axis is shown as, for example, times t ₁ , t ₂ ,..., T ₈ on the line L1 as being common in the horizontal axis direction, and a schematic example of the processing of each unit on this time axis is shown. It is shown. In the following description, the example of FIG. 3 will be referred to as appropriate.

The image capturing unit 1 captures an image and outputs the obtained image to the first estimating unit 3 and the presenting unit 6 together with time information (time stamp of image capturing time) as image capturing information. The imaging unit 1 can be realized by using a camera 201 such as a digital camera as hardware.

FIG. 4 is a flowchart of the operation of the image capturing unit 1 according to the embodiment. In step S11, it is determined whether or not the current time has reached the image capturing timing. If the current time has arrived, the process proceeds to step S12, and if not, the process returns to step S11 and this determination is repeated. In step S12, imaging is performed at this imaging timing, and imaging information in which a time stamp is associated with the obtained image is transferred to the first estimating unit 3 and the presenting unit 6 (that is, the first estimating unit 3 Then, the presentation unit 6 transfers the image data to the RAM 103 as an image buffer to be referred to for processing, and then returns to step S11. According to the flow of FIG. 4 described above, the image pickup by the image pickup unit 1 is repeated at each image pickup timing determined in step S11.

The image capturing timing in step S11 may be set at predetermined time intervals specified in advance, but the actual image capturing time (image capturing time reflected as time information in the image capturing information) is, for example, processing completion such as focusing of an image. It may be that there is a deviation from a predetermined imaging timing by waiting for. In the example of FIG. 3, the imaging information I ₁ , I ₂ ,..., I ₈ as shown on the line L2 is obtained at the imaging times t ₁ , t ₂ ,..., T ₈ as shown on the line L1. Then, the transfer is started, and the state in which the transfer is completed is shown on the line L3. This transfer is typically subject to non-uniform delay. The cause of this non-uniform delay is that other processes (such as background processes managed by the operating system other than the processes related to the present invention) are concurrently performed in the information terminal device 10 which is a general-purpose computer device 20. Including non-uniform load and the like caused by the implementation of. In the example of FIG. 3, due to this non-uniform delay, for example, the transfer required time of the image pickup information I ₃ at the image pickup time t ₃ is longer than the transfer required time of the image pickup information I ₂ at the image pickup time t ₂ .

As an additional process to the process of the image capturing unit 1 according to the flow of FIG. 4, when the delay (transfer required time) is large and the transfer is not completed even when the next image capturing timing (affirmative determination timing of step S11) is reached. The image pickup information that has not been transferred may be discarded. Even when such a discarding process partially occurs, the information terminal device 10 can continuously perform the posture estimation process.

In the case of an embodiment in which it is not necessary to use the imaging information in the presentation unit 6 (for example, in the case of an embodiment in which AR display is performed by a see-through HMD), the imaging information is output from the imaging unit 1 to the presentation unit 6. This may be omitted, and the imaging information may be output only to the first estimation unit 3.

The first estimation unit 3 configures the imaging unit 1 after detecting a predetermined target (imaging target) from the captured image I _n (n=1, 2,...) In the imaging information input from the imaging unit 1. The relative posture information P _n (n=1, 2,...) Of the imaging target with respect to the camera 201 is estimated, and this posture information P _n is output to the synthesis unit 5. This orientation information can be obtained in the form of a plane projective transformation matrix of the product of the translation component matrix and the rotation component matrix, given the camera parameters of the camera 201 as known. Any existing method may be used for this posture estimation process itself, for example, the method of Non-Patent Document 2 described above may be used as a black-and-white square marker (AR marker) as an imaging target, or an arbitrary pattern may be used. The method of Non-Patent Document 3 described above may be used for detecting an imaging target by extracting SIFT characteristic information, which is an example of natural characteristic information (a characteristic amount obtained from a characteristic point and its periphery), for the imaging target. Alternatively, although in FIG. 1 the flow of the data exchange is omitted, also be utilized sensor values s _m by the sensor unit 2 not only captured images I _n obtained by the imaging section 1 as an input the first estimation unit 3 Therefore, the posture may be estimated using the deep learning method described in Non-Patent Document 1 described above. Further, when the posture estimation processing is continuously performed, the target may be detected at a certain initial time and the target tracking that can be performed at a higher speed than the detection may be performed at the subsequent time.

FIG. 5 is a flowchart of the operation of the first estimation unit 3 according to the embodiment. In step S31, it is determined whether or not there is imaging information that has not been processed with respect to posture estimation and that has been transferred from the imaging unit 1. If it exists, the process proceeds to step S32, and if it does not exist. Returning to step S31, this determination is repeated until the imaging information satisfying the conditions appears. In step S32, the posture estimation process is performed on the captured image I _n (n=1, 2,...) Of the imaged information obtained in the affirmative determination immediately before that in step S31, and the obtained posture information P _n (n= 1, 2,...) Is output to the synthesizing unit 5, and then the process returns to step S31.

The processing by the first estimation unit 3 is to execute a plurality of posture estimation processing programs by a multithread in a dedicated processor 102 such as a CPU 101 and/or a GPU configured by a multi-core at the same time, thereby performing a plurality of posture estimations at the same time. You may implement. That is, a plurality of threads may be concurrently executed by executing the flow of FIG. 5 for each thread. In this case, the unprocessed determination in step S31 may be made by including whether or not the other threads other than the own thread in the multithread are unprocessed. In this case, if the posture estimation process is already started in another thread, but it is not completed at this time, it is determined that it should be excluded from the processing target in its own thread, and the same process may be duplicated between threads. You should avoid.

In the example of FIG. 3, when three processes by the first estimation unit 3 are simultaneously performed in parallel, on the lines L4, L5, and L6, the time range in which the individual posture estimation processes are performed is hatched. It is shown by showing as.

That is, on the line L4, the posture estimation process is started for the unprocessed captured image I ₁ that has been transferred between the times t ₁ and t ₂ by the first thread, and the posture is estimated between the times t ₃ and t _4. The estimation process is completed and the posture information P ₁ is obtained. Further, on the line L4, the posture estimation process is started by the first thread for the captured image I ₄ between times t ₄ and t ₅ , and the posture information P ₄ is obtained between times t ₆ and t ₇ . .. Similarly, on the line L5, the posture estimation process is started between the times t ₂ and t _{3 with} respect to the captured image I ₂ by the second thread, and the posture information P ₂ is obtained between the times t ₅ and t ₆ , The posture estimation process is started for the captured image I ₅ between times t ₅ and t ₆ , and the posture information P ₅ is obtained after time t ₈ . Similarly, on the line L6, the attitude estimation process is started between the times t ₃ and t _{4 with} respect to the captured image I ₃ by the third thread, and the attitude information P ₃ is obtained between the times t ₅ and t ₆ , The posture estimation process is started for the captured image I ₆ between times t ₆ and t ₇ , and the posture information P ₆ is obtained after time t ₈ .

As described above, it takes some time to estimate the posture information after the captured image that is the target of the posture estimation processing is obtained. In the example of FIG. 3, in the operation of the first estimation unit 3, the thread in the processing waiting state (the thread waiting for the affirmative judgment in step S31) starts the posture estimation processing every time the imaging information is input, and the imaging unit 1 The figure shows that it takes a time corresponding to 2-3 imaging intervals. Since the processing load changes depending on the content of the imaging information and the presence of the other background processing described above, it is unknown when the processing is completed at the time of execution.

The amount of data of the posture information P _n (n=1,2,...) Obtained as the captured image I _n (n=1,2,...) After the posture estimation process is completed is the data amount of the captured image. Since it is much smaller than that, unlike the case of a captured image that requires at least a certain transfer completion time, it is output to the synthesizing unit 5 almost at the same time as the obtained time (that is, the synthesizing unit 5 refers to it). It is output to the RAM 103 and written therein, and can be referred to in the synthesizing unit 5.

In the example of FIG. 3, the posture estimation processing is performed for all frames (all of the captured images I _n (n=1,2,...) on the time axis), but frames are thinned on the time axis. Alternatively, the posture estimation process may be performed. For example, by thinning out to approximately 1/3, it is possible to perform only the posture estimation processing indicated by the line L4 of only one thread of the lines L4, L5, and L6 for three threads.

At the time of thinning out, the first method may be used in which the objects to be judged in the posture estimation processing in step S31 are thinned out in advance at a fixed rate on the time axis and only the remaining objects are unprocessed in step S31. As in the case of multi-threading, those that have been started in other threads are processed.) And when there are multiple transferred ones, only the one that is closest to the current time is selected. A second method of excluding the object from the posture estimation target at the time of (in the case of multi-thread, excluding all threads) may be used, or the first method and the second method may be combined. Regarding the first method, the load status of the CPU 101 and/or the dedicated processor 102 is dynamically monitored by monitoring the load status of the CPU 101 and/or the dedicated processor 102 at regular intervals and increasing the thinning rate as the load increases. It may be optimized.

The sensor unit 2 outputs the measured value to the second estimation unit 4 as sensor information together with time information (time stamp of measurement time). As hardware that realizes the sensor unit 2, an acceleration sensor and/or an inertial sensor (a gyro sensor that measures a rotational angular velocity or a rotational motion), which is often provided as standard equipment in a mobile terminal, that measures acceleration or translational motion, etc. , A sensor 204 composed of one or more sensor devices.

Specifically, the sensor unit 2 acquires sensor information as a representation of the posture of the camera 201. For this purpose, for example, the sensor 204 as the hardware that realizes the sensor unit 2 is fixedly installed in the same housing as the camera 201, and the relative positional relationship between the sensor 204 and the camera 201 is the same. The positional relationship may be such that it does not change rigidly when it is fixed to the housing.

FIG. 6 is a flowchart of the operation of the sensor unit 2 according to the embodiment. In step S21, it is determined whether or not the current time has reached the measurement timing. If the current time has arrived, the process proceeds to step S22, and if not, the process returns to step S21, and this determination is repeated. In step S22, the sensor unit 2 performs measurement at this measurement timing, outputs sensor information in which a time stamp is associated with the obtained sensor value to the second estimation unit 4, and then returns to step S21. According to the flow of FIG. 6 described above, the sensor measurement by the sensor unit 2 is repeated at each measurement timing determined in step S21.

By repeating this, the sensor information is obtained in time series. The change between any two times t _a and t _{b of the} sensor information is a change in the attitude of the camera 201 as the hardware that realizes the imaging unit 1 measured between two times t _a and t _b , The same kind of information as the relative posture information obtained by the first estimation unit 3 is given. That is, as an example of the expression of the relative posture information (posture information as a change from the reference posture), it is obtained by the first estimation unit 3 in the form of a plane projective transformation matrix of the product of the translation component matrix and the rotation component matrix. As the same type of information as the above, the sensor information in time series can be obtained in the sensor unit 2.

The measurement timing in step S21 may be set at predetermined time intervals that are designated in advance. This measurement timing is preferably set such that the interval is finer (shorter) than the imaging timing by the imaging unit 1 in step S11. In FIG. 3, the sensor measurement is performed 7 times as shown on the line L7 between the times t ₁ and t ₂ forming one interval at the imaging timing, and the sensor values s ₁ , s ₂ ,..., S ₇ are obtained. The case where the measurement timing interval is set to be 6 times finer (1/6 times shorter) than the imaging timing interval is shown as an example.

In the example of FIG. 3, it is denoted m-th (m = 1, 2, ...) of the sensor value obtained by the measurement as s _m. Data other than the sensor value, for example, the captured images I _n (n=1,2,...) Shown on the lines L2, L3 correspond to the capturing time t _n , and the subscript indicates the n-th capturing time t. _Although it corresponds to _n , the subscript of the sensor value s _m corresponds to the number of measurements m by the sensor unit 2, and does not directly correspond to the imaging time t _n . (However, generally with respect to the imaging time t _n in the example of FIG. 3, "m = 6n-5" relationship that exists in the imaging time t _n roughly simultaneously m = 6n-5 th sensor value s _{6 N- In} addition, in FIG. 3, in order to avoid cluttering the display with respect to the sensor value s _m , not all of them but only those referred to in the main description Notation is given.

The sensor information s _m (m=1,2,...) Obtained after the measurement process is completed requires a transfer completion time of at least a certain time because its data amount is much smaller than the data amount of the captured image. Unlike the case of the captured image, the second estimation is output almost simultaneously with the obtained time to the second estimation unit 4 (that is, output to and written to the RAM 103 referred to by the second estimation unit 4). It will be available in Part 4.

The second estimation unit 4 uses the sensor information continuously input from the sensor unit 2 in time series, and the posture change that will change in the presentation processing interval of the presentation unit 6 in the future when viewed from the current time. Information is estimated for each presentation processing interval and output to the synthesis unit 5. As will be described later, the presentation unit 6 can realize AR display and the like by continuously performing the presentation process at a predetermined presentation process interval (frame rate).

In FIG. 3, as in the case of the first estimation unit 3 shown on the lines L4, L5, and L6, the time range in which the individual estimation processing by the second estimation unit 4 is performed is hatched on the line L8. This is indicated by the frame. In FIG. 3, as an example, the presentation processing interval of the presentation unit 6 is determined to match the imaging times t ₁ , t ₂ ,..., T ₈ by the imaging unit 1 and the processing by the second estimation unit 4 on the line L8. An example is shown.

That is, when the present time t _present is between the imaging times t ₂ and t ₃ (t ₂ <t _present <t ₃ ) and up to the eleventh sensor information s ₁₁ can be referred to, presentation in the future the time t _3, t posture change information [Delta] P _{3, 4} of between ₄ is a presenting process interval parts 6, the estimated starting at the current time t _now using up to 11 th sensor information s _11, then the current The estimation process is completed before the time reaches the future time t ₄ at the end of this interval. Similarly, when the present time t _present is between the imaging times t ₃ and t ₄ (t ₃ <t _present <t ₄ ) and up to the 17th sensor information s ₁₇ can be referred to, the future the posture change information [Delta] P _{4, 5} in between time t _4, t ₅ is a presentation process interval indication section 6, the estimated starting at the current time t _now using up to 17 th sensor information s _17, followed by The estimation process is completed before the current time reaches the future time t ₅ at the end of this interval.

Similarly, if the present time t _present is between the imaging times t ₄ and t ₅ (t ₄ <t _present <t ₅ ) and up to the 24th sensor information s ₂₄ can be referred to, the posture change information [Delta] P _{5, 6} in between time t _5, t ₆ is a presentation process interval indication section 6, the estimated starting at the current time t _now using up to 24 th sensor information s _24, followed by The estimation process is completed before the current time reaches the future time t ₆ at the end of this interval. Similarly, 31 th of sensor information current time t _current is obtained at time t ₆ In the vicinity immediately after the time t ₆ of between imaging time t _6, t ₇ (t ₆ <t _current <t ₇₎ Posture change between times t ₆ and t ₇ (present time t ₆ is past but time t ₇ is future), which is the presentation processing interval of the presenting unit 6 in the future when up to s ₃₁ can be referenced the information [Delta] P _{6, 7,} 31 th of sensor information using up to s ₃₁ starts estimated at the current time t _current, current time subsequent estimated before reaching the future time t ₇ at the end side of the spacing Complete the process. Similarly, if the present time t _present is between the imaging times t ₆ and t ₇ (t ₆ <t _present <t ₇ ) and up to the 36th sensor information s ₃₆ can be referred to, Posture change information ΔP _7,8 between times t ₇ and t ₈ which is the presentation processing interval of the presentation unit 6, starts estimation at the current time t _current using the _36th sensor information up to s ₃₆ , and then The estimation process is completed before the current time reaches the future time t ₈ on the end side of this interval.

In the second estimation unit 4, the posture change information estimated start using the sensor information that is referable up to the current time t _current as an input, to complete this process takes some time. (That is, when this completion time is t _completion , t _current <t _completion .) In the example of FIG. 3 as described above, the operation of the second estimation unit 4 shown on the line L8 is at the start of processing. The posture estimation process is executed using the sensor information up to, and it takes a time of about 0.5 to 0.8 imaging intervals. Unlike the case where the processing load may change significantly depending on the image content such as the case of processing including image processing in the first estimation unit 3, the processing load of the second estimation unit 4 changes so much depending on the content of the sensor information. Therefore, when the process starts, it is generally known when the process is completed.

Based on this consideration, the required time T _{second estimation} process completion in the second estimation unit 4, a required time T _Synthesis of completion of processing by the combining unit 5, required to carry out the presentation processing in the presentation unit 6 time T _{The presentation} (the time required to prepare the AR display information and the like required to perform the presentation processing for one frame) and ( _presentation T ₂ +T _synthesis +T _presentation ) presentation interval t _{the beginning} of, before the time going back by the added amount of the terminating time t _end at t _end, it is desirable to start the process in the second estimation unit 4. That is, in the current time t _current as a timing for starting the estimation process has been described more than the example of FIG. 3, it is desirable to set as the condition is satisfied the following equation (1). (Note that, in the following formula (1) and the example described above, the value of the time is assumed to be larger as the time on the future side is similar to the case where the value of the time is normally treated as such. The size is specified. The same applies to the following explanations.)
t _present ≤ t _terminal- (T _{second estimation} + T _synthesis + T _presentation ) (1)

Regarding the three required times T _{second estimation} , T _synthesis, and T _presentation in Expression (1), the actual values are experimentally collected in the actual environment in which the information terminal device 10 is used, and the actual results are obtained. A representative value or the like of the values may be used. For example, an average value or mode value may be used, or a value obtained by adding a predetermined margin to the average value or mode value may be used.

For the posture change estimation processing itself by the second estimation unit 4, any existing method for predicting future behavior of any type of time series data whose applicable target is not limited to posture data may be used, for example, a Kalman filter. (Kalman filter) or deep learning may be used.

FIG. 7 is a flowchart of the operation of the second estimation unit 4 according to the embodiment. In step S41, it is determined whether or not the current time has reached the estimated timing, and if it has arrived, the process proceeds to step S42, and if it has not arrived, the process returns to step S41, and step S41 is repeated until a positive determination is obtained. .. In step S42, using the sensor information as a history of the time on the series has become visible in until an affirmative decision obtained present time t _currently in step S41, the next presentation interval t _start in the presentation unit 6, t _termination The posture change information in 1 is estimated and output to the synthesis unit 5, and the process returns to step S41.

The determination in step S41 may be determined as satisfying at least the above equation (1). Specifically, it is possible to set the current time at which an affirmative determination is made in step S41 based on the following consideration, and the posture change information to be estimated in the next step S42.

Here, as the future for the second estimation unit 4 at the current time t _current, following presentation interval t _start in the presentation unit 6, t _terminated (at least the terminating time t of the start end time t _start and end side time t _{end The end point} is the future with respect to the present time t _present , and t _present <t _end ). The posture change information is estimated as a change prediction. It is generally assumed that the prediction accuracy of the future tends to decrease as the prediction becomes longer in the future. Therefore, in order to ensure the estimation accuracy of the posture change information with respect to the future, it is preferable to set the future as close as possible to the prediction target.

Therefore, the k-th (k=1, 2,...) Processing interval in the presentation unit 6 is set to the interval t start point _[k] , t _{end [k]} (k-th start time t start point _[k] and k-th end point). interval defined by side the time t _{end [k].} Thus, t _{start [k]} = t _{end [k-1]} related.) and when the current time t _current is the k-th presentation processing interval t _{start [k} of _], if it is in t _{end [k]} (t _{start [k]} <t _current <t _{end [k]),} the the following is closest future k + 1 th processing interval t _{start [k + 1 ]} , t _{present [k]} at the current time current time t _{present [k]} as the estimation start timing of the kth posture change information in step S41 so as to estimate the posture change information at the _{terminal [k+1]} processing interval t _{start [k],} it is desirable to set the allowed (synchronized by matching the interval timing) synchronization and t _{end [k].} That, k-th presentation processing interval length t _{end [k]} -t _{start [k]} is shorter than the predetermined value c (0 <c <t _{end [k]} -t _{start [k]} ... (2)) the set advance, the posture change information in estimating the start timing of the k-th posture change information in step S41 (the next (k + 1) th presentation processing interval t _{start [k + 1],} t _{end [k + 1]} The current time t _{present [k],} which is the estimated start timing of (3), may be set as in the following equation (3).
t _{present [k]} = t _{terminal [k]} -c …(3)

Note that the posture change information at the k+1-th presentation processing interval t _{starting point [k+1]} , t _{terminal [k+1]} in equation (3) is estimated to be started by the constant c within the range of equation (2). If (1) with “t _end =t _{end [k+1]} ” and “t _present =t _{present [k]} ” does not hold, that is, the current time is k th presentation processing interval t _{start [k],} the following k + 1-th presentation processing interval t _starting at some point in t _{end [k] [k + 1]} , the attitude at t _{end [k + 1]} When the change information is estimated, if the estimation process is not completed by the end time t _{terminal [k+1]} , the target for which the posture change information is estimated is the next k+ second presentation processing interval t start point _{[k+]. 2]} , t _{terminal [k+2]} . Even if you set in this way, if the formula (1) (“t _end = t _{end [k+2]} ” and “t _present = t _{present [k]} ” (1)) does not hold, the prediction target Is the posture change information at the further k+3, k+4,...th presentation processing interval, and the closest future presentation processing interval such that Eq. Good.

On the contrary, when the present time is within the k-th presentation processing interval t start point _[k] , t _{end [k]} , the posture change information estimation start timing t _{current [k]} set, attitude change the estimation target k-th presentation processing interval t _start information _[k], if the estimation processing until the end time t _{end [k]} as t _{end [k]} is completed (formula (1) Is satisfied even when “t _end =t _{end [k]} ” and “t _present =t _{present [k]} ”), it may be set as it is. That is, the estimation start timing t _{present [k]} of the posture change information is set at the time when the present time is within the k-th presentation processing interval t start _{end [k]} , t _{end [k]} , and the posture change information estimation target is set. The attitude change information at the k-th presentation processing interval t start point _[k] , t _{end [k] to} which the current time t _{present [k]} belongs may be used.

The synthesis unit 5 synthesizes the posture information obtained from the first estimation unit 3 and the posture change information obtained from the second estimation unit 4, synthesizes the posture information in the future seen from the present time, and presents it as synthetic posture information. Output to part 6. As will be described later, this future composite posture information is assumed to correspond to the present time, which is the presentation timing of the presentation unit 6, when it is used by the presentation unit 6 in the future after being composed by the composition unit 5. Will be used.

In one embodiment, the presentation timing of the presentation unit 6 is set so as to coincide with the imaging time of the imaging unit 1, and the latest timing that can be referred to at the time when the composition process of the composition unit 5 starts (=current time t _{composition start} ). and orientation information P _a of one or more posture change information [Delta] P _k from time t _a which is associated string to the posture information until the next presentation timing _{t i, k + 1 (k} = a, a + 1, , I-2, i-1) and the combined posture information Pe _i is calculated by the following equation (4). Here, as described above, the posture information P _a is estimated by the first estimating unit 3 from the captured image I _a time t _a is in the past when viewed from the _starting current time t, _synthetic unit 5 at _{the start} current time t _synthesis Can be referred to in. Further, the next presentation timing t _i is the future when viewed from the _{start of synthesis at the} current time t.

As already described, both the posture information P _a and the posture change information ΔP _k,k+1 can be obtained in the form of a plane projective transformation matrix. In equation (4), the product of these matrices is obtained, and thus the combined posture information Pe _i is also obtained in the form of a plane projective transformation matrix.

In the synthesizing unit 5, the corresponding posture change information ΔP _k,k+1 (k=a,a+1,...,i-2,i-1) is used even for the latest referable posture information P _a. ) Does not exist is not adopted. That is, in the synthesizing unit 5, the posture information P _a that can be referred to and the corresponding posture change information ΔP _k,k+1 (k=a,a+1,...,i-2,i-1) The posture information P _a is selected as the latest one from among the existing ones, and the combining process is performed according to the equation (4).

In the example of FIG. 3, the combining process of the combining unit 5 according to the equation (4) is shown on the line L9. As shown in equation (4), the combining process is an integration of the plane projective transformation matrix (size 3×3) and can be calculated at high speed (almost instantaneously). Therefore, on line L9, the values on lines L4, L5, and L6 Unlike the case of the processing example of the one estimation unit 3 or the second estimation unit 4 on the line L8, the processing time range is not indicated by the hatching frame, and the synthesized posture information Pe _i (i =4,5,6,7,8) only.

Specifically, each combined posture information Pe _i (i=7,8) is obtained on the line L9 as follows. At the present time t _{synthesis start} (t ₆ <t _{synthesis start} <t ₇ ) with respect to the future presentation time t ₇ , it is possible to refer to the posture information P _{3 at} time t ₃ at which there is corresponding posture change information. , By combining the posture change information ΔP _3,4, ΔP _4,5 , ΔP _5,6 , ΔP _6,7 from this time t ₃ to the future presentation time t ₇ by the equation (4). synthetic attitude information Pe ₇ in the presentation time t ₇ of the future can be obtained as shown in the following equation (4-7).
Pe ₇ = ΔP _6,7 ΔP _5,6 ΔP _4,5 ΔP _3,4 P ₃ …(4-7)

Similarly, the posture information at time t ₄ that can be referred to at the present time t _{synthesis start} (t ₇ <t _{synthesis start} <t ₈ ) with respect to the future presentation time t ₈ and at which corresponding posture change information exists P ₄ and the posture change information ΔP _4,5 , ΔP _5,6 , ΔP _6,7 , ΔP _7,8 from this time t ₄ to the future presentation time t ₈ are combined by equation (4). By doing so, the combined posture information Pe ₈ at the future presentation time t ₈ can be obtained as in the following Expression (4-8).
Pe ₈ = ΔP _7,8 ΔP _6,7 ΔP _5,6 ΔP _4,5 P ₄ …(4-8)

As for the other combined posture information Pe _i (i=4,5,6) on the line L9, the posture change information ΔP _1,2 , ΔP _2,3, etc. not shown on the line L8, and one of orientation information _{P 1, P 2, P 3} , the above (4-7), a suitable selection in a similar appropriate synthetic processing start timing t _{synthesis initiation} and in the example of (4-8) Then, it can be synthesized by the formula (4).

The presentation unit 6 uses the imaging information obtained from the imaging unit 1 and the synthetic posture information obtained from the synthesis unit 5 to generate presentation information according to the synthetic posture information, and then presents this presentation information to the imaging information. Present by superimposing. Here, regarding the generation of the presentation information according to the combined posture information, existing AR technology may be used to generate arbitrary content according to the application content realized by the information terminal device 10 as its use. For example, with respect to the object imaged in the imaging information, the spatial position according to the spatial position of the object and the attitude of the camera 201 that is imaging (that is, the attitude represented in the combined attitude information) is set. The presentation information may be generated by a computer graphics (CG) or the like of a signboard that occupies a commentary comment on the object.

The above-described embodiment is for the case where the presentation unit 6 is realized by using the normal display 202 instead of the see-through type display 203. In the embodiment in which the presentation unit 6 is realized by using the see-through display 203, the presentation unit 6 does not use the imaging information (corresponding to the view seen by the user through the see-through display 203) and the combined posture information Alternatively, only the presentation information generated according to the target object (the target object imaged by the camera 201) existing in the physical space may be displayed on the see-through display 203.

FIG. 8 is a flowchart of operations of the combination unit 5 and the presentation unit 6 according to the embodiment. In FIG. 8, steps S51, S52, and S53 are parts related to the operation of the combining unit 5, and steps S54 and S55 are parts related to the operation of the presentation unit 6.

In step S51, it is determined whether or not the current time has reached the timing of the synthesizing process in the synthesizing unit 5. If the current time has arrived, the process proceeds to step S52. If not, the process returns to step S51 and the determination in step S51. repeat. The timing at which an affirmative determination is obtained in step S51 is the synthesis start time t _{synthesis start} in the synthesis section 5 already described. In step S52, the posture information that can be used at the _start of the current time t combination for which the affirmative determination is obtained and the corresponding posture change information are acquired, and the process proceeds to step S53. In step S53, using the information acquired in step S52, the synthesizing unit 5 obtains the synthesized posture information by the above-mentioned equation (4) and outputs it to the presentation unit 6, and then proceeds to step S54.

In step S54, it is determined whether or not the present time (corresponding to immediately after the _start of the time t combination for which an affirmative determination is obtained in step S52) has reached the presentation timing in the presentation unit 6, and if it has arrived, step S55 If not, the process returns to step S54 to repeat the determination of step S54. In step S55, the presentation information is generated using the synthesized posture information corresponding to the reached current time, and then only the presentation information is displayed in the case of the see-through display 203, or this presentation information is displayed in the case of the normal display 202. After the presentation unit 6 presents the information superimposed on the imaging information, the process returns to step S51.

Note that the generation of the presentation information by the presentation unit 6 and the processing of further superimposing it on the imaging information are performed immediately after the synthesis unit 5 obtains the synthetic posture information in step S53, and already generated/superimposed in step S55. The presented information or the like may be presented as it is.

The presentation timing by the presentation unit 6 as the timing for obtaining an affirmative determination in step S54 can be set at a predetermined rate as already mentioned, and for example, can be matched with the imaging time (or imaging timing) by the imaging unit 1. Good. Incidentally, already mentioned k-th presentation processing interval t _start in the description of the second estimation unit 4 _[k], t _{end [k]} is intended to be defined by the presentation timing. That is, both the start time t start end _[k] and the end time t start end _[k] are presentation timings and are adjacent presentation timings.

The synthesis timing t, which is the time at which an affirmative determination is obtained in step S51, is _started in synchronization with the presentation timing in step S54 (synchronization by matching the timing intervals) and set as a predetermined time immediately before this presentation timing. You can That is, when the k-th combining timing is t _{combining start [k]} , this may be set in synchronization with the presentation timing as in the following equations (5A) to (5C).
t _{start point [k]} <t _{synthesis start [k]} <t _{end point [k]} …(5A)
t _{synthesis start [k]} = t _{termination [k]} -b (b is a constant within the range of the following formula (5C)) (5B)
0<b<t _{end [k]} -t start _[k] …(5C)

In the example of FIG. 3, the combined posture information Pe _i (i=4,5,..., 8) shown on the line L9 is not the above-mentioned combined timing t _{combined start [k]} , and this combined process is completed. It is shown at the time position. (Composite processing is instantaneous compared to image transfer, but as described in equation (1), it takes some time that can fluctuate.)

In the example of FIG. 3, a processing example by the presentation unit 6 is shown on the line L10. That is, the presentation timing by the presentation unit 6 is set in agreement with the imaging time t _i (i=4,5,..., 8), and immediately before this presentation timing t _i (i=4,5,..., 8). The presentation information A _i (i=4,5,..., 8) is generated using the corresponding synthetic posture information Pe _i (i=4,5,..., 8), and the presentation timing t _i (i=4 , 5,..., 8), the presentation unit 6 presents the presentation information A _i (i=4,5,..., 8).

As described above, according to the present invention exemplified as one embodiment thereof, the attitude of the camera 201 is obtained as attitude information with high accuracy from a past captured image that is as close as possible to the current time, and Since the future prediction is also applied to the output of the sensor 204 as the posture change information to obtain the change amount of the past posture information in the minute time until the current time, the posture of the camera 201 at the current time is obtained as the synthetic posture information. The posture of the camera 201 at the current time can be obtained with high accuracy and without delay. If the AR display is realized in the presenting unit 6 using this synthetic posture information, even if the see-through display 203 is used (when the view seen by the user is always the current state), there is no delay and high accuracy. AR display will be realized.

Even when the presentation unit 6 is realized by the normal display 202 that is not the see-through type, the captured image of the target on which the presentation information is superimposed can be regarded as the current time immediately after being captured (the image is captured by the camera 201. That is obtained by a preview display at the time), that is, the past captured image is used to obtain the posture information, but the present captured image is used as the target to which the presentation information is superimposed. Similar to the case of the model display 203, highly accurate AR display can be realized without delay at the current time.

That is, in the example of FIG. 3, the presentation information A _i (i=4,5,..., 8) at the presentation timing t _i (i=4,5,..., 8) on the line L10 is the same as this. In the presentation unit 6, an image superimposed on the captured image I _i (i=4,5,..., 8) captured at the capturing time t _i (i=4,5,..., 8), which is the time It should be presented as.

Hereinafter, additional description regarding other embodiments of the present invention will be given.

(1) The following is an embodiment in which the composition unit 5 can reduce the load of the process of combining the combined posture information when a certain condition is satisfied. As described above, the synthesizing unit 5 synthesizes the synthetic posture information repeatedly, but in the k-th synthesis process and the k+1-th subsequent synthesis process, the posture information P used in Equation (4) is used. _{When a} does not change, using the k-th combined posture information Pe _k that has already been obtained, it is possible to obtain the k+1-th combined posture information Pe _k+1 without calculating all products of equation (4). By accumulating only _one piece of posture change information ΔP _k,k+1 corresponding to the change amount, the k+1-th combined posture information Pe _k+1 can be simply calculated as in the following equation (6). Good.
Pe _k+1 = ΔP _k,k+1 Pe _k …(6)

If the posture information P _a used in the equation (4) does not change at the k-th time even after the k+2th time, the equation (6) may be repeatedly used.

Here, in the case where it is determined that the posture information P _a used in Expression (4) does not change in the k-th combining process and the k+1-th (and subsequent) combining processes that follow, The first and second cases of are mentioned. For the sake of explanation, the posture information used in the k-th and k+1-th synthesis processing is respectively a[k]-th and a[k+1]-th imaging times t _a[k] and t _{a[k +1]} (including the case where they are at the same time) is described as P _a[k] and P _a[k+1] as obtained from the captured images I _a[k] and I _a[k+1] To do. In the first case, when a[k]=a[k+1], that is, when the posture information to be used is the k-th time and the k+1-th time, the same imaging time t _a[k] =t _{a[ It} may have been obtained from a captured image of _[k+1] . In the first case, it is possible to judge automatically according to this definition. The first case corresponds to the case where captured images after this same capturing time t _a[k] =t _a[k+1] have been discarded, or this capturing time t _{a[ When the} captured image after _k] =t _a[k+1] is obtained, but when the estimation process of the posture information is not completed at the start point of the k+1-th synthesis process, ..

In the second case, a[k]≠a[k+1], and the captured images I _a[k] and I _a[k ] at different imaging times t _a[k] ≠t _{a[k+1] Although the} posture information P _a[k] and P _a[k+1] are obtained from _+1], it may be determined that the posture information does not change substantially equally. That is, this is the case where “P _a[k] ≈P _a[k+1] ” is determined. This judgment is made, for example, by the fact that the norm of difference between both matrices |P _a[k] -P _a[k+1] | (norm by sum of absolute value of each element of matrix, etc., below) is below threshold do it.

As a specific example of applying the formula (6), with respect to the examples of the above formulas (4-7) and (4-8), the k-th time (k=7-th time) is calculated by the formula (4-7), and k+ At the first time (k+1=8th time), unlike the example of FIG. 3, if the posture information P ₄ is not available, or the posture information P ₄ is available, but “P ₃ ≈P ₄ ” If it is determined, instead of multiplying a large number of equations (4-8), by multiplying the immediately preceding synthetic posture information Pe ₇ by the posture change information ΔP _7,8 in the following formula (7), the synthetic posture information It is possible to calculate Pe ₈ simply.
Pe ₈ = ΔP _7,8 Pe ₇ …(7)

(2) As described above, the imaging unit 1 repeatedly performs imaging at a predetermined imaging rate and obtains a captured image I _n (n=1, 2,...) On the time axis. Then, in the first estimation unit 3, all of the captured images I _n (n=1, 2,...) May be used as the estimation target of the posture information, or only the thinned images I _n (n=1, 2,...) At a fixed ratio by the first method already described. May be the target for estimating the posture information.

Here, the following is also possible as an embodiment of dynamically determining the thinning rate in the first method. In this embodiment, it is premised that the presentation timing of the presentation unit 6 as a target for the synthesis unit 5 to synthesize the synthetic posture information by synchronizing the timing is matched with the imaging time (or the imaging timing) of the imaging unit 1. .. In the first estimation unit 3, as an additional process, at the presentation timing, at the timing when the estimation process of step S31 is completed, or at any other predetermined timing, the same presentation timing t in the _past is combined with the _past . The combined posture information Pe _past obtained by the unit 5 and the posture information P _past which is already estimated by the first estimation unit 3 by capturing the captured image I _past by the imaging unit 1 at the past time t _past . , Among those that can be referred to and are closer to the current time are compared. That is, the process of evaluating the norm |Pe _past- P _past | of the difference between both matrices is performed.

Since the norm of the difference becomes equal to or less than the predetermined threshold value by this comparison process, both are substantially equal, that is, it is determined that “Pe _past ≈P _past ” means that the accuracy of the posture change information used in the combining unit 5 is high. It means high. The fact that the posture change information obtained based on the output of the sensor 204 has high accuracy is considered to be because the actual posture change of the camera 201 is small and the movement thereof is small. Therefore, even if the estimation processing rate of the posture information using the captured image by the first estimation unit 3 with high accuracy but high calculation load is reduced, the precision of the combined posture information obtained by the synthesis unit 5 can be maintained. Be expected.

Based on the above discussion, when it is determined that "Pe _past ≒ P _past" may be modified as in the first estimation unit 3 dynamically increasing the thinning ratio of the estimation process. More generally, the Pe _past and the P _previous and differences difference norm | Pe _past -P _past | evaluated as may be set large thinning ratio estimating process as the difference norm is small. The past time t _past as an evaluation target may be only the latest one available, or may be two or more and evaluated by a linear sum of difference norms.

In the example of FIG. 3, at time t ₄ , the combined posture information Pe ₄ is the product “Pe ₄ =ΔP _3,4 ΔP _2,3 ΔP _1,2 P ₁ ”according to equation (4) (as already described, Although [Delta] P _2,3 and [Delta] P _{1, 2} are not shown in the line L8, after the completion of the synthesis as is capable estimation at the appropriate time range), the in between times t ₆ and time t ₇ Since the one estimation unit completes the estimation of the posture information P ₄ at time t ₄ , for example, at time t ₇ immediately thereafter, the synthetic posture information Pe ₄ and the posture information P ₄ are compared, and if the degree of coincidence is low, the thinning rate is set to It is possible to set a dynamic thinning-out ratio such that the thinning-out ratio is reduced and the thinning-out ratio is maintained or increased if the degree of coincidence is high.

It should be noted that this embodiment is premised on that an object such as a square marker used for estimating the posture of the camera 201 in the first estimation unit 3 is stationary.

(3) at time t _{synthesis initiation} to begin the process of obtaining synthetic posture information synthesizing unit 5 by the equation (4), one or more posture change information [Delta] P _k until the next presentation timing t _{i, k + 1} (k =a,a+1,...,i-2,i-1), which is not the future predicted value by the estimation process of the second estimation unit 4, but the value actually measured by the sensor unit 2 has already been obtained. If so, the future prediction value may be replaced with the actual measurement value, and the equation (4) may be applied.

That is, assuming that a<r<i-1, up to k=a,a+1,...,r-1,r, the measured sensor value corresponding to the predicted value ΔP _k,k+1 is ΔP _k,k+1. It is obtained as _{[actual measurement]} , and when k=r+1,r+2,...,i-2,i- ₁ only the predicted value ΔP _k,k+1 is obtained, the equation (4 ), the combined posture information may be obtained by the following equation (8).

In addition, in this embodiment, as an additional process to the estimation process in step S42 by the second estimation unit 4, among the predicted values ΔP _k,k+1 already obtained in the past, the output from the sensor unit 2 (time t _k , output between t _k+1 ) and the measured value ΔP _{k,k+1 [measured]} that can be referred to, this measured value ΔP _{k,k+1 [measured]} Is output to the synthesizing unit 5.

For example, in the example of the above formula (4-7), of the four estimated posture change information ΔP _3,4, ΔP _4,5 , ΔP _5,6 , ΔP _6,7 at the start of the combining process. For the past three, if the measured posture change information ΔP _{3,4 [measured],} ΔP _{4,5 [measured]} , ΔP _{5,6 [measured]} is obtained and output, the formula (4-7 Instead of ), combined posture information may be obtained by the following equation (9).
Pe ₇ = ΔP _6,7 ΔP _{5,6 [measured]} ΔP _{4,5 [measured]} ΔP _{3,4 [measured]} P ₃ …(9)

(4) The posture information (posture information as a change from the reference posture) obtained by the first estimation unit 3 and the posture change information obtained by the second estimation unit 4 are converted corresponding to the posture change in the three-dimensional space. Has been described as an example in the form of a plane projective transformation matrix (size 3×3) performed in two-dimensional image coordinates (homogeneous coordinates), but other forms may be used. For example, the posture information and the posture change information represent the posture change in the three-dimensional space as it is, and the rotation component r _ij (1≦i,j≦3) and the translational component given by the following formula (10) are given. The shape of the external parameter M (size 4×4) of the camera 201 configured by t _X , t _Y , and t _Z may be used. Even when this external parameter M is used, the composition by the composition unit 5 can be performed almost instantaneously in the form of a product as in the case of the above formula (4) and other formulas. When the presentation information is generated in the presentation unit 6, a perspective projection matrix (camera matrix) is obtained by a mathematical relationship known in the CG field using a predetermined internal parameter of the camera 201 in addition to the external parameter M, and in the model space. The presentation information may be generated by projecting the defined three-dimensional CG model onto the image coordinates of the imaging unit 1 using this perspective projection matrix. Alternatively, using a known epipolar geometrical relationship from the external parameter M and the internal parameter, when a point on the same plane in space is imaged at two different camera positions, the two image coordinates of this point are It is also possible to obtain a plane projective transformation matrix as the transformation relationship and use this plane projective transformation matrix to generate the presentation information by the method already described.

10... Information terminal device, 1... Imaging unit, 2... Sensor unit, 3... First estimation unit, 4... Second estimation unit, 5... Synthesis unit, 6... Presentation unit

Claims (10)

An image capturing section that captures an image to obtain a captured image;
From the captured image, a first estimation unit that estimates orientation information of the imaging unit at a past time,
A sensor unit that continuously acquires a sensor value that reflects the attitude of the imaging unit,
A second estimation unit that estimates the posture change information toward the future time from the continuously acquired sensor value,
An information terminal device, comprising: a combining unit that obtains combined attitude information that combines the attitude information and the attitude change information. Further comprising a presentation unit that performs an augmented reality display at each display time using the synthetic posture information,
The information terminal device according to claim 1, wherein the second estimation unit estimates the posture change information toward a future time that is the display time. In the synthesizing unit, the posture change information estimated as being directed to a future time in the second estimating unit and the posture information are synthesized in advance to obtain synthetic posture information at the future time,
The information terminal device according to claim 2, wherein when the present time reaches the future time, the presenting unit displays the augmented reality by using the preliminarily combined synthetic posture information. 3. The second estimating unit starts the process of estimating the posture change information toward a future time which is the display time, at a time that is a certain interval or more past the future time. Or the information terminal device as described in 3. The fixed interval is a time required for estimating the posture change information by the second estimation unit, a time required for obtaining the synthesized posture information by the synthesis unit, and an augmented reality display by the presentation unit. The information terminal device according to claim 4, wherein the required time is added. The synthesis unit repeatedly obtains synthetic posture information by synthesizing posture information and posture change information that can be referred to at the time,
If it is determined that the attitude information that can be referenced at the current time does not change from the attitude information that can be referenced at the immediately previous time, the combined attitude information obtained at the immediately previous time and the current time from the immediately previous time 6. The information terminal device according to claim 1, wherein the combined posture information at the current time is obtained by synthesizing the posture change information up to and including. The image capturing unit obtains a captured image at each time,
In the synthesizing unit, synthetic posture information is obtained for each time as a future time,
In the first estimation unit, the posture information is estimated for all or part of the obtained captured images at each time, and
The first estimating unit repeatedly compares the posture information already estimated at the past time by the first estimating unit with the synthetic posture information obtained by the synthesizing unit as corresponding to the past time. Evaluate the degree of agreement between the two, and
In the first estimation unit, as the degree of matching is larger, only the thinned-out images with a larger thinning-out ratio among all the captured images at the respective times obtained are targets for estimating the posture information. The information terminal device according to any one of claims 1 to 6. At the second estimation unit, at each time, the posture change information is estimated toward a future time based on the time, and the posture change information already estimated at the past time is continuously detected by the sensor unit. If there is a corresponding actually measured posture change information by the sensor value acquired in, obtain the actually measured posture change information,
At each time, the synthesizing unit obtains the synthesized posture information by synthesizing the posture information and the posture change information which can be referred to at the time, and the posture change information which can be referred to at the time. In the case where the corresponding actually-measured posture change information acquired by the second estimation unit is present, the actually-measured posture change information is used instead of the posture change information estimated by the second estimation unit, The information terminal device according to claim 1, wherein the combined attitude information is obtained. The information terminal device according to any one of claims 1 to 8, wherein the sensor unit includes an acceleration sensor and/or an angular velocity sensor. A program causing a computer to function as the information terminal device according to any one of claims 1 to 9.

Images