JP2016110546A - Display device and display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To display a virtual object in an aspect having reality in response to motion of an object in real space.SOLUTION: A terminal device 1 comprises: an image acquisition unit 11 which acquires a captured image generated by capturing real space; an acceleration derivation unit 20 which derives image acceleration which is acceleration of an object in the captured image on the basis of difference between positions in a plurality of captured images of the same object displayed in the respective captured images and difference between times at which the respective captured images are captured; a terminal acceleration acquisition unit 31 which acquires terminal acceleration which is the measured acceleration of the terminal device 1; a determination unit 41 which derives corrected acceleration obtained by correcting the image acceleration by the terminal acceleration to determine a display aspect of a virtual object using the corrected acceleration; and a display unit 44 which displays, according to the display aspect, the virtual object in synchronization with the captured image in real space.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a display device and a display method using augmented reality technology.

  Augmented Reality (AR) technology for displaying a virtual object in synchronization with a real space image captured by an imaging device or the like is known. For example, Patent Document 1 discloses an information processing apparatus that reflects the gravity direction of the real space in the virtual space when the virtual object is displayed in synchronization with the image in the real space.

JP 2011-197777 A

  By the way, an object in the real space may change (move) in its own position. In such a case, there is a possibility that the virtual object cannot be displayed in a realistic manner simply by considering the direction of gravity in the real space as in the information processing apparatus of Patent Document 1. For example, in a real space, if a box containing a ball is moved up and down, the ball may jump out of the box. However, if the object in the real space is a box and the virtual object is a ball, the ball, which is a virtual object, always follows the box, which is an object in the real space, simply considering the direction of gravity. There is a risk that the display does not conform to the physical phenomenon. Further, there are cases where the position of an object displayed as an image changes relatively as the imaging apparatus moves. Even in such a case, the virtual object may not be displayed in a realistic manner in accordance with the actual movement of the object.

  The present invention has been made in view of the above problems, and an object thereof is to display a virtual object in a realistic manner in accordance with the movement of an object in the real space.

  A display device according to one embodiment of the present invention is a display device that displays a virtual object that is an object in a virtual space in synchronization with an image in the real space, and obtains a captured image generated by imaging the real space Based on the difference in position between the captured images of the same object displayed in each of the plurality of captured images acquired by the image acquiring means and the time at which each captured image was captured, Acceleration deriving means for deriving image acceleration that is the acceleration of the object in the captured image, acceleration obtaining means for obtaining device acceleration that is the measured acceleration of the display device, and image acquisition derived by the acceleration deriving means for obtaining the acceleration Determination of deriving a corrected acceleration corrected by the device acceleration acquired by the means and determining the display mode of the virtual object using the corrected acceleration Comprises a stage, the display mode determining means has determined, and display means for displaying to synchronize the virtual object on the captured image of the physical space, the.

  According to this display device, the image acceleration is derived and the device acceleration is acquired based on the difference in position between the captured images of the same object and the difference in time at which the captured images are captured. Then, a corrected acceleration obtained by correcting the image acceleration by the device acceleration is derived, and the virtual object is displayed synchronously with the captured image in a display mode based on the corrected acceleration. Thus, the virtual object is displayed according to the acceleration of the object in the real space by determining the display mode of the virtual object in consideration of the image acceleration that is the acceleration of the imaged object. Thereby, the virtual object can be displayed in a realistic manner in accordance with an actual physical phenomenon. Furthermore, since the corrected acceleration obtained by correcting the image acceleration by the device acceleration is used, the display mode of the virtual object is determined in consideration of the movement of the display device itself. For example, when image acceleration is derived from a plurality of captured images captured while the display device is moving, the position of the object displayed in the captured image is relatively changed due to the movement of the display device itself, and is derived. There is a risk that the image acceleration value deviates from the actual movement of the object. In this regard, by using the corrected acceleration obtained by correcting the image acceleration by the device acceleration, it is possible to display a virtual object in accordance with the movement of the object in the real space in consideration of the movement of the display device itself. As described above, the virtual object can be displayed in a realistic manner in accordance with the movement of the object in the real space.

  The determining means may derive the corrected acceleration by subtracting the device acceleration from the image acceleration. Thereby, the apparatus acceleration can be appropriately canceled from the image acceleration, and a corrected acceleration that appropriately reflects the movement of the actual object can be derived.

  The determining means may determine the display mode by setting the image acceleration and the device acceleration to 0 when the magnitude of the device acceleration is larger than a predetermined value. When the device acceleration is large, that is, when the display device is moving violently, it is considered that the reliability of the image acceleration value derived from the difference in the position of the object between the captured images is low. Therefore, when the device acceleration is larger than a predetermined value, the display mode is determined by setting the image acceleration and the device acceleration to 0, so that the display mode of the virtual object is determined based on information with low reliability. It can be avoided.

  The apparatus further comprises apparatus acceleration storage means for storing the apparatus acceleration acquired by the acceleration acquisition means in association with the time when the apparatus acceleration is measured, and the determining means derives the time when the image acceleration is derived by the acceleration deriving means. In addition, the image acceleration may be corrected using the device acceleration stored in the device acceleration storage means for a predetermined time before the derivation timing. The image acceleration is derived from a plurality of captured images captured at different times. Therefore, derivation requires an imaging interval (difference between the imaging time of the first captured image and the imaging time of the last captured image among the plurality of captured images). Also, calculation time is required to derive the image acceleration. Considering the imaging interval and calculation time, the derived image acceleration corresponds to the past device acceleration by a predetermined time from the derivation timing. For this reason, by correcting the image acceleration by using the past apparatus acceleration for a predetermined time from the derivation timing, it is possible to derive the corrected acceleration more appropriately reflecting the movement of the actual object.

  The determining unit may determine the display mode by setting the image acceleration and the device acceleration to 0 for a certain time longer than the predetermined time when the magnitude of the device acceleration is larger than a predetermined value. When the device acceleration is increased, the image acceleration and the device acceleration at the timing when the device acceleration is increased are displayed as 0 by setting the image acceleration and the device acceleration to 0 for a certain time longer than a predetermined time. Aspects can be determined. Thereby, it can avoid more reliably that the display mode of a virtual object is determined based on information with low reliability.

  The acceleration deriving unit includes an image storage unit that stores the captured image acquired by the image acquisition unit, and a position between the captured images of the same object displayed on each of the plurality of captured images stored in the image storage unit. Based on the difference and the difference in time at which each captured image was captured, speed calculation means for calculating the image speed, which is the speed of the object, speed storage means for storing the image speed calculated by the speed calculation means, and speed storage You may have an acceleration calculation means to calculate an image acceleration based on the difference of the several image speed memorize | stored in the means, and an image acceleration memory | storage means to memorize | store the image acceleration calculated by the acceleration calculation means. With such a configuration, the image acceleration can be derived from the captured image.

  By the way, the present invention can be described as an invention of a display apparatus as described below, in addition to the invention of a display device as described above. This is substantially the same invention only in different categories, and has the same operations and effects.

  That is, a display method according to another aspect of the present invention is a display method performed by a display device that displays a virtual object that is an object in a virtual space in synchronization with an image in the real space, and by imaging the real space. The step of acquiring the generated captured image and the difference in position between the captured images of the same object displayed in each of the plurality of captured images acquired in the step of acquiring the captured image, and each captured image is captured. Derived in the step of deriving the image acceleration that is the acceleration of the object in the captured image, the step of obtaining the device acceleration that is the measured acceleration of the display device, and the step of deriving the image acceleration Deriving the corrected acceleration obtained by correcting the image acceleration by the device acceleration acquired in the step of acquiring the device acceleration, Determining a display form of the virtual object using the positive acceleration, the display mode determined in the determining step, and displaying to synchronize the virtual object on the captured image of the physical space.

  According to the present invention, a virtual object can be displayed in a realistic manner in accordance with the movement of an object in real space.

It is a figure which shows the display image of the terminal device which concerns on this embodiment. It is a functional block diagram of the terminal device concerning this embodiment. It is a hardware block diagram of the terminal device which concerns on this embodiment. It is a table | surface which shows the acceleration table of the terminal acceleration memory | storage part of FIG. It is a figure which shows the image which derives | leads-out image acceleration from a captured image. 6A is a diagram showing an image of an object stored in the object storage unit of FIG. 2, and FIG. 6B is a table showing an object table of the object storage unit of FIG. It is a flowchart which shows a series of processes of the display method which the terminal device of FIG. 2 performs. It is a figure which shows the display image of the terminal device which concerns on a comparative example. It is a figure which shows the display image of the terminal device which concerns on a comparative example.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same reference numerals are used for the same or equivalent elements, and redundant descriptions are omitted.

  The display device according to the present embodiment is a device that displays a virtual object in synchronization with an image of a real space imaged by an imaging device by using augmented reality (AR) technology. More specifically, the display device according to the present embodiment calculates the acceleration of the object in the captured image from the motion of the captured object, and controls the virtual object with the calculated acceleration, thereby following the actual physical phenomenon. It is an apparatus that displays a virtual object in a form with reality.

  FIG. 1 is a diagram illustrating a display image of the terminal device according to the present embodiment. In the display image shown in FIG. 1A, a box R that is an object in the real space imaged by the imaging device, and a ball BV that is a virtual object displayed as if it is in the box R. It is displayed on the display of the terminal device. Then, when the box R is moved upward as shown in FIG. 1B, both the box R and the ball BV, which is a virtual object, are moved upward and displayed. Further, as shown in FIG. 1C, when the moving direction of the box R that has been moved upward is changed downward, the box R is moved downward and displayed. On the other hand, since the ball BV has an upward speed due to the upward movement shown in FIG. 1B, when the box R moves downward as shown in FIG. 1C. Is displayed so as to be flipped up away from the box R. Such a display is a realistic display in accordance with an actual physical phenomenon, and is realized by deriving the acceleration of the object in the captured image. Hereinafter, the function of the terminal device which is an example of the display device according to the present embodiment will be described in detail with reference to FIG.

  FIG. 2 is a functional block diagram of the terminal device according to the present embodiment. The terminal device 1 shown in FIG. 2 is a display device that displays an object (virtual object) in a virtual space in synchronization with an image in the real space. A virtual object is computer graphics (CG: computer graphics) created using a computer, for example. The real space image is, for example, a still image (captured image) captured by an imaging device. Displaying a virtual object in synchronization with an image in the real space means that the virtual object is superimposed and displayed at a predetermined position associated with the virtual object in the real space image. For example, in the display image shown in FIG. 1A, a position where a ball BV, which is a virtual object, is contained in a box R, which is an object in real space, is a predetermined position associated with the virtual object. The ball BV is displayed at the predetermined position.

  The terminal device 1 is a device that is used by a user and has an imaging function. The terminal device 1 is, for example, a mobile phone such as a smartphone, a tablet terminal, a notebook PC (Personal Computer), or the like. Functionally, the terminal device 1 has an image acquisition unit 11, an image storage unit 21, a speed calculation unit 22, a speed storage unit 23, an acceleration calculation unit 24, an image acceleration storage unit 25, and terminal acceleration acquisition. Unit 31, terminal acceleration storage unit 32, determination unit 41, object storage unit 42, image composition unit 43, and display unit 44.

  The terminal device 1 is configured by hardware shown in FIG. FIG. 3 is a hardware configuration diagram of the terminal device according to the present embodiment. As shown in FIG. 3, the terminal device 1 physically includes one or more CPUs (Central Processing Units) 101, a RAM (Random Access Memory) 102 and a ROM (Read Only Memory) 103 that are main storage devices, data Communication module 104 as a transmission / reception device, auxiliary storage device 105 such as a semiconductor memory, input device 106 that accepts user input such as a touch panel, output device 107 such as a display, imaging device 108 such as a camera, and acceleration of the terminal device 1 It is comprised as a computer provided with hardware, such as the acceleration sensor 109 which measures this. The imaging device 108 generates a captured image by imaging the real space. Each function of the terminal device 1 in FIG. 2 is such that one or a plurality of predetermined computer software is read on hardware such as the CPU 101 and the RAM 102, so that the communication module 104, the input device 106, and the output are controlled under the control of the CPU 101. This is realized by operating the device 107, the imaging device 108, and the acceleration sensor 109, and reading and writing data in the RAM 102 and the auxiliary storage device 105.

  Returning to FIG. 2, the functional configuration of the terminal device 1 will be described in detail. The image acquisition unit 11 functions as an image acquisition unit that acquires a captured image generated by being captured by the imaging device 108. The image acquisition unit 11 may acquire a captured image every time a captured image is generated by the imaging device 108, or may acquire a captured image from the imaging device 108 at a predetermined interval. . The meta information of the captured image acquired by the image acquisition unit 11 includes at least an imaging time (time when the image was captured). The image acquisition unit 11 stores the acquired captured image and meta information of the captured image in the image storage unit 21.

  The image storage unit 21 functions as an image storage unit that stores a plurality of captured images acquired by the image acquisition unit 11. The image storage unit 21 has a first storage area and a second storage area. The first storage area stores the latest captured image among the plurality of captured images acquired by the image acquisition unit 11 and meta information of the captured image. The second storage area stores a past captured image (with an imaging time older than the captured image) stored in the first storage area and meta information of the captured image. When storing a new captured image in the image storage unit 21, the image acquisition unit 11 stores the information stored in the first storage area in the second storage area, and then stores the new captured image and the captured image. The meta information of the image is stored in the first storage area.

  The speed calculation unit 22 is based on the difference in position between the captured images of the same object displayed in each of the plurality of captured images stored in the image storage unit 21 and the difference in time at which each captured image was captured. It functions as a speed calculation means for calculating the image speed, which is the speed of the object. In the following description, calculation of vector quantities such as speed and acceleration all indicate vector quantity calculations. The speed calculation unit 22 performs an object identification process, a motion vector derivation process, and an image speed derivation process.

  The object specifying process is a process of extracting a feature point in a captured image and specifying an object included in the captured image based on the feature point. The feature points are points that become features such as borders and corners of the image, and are, for example, a location where the color of the image changes greatly, a dark portion, a contour, or the like. In order to specify the object included in the captured image based on the feature points, for example, the extracted feature points are matched with the feature points stored in advance. The feature points stored in advance are associated with real objects. By the above matching, an object included in the captured image can be specified. The feature points associated with the object may be stored in the terminal device 1 or may be stored in an external database that can communicate with the terminal device 1. The speed calculation unit 22 includes a captured image stored in the first storage area of the image storage unit 21 (hereinafter sometimes referred to as a first image) and a captured image stored in the second storage area (hereinafter referred to as a second image). Object identification processing is performed on both of them (which may be described as images).

  The motion vector deriving process is a process of deriving the motion of the object specified in the object specifying process as a motion vector by comparing the first image and the second image. A motion vector represents a motion of an object from a reference image (frame) as a vector. The speed calculation unit 22 compares the position in the first image with the position in the second image for the object displayed in both the first image and the second image, and the position in the first image from the position in the second image. The amount and direction of movement up to are derived as motion vectors. The speed calculation unit 22 derives a motion vector for each of a plurality of feature points included in the same object. That is, a plurality of motion vectors are derived for the same object. Regarding the process of deriving motion vectors from multiple images, for example, `` Bruce D. Lucas and Takeo Kanade.An Iterative Image Registration Technique with an Application to Stereo Vision.International Joint Conferenceon Artificial Intelligence, pages 674-679, 1981. '' Etc. can be performed using the method described in the above.

  The image speed derivation process includes the meta information of the first image stored in the first storage area, the meta information of the second image stored in the second storage area, and the motion vector derived in the motion vector derivation process. This is a process for deriving an image speed that is the speed of an object in the captured image. The speed calculation unit 22 derives an imaging time difference that is a difference between the imaging time of the second image included in the meta information of the second image and the imaging time of the first image included in the meta information of the first image. . Further, the speed calculation unit 22 obtains one motion vector from the motion vectors of a plurality of feature points related to the same object, which are derived from the first image and the second image. The one motion vector is, for example, an average vector of motion vectors of a plurality of feature points (hereinafter sometimes referred to as an average vector). The speed calculation unit 22 derives the image speed from the imaging time difference and one motion vector. The speed calculation unit 22 stores the derived image speed and the time information of the image speed in the speed storage unit 23. Here, the time information of the image speed is, for example, an imaging time included in the meta information of the first image.

  Further, for the sake of simplicity, it is assumed that captured image generation by the imaging device 108 is periodically performed at the same interval (that is, when the imaging time difference is always constant), and the value obtained by multiplying the average vector by a constant is the image It may be speed. When the imaging time difference is not constant, the image speed is obtained by multiplying the number inversely proportional to the time difference, and when the position is updated, the position obtained by multiplying the image speed by the number proportional to the time difference is used as the position. What is necessary is just to add. In addition, when the depth of the start point of the motion vector can be estimated, an average value of values obtained by correcting the magnitude of the apparent motion vector by the depth by applying the inverse transformation of the perspective transformation may be used as the image speed. By correcting by depth, a highly accurate image speed is derived. For example, “Carlo Tomasi and Takeo Kanade. (November 1992).“ Shape and motion from image streams under orthography: a factorization method. ”. International Journal of The publicly known Tomasi-Kanada method described in Computer Vision, 9 (2): 137-154.

  The speed storage unit 23 functions as a speed storage unit that stores the image speed derived (calculated) by the speed calculation unit 22. The speed storage unit 23 has a third storage area and a fourth storage area. The third storage area stores the latest image speed among the image speeds derived by the speed calculation unit 22 and time information of the image speed. The fourth storage area stores the past image speed (the time information is older) than the image speed stored in the third storage area and the time information of the image speed. When storing the new image speed in the speed storage unit 23, the speed calculation unit 22 stores the information stored in the third storage area in the fourth storage area, and then stores the new image speed and the image. Speed time information is stored in the third storage area.

  The acceleration calculation unit 24 functions as an acceleration calculation unit that calculates an image acceleration, which is an acceleration of an object in the captured image, based on a difference between a plurality of image speeds stored in the speed storage unit 23. That is, the acceleration calculation unit 24 calculates the speed difference that is the difference between the image speed stored in the third storage area and the image speed stored in the fourth storage area, the time information stored in the third storage area, and the fourth information. A time difference which is a difference between time information stored in the storage area is obtained, and an image acceleration is derived (calculated) from the speed difference and the time difference. The acceleration calculation unit 24 stores the derived image acceleration and time information of the image acceleration in the image acceleration storage unit 25. Here, the time information of the image acceleration is, for example, information indicating the time information stored in the third storage area (that is, the imaging time of the first image) and the time when the image acceleration is derived (derivation timing). The

  The image acceleration storage unit 25 functions as an image acceleration storage unit that stores the image acceleration derived (calculated) by the acceleration calculation unit 24. The image acceleration storage unit 25 stores at least the latest image acceleration derived by the acceleration calculation unit 24 and time information of the image acceleration. Note that the image acceleration storage unit 25 may store a plurality of image accelerations and time information.

  The above-described image storage unit 21, speed calculation unit 22, speed storage unit 23, acceleration calculation unit 24, and image acceleration storage unit 25 constitute an acceleration deriving unit 20 that derives image acceleration. That is, the acceleration deriving unit 20 determines the difference in position between the captured images of the same object displayed in each of the plurality of captured images acquired by the image acquiring unit 11 and the difference in time at which the captured images are captured. Functions as an acceleration deriving means for deriving the image acceleration.

  The terminal acceleration acquisition unit 31 functions as an acceleration acquisition unit that acquires terminal acceleration (apparatus acceleration) that is the acceleration of the terminal device 1. The terminal acceleration acquisition unit 31 acquires, for example, acceleration measured by the acceleration sensor 109 at a predetermined interval (for example, several tens of milliseconds to several seconds). In addition, the terminal acceleration acquisition unit 31 may acquire acceleration from the acceleration sensor 109 at random instead of a constant interval. The terminal acceleration acquisition unit 31 acquires the acceleration measurement time from the acceleration sensor 109 together with the acceleration. The acceleration measured by the acceleration sensor 109 includes both terminal acceleration and gravitational acceleration. Therefore, the terminal acceleration acquisition unit 31 acquires a value obtained by subtracting the gravitational acceleration from the acceleration measured by the acceleration sensor 109 as the terminal acceleration.

  The terminal acceleration storage unit 32 functions as a device acceleration storage unit that stores the terminal acceleration acquired by the terminal acceleration acquisition unit 31 in association with a time stamp indicating the measurement time of the terminal acceleration. The terminal acceleration storage unit 32 associates not only the latest terminal acceleration among terminal accelerations acquired by the terminal acceleration acquisition unit 31 but also terminal accelerations acquired in the past from the latest terminal acceleration with time stamps. Is stored in the acceleration table. FIG. 4 is a table showing an acceleration table of the terminal acceleration storage unit 32 of FIG. In the example shown in FIG. 4, in the acceleration table T of the terminal acceleration storage unit 32, a time indicating 11: 41: 12.000 seconds to 12.060 seconds, which is the measurement time of the terminal acceleration acquired at intervals of 20 milliseconds. The terminal acceleration (0.1, 0.2, 0.0, 0.0) is stored corresponding to each stamp (11: 41: 12.000 to 12.060). That is, the acceleration table T stores the terminal acceleration acquired by the terminal acceleration acquisition unit 31 every 20 milliseconds.

  The terminal acceleration storage unit 32 has a buffer for storing the acceleration table T. The buffer may be a fixed size and in the form of a ring buffer. In this case, when a new terminal acceleration is stored by the terminal acceleration acquisition unit 31, the oldest terminal acceleration value stored in the buffer is overwritten with the new terminal acceleration stored in the buffer. May be.

  Returning to FIG. 2, the determination unit 41 derives a corrected acceleration obtained by correcting the image acceleration derived by the acceleration deriving unit 20 with the terminal acceleration stored in the terminal acceleration storage unit 32, and uses the corrected acceleration to generate a virtual acceleration. It functions as a determining unit that determines the display mode (specifically, the display position) of the object. That is, the determination unit 41 performs a corrected acceleration derivation process and a display mode determination process. Hereinafter, each process will be described in detail.

  First, the corrected acceleration deriving process will be described. The determination unit 41 derives a corrected acceleration by subtracting the terminal acceleration from the image acceleration. The determination unit 41 may derive the corrected acceleration in consideration of the time required for deriving the image acceleration. That is, the determination unit 41 specifies the time when the image acceleration is derived by the acceleration deriving unit 20 (more specifically, the acceleration calculating unit 24) as the deriving timing. Specifically, the determination unit 41 refers to the image acceleration storage unit 25 and uses the image acceleration time information (more specifically, information indicating the time (derivation timing) at which the image acceleration is derived) as the latest image acceleration. It is specified as the derivation timing. And the determination part 41 correct | amends an image acceleration by a terminal acceleration past for predetermined time rather than the derivation timing among the terminal accelerations memorize | stored in the terminal acceleration memory | storage part 32. FIG. The predetermined time is a time corresponding to a time required for deriving the image acceleration from the captured image (hereinafter sometimes referred to as a derived time lag).

FIG. 5 is a diagram illustrating an image for deriving the image acceleration from the captured image. As shown in FIG. 5, to derive an image acceleration _{a I} is at least different times t, t-1, captured t-2 are a plurality of captured images I (t), I (t -1) , I (t−2) is required. Therefore, the derived time lag includes an imaging interval for imaging a plurality of captured images I (t), I (t−1), and I (t−2). The imaging interval in this case is the time from time t-2 when the captured image I (t-2) is captured to time t that is the time when the captured image I (t) is captured. Further, as shown in FIG. 5, at least speeds V (t) and V (t-1) are calculated from the plurality of captured images I (t), I (t-1), and I (t-2). speed V (t), since V (t-1) from the image acceleration _{a I} is derived, the derived time lag includes the computation time. From the above, the derived time lag is a time including the above-described imaging interval and calculation time.

  This will be described in more detail with reference to FIG. For example, it is assumed that the predetermined time (that is, the derived time lag including the imaging interval and the calculation time) is 20 milliseconds, and the derived timing is 11: 41: 12.040 seconds. In this case, the determination unit 41 refers to the acceleration table T and determines the terminal acceleration at 11: 41: 12.020 seconds, which is 20 milliseconds past the derivation timing of 11: 41: 12.040 seconds, as the image acceleration. The corrected acceleration is derived by subtracting from the value. If the derived time lag is a constant value, recording of a time stamp in the acceleration table T may be omitted, and a certain number of old image acceleration values may be used.

  Next, display mode determination processing will be described. The determining unit 41 determines the display mode of the virtual object by updating the parameter value of the virtual object stored in the object storage unit 42 based on the corrected acceleration derived by the corrected acceleration deriving process. When the update of the parameter value of the virtual object is completed, the determination unit 41 outputs an update completion notification to the image composition unit 43. Here, the object storage unit 42 stores various parameter values and a real world synchronization flag for each virtual object in the captured image. The various parameters are, for example, physical calculations such as the position, posture, shape, speed, mass, friction coefficient between objects, elastic coefficient at the time of collision, propulsive force, attractive force, and electromagnetic force (details will be described later). Is the parameter used for.

  The real world synchronization flag is a flag indicating whether or not to display a virtual object as an object representing a part or all of an object in real space (that is, an object in a captured image). That is, for example, when a box is imaged, a virtual object representing a bottom surface that is a part of the box is an object representing a part of the object in the real space, so the real world synchronization flag of the virtual object is Y (Yes ). On the other hand, for example, a virtual object that represents a virtual ball that appears to be in the box when the box is imaged is not an object that represents an object in the real space. Is set to N (No). The virtual object whose real world synchronization flag is Y is displayed in alignment with a part or all of the object in the captured image represented by the virtual object. On the other hand, the virtual object whose real world synchronization flag is N changes its position and speed from moment to moment based on physical calculation (details will be described later) in consideration of parameter values such as velocity and collision determination between virtual objects, It is displayed in a mode corresponding to the change.

  6A is a diagram showing an image of an object stored in the object storage unit 42 in FIG. 2, and FIG. 6B is a table showing an object table in the object storage unit 42 in FIG. The object B shown in FIG. 6A is a virtual object that represents the bottom surface of the box included in the captured image. An object O shown in FIG. 6A is a virtual object representing a ball displayed as if it is in a box in the captured image. In the example of FIG. 6A, only the bottom surface of the box included in the captured image is a virtual object, but the entire box may be a virtual object.

  In the example shown in FIG. 6B, information regarding the object O and the object B, which are the virtual objects shown in FIG. 6A, is stored in the object table OT of the object storage unit. In the object table OT, as information related to the virtual object, an object ID that identifies the virtual object, a position / shape and speed that are parameters of the virtual object, and a real world synchronization flag are stored in association with each other. In the example shown in FIG. 6B, description will be made assuming that the position / shape and speed are stored as parameters. Each virtual object is assumed to be a rigid body, and the speed parameter value is set to one for each virtual object. The initial value of the speed is a zero vector.

The object O indicated by the object ID: O is a virtual object of a ball that is a spherical object, and the center coordinates (coordinates in a captured image indicating the center of the object O) p ₀ and the radius r are the position / shape parameter values. Has been. In the object O, the velocity v ₀ is a velocity parameter value. Further, since the object O is not an object representing an object (an object in the real space) in the captured image, the real world synchronization flag is set to N.

On the other hand, the object B indicated by the object ID: B is a virtual object on the bottom surface of the box which is a planar object, and apex coordinates (coordinates in the captured image indicating the apex of the object B) p ₁ , p ₂ , p ₃ , p ₄ is the parameter value of the position and shape. In the object B, the speed v _b is a speed parameter value. In addition, since the object B is an object that represents a part (bottom surface) of the box that is an object in the captured image (an object in the real space), the real world synchronization flag is set to Y.

  Here, the position and speed of the object O, which is a virtual object whose real world synchronization flag is N, is derived from physical calculation based on parameter values and corrected acceleration in consideration of collision determination between virtual objects. . Note that collision determination between virtual objects can be performed using a known collision determination technique, and thus detailed description thereof is omitted.

The determination unit 41 uses the velocity v ₀ (t) of the object O at time t as the velocity v ₀ (t−1) at time t−1 stored in the object table OT of the object storage unit 42, the gravitational acceleration g, and and derived from the corrected acceleration a _c, it can be determined, for example, by the following equation (1). Incidentally, the following (1) the gravitational acceleration g and the correction acceleration a _c in the expression is a value per unit time. The direction of the gravitational acceleration g may be measured by the acceleration sensor 109 or may be regarded as the downward direction of the terminal device 1. The determination unit 41 uses the derived velocity v ₀ (t) as a new parameter value for the velocity of the object O, and updates the object table OT.

In addition, the determination unit 41 uses the position p ₀ (t) of the object O at time t as the position p ₀ (t−1) at time t−1 and the time t− stored in the object table OT of the object storage unit 42. 1 and derived from the velocity v ₀ (t−1) of 1. For example, it can be obtained by the following equation (2). Note that the speed v ₀ in the following equation (2) is a value per unit time. The determination unit 41 uses the derived position p ₀ (t) as a new parameter value for the position of the object O, and updates the object table OT.

Thus, since the position p ₀ of the object O is obtained based on the corrected acceleration and the velocity v ₀ , the object O can be displayed in a realistic manner in accordance with an actual physical phenomenon. Note that the method for obtaining the speed and position of the object O is not limited to the simple method (Euler method) described above, and may be the Runge-Kutta method with higher accuracy.

  Here, when the magnitude of the terminal acceleration stored in the terminal acceleration storage unit 32 is larger than a predetermined value, the determination unit 41 sets both the image acceleration and the terminal acceleration to 0, that is, sets the corrected acceleration to 0, and the virtual object The display mode may be determined. The terminal acceleration magnitude is the absolute value of the terminal acceleration acquired by the terminal acceleration acquisition unit 31. The image acceleration derived while the terminal device 1 is moving violently is considered to have low reliability. Therefore, the predetermined value is, for example, a lower limit value of the terminal acceleration that is considered to reduce the reliability of the image acceleration value derived from the captured image. Further, when the determination unit 41 determines that the terminal acceleration stored in the terminal acceleration storage unit 32 is larger than a predetermined value, the determination unit 41 determines the predetermined time (that is, the imaging interval and The display mode of the virtual object may be determined by setting both the image acceleration and the terminal acceleration to 0, that is, the corrected acceleration to 0, for a fixed time longer than a derived time lag (a time including the calculation time).

  The image synthesis unit 43 synthesizes (synchronizes) the virtual object with the captured image in the real space. The image composition unit 43 acquires a captured image related to the image acceleration used for the update related to the update completion notification from the image storage unit 21, triggered by the update completion notification from the determination unit 41. Then, referring to the object table OT of the object storage unit 42, the virtual object whose real world synchronization flag is Y is specified, and the latest position of the virtual object is specified from the captured image. The image composition unit 43 updates the object table OT of the object storage unit 42 so that the position of the virtual object whose real world synchronization flag is Y becomes the latest position specified from the captured image. Thereafter, the image composition unit 43 refers to the object table OT of the object storage unit 42 and specifies the positions and shapes of all the stored virtual objects (virtual objects with the real world synchronization flag Y and N virtual objects). . Then, the image composition unit 43 synthesizes each virtual object with the acquired captured image so that the position / shape of each virtual object becomes the specified position / shape, thereby generating a composite image. The image composition unit 43 outputs the composite image to the display unit 44.

  The display unit 44 functions as a display unit that displays the virtual object in synchronization with the captured image of the real space according to the display mode determined by the determination unit 41. Specifically, the display unit 44 causes the output device 107 to display the composite image generated by the image composition unit 43.

  Next, a series of processing by the terminal device 1 will be described with reference to FIG. FIG. 7 is a flowchart showing a series of processing of the display method performed by the terminal device 1 of FIG. The process illustrated in FIG. 7 is started in response to the imaging device 108 generating a captured image after, for example, dedicated software (application) installed in the terminal device 1 is activated.

  First, the image acquisition unit 11 acquires a captured image generated by imaging the real space by the imaging device 108 (step S1, step of acquiring a captured image). The image acquisition unit 11 stores the acquired captured image and meta information of the captured image in the image storage unit 21. Subsequently, a motion vector is derived from the plurality of captured images stored in the image storage unit 21 by the speed calculation unit 22 (step S3, step of deriving image acceleration). Specifically, the speed calculation unit 22 performs the object identification process on both the first image and the second image, and then performs an operation on the objects displayed in both the first image and the second image. The position in one image is compared with the position in the second image, and the amount and direction of movement from the position in the second image to the position in the first image are derived as motion vectors.

  Subsequently, the speed calculation unit 22 derives the image speed based on the motion vector and the meta information of the first image and the meta information of the second image (step S5, step of deriving the image acceleration). The speed calculation unit 22 stores the derived image speed and the time information of the image speed in the speed storage unit 23. Subsequently, the acceleration calculation unit 24 derives the image acceleration based on the difference between the plurality of image speeds stored in the speed storage unit 23 (step S7, step of deriving the image acceleration). Specifically, the acceleration calculation unit 24 calculates the speed difference that is the difference between the image speed stored in the third storage area and the image speed stored in the fourth storage area, and the time information stored in the third storage area. And the time difference that is the difference between the time information stored in the fourth storage area is obtained, and the image acceleration is derived (calculated) from the speed difference and the time difference. The acceleration calculation unit 24 stores the derived image acceleration and time information of the image acceleration in the image acceleration storage unit 25.

  Subsequently, the terminal acceleration is acquired by the terminal acceleration acquisition unit 31 based on the acceleration measured by the acceleration sensor 109 (step S9, step of acquiring apparatus acceleration). The terminal acceleration acquisition unit 31 stores a terminal acceleration and a time stamp indicating the measurement time of the terminal acceleration in the terminal acceleration storage unit 32. Subsequently, the determination unit 41 derives a corrected acceleration obtained by correcting the image acceleration with the terminal acceleration (step S11, step of determining the display mode). Specifically, the determination unit 41 derives the corrected acceleration by subtracting the terminal acceleration from the image acceleration.

  Subsequently, the display unit of the virtual object is determined by the determination unit 41 based on the corrected acceleration (step S13, step of determining the display mode). Specifically, the determination unit 41 determines the display mode of the virtual object by updating the parameter value of the virtual object stored in the object storage unit 42 based on the corrected acceleration derived by the corrected acceleration deriving process. . When the update of the parameter value of the virtual object is completed, the determination unit 41 outputs an update completion notification to the image composition unit 43. Then, after the image synthesis unit 43 synthesizes the virtual object with the captured image of the real space, the display unit 44 displays the virtual object in synchronization with the captured image of the real space in the display mode determined by the determination unit 41. (Step S15, displaying step).

  Next, operations and effects of the terminal device 1 according to the present embodiment will be described.

  When a virtual object is displayed in synchronization with a captured image using AR technology, the position of the object in the real space may change (the object in the real space moves). In such a case, it is conceivable to display the virtual object so as to be aligned with the object in the real space in consideration of the collision determination and considering the gravity direction of the real space. For example, the terminal device 151 according to the comparative example illustrated in FIG. 8 is a terminal device that considers only the gravity direction of the real space without considering the acceleration of the imaged object. In the terminal device 151, a box R that is an object in the real space imaged by the imaging device and a ball BV that is a virtual object displayed as if inside the box R are displayed on the display of the terminal device. ing. Then, when the box R is moved upward as shown in FIG. 8B, both the box R and the ball BV, which is a virtual object, are moved upward and displayed. Further, as shown in FIG. 8C, when the moving direction of the box R that has been moved upward is changed downward, the box R is moved downward and displayed, and a ball that is a virtual object. BV follows the box R and is moved downward and displayed. However, when the box containing the ball in the real space is moved up and down, it is considered that the ball may jump out of the box. That is, as shown in the example of FIG. 8, when the collision determination is performed and the direction of gravity in the real space is taken into consideration, the ball BV, which is a virtual object, is displayed so as to always follow the box. There is a possibility that the display does not match the phenomenon.

  In this regard, according to the terminal device 1 of the present embodiment, image acceleration is derived based on the position difference between the captured images of the same object and the difference in time at which each captured image was captured by the acceleration deriving unit 20. At the same time, the terminal acceleration is acquired by the terminal acceleration acquisition unit 31. Then, the determination unit 41 derives a corrected acceleration obtained by correcting the image acceleration with the terminal acceleration, and the display unit 44 displays the virtual object in synchronization with the captured image in a display mode based on the corrected acceleration.

  Thus, the virtual object is displayed according to the acceleration of the object in the real space by determining the display mode of the virtual object in consideration of the image acceleration that is the acceleration of the imaged object. Thereby, the virtual object can be displayed in a realistic manner in accordance with an actual physical phenomenon. For example, as shown in FIG. 1C, when the moving direction of the box R that has been moved upward is changed downward, the box R is moved downward and displayed. On the other hand, since the ball BV has an upward speed due to the upward movement shown in FIG. 1B, when the box R moves downward as shown in FIG. 1C. Is displayed so as to be flipped up away from the box R. Such a display is a realistic display in accordance with an actual physical phenomenon, and is realized by considering the acceleration of the object in the captured image.

  For example, when image acceleration is derived from a plurality of captured images captured while the terminal device itself is moving, the position of the object displayed in the captured image is relatively changed due to the movement of the terminal device itself. There is a risk that the derived image acceleration value deviates from the actual movement of the object. For example, the terminal device 152 according to the comparative example illustrated in FIG. 9 is a terminal device that does not consider the movement (acceleration) of the terminal device itself. In the terminal device 152, a box R, which is an object in the real space imaged by the imaging device, and a ball BV, which is a virtual object displayed as if inside the box R, are displayed on the display of the display device. ing. When the box R is moved upward as shown in FIG. 9B, both the box R and the ball BV, which is a virtual object, are moved upward and displayed. In this case, if the position of the terminal device 152 itself is moved upward without changing the position of the box R in the real space, the position of the box displayed in the captured image changes relatively downward. As shown in FIG. 9C, the ball BV is displayed so as to be flipped up. Since the position of the box R does not change in the real space, the display of the ball BV (display that jumps up) is difficult to say as a realistic display in accordance with an actual physical phenomenon.

  In this regard, in the terminal device 1 of the present embodiment, the corrected acceleration obtained by correcting the image acceleration by the terminal acceleration is used, and therefore the display mode of the virtual object is determined in consideration of the movement of the terminal device 1 itself. Thus, even when a plurality of captured images captured while the terminal device 1 is moving, a virtual object that matches the movement of the object in the real space can be displayed.

  Further, in the terminal device 1 of the present embodiment, the determination unit 41 derives the corrected acceleration by subtracting the terminal acceleration from the image acceleration. Therefore, the terminal acceleration corresponding to the movement of the terminal device 1 is appropriately canceled from the image acceleration. Therefore, it is possible to derive a corrected acceleration that appropriately reflects the movement of an actual object.

  In the terminal device 1 of the present embodiment, the determination unit 41 determines the display mode with the image acceleration and the terminal acceleration set to 0 when the magnitude of the terminal acceleration is larger than a predetermined value. When the terminal acceleration is large, that is, when the terminal device 1 moves violently, it is considered that the reliability of the image acceleration value derived from the difference in the position of the object between the captured images is low. For this reason, when the terminal acceleration is larger than a predetermined value, the display mode is determined by setting the image acceleration and the terminal acceleration to 0, so that the display mode of the virtual object is determined based on information with low reliability. It can be avoided.

  Moreover, the terminal device 1 of this embodiment is provided with the terminal acceleration memory | storage part 32 which matches and memorize | stores the terminal acceleration acquired by the terminal acceleration acquisition part 31 with the time when the terminal acceleration was measured. The determining unit 41 specifies the time at which the image acceleration is derived by the acceleration deriving unit 20 (more specifically, the acceleration calculating unit 24) as a deriving timing, and among the terminal accelerations stored in the terminal acceleration storage unit 32 The image acceleration is corrected by using the past terminal acceleration for a predetermined time from the derivation timing. Since the image acceleration is derived from a plurality of captured images captured at different times, an imaging interval is required for the derivation. Also, calculation time is required to derive the image acceleration. Considering the imaging interval and calculation time, the derived image acceleration corresponds to the past terminal acceleration by a predetermined time from the derivation timing. For this reason, by correcting the image acceleration by using the past terminal acceleration for a predetermined time from the derivation timing, it is possible to derive the corrected acceleration more appropriately reflecting the movement of the actual object.

  Moreover, in the terminal device 1 of this embodiment, when the magnitude | size of a terminal acceleration is larger than a predetermined value, the determination part 41 calculates an image acceleration and a terminal acceleration for a fixed time longer than the said predetermined time. The display mode is determined as 0. When the terminal acceleration increases, the image acceleration and the terminal acceleration are displayed as 0 at least when the terminal acceleration increases by setting the image acceleration and the terminal acceleration to 0 for a certain time longer than a predetermined time. Aspects can be determined. Thereby, it can avoid more reliably that the display mode of a virtual object is determined based on information with low reliability.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment. For example, in the above embodiment, the image acquisition unit 11 acquires a captured image captured by the imaging device 108 of the terminal device 1, but is not limited thereto, and acquires a captured image captured outside the terminal device. May be. In addition, the determination unit 41 derives the corrected acceleration by subtracting the terminal acceleration from the image acceleration. However, for example, the corrected acceleration may be derived based on the image acceleration and the terminal acceleration by a calculation other than the subtraction.

  In the description of the processing flow using FIG. 7, it has been described that the terminal acceleration is acquired after the image acceleration is derived. However, the present invention is not limited to this, and the terminal acceleration is acquired before the image acceleration is derived. It may be a thing. Moreover, although the display position of the virtual object has been described as an example of the display mode of the virtual object determined by the determination unit 41 using the corrected acceleration, the display mode of the virtual object determined based on the corrected acceleration is not limited to this. For example, the determination unit 41 may determine the shape of the virtual object as the display mode of the virtual object using the corrected acceleration. As a specific example, a virtual object of a board (hereinafter referred to as a board object) whose real world synchronization flag is Y (Yes) and displayed on a board that is an object in a captured image, and a virtual board A soft ball virtual object whose real world synchronization flag is N (No) (hereinafter referred to as a ball object) displayed as being played by a board object in accordance with the movement of the object is displayed as a virtual object. An example will be described. The determining unit 41 smashes the shape of the ball object when it is flipped by the plate object as the image acceleration, which is the acceleration of the plate in the captured image, increases as the plate is vigorously swung in the real space and the corrected acceleration increases. The shape of the ball object may be determined so as to have a shape (a deformed shape). The display mode of the virtual object determined based on the corrected acceleration is not limited to the display position of the virtual object and the shape of the virtual object, and other display modes that enable the virtual object to be displayed in a realistic mode. It may be.

  DESCRIPTION OF SYMBOLS 1 ... Terminal device, 11 ... Image acquisition part, 20 ... Acceleration deriving part, 21 ... Image storage part, 22 ... Speed calculation part, 23 ... Speed storage part, 24 ... Acceleration calculation part, 25 ... Image acceleration storage part, 31 ... Terminal acceleration acquisition unit, 32 ... terminal acceleration storage unit, 41 ... determination unit, 44 ... display unit.

Claims (7)

A display device that displays a virtual object that is an object in a virtual space in synchronization with an image in a real space,
Image acquisition means for acquiring a captured image generated by imaging a real space;
Based on the difference in position between the captured images of the same object displayed in each of the plurality of captured images acquired by the image acquisition means and the difference in time at which each captured image was captured, the image in the captured image Acceleration deriving means for deriving image acceleration that is the acceleration of the object;
Acceleration acquisition means for acquiring a device acceleration which is a measured acceleration of the display device;
Determining means for deriving a corrected acceleration obtained by correcting the image acceleration derived by the acceleration deriving means with the device acceleration acquired by the acceleration acquiring means, and determining a display mode of the virtual object using the corrected acceleration. When,
A display device comprising: a display unit configured to display the virtual object in synchronization with the captured image in the real space according to the display mode determined by the determination unit.   The display device according to claim 1, wherein the determination unit derives the corrected acceleration by subtracting the device acceleration from the image acceleration.   The display device according to claim 1, wherein the determining unit determines the display mode by setting the image acceleration and the device acceleration to 0 when the magnitude of the device acceleration is larger than a predetermined value. Apparatus acceleration storage means for storing the apparatus acceleration acquired by the acceleration acquisition means in association with the time when the apparatus acceleration was measured;
The determining means specifies the time at which the image acceleration is derived by the acceleration deriving means as a derivation timing, and among the apparatus accelerations stored in the apparatus acceleration storage means, a predetermined amount of time from the derivation timing. The display device according to claim 1, wherein the image acceleration is corrected using the past device acceleration.   The determining means determines the display mode by setting the image acceleration and the apparatus acceleration to 0 for a predetermined time longer than the predetermined time when the magnitude of the apparatus acceleration is larger than a predetermined value. The display device according to claim 4. The acceleration deriving means includes
Image storage means for storing the captured image acquired by the image acquisition means;
The speed of the object based on the difference in position between the captured images of the same object displayed in each of the plurality of captured images stored in the image storage means and the difference in time at which each captured image was captured A speed calculating means for calculating the image speed,
Speed storage means for storing the image speed calculated by the speed calculation means;
Acceleration calculation means for calculating the image acceleration based on a difference between the plurality of image speeds stored in the speed storage means;
The display device according to claim 1, further comprising an image acceleration storage unit that stores the image acceleration calculated by the acceleration calculation unit. A display method performed by a display device that displays a virtual object that is an object in a virtual space in synchronization with an image in a real space,
Obtaining a captured image generated by imaging a real space;
Based on the difference in position between the captured images of the same object displayed in each of the plurality of captured images acquired in the step of acquiring the captured image and the difference in time at which each captured image was captured, the imaging Deriving an image acceleration that is an acceleration of the object in the image;
Obtaining a device acceleration which is a measured acceleration of the display device;
Deriving a corrected acceleration obtained by correcting the image acceleration derived in the step of deriving the image acceleration by the device acceleration acquired in the step of acquiring the device acceleration, and displaying the virtual object using the corrected acceleration Determining an aspect;
Displaying the virtual object in synchronization with the captured image in the real space according to the display mode determined in the determining step.

Images