JP7467748B1 - Display control device, display system and program - Google Patents

Abstract

[Problem] Even if the user is not in front of the display device, the user's viewpoint and the displayed image are matched to provide a sense of immersion. [Solution] A display control device includes a display control unit that controls the display of a display object on the display device, an image acquisition unit that acquires an image of the user's face captured by an imaging device rigidly connected to the display device, a viewpoint relative position calculation unit that calculates the relative positional relationship between the user's viewpoint and the display device, a gravity information acquisition unit that acquires information about acceleration in the gravity direction and its azimuth angle, an initial position information acquisition unit that acquires position information of the device's location, an acceleration information acquisition unit that acquires acceleration information, and a vector calculation unit that calculates the absolute position of a projection plane onto which the display device projects an image, calculates the absolute position coordinates of the user, and calculates the direction in which the projection plane faces, and the display control unit transforms the three-dimensional information of the display object based on the information calculated by the vector calculation unit. [Selected Figure] Figure 9

Description

The present invention relates to a display control device, a display system, and a program.

Conventionally, there has been technology that displays three-dimensional information of objects that exist in a virtual space on a display device such as a smartphone or tablet terminal device. By viewing an image displayed on a display device using such technology, a user can get the sensation that a virtual object is actually present through the display device. For example, as an example of a device using such technology, an image processing device that can synthesize three-dimensional information of an object in a virtual space with an image of the real world has been disclosed (see Patent Document 1).

JP 2022-113269 A

According to such technology, three-dimensional information of an object in a virtual space is displayed on the display device, assuming that the user views the display device from the front. However, there are cases where the user's viewpoint remains fixed and only the display device moves, or the angle of the display device changes. In such cases, since the user's line of sight is fixed, the user expects to continue to view the same virtual object through the display device. On the other hand, the control performed by the device assumes that the user is in front, so the image the user expects does not match the image actually displayed. Therefore, even if the user views a virtual object through the display device, the user cannot get the feeling that the object is actually there, and it is difficult to get a sense of immersion in the virtual world. In other words, the conventional technology has a problem that the user's viewpoint does not match the displayed image when the user is not in front of the display device.

The present invention has been made in consideration of these circumstances, and its purpose is to provide a display control device, display system, and program that allows the user's viewpoint to match the displayed image, even when the user is not in front of the display device, thereby enabling the user to feel immersed in the virtual world.

(1) One aspect of the present invention includes a display control unit that controls the display of a display object having three-dimensional information on a display device; an image acquisition unit that acquires an image captured by an imaging device rigidly connected to the display device, the image capturing an image of a user's face looking at the display object displayed on the display device; a viewpoint relative position calculation unit that calculates a relative positional relationship between the viewpoint of the user and the display device based on the acquired image; a gravity information acquisition unit that acquires information regarding acceleration in the gravity direction and its azimuth angle; an initial position information acquisition unit that acquires position information of the location of the device when a predetermined point is set as the origin; and an acceleration information acquisition unit that acquires acceleration information of the device. a vector calculation unit that calculates an absolute position of a projection plane onto which the display device projects an image, calculates absolute position coordinates of the user's viewpoint based on the calculated absolute position of the projection plane and the relative positional relationship between the user's viewpoint and the display device calculated by the viewpoint relative position calculation unit, and calculates a vector of a direction in which the projection plane faces based on the calculated absolute position of the projection plane, gravity information acquired by the gravity information acquisition unit, and the acceleration information acquired by the acceleration information acquisition unit, and the display control unit is a display control device that controls the display on the display device so as to perspectively project an image obtained by deforming three-dimensional information of the object to be displayed onto a projection plane determined by the absolute position of the projection plane onto which the display device projects an image and the vector of the direction in which the projection plane faces, with the absolute position coordinates of the user's viewpoint as a focus.
(2) Also, one aspect of the present invention is that in the display control device described above in (1), the viewpoint relative position calculation unit calculates the relative positional relationship between the user's viewpoint and the display device based on attributes of the user that have been acquired in advance.
(3) Also, one aspect of the present invention is that in the display control device of (1) described above, the viewpoint relative position calculation unit measures the distance between the user's pupils based on the image acquired by the image acquisition unit, and calculates the relative positional relationship between the user's viewpoint and the display device based on the measured distance between the user's pupils.
(4) Also, in one aspect of the present invention, in the display control device of (3) described above, the viewpoint relative position calculation unit measures the interpupillary distance and interpupillary angle of view of the user based on the image acquired by the image acquisition unit, and calculates the relative positional relationship between the user's viewpoint and the display device based on the measured interpupillary distance and interpupillary angle of view of the user.
(5) Also, one aspect of the present invention is that in the display control device described above in (3), the viewpoint relative position calculation unit calculates the relative positional relationship between the user's viewpoint and the display device based on the user's interpupillary distance stored in advance and the user's interpupillary distance measured from the image acquired by the image acquisition unit.
(6) Also, one aspect of the present invention is that in any of the display control devices described above in (1) to (5), the viewpoint relative position calculation unit updates the relative positional relationship between the user's viewpoint and the display device in response to a change in the ratio of the sizes of the user's left and right pupils or irises obtained based on the image acquired by the image acquisition unit.
(7) Furthermore, one aspect of the present invention is a display control device according to any one of (1) to (6) above, wherein the display control unit displays a plurality of the display objects on the display device, and further includes an error assigning unit that assigns different errors to each of the plurality of display objects based on a virtual position of each of the plurality of display objects, and the display control unit controls the display of the plurality of display objects on the display device by varying the relative relationship between the display objects and the background within the range of the assigned errors.
(8) Also, one aspect of the present invention is that in the display control device described above in (7), the display control unit controls the display of a plurality of display objects on the display device by varying the position at which the display objects are displayed within a given error range.
(9) Also, one aspect of the present invention is that in the display control device described above in (7), the display control unit controls the display of a plurality of display objects on the display device by varying the position at which the display objects are displayed in a predetermined direction within a given error range.
(10) Also, one aspect of the present invention is that in the display control device of (1) described above, the display control unit displays on the display device a plurality of display objects that are each located at a different distance from the display device, and the plurality of display objects are displayed at different display timings depending on their distance from the display device.
(11) Another aspect of the present invention is a display system comprising: a display control device according to any one of (1) to (10) described above; an imaging device that captures an image of a user's face and provides the captured image to the image acquisition unit; and the display device controlled by the display control unit.
(12) Also, one aspect of the present invention is a method for controlling a computer to display a display object having three-dimensional information on a display device, comprising: a display control step of controlling a computer to display a display object having three-dimensional information on a display device; an image acquisition step of acquiring an image captured by an imaging device rigidly connected to the display device, the image capturing an image of a user's face looking at the display object displayed on the display device; a viewpoint relative position calculation step of calculating a relative positional relationship between the viewpoint of the user and the display device based on the acquired image; a gravity information acquisition step of acquiring information regarding acceleration in the gravity direction and its azimuth angle; an initial position information acquisition step of acquiring position information of the location of the device when a predetermined point is set as the origin; an acceleration information acquisition step of acquiring acceleration information of the device; and a pre-determined position based on the position information acquired by the initial position information acquisition step and the acceleration information acquired by the acceleration information acquisition step. a vector calculation step of calculating an absolute position of a projection plane onto which the display device projects an image, calculating absolute position coordinates of the user's viewpoint based on the calculated absolute position of the projection plane and the relative positional relationship between the user's viewpoint and the display device calculated in the viewpoint relative position calculation step, and calculating a vector of the direction in which the projection plane faces based on the calculated absolute position of the projection plane, gravity information acquired in the gravity information acquisition step, and the acceleration information acquired in the acceleration information acquisition step, wherein the display control step is a program that controls the display on the display device so that an image obtained by deforming three-dimensional information of the object to be displayed is perspectively projected onto a projection plane determined by the absolute position of the projection plane onto which the display device projects an image and the vector of the direction in which the projection plane faces, with the absolute position coordinates of the user's viewpoint as a focus.

According to the present invention, even if the user is not in front of the display device, the user's viewpoint can be aligned with the displayed image, providing the effect of creating a sense of immersion in the virtual world.

11A and 11B are diagrams for explaining changes in an image displayed by a display system according to an embodiment. 1A to 1C are diagrams for explaining virtual objects displayed by the display system according to the embodiment. FIG. 11 is a first diagram for explaining an example of a process for analyzing a user's viewpoint based on an image captured by the display system according to the present embodiment. 13 is a second diagram for explaining an example of a process for analyzing a user's viewpoint based on an image captured by the display system according to the present embodiment. FIG. FIG. 1 is a diagram for explaining an image of a display system according to an embodiment of the present invention. 10A and 10B are diagrams for explaining excess and deficiency of information analyzed by the display system according to the embodiment. 5A and 5B are diagrams for explaining vectors calculated by the display system according to the embodiment. FIG. 2 is a block diagram showing an example of a functional block for realizing the display system according to the present embodiment. 1 is a functional configuration diagram showing an example of a functional configuration of a display control device according to an embodiment of the present invention. FIG. 11 is a functional configuration diagram showing a modified example of the functional configuration of the display control device according to the embodiment. 11A and 11B are diagrams for explaining how to handle a case where the user's face is tilted in the display system according to the present embodiment. 1 is a block diagram showing an example of an internal configuration of a display control device according to an embodiment of the present invention. 11 is a diagram for explaining a change in coordinate axes in response to a change in the inclination of a smartphone when attempting to display a virtual object on the smartphone using augmented reality or mixed reality technology according to the related art. FIG. FIG. 13 is a diagram for explaining changes in coordinate axes in response to changes in the inclination of smart glasses when attempting to display a virtual object on the smart glasses using virtual reality technology according to the related art. 11A and 11B are diagrams for explaining a change in the appearance of a virtual object when the smartphone is tilted in a case where the virtual object is displayed on the smartphone using conventional technology.

The display control device, display system, and program according to the present invention will be described in detail below with reference to the accompanying drawings, showing preferred embodiments. Note that the present invention is not limited to these embodiments, and includes various modifications or improvements. In other words, the components described below include those that a person skilled in the art can easily imagine, and those that are substantially the same, and the components described below can be combined as appropriate. In addition, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the present invention. In addition, in the drawings below, the scale and number of each structure may be different from the scale and number of the actual structure in order to make each structure easier to understand. In addition, due to the available characters, those in the formulas shown below that are accompanied by vector symbols (variables expressed in English letters or Greek letters) may be described as "vectors (English letters or Greek letters)" in the text of this specification.

[Embodiment]
Hereinafter, an embodiment of the present invention will be described with reference to the drawings. First, the matters that are the premise of the embodiment of the present invention will be described. The display control device, display system, and program according to the present invention can be used in technologies such as Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR). The display control device, display system, and program according to the present invention can display a virtual object by superimposing it on an image (or video, hereinafter simply referred to as an image) of the real world. The image of the real world is an image of the world that the user actually sees, and may be captured by an imaging device. In addition, features may be extracted from an image captured by an imaging device, and a virtual object may be superimposed according to the extracted features. In addition, according to this embodiment, the position and direction of the device are detected based on position information, orientation information, gyro information, and the like, and the display is controlled based on the detected position and direction. By performing such a display, the user can obtain a sense of immersion.

However, this embodiment is not limited to this example, and it is also possible to display only virtual objects. A virtual object is anything that has three-dimensional information, and broadly includes objects, backgrounds, and the like. In the following explanation, an embodiment in which a virtual object is simply displayed will be explained. The background in this case may be from the real world or from the virtual world.

The technology according to this embodiment can be applied to devices with displays such as smartphones and tablet terminals, as well as wearable devices such as smart glasses and smart goggles. In the following explanation, an embodiment using a smartphone will be described as an example.

First, we will explain in detail the problems with the conventional technology with reference to Figures 13 to 15.

Figure 13 is a diagram for explaining changes in coordinate axes according to changes in the inclination of a smartphone when attempting to display a virtual object on the smartphone using augmented reality or mixed reality technology according to the conventional technology. With reference to the figure, an example of a case where a user U views a virtual object 92 using a device 91 capable of displaying virtual objects will be described. The device 91 may be, for example, a smartphone.

Figure 13 (A) shows an example in which the user U holds the device 91 at a first angle. In this case, an object 92 exists at the point where the user U's viewpoint is connected to the device 91. The device 91 captures an image or video of the real world and displays the object 92 by superimposing it on the captured image or video. The display of the object 92 is drawn, for example, based on a reference feature on the captured image. By looking at the display of the device 91 from the front, the user U can get the feeling that the object 92 exists in the real world. However, according to conventional technology, this is not the case when the angle of the device 91 is tilted by the user U and the user views the object 92 from a position that is not in front of the display.

FIG. 13(B) shows an example in which the user U holds the device 91 at a second angle. In the example shown in FIG. 13(B), the relationship between the coordinate axes of the real world and the coordinate axes of the device 91 is different from the example shown in FIG. 13(A). As a result of the device 91 being tilted by the user U, the coordinate axes of the device 91 are also tilted, and the object 92 falls outside the imaging angle of view of the device 91. In this case, even though the object 92 should exist at the point where the user U's viewpoint and the device 91 are connected, the object 92 is not displayed on the display of the device 91. Therefore, the user U is unable to feel immersed in the virtual space in which the virtual object is displayed, depending on the tilt of the device 91.

Figure 14 is a diagram for explaining the change in coordinate axes according to the change in the inclination of smart glasses when a virtual object is to be displayed on the smart glasses using virtual reality technology according to the conventional technology. The example shown in Figure 14 is an example of a case where smart glasses are used. In this case, as in the case shown in Figure 13, the user U has multiple issues, such as difficulty in achieving a sense of immersion in the virtual space in which the virtual object is displayed, and issues in achieving a sense of immersion. With reference to the same figure, an example of a case where the user U views a virtual object 92 using a device 93 capable of displaying virtual objects will be described. The device 93 may be, for example, smart glasses that the user U can wear.

FIG. 14(A) shows an example of a case where a user U wears a device 93 and views a virtual object 92 at a first angle. FIG. 14(B) shows an example of a case where a user U wears a device 93 and views a virtual object 92 at a second angle. In the case of smart glasses, the device 93 performs perspective drawing with a fixed operator's viewpoint as a focus. In the case of smart glasses, the viewpoint and the coordinate axis of the device are fixed, so as shown in FIG. 13, a situation does not occur in which only the coordinate axis of the device changes while the viewpoint of the user U is fixed. Therefore, in the case of smart glasses, there is an effect that a greater sense of immersion can be obtained compared to a smartphone. However, smart glasses have a safety issue because the field of view is blocked. In addition, smart glasses require a lot of effort to wear and a lot of effort to calibrate. Furthermore, there is a problem that it is not easy to miniaturize smart glasses with current technology. If smart glasses are not small, a sense of discomfort occurs when wearing them, making it difficult to obtain a sense of immersion in the virtual world. For these reasons, it has to be said that the penetration rate of smart glasses is low compared to that of smartphones.

Figure 15 is a diagram for explaining how a virtual object changes when the smartphone is tilted when the virtual object is displayed on the smartphone using conventional technology. With reference to the figure, the change in the appearance of the virtual object when the smartphone is tilted in conventional technology will be explained. In the illustrated example, a device 91 capable of displaying virtual objects displays, of multiple virtual objects 92-1, object 92-2, and object 92-3, an object that exists in the direction in which the device 91 is facing.

FIG. 15(A) shows an example of a case where the user U holds the device 91 at a first angle. In this case, the device 91 faces the direction in which the object 92-2 exists, and therefore displays an image 92-2A of the object 92-2. FIG. 15(B) shows an example of a case where the user U tilts only the device 91 while keeping the viewpoint of the user U fixed. As shown in FIG. 15(B), when the device 91 is tilted to the right, the object 92-2 and the object 92-3 exist in front of the device 91. Therefore, the device 91 displays an image 92-2A of the object 92-2 and an image 92-3A of the object 92-3. However, only the object 92-2 should exist in the line of sight of the user U through the device 91, and if such a display were made, the user U would not be able to feel immersed in the virtual world.

Next, the display control device, display system, and program according to the present invention will be described with reference to Figs. 1 to 12. In the following description, an example of a case in which the display control device, display system, and program according to this embodiment are applied to a smartphone SP will be described. However, this embodiment is not limited to this example, and can be widely applied to devices that have a display and display images to a user. The smartphone SP shown below as an example is a device that can display virtual objects. First, an overview of the display control device, display system, and program according to this embodiment will be described with reference to Figs. 1 and 2.

Figure 1 is a diagram for explaining changes in an image displayed by a display system according to one embodiment. With reference to the figure, a change in the appearance of a virtual object in relation to the viewpoint of a user U when a smartphone SP is tilted in a display system according to this embodiment will be explained. In the example shown, the smartphone SP displays, of multiple virtual objects OBJ1, object OBJ2, and object OBJ3, an object that exists in the direction in which the smartphone SP is facing. The positional relationship between the positions of these virtual objects relative to each other on the coordinate axes of the virtual world is predetermined.

In the positional relationship shown in FIG. 1(A), the smartphone SP faces the direction in which the object OBJ2 exists. Therefore, the smartphone SP displays an image OBJ2A of the object OBJ2. Here, in FIG. 1(B), there is no change in the positional relationship between the user U and the center of gravity of the smartphone SP, but the direction in which the smartphone SP faces changes as the smartphone SP rotates to the right. According to the conventional technology, the display object was determined by detecting only the rotation of the smartphone SP, so that the display object changed as the smartphone SP rotated. However, according to the present embodiment, in addition to detecting the rotation of the smartphone SP, information such as the user U's viewpoint is also taken into consideration, and display is performed according to the positional relationship between the user U's viewpoint, the smartphone SP, and the virtual object.

Here, FIG. 1(C) shows an example of information displayed by the smartphone SP when the user U moves from the positional relationship shown in FIG. 1(B) to the positional relationship shown in FIG. 1(C), there is no change in the positional relationship between the smartphone SP and the virtual object. However, in FIG. 1(C), the user U changes his/her viewpoint and looks into the screen from the left side (from the front of the smartphone SP). In such a case, the smartphone SP displays objects OBJ2 and OBJ3 that are in the line of sight of the user U through the smartphone SP. That is, according to this embodiment, by taking into account information such as the user U's viewpoint, a display is performed according to the positional relationship between the user U's viewpoint, the smartphone SP, and the virtual object, so that the user U can more easily get a sense of immersion in the virtual world. The relative positions of multiple rendered objects change in response to the movement of the viewpoint, providing motion-based clues (motion parallax) related to depth perception, resulting in depth perception that is independent of other perceptual psychological clues (e.g. binocular disparity), and providing a sense of three-dimensionality for the object.

2 is a diagram for explaining a virtual object displayed by the display system according to the present embodiment. The figure shows the positional relationship between the user U, the smartphone SP, and the virtual object OBJ. The figure also shows the coordinate axes of the real world in which the user U exists and the coordinate axes of the smartphone SP. The coordinate axes of the smartphone SP can be said to be the depiction reference coordinate axes for depicting a virtual object on the smartphone SP. FIG. 2(A) shows an example of a case in which the user U holds the smartphone SP at a first tilt, and FIG. 2(B) shows an example of a case in which the user U holds the smartphone SP at a second tilt. According to this embodiment, as shown in the figure, even if the tilt of the smartphone SP changes, the coordinate axes of the smartphone SP remain fixed. That is, according to this embodiment, it is possible to draw the object OBJ so that it is transparent through the screen based on the viewpoint of the user U, regardless of the position or angle of the screen. Furthermore, by adopting such a configuration, the user U can experience as if the virtual object OBJ is fixed in the space of the real world.

Next, the operating principles of the display control device 10, display system 1, and program according to this embodiment will be described with reference to Figures 3 to 7.

FIG. 3 is a first diagram for explaining an example of an analysis process of a user's viewpoint based on an image captured by the display system according to the present embodiment. FIG. 4 is a second diagram for explaining an example of an analysis process of a user's viewpoint based on an image captured by the display system according to the present embodiment. First, the display system 1 captures the face of the user U and analyzes the position of the user U's pupil. In the figure, an image IMG in which the face of the user U is captured is shown. The image IMG may be one of consecutive frame images included in the moving image data. The display system 1 analyzes the image IMG and specifies the positions of the right eye ER and the left eye EL of the user U. The display system 1 calculates the distance _Dbase from the right eye ER to the left eye EL of the user U based on the specified positions of the right eye ER and the left eye EL of the user U. The distance Dbase from the right eye ER to the left eye EL of the user U may be calculated by image processing, or may be measured by the user U himself and input in advance.

The display system 1 also specifies an angle η between a line connecting the center O of the image IMG to the center of the pupil and a reference line of the image IMG. The display system 1 also specifies an interpupillary angle of view φ. Here, when the reference length L ₀ =D _base /sin(φ/2), a point that is a point away from the center COGR of the right eye ER and the center COGL of the left eye EL by the reference length L ₀ and is at the midpoint of the distance D _base from the right eye ER to the left eye EL of the user U can be defined as point P. The interpupillary angle of view φ is an angle between a line connecting the point P and the center COGR of the right eye ER and a line connecting the point P and the center COGL of the left eye EL. The display system 1 also specifies an angle of view θ from the center O of the image IMG to the center of the pupil. In addition, in order to specify these angles and calculate the numerical values, a known image processing technique such as an object detection technique may be used.

5 is a diagram for explaining an image of the display system according to the present embodiment. With reference to the figure, a calculation process of a vector V indicating the coordinates of the perspective plane in the virtual space, a plane vector W of the perspective plane in the virtual space, and a vector U indicating the viewpoint coordinates in the virtual space performed by the display system 1 will be described. The perspective plane is the display surface of a display (display device) equipped in the smartphone SP. In the calculation of the vector U indicating the viewpoint coordinates in the virtual space, the vector V indicating the coordinates of the perspective plane in the virtual space, and the plane vector W of the perspective plane in the virtual space, in addition to the distance D _base from the right eye ER to the left eye EL of the user U described above, the angle η that is the angle between the line connecting the center of gravity of the pupil and the reference line of the image IMG, the interpupillary angle of view φ, and the angle of view θ from the center O of the image IMG to the center of the pupil, a vector S indicating the initial coordinates in the virtual space, and a vector ζ indicating the angle between the display device and the direction of gravity are used.

The vector S indicating the initial coordinates in the virtual space is the initial coordinates (reference coordinates) in the virtual space. The virtual object to be displayed is generated based on the vector S indicating the initial coordinates in the virtual space. The vector S indicating the initial coordinates in the virtual space may be determined in advance and stored in a predetermined storage unit.

The vector ζ indicating the angle between the display and the direction of gravity is obtained from the gravity sensor Gs. The gravity sensor Gs is provided in the smartphone SP and detects the acceleration in the direction of gravity. The gravity sensor Gs also detects the azimuth angle of the direction of gravity. The display system 1 obtains information on the acceleration in the direction of gravity and its azimuth angle from the gravity sensor Gs, and calculates the vector ζ indicating the angle between the display and the direction of gravity based on the obtained information.

The following formula (1) may be used to calculate vector U, which indicates the viewpoint coordinates in the virtual space.

Here, f in equation (1) is a coordinate transformation function. The coordinate transformation function f is a coordinate transformation function that determines the position of vector U, which indicates the viewpoint coordinate in the virtual space, relative to vector S, which indicates the initial coordinate in the virtual space, from the relative positional relationship between the camera and the pupil.

The vector V indicating the coordinates of the perspective plane in the virtual space may be calculated using the following formula (2):

ここで、（２）式におけるｇは、座標変換関数である。座標変換関数ｇは、仮想空間内初期座標を示すベクトルＳに対して、仮想空間内透視面の座標を示すベクトルＶがどの位置にあるのかを、決める座標変換関数である。座標変換関数ｇは、仮想空間内初期座標を示すベクトルＳに対して、仮想空間内透視面の座標を示すベクトルＶがどの位置にあるのかを、加速度センサにより得られる情報に基づいて算出するものであってもよい。

Here, g in formula (2) is a coordinate transformation function that determines the position of vector V indicating the coordinates of the perspective plane in virtual space relative to vector S indicating the initial coordinates in virtual space. Coordinate transformation function g may calculate the position of vector V indicating the coordinates of the perspective plane in virtual space relative to vector S indicating the initial coordinates in virtual space based on information obtained by an acceleration sensor.

The vector W, which indicates the plane vector of the perspective surface in the virtual space, may be calculated using the following formula (3).

ここで、（３）式におけるｈは、座標変換関数である。座標変換関数ｈは、仮想空間内初期座標を示すベクトルＳに対して仮想空間内透視面の平面ベクトルを示すベクトルＷがどの方向にあるのかを重力センサーから決める座標変換関数である。座標変換関数ｈは、仮想空間内初期座標を示すベクトルＳに対して仮想空間内透視面の平面ベクトルを示すベクトルＷがどの方向にあるのかを、更にジャイロセンサにより得られる情報に基づいて算出するものであってもよい。

Here, h in formula (3) is a coordinate transformation function. The coordinate transformation function h is a coordinate transformation function that determines, from a gravity sensor, the direction of vector W, which indicates the plane vector of the perspective surface in virtual space, relative to vector S, which indicates the initial coordinate in virtual space. The coordinate transformation function h may also calculate, based on information obtained by a gyro sensor, the direction of vector W, which indicates the plane vector of the perspective surface in virtual space, relative to vector S, which indicates the initial coordinate in virtual space.

The display system 1 perspectively renders a three-dimensional object in a virtual space on a plane represented by vector V indicating the coordinates of the perspective plane in the virtual space and plane vector W of the perspective plane in the virtual space, with vector U indicating the viewpoint coordinates in the virtual space as the focus.

In addition, in order to calculate the vector U indicating the viewpoint coordinates in the virtual space, the vector V indicating the coordinates of the perspective plane in the virtual space, and the plane vector W of the perspective plane in the virtual space, a base line length _L0 as shown in the figure may be used instead of using the distance _Dbase from the right eye ER to the left eye EL of the user U. In order to calculate the base line length _L0 , a distance sensor such as a LiDAR (Light Detection and Ranging) or ToF (Time of Flight) sensor may be used.

When calculating a vector U indicating the viewpoint coordinates in the virtual space, a vector V indicating the coordinates of the perspective plane in the virtual space, and a plane vector W of the perspective plane in the virtual space using the base line length _L0 , the following equations (4), (5), and (6) are used.

FIG. 6 is a diagram for explaining the excess and deficiency of information analyzed by the display system according to the present embodiment. With reference to the figure, the excess and deficiency of the amount of information used by the display system 1 according to the present embodiment will be explained. The vectors shown in the figure are unknown information required by the display system 1. The known information includes a vector ζ indicating the angle between the display device and the direction of gravity, and a distance D _base from the right eye ER to the left eye EL of the user U, and the number of pieces of information is 4. The information to be acquired includes an angle η between a line connecting the centers of gravity of the pupils and a reference line of the image IMG, an interpupillary angle of view φ, an angle of view θ from the center O of the image IMG to the center of the pupil, and a vector ζ indicating the angle between the display device and the direction of gravity, and the number of pieces of information is 6. That is, the number of pieces of known information among the information used by the display system 1 according to the present embodiment is 10.

On the other hand, the information (unknown information) that the display system 1 must determine is a vector (three-dimensional) indicating the viewpoint coordinates in the virtual space, a vector (three-dimensional) indicating the perspective plane coordinates in the virtual space, and a plane vector (three-dimensional) of the perspective plane in the virtual space, and the amount of unknown information is 9.

Therefore, since there are 9 unknowns compared to 10 knowns, it can be said that there is an excess of information. However, since the baseline length L ₀ (one piece of information) is calculated using two pieces of information, the interpupillary distance and the interpupillary angle of view, the known number is 9, and therefore there is no excess or deficiency of information, which is the minimum configuration.

7 is a diagram for explaining the vectors calculated by the display system according to this embodiment. With reference to the same figure, the calculation method of the unknowns (three pieces of three-dimensional information) described above will be described. Note that the conditions for the method described below include, first, that the display surface (projection surface) of the display device, the selfie camera that captures the face of the user U, and the gravity sensor are rigidly coupled, second, that the display surface, the selfie camera focus, and the gravity sensor are arranged in approximately the same plane, and third, that the distance between the display surface, the selfie camera focus, and the gravity sensor is sufficiently smaller than the viewpoint relative position E _L.

As shown in the figure, the unknowns (three pieces of three-dimensional information) required for display by the display system 1 can be calculated by combining known vectors. For example, the absolute viewpoint position Ez can be calculated by the following equation (7).

Note that P and Q in equation (7) are correction values based on the coupling angle between the selfie camera axis and the gravity sensor axis.

In addition, regarding the gravity orientation sensor input G _θ , the initial or periodical measurement value may be set as G _θ0 , and G _θ = G _θ0 + Δ _θ may be set.

When calculating the unknowns (three pieces of three-dimensional information) by combining known vectors, they can be found using the following equations (8), (9), and (10).

The vector W shown in equation (10) can also be calculated using equation (11).

Next, with reference to Figures 8 to 12, we will explain an example of a specific configuration for realizing the display system 1 described above.

FIG. 8 is a block diagram showing an example of a functional block for realizing the display system according to this embodiment. An example of the functional configuration of the display system 1 will be described with reference to the diagram. The display system 1 includes an imaging device 2, a gravity sensor 3, an acceleration sensor 4, a storage device 5, a display device 6, and a display control device 10. Note that the display system 1 may include a gyro sensor and the like in addition to the configuration shown in the figure.

The imaging device 2 captures an image of the face of the user U. That is, the imaging device 2 may be an in-camera (selfie camera) attached to a smartphone SP or the like. The user U is a user of the display system 1. Specifically, the imaging device 2 may be a CCD camera using a CCD (Charge Coupled Devices) image sensor, or a CMOS camera using a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

The gravity sensor 3 detects acceleration in the direction of gravity. The gravity sensor 3 also detects the relative orientation of the device with respect to an absolute reference such as the earth's axis.

The acceleration sensor 4 detects the acceleration of the device itself. The acceleration sensor 4 may be capable of detecting gravitational acceleration, in which case the gravity sensor 3 and the acceleration sensor 4 may have the same configuration. The acceleration sensor 4 detects acceleration in three axial directions (X-axis, Y-axis, and Z-axis).

The storage device 5 stores parameter information used in the calculation. The parameter information includes information on the attributes of the user U and information on the initial position of the display system 1. The information stored in the storage device 5 may be received from the user U by an operation reception unit (not shown), or may be updated by the display control device 10.

The display device 6 displays various information according to the control of the display control device 10. The display device 6 acquires a control signal from the display control device 10 and displays information according to the acquired control signal. The display device 6 may be, for example, a liquid crystal display, an organic EL (Electroluminescence) display, or the like. It is preferable that the imaging device 2 and the display device 6 are rigidly connected to each other.

The display control device 10 acquires information from the imaging device 2, the gravity sensor 3, the acceleration sensor 4, the storage device 5, etc., generates information to be displayed on the display device 6 based on the acquired information, and controls the display on the display device 6. An example of the functional configuration of the display control device 10 will be described in detail below with reference to FIG. 9.

Figure 9 is a functional configuration diagram showing an example of the functional configuration of a display control device according to this embodiment. The display control device 10 includes an image acquisition unit 11, a viewpoint relative position calculation unit 12, a gravity information acquisition unit 13, a display device angle calculation unit 14, an acceleration information acquisition unit 15, an initial position information acquisition unit 16, a vector calculation unit 17, and a display control unit 18. Each of these functional units is realized, for example, using electronic circuits. Furthermore, each functional unit may include internal storage means such as a semiconductor memory or a magnetic hard disk device, as necessary. Furthermore, each function may be realized by a computer having a CPU (Central Processing Unit) and software. In addition, all or part of each functional unit may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field-Programmable Gate Array). In addition, all or part of each functional unit may be realized by a combination of software and hardware.

The image acquisition unit 11 acquires an image captured by the imaging device 2. The image acquired by the image acquisition unit 11 is an image of the face of the user U of the display system 1. The image acquisition unit 11 may acquire successive frame images included in the video data captured by the imaging device 2.

The viewpoint relative position calculation unit 12 calculates the relative positional relationship between the viewpoint of the user U and the display device 6 based on the image acquired by the image acquisition unit 11. The viewpoint relative position calculation unit 12 is equipped with, for example, a pupil position identification unit 121, an interpupillary angle of view identification unit 122, and a viewpoint relative position identification unit 123, thereby calculating the relative positional relationship between the viewpoint of the user U and the display device 6. Note that the specific calculation method of the viewpoint relative position calculation unit 12 is not limited to this example, and other methods may be used.

The pupil position identification unit 121 identifies the position coordinates of the left eye EL and the right eye ER of the user U based on the image acquired by the image acquisition unit 11. By identifying the position coordinates of the left eye EL and the right eye ER of the user U, the interpupillary distance can be identified. In addition, a known image processing technique such as an object detection technique may be used to identify the position coordinates. Next, the interpupillary angle of view identification unit 122 detects the interpupillary angle of view φ based on the identified position coordinates of the left eye EL and the right eye ER of the user U. Furthermore, the viewpoint relative position identification unit 123 identifies the relative position between the viewpoint of the user U and the display device 6 based on the identified interpupillary distance, the detected interpupillary angle of view φ, and the image acquired by the image acquisition unit 11. In addition, the method of identifying the relative position between the viewpoint of the user U and the display device 6 can be performed by the method described with reference to FIG. 3 to FIG. 7.

The relative position between the user U's viewpoint and the display device 6 may be specified based on the attributes of the user U. The attributes of the user U may be, for example, the actual value of the interpupillary distance, gender, age, race, etc. The attributes of the user U are input in advance based on the operation of the user U and stored in a predetermined storage unit. In this case, the viewpoint relative position calculation unit 12 calculates the relative positional relationship between the user U's viewpoint and the display device 6 based on the attributes of the user U acquired in advance. For example, the viewpoint relative position calculation unit 12 may calculate the relative positional relationship between the user U's viewpoint and the display device 6 based on the interpupillary distance of the user U stored in advance and the interpupillary distance of the user U measured from the image acquired by the image acquisition unit 11.

The gravity information acquisition unit 13 acquires information from the gravity sensor 3 regarding the acceleration in the direction of gravity and the relative orientation of the device with respect to an absolute reference such as the earth's axis.

The display device angle calculation unit 14 calculates the angle of the display device 6 based on the acceleration in the direction of gravity acquired by the gravity information acquisition unit 13 and information related to the azimuth angle. Here, since the display device 6 is rigidly coupled to the gravity sensor 3, the display device angle calculation unit 14 is able to calculate the angle of the display device 6 based on the acceleration in the direction of gravity acquired by the gravity information acquisition unit 13 and information related to the relative orientation of the device with respect to an absolute reference such as the earth's axis.

The acceleration information acquisition unit 15 acquires acceleration information of the device itself from the acceleration sensor 4. Specifically, the acceleration information acquisition unit 15 acquires acceleration information in three axial directions (X-axis, Y-axis, and Z-axis).

The initial position information acquisition unit 16 acquires the acceleration information of the own device acquired by the acceleration information acquisition unit 15, and acquires the origin coordinates OC, which are three-dimensional coordinate information of a specified point. The initial position information acquisition unit 16 acquires the position information of the own device when the origin coordinates OC are set as the origin. The initial position information is the initial position vector S (Ix, Iy, Iz) described in FIG. 7.

The vector calculation unit 17 calculates unknown information based on known information. The unknown information calculated by the vector calculation unit 17 is the three three-dimensional vectors described with reference to Figures 6 and 7. Specifically, the unknown information calculated by the vector calculation unit 17 is the absolute position of the projection plane onto which the display device 6 projects an image (perspective plane position vector), the absolute position coordinates of the user U (viewpoint position vector), and the vector of the direction in which the projection plane faces (perspective plane vector). A specific calculation method will be described below.

The absolute position of the projection plane onto which the display device 6 projects an image (perspective plane position vector) is the projection plane absolute position vector V (Px _{, Py, Pz} ) shown in Fig. 7. This vector is calculated based on the position information acquired by the initial position information acquisition unit 16 and the acceleration information acquired by the acceleration information acquisition unit 15.

The absolute position coordinates (viewpoint position vector) of the user U are the viewpoint absolute position vector U (E _x, E _y, E _z ) shown in Fig. 7. This vector is calculated based on the calculated projection plane absolute position vector V (P _x, P _y, P _z ) and the relative positional relationship between the viewpoint of the user U and the display device 6 calculated by the viewpoint relative position calculation unit 12.

The vector of the direction in which the projection plane faces (perspective plane vector) is the projection plane vector (Pθ _, PΦ _{, PΨ} ) shown in Fig. 7. This vector is calculated based on the projection plane absolute position vector V (Px _{, Py, Pz} ), the gravity information acquired by the gravity information acquisition unit 13, and the acceleration information acquired by the acceleration information acquisition unit 15.

The display control unit 18 transforms the three-dimensional information of the object to be displayed based on the information calculated by the vector calculation unit 17, and controls the display on the display device 6.

By adopting such a configuration, the display system 1 deforms and displays the display target, which is a virtual object, according to the position of the user U's face, the inclination of the smartphone SP, etc. Therefore, the user U can have the experience of looking into a virtual space through a window without using a wearable device. Furthermore, the sense of immersion is enhanced by the motion parallax created by the user's subtle movements.

Here, according to the embodiment described above, if the positional relationship between the user U's viewpoint and the smartphone SP does not change, the display of the virtual object remains fixed on the display device 6. If there is no movement on the screen, it may be difficult to grasp the three-dimensional positional relationship, and it may be difficult to achieve a sense of immersion in the virtual space. Therefore, below, as a modified example of the display control device 10 according to this embodiment, a display control device 10A that can achieve a sense of immersion in the virtual space even if the positional relationship between the user U's viewpoint and the smartphone SP does not change will be described.

FIG. 10 is a functional configuration diagram showing a modified example of the functional configuration of a display control device according to this embodiment. An example of the functional configuration of the display control device 10A will be described with reference to the same diagram. In the description of the display control device 10A, the same components as those in the display control device 10 are given the same reference numerals and may not be described. The display control device 10A differs from the display control device 10 in that it further includes an error assignment unit 19.

The error assigning unit 19 assigns a different error to each of the multiple display objects based on the virtual position of each of them. The error may be given as distance information, for example. The error assigning unit 19 may, for example, assign a large error to an object that exists near the calculated viewpoint position, and a small error to an object that exists far from the calculated viewpoint position. Conversely, the error assigning unit 19 may assign a small error to an object near the smartphone SP, and a large error to an object that exists far from the smartphone SP.

The display control unit 18 controls the display of multiple display objects on the display device 6 by varying the relative relationship between the display objects and the background within the error range assigned by the error assigning unit 19. That is, according to this embodiment, the relative relationship between the display objects and the background is varied depending on the distance from the smartphone SP to the position where the virtual object is located. By varying the relative relationship, the user U can get the sensation that the virtual object is trembling. Furthermore, because the amount of trembling varies depending on the distance, the user U can easily grasp the three-dimensional positional relationship and can be immersed in the three-dimensional virtual world.

The change in the relative relationship between the display objects and the background may be achieved by changing the position information of the virtual objects, or by changing the position information of the background. When the change is achieved by changing the position information of the virtual objects, the display control unit 18 controls the display of the multiple display objects on the display device 6 by changing the positions at which the display objects are displayed within the given error range.

The change in the relative relationship between the display object and the background is preferably performed in a specific direction. The specific direction is preferably either the horizontal direction or the vertical direction. For example, it is possible to perform the change in a mixture of the vertical direction and the horizontal direction, but excessive change may be a burden to the user U and may cause the user U to feel drunk.

In addition, when multiple display objects are displayed on the display device 6 and the display is deformed by movement of either the absolute viewpoint position, the display absolute position, or the display orientation vector, it is preferable to delay updating the display by a delay time of different lengths for each. By adopting such a configuration, it is possible to give the illusion of a larger displacement than in reality, making it easier to perceive depth. In this case, the error imparting unit 19 may impart a delay time of different lengths for each.

In addition, when the display positions of objects change due to a change in vector, a more realistic change may be expressed by displaying them with different delay times. For example, in FIG. 1C, when the user U moves leftward, the object OBJ2 in front of the user U and the object OBJ3 behind the user U appear to the user as if the object in front moves faster and the object in the back moves slower, as a physical law. Therefore, following such physical laws, the smartphone SP may also display the objects in front and behind with a delay in display timing. In this case, the display control unit 18 displays multiple display objects that are at different distances from the display device 6 on the display device 6, and the multiple display objects are displayed with different display timings depending on their distances from the display device 6.

11 is a diagram for explaining the concept of the case where the user's face is tilted in the display system according to this embodiment. With reference to the same figure, the processing of the change in apparent distance of the interpupillary angle of view due to the left-right tilt of the face will be explained. FIG. 11(A) shows the interpupillary angle of view φ when the user U looks at the display device 6 from the front. FIG. 11(B) shows the interpupillary angle of view φ when the user U looks at the display device 6 from an angle (sideways). In this way, the interpupillary angle of view φ changes depending on the tilt of the user U's face. In addition, the interpupillary distance also varies depending on the tilt of the user U's face. In order to solve such a problem, the relative position between the user U's viewpoint and the display device 6 may be specified based on the ratio of the size of the user U's left and right pupils or irises. In this case, the viewpoint relative position calculation unit 12 may update the relative positional relationship between the user U's viewpoint and the display device 6 in accordance with the change in the ratio of the size of the user U's left and right pupils or irises obtained based on the image acquired by the image acquisition unit 11.

The relative position between the viewpoint of the user U and the display device 6 may be determined using a three-dimensional facial model on which the positions of features such as the eyes, nose, and mouth are arranged. The simplest three-dimensional facial model can be represented by a spherical model, but any other suitable model may be used. In this case, the positions of features such as the eyes, nose, and mouth are arranged on the three-dimensional facial model, and the relative positional relationship can be updated by calculating the position and angle of the three-dimensional facial model that best matches the apparent positions.

In addition, in actual use cases, there may be cases where two people look into the display device 6 at the same time. In such cases, iris authentication technology may be used to identify one of the users U, and a display appropriate for the identified user U may be provided.

FIG. 12 is a block diagram showing an example of the internal configuration of the display control device 10 of this embodiment. At least some of the functions of the display control device 10 can be realized using a computer. As shown in the figure, the computer is configured to include a central processing unit 901, a RAM 902, an input/output port 903, input/output devices 904 and 905, etc., and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902, etc. According to each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic operations and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. Note that RAM is an abbreviation for "random access memory." The input/output port 903 is a port through which the central processing unit 901 exchanges data with external input/output devices, etc. The input/output devices 904 and 905 are input/output devices. The input/output devices 904 and 905 exchange data with the central processing unit 901 via the input/output port 903. The bus 906 is a common communication path used inside the computer. For example, the central processing unit 901 reads and writes data from the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input/output port via the bus 906. Also, all or part of each functional unit provided in the display control device 10 may be realized using hardware such as an ASIC, a PLD, or an FPGA. Also, all or part of each functional unit may be realized by a combination of software and hardware.

[Summary of the embodiment]
According to this embodiment, the display control device 10 is provided with a display control unit 18, which controls the display of a display object, which is a virtual object having three-dimensional information, on the display device 6; by providing an image acquisition unit 11, it acquires an image captured by an imaging device 2 rigidly connected to the display device 6, which image captures an image of the face of a user U looking at a display object displayed on the display device 6; by providing a viewpoint relative position calculation unit 12, it calculates the relative positional relationship between the viewpoint of the user U and the display device 6 based on the acquired image; by providing a gravity information acquisition unit 13, it acquires information regarding acceleration in the direction of gravity and its azimuth angle; by providing an initial position information acquisition unit 16, it acquires position information at which the device is located when a specified point is set as the origin; and by providing an acceleration information acquisition unit 15, it acquires acceleration information of the device.

The display control device 10 calculates the absolute position (perspective plane position vector) of the projection plane onto which the display device 6 projects the image based on the position information acquired by the initial position information acquisition unit 16 and the acceleration information acquired by the acceleration information acquisition unit 15, calculates the absolute position coordinates (viewpoint position vector) of the user U based on the calculated absolute position of the projection plane and the relative positional relationship between the user U's viewpoint and the display device 6 calculated by the viewpoint relative position calculation unit 12, and calculates a vector (perspective plane vector) in the direction in which the projection plane faces based on the calculated absolute position of the projection plane, the gravity information acquired by the gravity information acquisition unit 13, and the acceleration information acquired by the acceleration information acquisition unit 15. The display control unit 18 also transforms the three-dimensional information of the display object based on the information calculated by the vector calculation unit 17, and controls the display on the display device 6. By adopting such a configuration, even if the user U is not in front of the display device 6, the display control device 10 can match the viewpoint of the user U with the display image, allowing the user to feel immersed.

Furthermore, according to the above-described embodiment, the viewpoint relative position calculation unit 12 calculates the relative positional relationship between the viewpoint of the user U and the display device 6 based on the attributes of the user U acquired in advance. Therefore, according to the display control device 10, it is possible to accurately calculate the relative positional relationship between the viewpoint of the user U and the display device 6. Therefore, the display control device 10 can accurately match the viewpoint of the user U with the display image, thereby providing a more immersive feeling.

Furthermore, according to the above-described embodiment, the viewpoint relative position calculation unit 12 measures the distance between the pupils of the user U based on the image acquired by the image acquisition unit 11, and calculates the relative positional relationship between the viewpoint of the user U and the display device 6 based on the measured distance between the pupils of the user U. Thus, the display control device 10 can accurately match the viewpoint of the user U with the displayed image, providing a more immersive feeling.

Furthermore, according to the above-described embodiment, the viewpoint relative position calculation unit 12 measures the interpupillary distance and interpupillary angle of view of the user U based on the image acquired by the image acquisition unit 11, and calculates the relative positional relationship between the viewpoint of the user U and the display device 6 based on the measured interpupillary distance and interpupillary angle of view of the user U. Thus, the display control device 10 can accurately match the viewpoint of the user U with the displayed image, providing a more immersive feeling.

Furthermore, according to the above-described embodiment, the viewpoint relative position calculation unit 12 calculates the relative positional relationship between the viewpoint of the user U and the display device 6 based on the interpupillary distance of the user U stored in advance and the interpupillary distance of the user U measured from the image acquired by the image acquisition unit 11. Therefore, when the interpupillary distance of the user U is known, the display control device 10 can accurately match the viewpoint of the user U with the displayed image to provide a more immersive feeling.

Furthermore, according to the above-described embodiment, the viewpoint relative position calculation unit 12 updates the relative positional relationship between the viewpoint of the user U and the display device 6 in accordance with a change in the ratio of the sizes of the left and right pupils or irises of the user U obtained based on the image acquired by the image acquisition unit 11. Therefore, even if the face of the user U is tilted, the display control device 10 can accurately match the viewpoint of the user U with the displayed image, thereby providing a more immersive feeling.

Furthermore, according to the above-described embodiment, in the display control device 10A, the display control unit 18 displays a plurality of display objects on the display device 6. The display control device 10A further includes an error imparting unit 19 that imparts different errors to the plurality of display objects based on the virtual positions of the respective display objects, and the display control unit 18 controls the display of the plurality of display objects on the display device 6 by varying the relative relationship between the display objects and the background within the range of the imparted errors. In other words, the display control device 10A intentionally moves the display objects in a jiggling manner to provide a visual effect that makes them appear more three-dimensional, and can give the user U a sense of immersion in the virtual world.

Furthermore, according to the above-described embodiment, the display control unit 18 controls the display of multiple display objects on the display device 6 by varying the position at which the display objects are displayed within the range of error given by the error giving unit 19. That is, the display control unit 18 varies the relative relationship between the display objects and the background by varying the position of the display objects among the background and the display objects. Therefore, the display control device 10A intentionally moves the display objects against the background in a jiggling manner, thereby creating a visual effect that makes the objects appear more three-dimensional, and can give the user U a sense of immersion in the virtual world.

Furthermore, according to the above-described embodiment,
The display control unit 18 controls the display of a plurality of display objects on the display device 6 by varying the position at which the display object is displayed in a predetermined direction within the range of error given by the error giving unit 19. According to this embodiment, the display control device 10A does not cause shaking by mixing up, down, left, and right, so that the user U is unlikely to feel dizzy even when looking at the screen displayed by the display control device 10A.

As a result, it will be possible to improve the quality of services that present virtual three-dimensional information to users via smartphones, tablet devices, etc., thereby contributing to Goal 8 of the United Nations-led Sustainable Development Goals (SDGs) which is to "promote inclusive and sustainable economic growth, employment and decent work for all."

The above describes an embodiment of the present invention in detail with reference to the drawings, but the specific configuration is not limited to this embodiment, and includes design modifications and the like that do not deviate from the gist of the present invention.

In addition, a computer program for implementing the functions of each of the above-mentioned devices may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed. Note that the term "computer system" may include hardware such as an OS and peripheral devices.
In addition, the term "computer-readable recording medium" refers to a flexible disk, a magneto-optical disk, a ROM, a writable non-volatile memory such as a flash memory, a portable medium such as a DVD (Digital Versatile Disc), or a storage device such as a hard disk built into a computer system.

Furthermore, "computer-readable recording medium" also includes a medium that holds a program for a certain period of time, such as a volatile memory (e.g., DRAM (Dynamic Random Access Memory)) inside a computer system that becomes a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. The above program may also be transmitted from a computer system that stores the program in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
The program may be a program for implementing some of the above-mentioned functions, or may be a so-called differential file (differential program) that can implement the above-mentioned functions in combination with a program already recorded in the computer system.

1...display system, 2...imaging device, 3...gravity sensor, 4...acceleration sensor, 5...storage device, 6...display device, 10...display control device, 11...image acquisition unit, 12...viewpoint relative position calculation unit, 13...gravity information acquisition unit, 14...display device angle calculation unit, 15...acceleration information acquisition unit, 16...initial position information acquisition unit, 17...vector calculation unit, 18...display control unit, 19...error assignment unit, 121...pupil position determination unit, 122...interpupillary angle of view determination unit, 123...viewpoint relative position determination unit, OC...origin coordinates, DOI...display object information

Claims (12)

a display control unit that controls displaying a display object having three-dimensional information on a display device;
an image acquisition unit that acquires an image captured by an imaging device rigidly connected to the display device, the image being an image of a face of a user looking at the display object displayed on the display device;
a viewpoint relative position calculation unit that calculates a relative positional relationship between the viewpoint of the user and the display device based on the acquired image;
a gravity information acquisition unit that acquires information about the acceleration in the gravity direction and its azimuth angle;
an initial position information acquisition unit that acquires position information of the location of the own device when a predetermined point is set as an origin;
an acceleration information acquisition unit that acquires acceleration information of the own device;
a vector calculation unit that calculates an absolute position of a projection plane onto which the display device projects an image, based on the position information acquired by the initial position information acquisition unit and the acceleration information acquired by the acceleration information acquisition unit, calculates absolute position coordinates of the user's viewpoint, based on the calculated absolute position of the projection plane and the relative positional relationship between the user's viewpoint and the display device calculated by the viewpoint relative position calculation unit, and calculates a vector of the direction in which the projection plane faces, based on the calculated absolute position of the projection plane, the gravity information acquired by the gravity information acquisition unit, and the acceleration information acquired by the acceleration information acquisition unit;
Equipped with
The display control unit controls the display on the display device so that an image obtained by deforming three-dimensional information of the object to be displayed is perspectively projected onto a projection plane determined by the absolute position of the projection plane onto which the display device projects an image and a vector of the direction in which the projection plane faces, with the absolute position coordinates of the user's viewpoint as a focus. The display control device according to claim 1 , wherein the viewpoint relative position calculation unit calculates the relative positional relationship between the viewpoint of the user and the display device based on attributes of the user acquired in advance. The display control device according to claim 1 , wherein the viewpoint relative position calculation unit measures a distance between the user's pupils based on the image acquired by the image acquisition unit, and calculates a relative positional relationship between the user's viewpoint and the display device based on the measured distance between the user's pupils. 4. The display control device according to claim 3, wherein the viewpoint relative position calculation unit measures an interpupillary distance and an interpupillary angle of view of the user based on the image acquired by the image acquisition unit, and calculates a relative positional relationship between the viewpoint of the user and the display device based on the measured interpupillary distance and interpupillary angle of view of the user. The display control device according to claim 3 , wherein the viewpoint relative position calculation unit calculates a relative positional relationship between the viewpoint of the user and the display device based on a pre-stored interpupillary distance of the user and an interpupillary distance of the user measured from the image acquired by the image acquisition unit. The display control device according to claim 1 , wherein the viewpoint relative position calculation unit updates the relative positional relationship between the viewpoint of the user and the display device in accordance with a change in the ratio of the sizes of the left and right pupils or irises of the user obtained based on the image acquired by the image acquisition unit. The display control unit displays a plurality of the display objects on the display device,
an error providing unit that provides different errors based on the virtual positions of the plurality of display objects,
The display control device according to claim 1 , wherein the display control unit controls the display of the plurality of display objects on the display device by varying a relative relationship between the display objects and a background within a given error range. The display control device according to claim 7 , wherein the display control unit controls the display of the plurality of display objects on the display device by varying positions at which the display objects are displayed within a given error range. The display control device according to claim 7 , wherein the display control unit controls the display of the plurality of display objects on the display device by varying the positions at which the display objects are displayed in a predetermined direction within a given error range. the display control unit displays, on the display device, a plurality of the display objects that are present at different distances from the display device;
The display control device according to claim 1 , wherein the plurality of display objects are displayed at different display timings depending on a distance from the display device. A display control device according to any one of claims 1 to 10,
the imaging device that captures an image of a user's face and provides the captured image to the image acquisition unit;
and the display device controlled by the display control unit. On the computer,
a display control step of controlling the display of a display object having three-dimensional information on a display device;
an image capturing step of capturing an image captured by an imaging device rigidly connected to the display device, the image being an image of a face of a user looking at the display object displayed on the display device;
a viewpoint relative position calculation step of calculating a relative positional relationship between the viewpoint of the user and the display device based on the acquired image;
a gravity information acquiring step of acquiring information about an acceleration in a gravity direction and an azimuth angle thereof;
an initial position information acquisition step of acquiring position information of a location where the own device is located when a predetermined point is set as an origin;
an acceleration information acquisition step of acquiring acceleration information of the own device;
a vector calculation step of calculating an absolute position of a projection plane onto which the display device projects an image, based on the position information acquired in the initial position information acquisition step and the acceleration information acquired in the acceleration information acquisition step, calculating absolute position coordinates of the user's viewpoint, based on the calculated absolute position of the projection plane and the relative positional relationship between the user's viewpoint and the display device calculated in the viewpoint relative position calculation step, and calculating a vector of a direction in which the projection plane faces, based on the calculated absolute position of the projection plane, the gravity information acquired in the gravity information acquisition step, and the acceleration information acquired in the acceleration information acquisition step;
A program for executing
The display control step is a program for controlling the display on the display device so that an image obtained by deforming the three-dimensional information of the object to be displayed is perspectively projected onto a projection plane determined by the absolute position of the projection plane onto which the display device projects the image and a vector of the direction in which the projection plane faces, with the absolute position coordinates of the user's viewpoint as a focus.

Images