JP2021131741A - Ar display control device, program therefor, and ar display system - Google Patents

Abstract

To provide an AR display control device in which an uneasy feeling in space sharing can be reduced.SOLUTION: An AR display control device 5 includes a visual-line person information receiving unit 54 that receives partner's visual-line person information, a mode determination unit 55 that determines a redirection mode, a real mode, or a recovery motion mode on the basis of the visual-line person information, a redirection mode display control unit 57A that displays the partner on an AR glass 4 in the redirection mode, a real mode display control unit 57B that displays the partner on the AR glass 4 in the real mode, and a recovery motion mode display control unit 57c that displays the partner on the AR glass 4 in the recovery motion mode.SELECTED DRAWING: Figure 8

Description

The present invention relates to an AR display control device for displaying a partner on an AR display device, a program thereof, and an AR display system.

In recent years, research on virtual reality (VR) and augmented reality (AR) has been conducted. For example, a technique has been proposed in which a user's alter ego (avatar) is displayed in a three-dimensional virtual space and the users share the three-dimensional virtual space (Patent Documents 1 and 2). The methods described in Patent Documents 1 and 2 align the lines of sight of each other's alter ego in order to promote dialogue in a three-dimensional virtual space.

Japanese Unexamined Patent Publication No. 9-6985 Japanese Unexamined Patent Publication No. 2003-150978

The techniques described in Patent Documents 1 and 2 described above cannot handle a live-action model of a user, and it is also difficult for users to watch a television program while sharing a three-dimensional virtual space. For this reason, it is realistic to use an AR display device such as AR glasses for space sharing in viewing TV programs. That is, if the other party is displayed on the user's AR display device when watching the TV program, space sharing in watching the TV program can be realized.

In addition, the inventors of the present application are also conducting research on space sharing. For example, when two people are watching a TV program and the other person is in front of the user (for example, when they are in sight), the impression that "two people are watching the TV program" is strong. It is considered to be.

Here, consider a case where the AR display device displays the other party forward so as to be in the user's field of view. In this case, even if the other party turns around in the real world, the other party is not completely turned around because the other party is located in front of the user on the user's AR display device. In this way, in the AR display device, the display is displayed in a state where the other party turns around only halfway, which gives the user a sense of discomfort.

An object of the present invention is to provide an AR display control device and a program thereof, which can reduce a sense of discomfort in space sharing, and an AR display system.

In order to solve the above problems, in the AR display control device according to the present invention, the AR display device worn by the first person who watches the video display device has a second person who watches another video display device. It is an AR display control device that displays so as to be on the side of the video display device to be viewed by the person in question. It is configured to include a motion mode display control unit.

According to such a configuration, the line-of-sight information receiving unit receives line-of-sight information indicating whether the second person is looking at the first person or the second person looks away from the first person. ..
Based on the line-of-sight information, the mode determination unit determines that if the second person continues to look at the first person for a predetermined time or longer, the mode determination unit determines that the direction mode is determined, and the second person looks away from the first person. If this is the case, it is determined to be the real mode, and if the second person continues to look away from the first person for a predetermined time or longer after being determined to be the redirection mode, it is determined to be the return motion mode.

When the redirection mode display control unit determines that the redirection mode is determined, the AR display device displays the second person so as to turn around the first person.
When the real mode display control unit determines that the real mode is set, the real mode display control unit displays the second person in the actual posture on the AR display device.
When the return motion mode display control unit determines that the return motion mode is determined, the AR display device displays the second person who is turning to the first person so that the second person returns to the actual posture.
As described above, in the AR display device, even when the second person is displayed in front of the first person, the second person turns around the first person.

Further, in order to solve the above-mentioned problems, the AR display system according to the present invention uses a photographing camera for photographing a second person viewing another image display device and a photographing image captured by the photographing camera to obtain the second person. The configuration includes a three-dimensional model / attitude information generation device for generating the three-dimensional model and attitude information, and the AR display control device described above.

According to such a configuration, the AR display control device is such that the AR display device worn by the first person who watches the video display device is on the side of the video display device where the first person watches the second person. Display on.
As described above, in the AR display system, even when the second person is displayed in front of the first person, the second person turns around the first person.

The present invention can also be realized by a program for causing the computer to function as the AR display control device described above.

According to the present invention, even when the second person is displayed in front of the first person, the second person turns around the first person, so that a sense of discomfort in space sharing is reduced. can do.

It is the schematic of the AR display system which concerns on 1st Embodiment. It is explanatory drawing explaining the space sharing in the viewing of a television program, (a) shows the display method of own AR glass, and (b) shows the display method of the AR glass of the other party. It is explanatory drawing explaining the state which the other party turned around, (a) shows the display method of own AR glass, (b) shows the display method of the AR glass of the other party. It is explanatory drawing explaining redirection, (a) shows the display method of own AR glass, and (b) shows the display method of the AR glass of the other party. (A) is an explanatory diagram for explaining space sharing in watching a television program, (a) is an explanatory diagram for explaining a state in which the other party turns around, and (c) is an explanatory diagram for explaining redirection. .. It is an overall block diagram of the AR display system which concerns on 1st Embodiment. It is a block diagram which shows the structure of the capture apparatus which concerns on 1st Embodiment. It is a block diagram which shows the structure of the AR display control apparatus which concerns on 1st Embodiment. (A) to (c) are explanatory views explaining changes in the position of the head and the posture of the body when the person turns around. It is a state transition diagram which shows the operation of the AR display control device of FIG. It is explanatory drawing explaining an example of motion path information. (A) and (b) are explanatory views explaining the motion path in the direction mode display control unit and the return motion mode display control unit. It is an overall block diagram of the AR display system which concerns on 2nd Embodiment. It is an overall block diagram of the AR display system which concerns on 3rd Embodiment.

Hereinafter, each embodiment of the present invention will be described with reference to the drawings. However, the embodiments described below are for embodying the technical idea of the present invention, and the present invention is not limited to the following unless otherwise specified. Further, in each embodiment, the same means may be designated by the same reference numerals and description thereof may be omitted.

(First Embodiment)
[Overview of AR display system]
The outline of the AR display system 1 will be described with reference to FIGS. 1 to 5.
As shown in FIG. 1, the AR display system 1 realizes space sharing in viewing a television program by using an AR glass (AR display device) 4. Here, TV shows on AR-glass _{4 (4} A, _{4 B)} equipped with two persons _{H (H} A, _{H B)} are arranged in different locations video display device _{6 (6} A, _{6 B)} I'm watching.

In this case, AR display system 1, as shown in FIG. 2 (a), the person to person AR glass _{4 A} (the first person) _{H A} is mounted (second person) _{H B,} the person _{H A} viewed from and displays the CG synthesizing the side of the image display device 6 _a. Therefore, as shown in FIG. 5A, the person H _B is displayed in front of the AR glass 4 _{A worn by the person H A.} Similarly, AR display system 1, as shown in FIG. 2 (b), a person _{H A} in AR glass _{4 B} the person _{H B} is mounted, the image display device as viewed from the person _{H B} (other video display device) ₆ and CG synthesis on the side of the _B display.
In FIGS. 2 to 4, the person H wearing the AR glass 4 is shown by a solid line, and the person H displayed by CG synthesis is shown by a broken line.

Here, consider the case where the person H _B turns to the person H _A. As shown in FIG. 3 (b), AR glass _{4 B} in the person _{H A} person _{H B} are displayed in front. Therefore, the movement (head rotation amount) when the _{person H B} turns to the person H _A becomes small, but the head of the person H _B faces the direction of the _{person H A.} Meanwhile, as shown in FIG. 3 (a), since the person _{H B} even AR glass _{4 A} person _{H A} is displayed on the front, as shown in FIG. 5 (b), the person _{H B} is the person H _A is not completely turned around. That is, the amount of head rotation when the _{person H B} turns around the person H _A is insufficient, and the head of _{the person H B} does not face the direction of the _{person H A.} Therefore, in the AR display system 1, as shown in FIGS. 4A and 5C, the redirection process is performed _{so that the person H B} turns around the person H _{A completely.}

In the shared space in view of the television program, the display position of the person H _B (also similar display position of the person H _A) which can be arbitrarily set as long as the front of the person H _A. For example, the person H _B is displayed 1 meter forward when _{viewed from the person H A. At this time, the person H B} may be displayed diagonally in front of the person H _A so as not to interfere with the viewing of the television program.
Thereafter, the person _H A, "me" one of _{H B,} the person _H A, the other _{H B} may be referred to as "other party".

[AR display system configuration]
The configuration of the AR display system 1 will be described with reference to FIGS. 1 and 6.
As shown in FIGS. 1 and 6, the AR display system 1 includes a camera 2, a capture device 3, an AR glass 4, and an AR display control device 5. In this AR display system 1, the camera 2 and the capture device 3 are connected via a cable. Further, in the AR display system 1, the AR display control device 5 is built in the AR glass 4, and the capture device 3 and the AR display control device 5 are connected to each other via a network NW such as the Internet.

Here, AR display system 1, two persons _H A, so as to correspond to _{H B,} and two cameras _{2 (2} A, _{2 B),} the capture of the two devices _{3 (3} A, _{3 B} ), Two AR glasses 4 (4 _A , 4 _B ), and two AR display control devices 5 (5 _A , 5 _B ). The camera 2, the capture device 3, the AR glass 4, and the AR display control device 5 all have the same configuration. Further, _{the video display device 6 viewed by the person HA} is referred to as the video display device 6 _A, and the video display device 6 viewed by the person H _B is referred to as the video display device 6 _B.

The camera 2 is a general camera that captures the person H, and outputs the captured image of the person H to the capture device 3. This captured image is necessary for the capture device 3 to generate the 3DCG model and bone information of the person H. Here, the camera 2 _A outputs the captured image of the person _HA to the capture device 3 _A , and the camera 2 _B outputs the captured image of the person H _{B to the capture device 3 B.} For example, the camera 2 is arranged in the upper center of the image display device 6 so as to face the person H. Further, although the camera 2 is composed of one general RGB camera, it may be a stereo camera or an RGB camera with a depth camera.

The capture device 3 uses the captured image of the person H input from the camera 2 to generate the 3DCG model of the person H and the bone information of the person H. This 3DCG model is a three-dimensional model of the person H. Further, the bone information is the position of each joint of the person H, that is, the posture information representing the posture of the person H. Then, the capture device 3 transmits the generated 3DCG model of the person H and the bone information to the AR display control device 5 via the network NW.

Here, your 3DCG model and bone information are required to display yourself on the AR glass 4 of the other party. Therefore, the capture device 3 _A continues to transmit the 3DCG model and bone information of the person _HA to the AR display control device 5 _{B of the person H B.} Similarly, capture device _{3 B} is a 3DCG model and bone information of the person _{H B,} continues to transmit to the AR display control unit _{5 A} person _{H A.}
The configuration of the capture device 3 will be described later.

The AR glass 4 includes an AR display 41 like a general AR glass, and also has an AR display control device 5 (FIG. 8). Further, the AR glass 4 includes a line-of-sight sensor 40 that detects the line of sight of the person H, and outputs the line-of-sight detection result of the line-of-sight sensor 40 to the AR display control device 5.

The AR display control device 5 displays the other party to be viewed on the other image display device 6 on the side of the image display device 6 on the AR glass 4 worn by the user who views the image on the image display device 6. Further, the AR display control device 5 uses the 3DCG model and bone information of the other party received from the capture device 3 to perform a redirection process so that the other party turns around completely.

Here, in the AR display control device 5, the line-of-sight person information (line-of-sight information) is required to determine the necessity of the redirection process. This line-of-sight person information is information indicating whether the other party is looking at himself / herself or whether the other party has turned his / her eyes away from himself / herself. Therefore, the AR display control device 5 continues to receive the other party's line-of-sight person information from the other party's AR display control device 5 via the network NW.

To summarize the above, AR display control unit _{5 A} person _{H A,} using 3DCG model and bone information of the person _{H B} received from the capture device _{3 B,} a person _{H B} to AR glass _{4 A} person _{H A} Display it. At this time, the AR display control device 5 _A determines the necessity of the redirection process based on the line-of-sight person information of the person H _B received from the AR display control device 5 _{B , and the person H B} completely turns around the person H _A. Redirection processing is performed as described above. Further, the AR display control device 5 _A generates the line-of-sight person information of the person _HA and transmits it _{to the AR display control device 5 B.}

Similarly, AR display control unit _{5 B} of the person _{H B,} using 3DCG model and bone information of the person _{H A} received from the capture device _{3 A,} displays the person _{H A} in AR glass _{4 B} of the person _{H B} Let me. At this time, the AR display control device 5 _B determines the necessity of the redirection process based on the line-of-sight person information of the person _HA received from the AR display control device 5 _{A , and the person HA} completely turns around the person H _B. Redirection processing is performed as described above. Further, the AR display control device 5 _B generates the line-of-sight person information of the person H _B and transmits it _{to the AR display control device 5 A.}
The configuration of the AR display control device 5 will be described later.

The video display device 6 is, for example, a general broadcast receiver for displaying a television program. Further, the video display device 6 may display video content other than the TV program (for example, video content distributed on the network NW). Although the image display device 6 is not included in the AR display system 1, it is shown because it is closely related to the AR display system 1.

[Capture device configuration]
The configuration of the capture device 3 will be described with reference to FIG. 7. In FIG. 7, the network NW is not shown.
The capture device 3 includes a captured video input unit 30, a 3DCG model generation unit 31, a bone information generation unit 32, and a 3DCG / bone information transmission unit 33.

The captured image input unit 30 inputs its own photographed image from the camera 2. Then, the captured image input unit 30 outputs the input own photographed image to the 3DCG model generation unit 31.

The 3DCG model generation unit 31 generates its own 3DCG model by using its own photographed image input from the photographed image input unit 30.
Here, the 3DCG model generation unit 31 can generate a 3DCG model by any method. When the camera 2 is composed of one RGB camera, the 3DCG model generation unit 31 can generate a 3DCG model by deep learning. For example, the 3DCG model generation unit 31 may generate a 3DCG model from each frame included in its own captured image by a neural network trained with a photograph of a person. When the camera 2 is a stereo camera, the 3DCG model generation unit 31 can obtain depth information from the stereo image and generate a 3DCG model from the depth information. Further, when the camera 2 is an RGB camera with a depth camera, the 3DCG model generation unit 31 can generate a 3DCG model in the same manner as the stereo camera because the image of the depth camera represents the depth information.
Then, the 3DCG model generation unit 31 outputs the generated own 3DCG model to the bone information generation unit 32 and the 3DCG / bone information transmission unit 33.

The bone information generation unit 32 generates its own bone information using its own 3DCG model input from the 3DCG model generation unit 31. That is, the bone information generation unit 32 generates bone information corresponding to the 3DCG model in order to operate the 3DCG model. Here, the bone information generation unit 32 can generate bone information by an arbitrary method (for example, the quick rig described in Reference 1). Then, the bone information generation unit 32 outputs the generated own bone information to the 3DCG / bone information transmission unit 33.

Reference 1: Maya (registered trademark), [online], Autodesk Co., Ltd., [Searched on February 13, 2nd year of Reiwa], Internet <URL: https://www.autodesk.co.jp/products/maya/ overview>

The 3DCG / bone information transmission unit 33 transmits its own 3DCG model input from the 3DCG model generation unit 31 and its own bone information input from the bone information generation unit 32 to the AR display control device 5 on the other side. It is a thing.
Here, the 3DCG / bone information transmission unit 33 can transmit the 3DCG model and bone information at an arbitrary frequency. For example, if the 3DCG / bone information transmission unit 33 reduces the transmission frequency of the 3DCG model, the bandwidth of the network NW can be saved. On the other hand, when the 3DCG / bone information transmission unit 33 increases the frequency of transmitting bone information, the posture change of the 3DCG model becomes smooth.

[Configuration of AR display control device]
The configuration of the AR display control device 5 will be described with reference to FIG. In FIG. 8, the network NW is not shown.

As shown in FIG. 8, the AR display control device 5 includes a line-of-sight input unit 50, a line-of-sight determination processing unit 51, a line-of-sight person information transmission unit (line-of-sight information transmission unit) 52, and a 3DCG / bone information reception unit (three-dimensional). Model posture information receiving unit) 53, line-of-sight person information receiving unit (line-of-sight information receiving unit) 54, mode determination unit 55, motion path information recording unit 56, redirection mode display control unit 57A, and real mode display control unit. It includes a 57B, a return motion mode display control unit 57C, and a virtual display unit 58.

The line-of-sight input unit 50 inputs its own line-of-sight detection result from the line-of-sight sensor 40. Then, the line-of-sight input unit 50 outputs the input own line-of-sight detection result to the line-of-sight determination processing unit 51.

The line-of-sight determination processing unit 51 determines whether or not oneself is looking at the other party based on the line-of-sight detection result of oneself input from the line-of-sight input unit 50.
Here, the line-of-sight determination processing unit 51 inputs information on the partner area displayed on the AR glass 4 from the virtual display unit 58 described later, and inputs its own line-of-sight detection result from the line-of-sight input unit 50. NS. In this case, the line-of-sight determination processing unit 51 determines whether or not he / she is looking at the other party depending on whether or not his / her line of sight wearing the AR glass 4 overlaps the other party area displayed on the AR glass 4. .. That is, the line-of-sight determination processing unit 51 determines that he / she is looking at the other party when his / her line of sight overlaps the other party's area. On the other hand, when the line-of-sight determination processing unit 51 does not overlap the other party's area, the line-of-sight determination processing unit 51 determines that he / she has turned his / her eyes away from the other party.
After that, the line-of-sight determination processing unit 51 outputs the determination result of whether or not he / she is looking at the other party to the line-of-sight person information transmission unit 52 as the line-of-sight person information.

The line-of-sight person information transmission unit 52 transmits its own line-of-sight person information input from the line-of-sight determination processing unit 51 to the AR display control device 5 on the other side via the network NW.

The 3DCG / bone information receiving unit 53 receives the other party's 3DCG model and bone information from the other party's capture device 3 via the network NW. Then, the 3DCG / bone information receiving unit 53 outputs the received 3DCG / bone information of the other party to the mode determination unit 55.

The line-of-sight person information receiving unit 54 receives the line-of-sight person information of the other party from the AR display control device 5 of the other party via the network NW. Then, the line-of-sight person information receiving unit 54 outputs the received line-of-sight person information of the other party to the mode determination unit 55.

The mode determination unit 55 determines the reduction mode, the real mode, and the return motion mode based on the line-of-sight person information of the other party input from the line-of-sight person information receiving unit 54.
Specifically, the mode determination unit 55 determines that the direction is a redirection mode when the other party keeps looking at himself / herself for a predetermined time or longer. In this case, the mode determination unit 55 commands the redirection mode display control unit 57A, which will be described later, to redirection mode, and outputs 3DCG / bone information.
In addition, the mode determination unit 55 determines that the mode is real mode when the other party looks away from himself / herself. In this case, the mode determination unit 55 commands the real mode display control unit 57B, which will be described later, to perform the real mode, and outputs 3DCG / bone information.
Further, the mode determination unit 55 determines the return motion mode when the other party keeps looking away from himself / herself for a predetermined time or more after the determination of the redirection mode. In this case, the mode determination unit 55 commands the return motion mode display control unit 57C, which will be described later, to return motion mode, and outputs 3DCG / bone information.

The motion path information recording unit 56 is a storage device such as a memory, an HDD (Hard Disk Drive), or an SSD (Solid State Drive) that records motion path information in advance. This motion path information is referred to by the direction mode display control unit 57A and the return motion mode display control unit 57C. The details of the motion path will be described later.

The redirection mode display control unit 57A controls the virtual display unit 58 so that when the mode determination unit 55 determines the redirection mode, the other party is displayed in the redirection mode. This redirection mode is a mode in which the AR glass 4 is displayed so that the other party turns around completely. At this time, the redirection mode display control unit 57A displays the 3DCG model of the other party in a predetermined motion path so as to turn around naturally.

Here, as shown in FIG. 9 (a) ~ (c) , if the other turn around, rather than simply the angle of the head changes, the center position of the head (gravity center position) P _C and body posture Also changes. In FIG. 9, the direction of the head is shown by a chain line. Therefore, it is preferable that the redirection mode display control unit 57A follows the body (upper body of the shoulder, arm, etc.) of the 3DCG model to the head so that the opponent turns around naturally. For example, in the redirection mode display control unit 57A, the body of the 3DCG model can be made to follow the head by using IK (Inverse Kinematics).

The real mode display control unit 57B controls the virtual display unit 58 so that the other party is displayed in the real mode when the mode determination unit 55 determines the real mode. This real mode is a mode in which the opponent is displayed on the AR glass 4 in the actual posture. That is, the real mode display control unit 57B displays the 3DCG model of the other party in the posture of the bone information.

The return motion mode display control unit 57C controls the virtual display unit 58 so that the other party is displayed in the return motion mode when the mode determination unit 55 determines that the return motion mode is set. This return motion mode is a mode in which the AR glass 4 is displayed so that the opponent who is turning to himself / herself returns to the actual posture. At this time, the return motion mode display control unit 57C displays the 3DCG model of the other party so as to return to the actual posture with a motion path opposite to the motion path of the direction mode display control unit 57A.

The virtual display unit 58 CG-synthesizes and displays the other party on the AR glass 4 (AR display 41) according to the control from the direction mode display control unit 57A, the real mode display control unit 57B, and the return motion mode display control unit 57C. It is something to do. That is, the virtual display unit 58 virtually displays the opponent on the AR glass 4 in any of the redirection mode, the real mode, and the return motion mode. Further, the virtual display unit 58 outputs the information of the partner area displayed on the AR glass 4 to the line-of-sight determination processing unit 51.

[Operation of AR display control device]
The operation of the AR display control device 5 will be described with reference to FIG.
Hereinafter, will be explained as an example the operation of AR display control unit _{5 A} when displaying a person _{H B} to AR glass _{4 A} person _{H A.} In addition, person H _A may be described as "self" and person H _B may be described as "other party".

First, the overall operation of the AR display system 1 is organized.
_{The camera 2 B} on the other side shoots the person H _{B, and the capture device 3 B} on the other side generates the 3DCG model and bone information of the person H _B using the captured image of the _{camera 2 B.} Then, the capture device 3 _B on the other side transmits the 3DCG model and bone information of the person H _B to its own AR display control device 5 _A. Further, the AR display control device 5 _B on the other side transmits the line-of-sight person information of the person H _{B to its own AR display control device 5 A.}

Then, his AR display control unit _{5 A,} using the 3DCG model and bones information gaze person information of the person _{H B,} displays the person _{H B} to AR glass _{4 A} person _{H A.} In this case, AR display control unit 5 _A performs the operations discussed below.

As shown in FIG. 10, the mode determination unit 55 of the AR display control unit _{5 A} transitions from the initial state S0 to the real mode state S1. For example, or reset at power-on of the AR glasses _{4 A,} AR display control unit _{5 A} is the initial state.

In the real mode state S1, the mode determination unit 55 commands the real mode display control unit 57B to perform the real mode. Then, real-mode display control unit 57B displays the person _{H B} in real mode AR glass _{4 A} person _{H A.}

Further, in the real mode state S1, the mode determination unit 55, based on the line-of-sight personal information of the person H _B, determines the person H _B is whether looked at herself.
When the person H _B looks at himself / herself, the mode determination unit 55 transitions to the real mode (determination process) state S2.

In real mode (determination processing) state S2, the mode determining unit 55, a person on the basis of the visual axis personal information of H _B, the time t ₁ the person H _B continues watching yourself, thresholds T ₁ or a preset It is determined whether or not it is. The threshold value T ₁ can be set by any value (for example, the threshold value T ₁ = 1 second).
When the time t ₁ is equal to or greater than the threshold value T ₁ , the mode determination unit 55 transitions to the redirection mode state S3.

Further, in the real mode (determination processing) state S2, the mode determining unit 55, based on the line-of-sight personal information of the person H _B, it determines the person H _B is whether looked away from him.
When the person H _B looks away from himself / herself, the mode determination unit 55 transitions to the real mode state S1.
When the person H _B is not looking away from himself and the time t ₁ is less than the threshold value T ₁ , the mode determination unit 55 maintains the state of the real mode (determination process) state S2.

In the redirection mode state S3, the mode determination unit 55 commands the redirection mode display control unit 57A to perform the redirection mode. Then, the redirection mode display control unit 57A displays a person _{H B} redirection mode AR glass _{4 A} person _{H A.}

Further, in the redirection mode state S3, the mode determination unit 55, based on the line-of-sight personal information of the person H _B, it determines the person H _B is whether looked away from him.
When the person H _B looks away from himself / herself, the mode determination unit 55 transitions to the redirection mode (determination process) state S4.
When the person H _B is looking at himself / herself, the mode determination unit 55 maintains the state of the redirection mode state S3.

In Redirection mode (determination processing) state S4, the mode determination unit 55, based on the line-of-sight personal information of the person H _B, the person H _B is the time t ₂ which continues to look away from his, preset threshold T Determine if it is _{2 or more.} The threshold value T ₂ can be set by any value (for example, the threshold value T ₂ = 1 second).
When the time t ₂ is equal to or greater than the threshold value T ₂ , the mode determination unit 55 transitions to the return motion mode state S5.

Further, in the redirection mode (determination processing) state S4, the mode determination unit 55, based on the line-of-sight personal information of the person H _B, it determines the person H _B is whether looked at herself.
When the person H _B looks at himself / herself, the mode determination unit 55 transitions to the redirection mode state S3.
When the person H _B is not looking at himself and the time t ₂ is less than the threshold value T ₂ , the mode determination unit 55 maintains the state of the redirection mode (determination process) state S4.

In the return motion mode state S5, the mode determination unit 55 commands the return motion mode display control unit 57C to perform the return motion mode. Then, the return motion mode display control unit 57C displays the person _{H B} in return motion mode to the AR glasses _{4 A} person _{H A.}
When the return motion is completed, the mode determination unit 55 transitions to the real mode state S1.

Since work with similar AR display control unit _{5 B} when displaying a person _{H A} in AR glass _{4 B} of the person _{H B,} the description thereof is omitted.

[Motion path]
The motion paths in the direction mode display control unit 57A and the return motion mode display control unit 57C will be described with reference to FIGS. 11 and 12.

Redirection mode display control unit 57A is, from the actual position the head of the person _{H B} (head angle theta ₁₎ to an angle (head angle theta ₂₎ that turn around a person _{H A,} the person _{H B} of 3DCG model person H _{Display A} continuously so as to turn around. At this time, the redirection mode display control unit 57A determines the motion path _{from the head angle θ 1} to the head angle θ ₂ by using the motion path information of the motion path information recording unit 56.

As shown in FIG. 11, the motion path information MP includes the head position (x _θ , y _θ , z _θ ) and the head rotation amount (rx _θ , ry _θ , rs _{) for each head angle θ of the person H B. This} is information indicating θ). _{That is, the motion path information MP includes the head position (x θ} , y _θ , z _θ ) and the head rotation amount (rx _θ , ry _θ , rz θ) from the front (angle θ = 0 °) to the head angle _θ. ) Indicates the amount of change. In the motion path information MP of FIG. 11, the head angle θ is at intervals of 30 °, but the interval of the head angles θ is not particularly limited.

As shown in FIG. 12 (a), the AR display control unit _{5 A,} the actual posture head position of the person _{H B} in (head angle _{θ 1) (x, y,} z) and head rotation amount (rx , Ry, rz) are always known. Therefore, the redirection mode display control unit 57A sets the head position (x, y, z) and the head rotation amount (rx, ry, raz) of the person H _{B in the actual posture (head angle θ 1) to the person.} By converting the coordinates to the coordinate system based on _{HA , as shown in FIG. 12B, the angle at which the person HA} is turned (head angle θ ₂ ) can be calculated. This coordinate transformation is the same as when displaying _{the person H B} in the AR glass 4A of the person _HA. The direction of the head of the person H _B is represented by the direction ry with the Y axis as the vertical (the direction of the image display device 6 _{B as the front of the person H B} (upward in FIG. 12B) is 0. °).

Then, the redirection mode display control unit 57A displays on the AR glass 4 _A so that the person H _B turns around the person H _{A with the motion path corresponding to the calculated head angle θ 2} . Hereinafter, the determination of the motion path will be described with two specific examples.

<Determination of motion path: 1st example>
In the method of this first example, the time t = 0 determined to be the redirection mode is set, and the motion path at each time t is determined so that the person H _{B at the time t = T 3} completely turns around the person H _A. (However, 0 ≦ t ≦ T ₃ ).

Here, the function for calculating the head angle θ _t at each time t is expressed by the following equation (1). Further, _{in order to redirect the head of the person H B} more naturally, the quadratic interpolation function shown in the following equation (2) may be used.

First, the redirection mode display control unit 57A calculates _{the head angle θ t} at each time t from the _{head angles θ 1} and θ ₂ by using the function of the formula (1) or the formula (2). The head angle θ _t is calculated in a coordinate system based on _{the person H B} before the coordinate conversion. Then, the redirection mode display control unit 57A, from the motion path information MP of FIG. 11, head position (x _t , y _t , z _t ) and head of the person H _{B corresponding to the head angle θ t at each time t.} The amount of partial rotation (rx _t , ry _t , rz _t ) is obtained. After that, the redirection mode display control unit 57A sets the head position (x _t , y _t , z _t ) and head rotation amount (rx _t , ry _t , rz _{t ) of the person H B} at each time t to the person H. Coordinates are converted to a coordinate system based on _A.

Furthermore, in some cases will move a person H _A in the redirection mode, the angle (head angle theta ₂₎ that turn around a person H _A changes. In this case, the redirection mode display control unit 57A periodically _{recalculates the head angles θ 1} and θ ₂ to obtain the head angle θ _t, and the head position (x _t , y _t , z) of the person H _{B. t} ) and the amount of head rotation (rx _t , ry _t , rz _t ) may be calculated.

<Determination of motion path: 2nd example>
In the first example of the procedure described above, immediately upon entering the redirection mode, the head position of the person _{H B (x, y, z} ) and head rotation amount (rx, ry, rz) a motion path information MP It will be set with the value of, and there is a possibility that the posture will change suddenly and give a sense of discomfort.

Therefore, in the second example of the method, the head position of the person _{H B} in the actual position (x, y, z) and head rotation amount (rx, ry, rz) from gradually the motion path information MP head Bring it closer to the position (x _θ , y _θ , z _θ ) and the amount of head rotation (rx _θ , ry _θ , rz _θ).

For the sake of simplicity, we will focus on the x-coordinate of the head position. The redirection mode display control unit 57A calculates the x'coordinate of the head position in the next frame from the x coordinate of the head position in the current frame by using the following equation (3). This equation (3) is a recurrence equation with a preset constant k. As the value of this constant k is increased, the motion path information MP _{slowly approaches the head position (x θ} , y _θ , z _θ ) and the head rotation amount (rx _θ , ry _θ , rz _θ ) (however, however). k ≧ 1). Further, when the constant k = 1 is set, the same procedure as in the first example is obtained.
The y-coordinate and z-coordinate of the head position and the amount of rotation of the head (rx, ry, rz) can also be calculated by the same recurrence formula as the x-coordinate of the head position.

_{The return motion mode display control unit 57C displays on the AR glass 4 A so that the person H B} who is turning to the _{person H A} returns to the actual posture with a motion path opposite to the motion path in the direction mode display control unit 57A. Just let me do it. Here, since the return motion mode display control unit 57C can determine the motion path by the same method as the reduction mode display control unit 57A, the description thereof will be omitted.
Further, since the motion path of the same when displaying the person H _A in AR glass 4 _B of the person H _B, the description thereof is omitted.

[Action / Effect]
As described above, in the AR display system 1 according to the first embodiment, even when the other party is displayed in front of the AR glass 4, the other party is in a state of completely turning around, so that the space is available. It is possible to reduce the sense of discomfort in sharing.
Further, since the AR display system 1 has the same device configuration on the own side and the other side, the system configuration can be simplified.

(Second Embodiment)
[AR display system configuration]
With reference to FIG. 13, the configuration of the AR display system 1B according to the second embodiment will be described as being different from the first embodiment.

As shown in FIG. 13, the AR display system 1B according to the second embodiment is different from the first embodiment in that the capture device 3B is used as a server. That, AR display system 1B includes two cameras _{2B (2B A, 2B B)} , and one of the capture device 3B, two AR-glass ₄ and ₍₄ A, _{4 B),} two AR display It is equipped with a control device 5 (5 _A , 5 _B).

The camera 2B is a general network camera that transmits the captured image of the person H to the capture device 3B via the network NW. At this time, the camera 2B may add the identification information of the photographed person H to the photographed image. Referring to this identification information, capture device 3B described later can be identified which of the captured image of the person H _A or person H _B.

The capture device 3B uses the captured image of the person H input from the camera 2B to generate the 3DCG model of the person H and the bone information of the person H. In other words, capture apparatus 3B, the person _H A, generates a 3DCG model and bones information _{H B.} For example, the capture device 3B can be configured by a general server or a cloud server arranged on the network NW. The capture device 3B refers to the identification information, the 3DCG model and bone information of the person _{H A} sends to the AR display control unit _{5 B,} the 3DCG model and bone information of the person _{H B} to AR display control unit _{5 A} Send.
Since the capture device 3B is the same as the first embodiment in other respects, further description thereof will be omitted.

[Action / Effect]
As described above, the AR display system 1B according to the first embodiment can reduce the discomfort in space sharing as in the first embodiment.
Further, the AR display system 1B can be realized by one capture device 3B by converting the capture device 3B into a server, and the system configuration can be further simplified.

(Third Embodiment)
[AR display system configuration]
With reference to FIG. 14, the configuration of the AR display system 1C according to the third embodiment will be described as being different from the first embodiment.

As shown in FIG. 14, the AR display system 1C according to the third embodiment is different from the first embodiment in that the capture device 3C is built in the AR glass 4C and the captured image is transmitted and received. That, AR display system 1C includes two cameras 2C _(2C A, 2C _B) and, two capture device 3C _(3C A, 3C _B) and, two AR-glass _{4C (4C} A, _4C B) And two AR display control devices 5 (5 _A , 5 _B ).

The camera 2C is a general network camera that transmits the captured image of the person H to the capture device 3C built in the AR glass 4C on the other side via the network NW. In other words, the camera _{2 A} is a photographic image of the person _{H A} is the person _{H B} transmits to the capture device _{3 B} of AR glass 4C _B to be mounted. Further, the camera _{2 B} transmits the captured image of the person _{H B} to capture device _{3 A} of AR glass 4C _A the person _{H A} is attached.

The capture device 3C receives a photographed image of the person H from the camera 2C via the network NW, and uses the received photographed image to generate a 3DCG model and bone information of the person H. Further, the capture device 3C is built in the AR glass 4C.
Since the other points, the capture device 3C and the AR glass 4C are the same as those in the first embodiment, further description thereof will be omitted.

[Action / Effect]
As described above, the AR display system 1C according to the third embodiment can reduce the discomfort in space sharing as in the first embodiment.
Further, since the AR display system 1C transmits / receives a captured image having a smaller amount of data than the 3DCG model and bone information, the communication band of the network NW can be saved.

(Modification example)
Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and includes design changes and the like within a range that does not deviate from the gist of the present invention.
In each of the above-described embodiments, the AR display device has been described as being an AR glass, but the present invention is not limited thereto. For example, the AR display device may be a smartphone having an AR display function.
In each of the above-described embodiments, the AR display control device has been described as being built in the AR glass, but the AR display control device may be provided outside the AR glass.

In each of the above-described embodiments, the AR display device has been described as hardware, but the present invention is not limited thereto. For example, the present invention can also be realized by a program that operates hardware resources such as a CPU, memory, and hard disk of a computer as the AR display device described above. These programs may be distributed via a communication line, or may be written and distributed on a recording medium such as a CD-ROM or a flash memory.

1-1C AR display system 2-2C Camera 3-3C Capture device 4, 4C AR glass 5 AR display control device 6 Video display device 30 Captured video input unit 31 3DCG model generation unit 32 Bone information generation unit 33 3DCG / bone information transmission Unit 40 Line-of-sight sensor 41 AR display 50 Line-of-sight input unit 51 Line-of-sight determination processing unit 52 Line-of-sight person information transmission unit (line-of-sight information transmission unit)
53 3DCG / bone information receiver (3D model posture information receiver)
54 Line-of-sight person information receiving unit (line-of-sight information receiving unit)
55 Mode determination unit 56 Motion path information recording unit 57A Direction mode display control unit 57B Real mode display control unit 57C Return motion mode display control unit 58 Virtual display unit

Claims (7)

The AR display device worn by the first person viewing the video display device displays the second person viewing the other video display device so as to be on the side of the video display device viewed by the first person. AR display control device
A line-of-sight information receiving unit that receives line-of-sight information indicating whether the second person is looking at the first person or whether the second person has turned away from the first person.
When the second person continues to look at the first person for a predetermined time or longer based on the line-of-sight information, it is determined that the direction mode is set, and the second person looks away from the first person. If this is the case, it is determined to be the real mode, and if the second person continues to look away from the first person for a predetermined time or longer after being determined to be the reduction mode, the mode is determined to be the return motion mode. Judgment unit and
When the redirection mode is determined, the redirection mode display control unit that causes the AR display device to display the second person so as to turn around the first person.
When the real mode is determined, the real mode display control unit that displays the second person in the actual posture on the AR display device, and the real mode display control unit.
When the return motion mode is determined, the return motion mode display control unit that causes the AR display device to display the second person who is turning to the first person so that the second person returns to the actual posture.
An AR display control device comprising. A motion path information recording unit for pre-recording motion path information indicating the head position and the amount of head rotation for each head angle of the second person is further provided.
The redirection mode display control unit converts the head position and the amount of rotation of the head of the second person in the actual posture into coordinates, so that the head of the second person when the first person is turned around The first aspect of claim 1, wherein the angle is calculated, and the AR display device is displayed so that the second person turns around the first person with a motion path corresponding to the calculated head angle. AR display control device. The return motion mode display control unit is a motion path opposite to the motion path in the direction mode display control unit, and the AR display is performed so that the second person turning to the first person returns to the actual posture. The AR display control device according to claim 2, wherein the AR display control device is displayed on the device. A three-dimensional model of the second person and a three-dimensional model posture information receiving unit for receiving posture information are further provided.
The redirection mode display control unit, the real mode display control unit, and the return motion mode display control unit use the three-dimensional model and posture information of the second person to display the second person on the AR display device. The AR display control device according to any one of claims 1 to 3, wherein the AR display control device is displayed. Whether or not the line of sight of the first person overlaps the area of the second person displayed on the AR display device based on the line-of-sight detection result of the first person input from the AR display device. A line-of-sight determination processing unit that determines whether or not the first person is looking at the second person, and
A line-of-sight information transmitting unit that transmits the determination result of the line-of-sight determination processing unit as the line-of-sight information to another AR display device worn by the second person.
The AR display control device according to any one of claims 1 to 4, further comprising. A program for causing a computer to function as the AR display control device according to any one of claims 1 to 5. A shooting camera that shoots a second person watching another video display device,
A three-dimensional model / posture information generator that generates a three-dimensional model and posture information of the second person using the images taken by the shooting camera.
The AR display control device according to any one of claims 1 to 5 is provided.
The AR display control device uses the three-dimensional model and posture information of the second person to attach the second person to the AR display device worn by the first person who views the video display device. An AR display system characterized in that it is displayed so as to be on the side of a video display device for viewing by a person.

Images