WO2023073991A1 - Information processing system, server device, information processing method, and program - Google Patents

Information processing system, server device, information processing method, and program Download PDF

Info

Publication number
WO2023073991A1
WO2023073991A1 PCT/JP2021/040270 JP2021040270W WO2023073991A1 WO 2023073991 A1 WO2023073991 A1 WO 2023073991A1 JP 2021040270 W JP2021040270 W JP 2021040270W WO 2023073991 A1 WO2023073991 A1 WO 2023073991A1
Authority
WO
WIPO (PCT)
Prior art keywords
character
avatar
information processing
npc
interaction
Prior art date
Application number
PCT/JP2021/040270
Other languages
French (fr)
Japanese (ja)
Inventor
崇裕 松元
誉宗 巻口
Original Assignee
日本電信電話株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日本電信電話株式会社 filed Critical 日本電信電話株式会社
Priority to PCT/JP2021/040270 priority Critical patent/WO2023073991A1/en
Publication of WO2023073991A1 publication Critical patent/WO2023073991A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics

Definitions

  • One aspect of the present invention relates to an information processing system for human-computer interaction, a server device, an information processing method, and a program used in this system.
  • VR virtual reality
  • AR augmented reality
  • MR mixed reality
  • VR applications are often discussed not only with so-called VR games, but also with the concept of a VR world that provides a new viewing style for broadcasting and video content.
  • a high touch is an example of an interaction based on the emotional expression of content. Focusing on high-touch, there is a technology that realizes remote high-touch using a device that transmits images and vibrations of two people in a distant place by communication other than the VR space. Also, there is known a technique for achieving a high touch between a person and a telepresence robot remotely controlled by the person (see, for example, Non-Patent Document 1).
  • Non-Patent Document 1 discloses a technology in which avatars high-five each other on the premise that people intervene in real time.
  • this technology does not relate to high touch between a human-operated avatar and a computer-operated avatar. Therefore, in order to naturally perform emotional expressions and interactions such as high-five actions between human-operated avatars and computer-operated avatars, how should computer-operated avatars be controlled? A technology is required to solve the problem.
  • This invention has been made with a focus on the above circumstances, and its purpose is to provide a technology that can naturally convey to people the start of emotional expression and interaction with content.
  • An information processing system is an information processing system that allows a first character that reflects human intentions and a second character that is different from the first character to interact with each other.
  • This information processing system includes an information processing terminal and a server device capable of communicating with the information processing terminal.
  • the server device includes a determination unit and a requested operation control unit.
  • a determination unit calculates distance information between the first character and the second character, and based on the distance information, expresses emotions between the first character and the second character with respect to the presented content. determine the likelihood of interaction based on
  • the requested action control unit controls the action of the second character to perform the interaction when it is determined that the interaction is possible.
  • FIG. 1 is a diagram showing the overall configuration of an information processing system according to one embodiment of the present invention.
  • FIG. 2 is a diagram showing an example of data flow in the information processing system according to one embodiment of the present invention.
  • FIG. 3 is a block diagram showing the software configuration of the server device according to one embodiment of the present invention.
  • FIG. 4 is a flow chart showing an example of a processing procedure of the processor 20 relating to high-touch control.
  • FIG. 5 is a diagram for explaining an implementation example of the high touch possible distance determination unit 24.
  • FIG. 6 is a diagram for explaining an implementation example of the high-touch request motion generator 25.
  • FIG. 7 is a flow chart showing an example of a processing procedure by the NPC avatar control section 27.
  • FIG. 8 is a diagram showing an example of pre-embedded data in the content embedding method of the high-touch activity calculation processing unit 28.
  • FIG. 8 is a diagram showing an example of pre-embedded data in the content embedding method of the high
  • Emotional expressions and interactions with content are emotional expressions such as fist pumps and smiles.
  • a physical action that can be achieved by a single person (stand) of a person, CG avatar, or robot we will focus on actions such as high-fives and hugs that can be achieved by the body action of two or more people, CG avatars, and robots (stand).
  • the emotional expression and interaction with respect to the content is an action such as a handshake that can only be realized by the body action of two or more people, it is generally classified as a simple interaction if it does not correspond to the emotional expression.
  • three elements (A), (B), and (C) are used for NPC agents to express emotions and interact with content.
  • FIG. 1 is a diagram showing an example of an information processing system according to an embodiment of the invention.
  • the system shown in FIG. 1 includes a server 10 as a server device and virtual reality (VR) terminals 100-1, 100-2, . , collectively referred to as “VR terminal 100”).
  • VR terminal 100 virtual reality
  • the network NW is composed of, for example, a relay network and a plurality of access networks for accessing this relay network.
  • a relay network a public network such as the general Internet or a closed network that is controlled so that it can be accessed only from limited devices is used.
  • the access network for example, a wireless LAN (Local Area Network), mobile phone network, wired telephone network, FTTH (Fiber To The Home), CATV (Cable Television) network is used.
  • the VR terminal 100 is a head-mounted display type information processing terminal, acquires space information representing the real space as 3D information, and allows the user to experience a virtual space constructed based on the acquired 3D information.
  • the server 10 communicates with the VR terminals 100 and generates and manages information for sharing a virtual space with a plurality of VR terminals 100. For example, an operator who develops/provides games and applications operates and manages the server. It is a server computer that
  • FIG. 2 is a diagram showing an example of data flow in an information processing system according to one embodiment of the present invention.
  • a CG avatar operated by a person hereinafter referred to as a human avatar
  • a CG avatar operated by a computer hereinafter referred to as an NPC avatar
  • a VR viewing space 1 is a virtual space in which two or more human-controlled CG avatars (human avatars) and computer-controlled CG avatars (NPC avatars) can be viewed side by side. be. Furthermore, it is assumed that the VR space is a 3D polygon space in which moving images are reproduced.
  • the VR viewing space always outputs the entire 3D polygon space information as the VR space information to the VR space image presenting unit 21 . It also outputs the video playback time t in the VR space to the high-touch activity calculation processing unit 28 .
  • CG avatars (human avatar 2-1, NPC avatar 2-2) are human-shaped CG controlled by the human avatar control unit 22 or the NPC avatar control unit 27 in the VR viewing space 1. That is, the human avatar 2-1 and the NPC avatar 2-2 both hold anthropomorphic 3D polygon information as CG avatars and current head direction, head position, and hand positions as information.
  • This is a program that can move the head and hands of a 3D polygon within the VR viewing space 1. All or part of the head direction, head position, and hand position information held by each avatar can be sent to the VR space image presentation unit 21, the eye contact action generation/judgment unit 23, and the high touch possible. It is always output to the distance determination section 24 , the high touch request motion generation section 25 , and the high touch result determination section 26 .
  • the VR display device 101 is a video display device that presents the visual field video of the human avatar of the VR space video presentation unit 21 to the person.
  • the VR display device 101 is a device that outputs the display image output from the VR space image presentation unit 21 to a monitor and presents the image to a person.
  • the head position/direction control device 102 is a device provided in the VR display device 101, and is a device for controlling the head position/direction of the human avatar 2-1.
  • the head position/direction control device 102 measures the head position/direction of a person in the real world, and always provides the real-world position (three-dimensional) and direction information (three-dimensional) of the head, which are the measurement results, to the person. It is a device that outputs to the avatar control unit 22 .
  • the hand position control device 103 is mainly a HMD controller or a body position measuring device such as Kinect, and controls the hand position of the human avatar 2-1.
  • the hand position control device 103 measures the positions of both the right and left hands of a person in the real world, and always sends the position coordinates (three-dimensional) of the hands, which are the measurement results, to the human avatar control unit 22. It is an output device.
  • A is the head position/direction information of the CG avatar.
  • B is the head position information of the CG avatar.
  • C is the head position information of the CG avatar.
  • D is hand position information of the CG avatar.
  • E is the head position/direction information of the CG avatar.
  • F is the head position information of the CG avatar.
  • G is the head position information of the CG avatar.
  • H is hand position information of the CG avatar.
  • I is the head direction control information.
  • J is the eye contact correctness determination result.
  • K is propriety determination information (binary).
  • L is hand position information control information.
  • M is high touch completion information.
  • N is control information for the head position/direction and hand position of the CG avatar.
  • O is the estimation result (binary) of the high-touch activity.
  • P is the video playback time t.
  • Q is the VR spatial information.
  • R is the head position/orientation information of the CG avatar.
  • T is control information for the head position/direction and hand position of the CG avatar.
  • U is the real-world position/orientation information of the head.
  • V is the real-world position information of the hand.
  • the VR space image presentation unit 21 uses the entire 3D polygon space input from the VR viewing space 1 and the head position/head direction information in the VR space of the human avatar input from the human avatar 2-1 to An image seen when a pseudo camera is placed in the VR viewing space from the head position toward the head direction is always output as a display image.
  • the human avatar control unit 22 converts the real-world head position, head direction, and hand position coordinates sent from the head position/direction control device 102 and hand position control device 103 into coordinates in the VR viewing space. Then, the converted coordinates are constantly output to the human avatar 2-1.
  • the human avatar control unit 22 generates a CG avatar (a human avatar 2-1, an NPC avatar, and a 2-2) is a program for controlling the CG. Algorithms that determine the overall positions of the elbows, shoulders, etc. of the CG avatar from the designated head and hand positions can be realized by a general technique. It is assumed that there is one avatar controlled by the human avatar control unit 22 and one or more avatars controlled by the NPC avatar control unit 27 .
  • the VR space image presenting unit 21 generates contents such as images when the VR viewing space 1 is viewed from the head position and head direction of the CG avatar (human avatar 2-1, NPC avatar 2-2), and displays the VR space. It is a program to be output to the display device 101 .
  • the NPC avatar control unit 27 is a program that controls the NPC avatar 2-2 by designating the head position, head direction, and position information of both hands.
  • the high-touch activity calculation processing unit 28 is a program that calculates the likelihood of high-touch occurrence (high-touch activity) between the NPC avatar 2-2 and the human avatar 2-1.
  • the eye contact action generation/judgment unit 23 is a program that calculates the head direction of the NPC avatar 2-2 so that the NPC avatar 2-2 makes eye contact. Also, the eye contact behavior generation/determination unit 23 is a program for determining that eye contact has been successful.
  • the high-touchable distance determination unit 24 is a program for determining whether the person and the NPC avatar 2-2 are within a high-touchable distance.
  • the high touch request motion generation unit 25 is a program that controls the positions of both hands of the NPC avatar 2-2 so that the NPC avatar 2-2 generates a motion requesting a high touch.
  • the high touch result determination unit 26 is a program that determines whether or not the high touch was successful.
  • FIG. 3 is a functional block diagram showing an example of the server 10 shown in FIG.
  • the server 10 shown in FIG. 3 includes a processor 20 such as a CPU, a memory 30, and a communication interface 11, for example.
  • the communication interface 11 includes, for example, one or more wired or wireless communication interface units, and communicates with the VR terminal 100 according to the communication protocol defined by the network NW.
  • a wired interface for example, a wired LAN (Local Area Network) is used.
  • a wireless interface an interface that adopts a low-power wireless data communication standard such as wireless LAN or Bluetooth (registered trademark) is used.
  • the memory 30 uses, as a storage medium, a combination of, for example, a non-volatile memory such as an HDD or an SSD that can be written and read at any time, and a volatile memory such as a RAM. It is used to store various data that has been processed.
  • a non-volatile memory such as an HDD or an SSD that can be written and read at any time
  • a volatile memory such as a RAM. It is used to store various data that has been processed.
  • the processor 20 has, as processing functions related to the embodiment, a VR space image presentation unit 21, a human avatar control unit 22, an eye contact action generation/judgment unit 23, a high-touch possible distance judgment unit 24, and a high-touch request motion generation unit shown in FIG. 25 , a high touch result determination unit 26 , an NPC avatar control unit 27 , and a high touch activity calculation processing unit 28 .
  • FIG. 4 is a flowchart showing an example of the processing procedure of the processor 20 related to high-touch control.
  • the processor 20 calculates the high-touch activity level h of the VR viewing space and determines whether the high-touch activity level h is greater than a certain value (step S1). If Yes in step S1, the processor 20 determines whether or not the NPC avatar exists within the high touch possible distance of the human avatar (step S2).
  • step S2 the processor 20 performs eye contact control on the head of the NPC avatar (step S3).
  • step S3 the processor 20 determines whether the eye contact determination result is True while the high-touch activation level h is greater than the predetermined determination value T (step S4). If it is step S4, the processor 20 causes the NPC avatar to perform a high touch request action (step S5). Next, the processor 20 determines whether or not the high touch determination has become true for a certain period of time t (step S6). If Yes in step S6, the processor 20 performs high-touch effects (control of NPC avatars, operation of human avatars, vibration of user controllers, touch sounds, etc.), and ends high-touch control (step S7).
  • the high-touch activity calculation processing unit 28 As a method of calculating the high-touch activity level h at the playback time t of the content in the VR viewing space 1, for example, the following three methods can be cited.
  • the high-touch activity calculation processing unit 28 defines a threshold T for the high-touch activity, and determines that the high-touch activity is active when the high-touch activity h according to each method satisfies h>T, and is otherwise inactive. .
  • the eye contact behavior generation/determination unit 23 As one method for realizing the eye contact behavior generation/determining unit 23, a head position/direction control device 102 and a hand position control device 103 that determine the head position/direction of the human avatar, and the head position/direction of the NPC avatar. and how to determine that the eye contact was successful.
  • the generation of eye contact behavior is defined as control to control the head direction of the NPC avatar based on some processing so that the human avatar and the NPC avatar head direction (face front direction) or line-of-sight direction face each other. do.
  • control for determining successful eye contact is defined as the control for returning the success or failure of eye contact as a binary value based on the head direction or line-of-sight direction of the human avatar and the NPC avatar.
  • FIG. 5 is a diagram for explaining an implementation example of the high touch possible distance determination unit 24.
  • the unit vector of the current head direction (the front direction of the face) is (Ex u , Ey u , Ez u )
  • the human avatar Let the unit vector from the center of the head toward the center of the head of the NPC avatar be (Ex' u , Ey' u , Ez' u ), then (Ex u , Ey u , Ez u ) and (Ex' u ,
  • the angle formed by two vectors Ey′ u , Ez′ u ) is defined as ⁇ u.
  • FIG. 6 is a diagram for explaining an implementation example of the high-touch request motion generation unit 25.
  • the height component of the head center coordinates of the NPC avatar is 0 (X n , Y n 0) to (X u , Y u , 0), which is the height component 0 of the head center coordinates of the human avatar.
  • the unit vector A is rotated by ⁇ r in the right-hand horizontal direction as viewed from the NPC avatar, and the unit vector Al is rotated by ⁇ l in the left-hand horizontal direction.
  • RAr and RAl multiplied by the length R of the defined fixed length.
  • the unit vector of the current head direction is (Ex n , Eyn , Ez n ), and from the center of the NPC avatar's head
  • the unit vector toward the head center direction of the human avatar is ( Ex'n , Ey'n , Ez'n )
  • Exn , Eyn , Ezn is defined as ⁇ n.
  • the head direction of the NPC avatar is controlled so that ( Ex''n , Ey''n , Ez''n ) becomes a new head direction vector in the head center coordinates of the NPC avatar.
  • the NPC avatar can also implement an eye contact behavior in which the user avatar's head is turned back.
  • a successful eye contact is defined when ⁇ ′′ n in the eye contact behavior generation method is less than a predefined angle ⁇ eye_contact and is greater than a specified time t eye_contact .
  • the high touch possible distance determination unit 24 An implementation example of the high touch possible distance determination unit 24 will be described.
  • an example of a method of determining whether a high touch is possible distance is shown from the head position of a human avatar and the head position of an NPC avatar.
  • the human avatar's head center coordinates are (X u , Yu , Z u ) and the NPC character's head center coordinates are (X n , Y n , Z n )
  • the distance D between the two coordinates is given by the formula (1) ).
  • the high-touchable distance determining unit 24 is implemented as a program that returns binary information indicating whether or not the two are within a high-touchable distance from some kind of positional information about the human avatar and some kind of positional information about the NPC avatar.
  • the high-touch request motion generation unit 25 generates a high-touch request motion of the NPC from the head position/hand position of the human avatar and the head position/hand position of the NPC avatar.
  • the high-touch request motion generation unit 25 is defined to output the coordinates of the positions of both hands of the NPC avatar, and the NPC avatar control unit 27 controls the motion of the NPC avatar according to the coordinate positions of both hands output by the high-touch request motion generation unit 25. By generating it, the NPC avatar performs a high touch request action.
  • the processing of the NPC avatar control unit 27 moves the center coordinates of the current right and left hands of the NPC avatar toward the TR and TL obtained above. Both hands of the above NPC avatar are stretched out toward the human avatar and stand still at a certain height, and motions requiring a high touch can be generated and executed.
  • the high touch result determination unit 26 is defined as outputting a binary value indicating whether or not the high touch was successful between the human avatar and the NPC avatar.
  • FIG. 7 is a flowchart showing an example of the processing procedure by the NPC avatar control unit 27.
  • the NPC avatar control unit 27 determines whether or not the high touch activity input from the high touch activity calculation processing unit 28 is active (step S11). If Yes in step S11, the NPC avatar control unit 27 determines whether the determination result of whether or not the high-touchable distance is within the high-touchable distance by the high-touchable distance determination unit 24 is (True) (step S12).
  • step S12 the NPC avatar control unit 27 controls the head direction (eye direction) of the human avatar 2-1 based on the head direction data of the NPC avatar output from the eye contact behavior generating/determining unit 23. contact control) is performed (step S13).
  • the NPC avatar control unit 27 confirms that the high touch activity input from the high touch activity calculation processing unit 28 is in an active state (True) and the eye contact determination result by the high touch possible distance determination unit 24 is True. (Step S14). If Yes in step S ⁇ b>14 , the NPC avatar control unit 27 receives the positional coordinate information of both hands of the NPC avatar from the high touch request motion generation unit 25 . Then, the NPC avatar control unit 27 gradually changes the coordinates of both hands of the NPC avatar 2-2 in the direction of the received coordinates, stops after reaching the coordinates, and executes the high touch request operation (step S15).
  • the NPC avatar control unit 27 determines whether or not the high touch determination result input from the high touch result determination unit 26 is True within a certain period of time t after the high touch request operation is completed (step S16). If Yes in step S16, the NPC avatar control unit 27 ends the high-touch control. (Step S17).
  • FIG. 8 is a diagram showing an example of pre-embedded data in the content embedding method of the high-touch activity calculation processing unit 28. As shown in FIG. As for the high-touch activity calculation processing unit 28, an example based on the [content embedding method] is shown.
  • the high touch activity calculation processing unit 28 constantly receives the video playback time t from the VR viewing space 1 . It is also assumed that the high-touch activity level calculation processing unit 28 has pre-embedded data that uniquely determines the activity level h at each moving image time t, as shown in FIG. Then, the high-touch activity calculation processing unit 28 outputs True to the NPC avatar control unit 27 if the activity h at the current video time t is 1, and False if it is -1.
  • the eye contact behavior generation/determining unit 23 as described in (Method of generating eye contact behavior), based on the head position/direction information of the human avatar and the head position/direction information of the NPC avatar, Head direction information of the NPC avatar at the time of control is output to the NPC avatar control unit 27 .
  • the eye contact action generation/judgment unit 23 performs the eye contact correct/incorrect judgment shown in (regarding the eye contact success judgment method) at the timing of outputting information to the NPC avatar control unit 27, and the judgment result is True/ False is output to the NPC avatar control unit 27 .
  • the high touch possible distance determination unit 24 always refers to the head position information of the human avatar and the head position information of the NPC avatar. Then, determination as to whether or not the high-touchable distance is reached is always performed, and the determination result True/False is output to the NPC avatar control unit 27 .
  • the high touch request motion generation unit 25 always refers to the head position information of the human avatar and the head position information of the NPC avatar. Then, the high-touch request motion generation unit 25 constantly outputs control information of both-hands position information corresponding to the positions of the NPC avatar's hands at the time of the high-touch request to the NPC avatar control unit 27 .
  • the high touch result determination unit 26 always refers to the hand position information of the human avatar and the hand position information of the NPC avatar. Then, the judgment result as to whether or not the high touch is performed is always output to the NPC avatar control unit 27 with True/False.
  • the distance between the viewer and the computer avatar, the likelihood of eye contact, and high-touch occurrence are calculated while the content is being viewed, and these values satisfy certain conditions.
  • the avatar of the viewer and the avatar of the computer are controlled so that they touch each other. This allows a high-five action between a human avatar and an NPC avatar without an explicit presentation action to initiate the high-five.
  • the NPC agent urges the human agent to express emotions and interact with the content, such as high fives
  • the NPC agent presents utterances and texts such as "Let's give a high five" or CG icons associated with high fives. can be considered.
  • NPC agents can naturally express emotions and initiate interactions with content such as high fives using only non-verbal body movements. If the NPC agent cannot naturally convey the intention of starting the high-five to the person through non-verbal body movements, not only will the person ignore the start of the high-five, but the person will feel unnatural and uncomfortable with the NPC agent's movement and watch the content. with an NPC agent.
  • the NPC agent can naturally convey to the person the start of emotional expression and interaction with respect to the content, such as a high touch, using only non-verbal body movements, and can start the emotional expression and interaction with the content. become.
  • the present invention is not limited to the above embodiments as they are.
  • the human avatar 2-1 and the NPC avatar 2-2 can interact not only in a virtual space implemented by a computer (server 10), but also in real space.
  • the human avatar 2-1 and the NPC avatar 2-2 can also interact in an environment in which augmented reality is superimposed on the real space.
  • the human avatar 2-1 may be a person itself, and the NPC avatar 2-2 may be a real-world robot or the like.
  • the human avatar 2-1 is an example of a first character that reflects human intentions
  • the NPC avatar 2-2 is an example of a second character different from the first character.
  • the present invention can be embodied by modifying the constituent elements without departing from the gist of the present invention.
  • various inventions can be formed by appropriate combinations of the plurality of constituent elements disclosed in the above embodiments. For example, some components may be omitted from all components shown in the embodiments.
  • constituent elements of different embodiments may be combined as appropriate.
  • Reference Signs List 1 VR viewing space 2-1 Human avatar 2-2 NPC avatar 10 Server 11 Communication interface 20 Processor 21 VR space video presentation unit 22 Human avatar control unit 23 Eye contact action generation/judgment unit 24 High touch possible distance determination unit 25 High touch request motion generation unit 26 High touch result determination unit 27 NPC avatar control unit 28 High touch activity calculation processing unit 30 Memory 100 VR terminal 100-1 Virtual reality terminal 100-2 Virtual reality terminal 101 VR display device 102 Head position/direction control device 103 Hand position control device.

Abstract

The information processing system according to one aspect of the present invention causes a first avatar operated by a person and a second avatar operated by a computer to interact with each other. This information processing system comprises an information processing terminal and a server device capable of communicating with the information processing terminal. The server device comprises a determination unit and a requested operation control unit. The determination unit calculates information on a distance between a first character and a second character, and determines, on the basis of the distance information, a possibility of an interactive operation based on an emotional expression between the first character and the second character for presented content. When it is determined that it is possible to perform the interactive operation, the requested operation control unit controls the operation of the second character to execute the interactive operation.

Description

情報処理システム、サーバ装置、情報処理方法およびプログラムInformation processing system, server device, information processing method and program
　この発明の一態様は、人とコンピュータとの相互作用のための情報処理システムと、このシステムで使用されるサーバ装置、情報処理方法およびプログラムに関する。 One aspect of the present invention relates to an information processing system for human-computer interaction, a server device, an information processing method, and a program used in this system.
　仮想現実（ＶＲ：Virtual Reality）、拡張現実（ＡＲ：Augmented Reality）、あるいはこれらを統合した複合現実（ＭＲ：Mixed Reality）に関する技術が知られている。ヘッドマウンドディスプレイ（ＨＭＤ）を用いて、仮想空間上に様々なキャラクターを登場させて楽しむ没入型のＶＲアプリケーションがある。　
　人そのもの、または人により操作されるアバターは、人の意思を反映するキャラクターの一例である。別の人やその人が操作するアバター、またはコンピュータにより操作されるアバター等は、これとは異なるキャラクターの一例である。近年、このような複数のキャラクターを、コンピュータを用いて相互作用させるための技術が注目されている。　
　ＶＲアプリケーションは、いわゆるＶＲゲームだけでなく、放送やビデオコンテンツの新しい視聴スタイルを提供するＶＲワールドという概念とも絡めて議論されることが多い。この種の技術では、手の位置を取得するコントローラや、ＨＭＤのジャイロセンサ等で人の頭部や手の位置・方向情報を取得し、ＶＲ空間のＣＧアバターを制御することが一般的である。人が制御するＣＧアバターであれば、人同士がＶＲ空間のＣＧアバターの非言語動作を逐次制御して、自然にハイタッチを開始・実行できるＶＲ空間上でのコンテンツ視聴は実現されている。 Technologies related to virtual reality (VR), augmented reality (AR), and mixed reality (MR) are known. There is an immersive VR application that uses a head mounted display (HMD) to make various characters appear in a virtual space and enjoy them.
A person or an avatar operated by a person is an example of a character that reflects a person's will. Another person, an avatar controlled by that person, or an avatar controlled by a computer are examples of different characters. In recent years, attention has been paid to techniques for allowing a plurality of such characters to interact using a computer.
VR applications are often discussed not only with so-called VR games, but also with the concept of a VR world that provides a new viewing style for broadcasting and video content. In this type of technology, it is common to acquire information on the position and direction of a person's head and hands with a controller that acquires the position of hands, a gyro sensor of an HMD, etc., and control a CG avatar in a VR space. . In the case of CG avatars controlled by humans, it is possible to view content in a VR space in which people can sequentially control the non-verbal actions of the CG avatars in the VR space and naturally initiate and perform high touches.
　ハイタッチは、コンテンツに対する感情表現に基づく相互動作の一例である。ハイタッチに着目すると、ＶＲ空間以外では、通信によって離れた場所にいる２者の映像と振動を伝達するデバイスを用いてリモートハイタッチを実現する技術がある。また、人と人間が遠隔操作するテレプレゼンスロボットとの間でハイタッチを実現する技術が知られている（例えば、非特許文献１を参照）。 A high touch is an example of an interaction based on the emotional expression of content. Focusing on high-touch, there is a technology that realizes remote high-touch using a device that transmits images and vibrations of two people in a distant place by communication other than the VR space. Also, there is known a technique for achieving a high touch between a person and a telepresence robot remotely controlled by the person (see, for example, Non-Patent Document 1).
　既存の技術はＣＧアバター、映像・振動伝送デバイス、テレプレゼンスロボットを介することで、同空間に存在しない２者のハイタッチを実現する。しかし、ＣＧアバターやテレプレゼンスロボットの制御、および映像・振動伝送デバイスに表示される映像の制御は、両方に人がリアルタイムに介在することが前提になる。このため、人エージェントとＮＰＣエージェントとを、ＶＲ空間で自然な非言語動作を通じて相互作用させるにあたり、コンテンツに対する感情表現の開始をＮＰＣエージェントから促す場合、ＮＰＣエージェントをどのように制御すればよいかという課題がある。 Existing technology uses CG avatars, video/vibration transmission devices, and telepresence robots to achieve a high touch between two people who do not exist in the same space. However, the control of CG avatars and telepresence robots and the control of images displayed on image/vibration transmission devices are based on the premise that humans intervene in both in real time. For this reason, when human agents and NPC agents interact through natural non-verbal actions in the VR space, when the NPC agents prompt the start of emotional expression for content, how should the NPC agents be controlled? I have a problem.
　非特許文献１には、相互に人がリアルタイムに介在することを前提としてアバター同士がハイタッチする技術が開示される。しかしこの技術は、人が操作するアバターとコンピュータが操作するアバターとのハイタッチに関する技術ではない。このため、人が操作するアバターとコンピュータが操作するアバターとの間で、ハイタッチ動作のような感情表現及び相互動作を自然に行うためには、コンピュータが操作するアバターをどのように制御すればよいかという課題を解決する技術が求められる。 　Non-Patent Document 1 discloses a technology in which avatars high-five each other on the premise that people intervene in real time. However, this technology does not relate to high touch between a human-operated avatar and a computer-operated avatar. Therefore, in order to naturally perform emotional expressions and interactions such as high-five actions between human-operated avatars and computer-operated avatars, how should computer-operated avatars be controlled? A technology is required to solve the problem.
　この発明は上記事情に着目してなされたもので、その目的とするところは、コンテンツに対する感情表現かつ相互動作の開始を自然に人に伝えることのできる技術を提供することにある。 This invention has been made with a focus on the above circumstances, and its purpose is to provide a technology that can naturally convey to people the start of emotional expression and interaction with content.
　この発明の一態様に係る情報処理システムは、人の意思を反映する第１キャラクターと、前記第１キャラクターとは異なる第２キャラクターとを相互作用させる情報処理システムである。この情報処理システムは、情報処理端末と、情報処理端末との間で通信が可能なサーバ装置とを具備する。サーバ装置は、判定部と、要求動作制御部とを備える。判定部は、第１キャラクターと前記第２キャラクターとの間の距離情報を算出し、この距離情報に基づいて、提示されたコンテンツに対する前記第１キャラクターと前記第２キャラクターとの間での感情表現に基づく相互動作の可能性を判定する。要求動作制御部は、前記相互動作が可能と判定された場合に、前記第２キャラクターの動作を制御して前記相互動作を実行させる。 An information processing system according to one aspect of the present invention is an information processing system that allows a first character that reflects human intentions and a second character that is different from the first character to interact with each other. This information processing system includes an information processing terminal and a server device capable of communicating with the information processing terminal. The server device includes a determination unit and a requested operation control unit. A determination unit calculates distance information between the first character and the second character, and based on the distance information, expresses emotions between the first character and the second character with respect to the presented content. determine the likelihood of interaction based on The requested action control unit controls the action of the second character to perform the interaction when it is determined that the interaction is possible.
　この発明の一態様によれば、コンテンツに対する感情表現かつ相互動作の開始を自然に人に伝えることのできる技術を提供することができる。 According to one aspect of the present invention, it is possible to provide a technology capable of naturally conveying to a person the start of emotional expression and interaction with respect to content.
図１は、この発明の一実施形態に係る情報処理システムの全体構成を示す図である。FIG. 1 is a diagram showing the overall configuration of an information processing system according to one embodiment of the present invention. 図２は、この発明の一実施形態に係る情報処理システムでのデータの流れの一例を示す図である。FIG. 2 is a diagram showing an example of data flow in the information processing system according to one embodiment of the present invention. 図３は、この発明の一実施形態に係るサーバ装置のソフトウェア構成を示すブロック図である。FIG. 3 is a block diagram showing the software configuration of the server device according to one embodiment of the present invention. 図４は、ハイタッチ制御に係わるプロセッサ２０の処理手順の一例を示すフローチャートである。FIG. 4 is a flow chart showing an example of a processing procedure of the processor 20 relating to high-touch control. 図５は、ハイタッチ可能距離判定部２４の実現例について説明するための図である。FIG. 5 is a diagram for explaining an implementation example of the high touch possible distance determination unit 24. As shown in FIG. 図６は、ハイタッチ要求動作生成部２５の実現例について説明するための図である。FIG. 6 is a diagram for explaining an implementation example of the high-touch request motion generator 25. As shown in FIG. 図７は、ＮＰＣアバター制御部２７による処理手順の一例を示すフローチャートである。FIG. 7 is a flow chart showing an example of a processing procedure by the NPC avatar control section 27. As shown in FIG. 図８は、ハイタッチ活性度計算処理部２８のコンテンツ埋め込み方式における事前埋め込みデータの一例を示す図である。FIG. 8 is a diagram showing an example of pre-embedded data in the content embedding method of the high-touch activity calculation processing unit 28. As shown in FIG.
　以下、図面を参照してこの発明に係わる実施形態を説明する。実施形態では、スポーツ映像・音楽ライブ映像などのコンテンツ視聴時に、人または人が操作するＣＧアバターまたはロボット（人エージェント）と、コンピュータが操作するＣＧアバターまたはロボット（ＮＰＣエージェント）との間で、ハイタッチやハグなどのコンテンツに対する感情表現かつ相互動作を、発話・テキスト・アイコンなどの明示的なハイタッチ開始意図の提示なしに、非言語身体動作のみでＮＰＣエージェントからコンテンツに対する感情表現かつ相互動作の開始意図を人エージェント側に伝えて開始するための制御技術を開示する。 Hereinafter, embodiments according to the present invention will be described with reference to the drawings. In the embodiment, when viewing content such as sports videos and music live videos, high touch is performed between a human or a human-operated CG avatar or robot (human agent) and a computer-operated CG avatar or robot (NPC agent). Emotional expression and interaction with respect to content such as hugging, etc. Intention to initiate emotional expression and interaction with content from NPC agents only with non-verbal body movements without presentation of explicit intention to initiate high touch such as utterance, text, icon, etc. is disclosed to the human agent side to start.
　ここでは、スポーツ観戦や音楽ライブ映像といったコンテンツを複数人で視聴する環境において、それがＶＲ空間である場合には視聴者が操作するアバターとコンピュータにより操作されるアバターとの間での感情表現や相互動作を自然にできるように促す技術について説明する。詳しくは、動画コンテンツ視聴を伴うＶＲ空間において、人間により操作されるＣＧアバターと、コンピュータにより操作されるＣＧアバターのハイタッチを自然に実現するための操作手法について説明する。 Here, in an environment where content such as watching sports or live music video is viewed by multiple people, if it is a VR space, emotional expressions and interactions between an avatar operated by a viewer and an avatar operated by a computer will be considered. Technology that encourages natural interaction is described. More specifically, an operation method for naturally realizing a high touch between a human-operated CG avatar and a computer-operated CG avatar in a VR space accompanied by video content viewing will be described.
　コンテンツに対する感情表現かつ相互動作は、ガッツポーズや笑顔などの感情表現である。ここでは、人・ＣＧアバター・ロボットの１人［台］で成立してしまう身体動作ではなく、ハイタッチやハグなどの人・ＣＧアバター・ロボットが２人［台］以上の身体動作によって成立する動作を指す。またコンテンツに対する感情表現かつ相互動作は、握手などの２人［台］以上の身体動作によって初めて成立する動作であっても、一般に感情表現には該当しないものは単なる相互動作として区別する。 Emotional expressions and interactions with content are emotional expressions such as fist pumps and smiles. Here, instead of a physical action that can be achieved by a single person (stand) of a person, CG avatar, or robot, we will focus on actions such as high-fives and hugs that can be achieved by the body action of two or more people, CG avatars, and robots (stand). point to In addition, even if the emotional expression and interaction with respect to the content is an action such as a handshake that can only be realized by the body action of two or more people, it is generally classified as a simple interaction if it does not correspond to the emotional expression.
　実施形態では、ＮＰＣエージェントがコンテンツに対する感情表現かつ相互動作を実施するにあたり、（Ａ）、（Ｂ）、および（Ｃ）の３つの要素を利用する。 In the embodiment, three elements (A), (B), and (C) are used for NPC agents to express emotions and interact with content.
　（Ａ）人エージェントとＮＰＣエージェントの距離情報
　（Ｂ）人エージェントとＮＰＣエージェントのアイコンタクト状態
　（Ｃ）視聴しているコンテンツの状態（コンテンツに対する感情表現かつ相互動作の発生活性度）
　図１は、この発明の一実施形態に係る情報処理システムの一例を示す図である。図１に示されるシステムは、ネットワークＮＷを介して互いに通信可能な、サーバ装置としてのサーバ１０と、情報処理端末としての仮想現実（ＶＲ）端末１００－１，１００－２，・・・（以下、まとめて「ＶＲ端末１００」と称する）とを備える。 (A) Distance information between the human agent and the NPC agent (B) Eye contact state between the human agent and the NPC agent (C) State of the content being viewed (emotional expression for the content and activation level of interaction)
FIG. 1 is a diagram showing an example of an information processing system according to an embodiment of the invention. The system shown in FIG. 1 includes a server 10 as a server device and virtual reality (VR) terminals 100-1, 100-2, . , collectively referred to as “VR terminal 100”).
　ネットワークＮＷは、例えば中継網と、この中継網に対しアクセスするための複数のアクセス網とから構成される。中継網としては、一般的なインターネットのような公衆網や限られた機器などからのみアクセスできるよう制御された閉域網が用いられる。アクセス網としては、例えば、無線ＬＡＮ（Local Area Network）、携帯電話網、有線電話網、ＦＴＴＨ（Fiber To The Home）、ＣＡＴＶ（Cable Television）網が用いられる。 The network NW is composed of, for example, a relay network and a plurality of access networks for accessing this relay network. As the relay network, a public network such as the general Internet or a closed network that is controlled so that it can be accessed only from limited devices is used. As the access network, for example, a wireless LAN (Local Area Network), mobile phone network, wired telephone network, FTTH (Fiber To The Home), CATV (Cable Television) network is used.
　ＶＲ端末１００は、ヘッドマウントディスプレイ型の情報処理端末であり、現実空間を表す空間情報を３Ｄ情報として取得し、取得した３Ｄ情報に基づいて構築された仮想空間をユーザに体験させる。 The VR terminal 100 is a head-mounted display type information processing terminal, acquires space information representing the real space as 3D information, and allows the user to experience a virtual space constructed based on the acquired 3D information.
　サーバ１０は、ＶＲ端末１００と通信し、仮想空間を複数のＶＲ端末１００に共有させるための情報の生成および管理を行うもので、例えば、ゲームやアプリケーションなどを開発／提供する事業者が運営管理するサーバコンピュータである。 The server 10 communicates with the VR terminals 100 and generates and manages information for sharing a virtual space with a plurality of VR terminals 100. For example, an operator who develops/provides games and applications operates and manages the server. It is a server computer that
　図２は、この発明の一実施形態に係る情報処理システムでのデータの流れの一例を示す図である。ＶＲ空間で人の操作するＣＧアバター（以下、人アバター）とコンピュータにより操作されるＣＧアバター（以下、ＮＰＣアバター）との間でハイタッチする場合について説明する。 FIG. 2 is a diagram showing an example of data flow in an information processing system according to one embodiment of the present invention. A case of high-touching between a CG avatar operated by a person (hereinafter referred to as a human avatar) and a CG avatar operated by a computer (hereinafter referred to as an NPC avatar) in a VR space will be described.
　図２において、ＶＲ視聴空間１は、人が制御するＣＧアバター（人アバター）とコンピュータにより制御されるＣＧアバター（ＮＰＣアバター）とが、２体以上並んでコンテンツを視聴することができる仮想空間である。さらにＶＲ空間内では動画が再生されている３Ｄポリゴン空間とする。ＶＲ視聴空間は、ＶＲ空間映像提示部２１に常時ＶＲ空間情報として３Ｄポリゴン空間情報全体を出力する。またハイタッチ活性度計算処理部２８に対してＶＲ空間における動画再生時刻ｔを出力する。 In FIG. 2, a VR viewing space 1 is a virtual space in which two or more human-controlled CG avatars (human avatars) and computer-controlled CG avatars (NPC avatars) can be viewed side by side. be. Furthermore, it is assumed that the VR space is a 3D polygon space in which moving images are reproduced. The VR viewing space always outputs the entire 3D polygon space information as the VR space information to the VR space image presenting unit 21 . It also outputs the video playback time t in the VR space to the high-touch activity calculation processing unit 28 .
　ＣＧアバター（人アバター２－１、ＮＰＣアバター２－２）は、ＶＲ視聴空間１において、人アバター制御部２２またはＮＰＣアバター制御部２７によって制御される人型のＣＧである。すなわち人アバター２－１、ＮＰＣアバター２－２は、いずれもＣＧアバターとしての擬人的な３Ｄポリゴン情報および現在の頭部方向・頭部位置・手位置を情報として保持し、それに従ってＣＧアバターの３Ｄポリゴンの頭部・手をＶＲ視聴空間１内で動かすことができるプログラムである。各アバターが保持している頭部方向・頭部位置・手位置の情報については、全ての情報、または一部の情報がＶＲ空間映像提示部２１、アイコンタクト行動生成・判定部２３、ハイタッチ可能距離判定部２４、ハイタッチ要求動作生成部２５、および、ハイタッチ結果判定部２６に対して常時出力される。 CG avatars (human avatar 2-1, NPC avatar 2-2) are human-shaped CG controlled by the human avatar control unit 22 or the NPC avatar control unit 27 in the VR viewing space 1. That is, the human avatar 2-1 and the NPC avatar 2-2 both hold anthropomorphic 3D polygon information as CG avatars and current head direction, head position, and hand positions as information. This is a program that can move the head and hands of a 3D polygon within the VR viewing space 1. All or part of the head direction, head position, and hand position information held by each avatar can be sent to the VR space image presentation unit 21, the eye contact action generation/judgment unit 23, and the high touch possible. It is always output to the distance determination section 24 , the high touch request motion generation section 25 , and the high touch result determination section 26 .
　図２において、ＶＲ表示装置１０１は、ＶＲ空間映像提示部２１の人アバターの視界映像を人に提示する、映像表示装置である。つまりＶＲ表示装置１０１は、ＶＲ空間映像提示部２１から出力された表示映像をモニターに出力して、人に映像を提示する装置である。 In FIG. 2, the VR display device 101 is a video display device that presents the visual field video of the human avatar of the VR space video presentation unit 21 to the person. In other words, the VR display device 101 is a device that outputs the display image output from the VR space image presentation unit 21 to a monitor and presents the image to a person.
　頭部位置方向制御装置１０２は、ＶＲ表示装置１０１に備わるデバイスであり、人アバター２－１の頭部位置・方向を制御する装置である。つまり頭部位置方向制御装置１０２は、現実世界の人の頭部位置・方向を計測して、計測結果である頭部の実世界位置（３次元）、方向情報（３次元）を常時、人アバター制御部２２に出力する装置である。 The head position/direction control device 102 is a device provided in the VR display device 101, and is a device for controlling the head position/direction of the human avatar 2-1. In other words, the head position/direction control device 102 measures the head position/direction of a person in the real world, and always provides the real-world position (three-dimensional) and direction information (three-dimensional) of the head, which are the measurement results, to the person. It is a device that outputs to the avatar control unit 22 .
　手部位置制御装置１０３は、主にＨＭＤ用のコントローラや、Ｋｉｎｅｃｔなどの身体位置計測装置であり、人アバター２－１の手の位置を制御する。つまり手部位置制御装置１０３は、現実世界の人の両手の位置を右と左のそれぞれを計測して、計測結果である手部の位置座標（３次元）を常時、人アバター制御部２２に出力する装置である。 The hand position control device 103 is mainly a HMD controller or a body position measuring device such as Kinect, and controls the hand position of the human avatar 2-1. In other words, the hand position control device 103 measures the positions of both the right and left hands of a person in the real world, and always sends the position coordinates (three-dimensional) of the hands, which are the measurement results, to the human avatar control unit 22. It is an output device.
　図２において、Ａは、ＣＧアバターの頭部位置／方向情報である。Ｂは、ＣＧアバターの頭部位置情報である。Ｃは、ＣＧアバターの頭部位置情報である。Ｄは、ＣＧアバターの手位置情報である。これらは、人アバター２－１から出力される。 In FIG. 2, A is the head position/direction information of the CG avatar. B is the head position information of the CG avatar. C is the head position information of the CG avatar. D is hand position information of the CG avatar. These are output from the human avatar 2-1.
　Ｅは、ＣＧアバターの頭部位置／方向情報である。Ｆは、ＣＧアバターの頭部位置情報である。Ｇは、ＣＧアバターの頭部位置情報である。Ｈは、ＣＧアバターの手位置情報である。これらは、ＮＰＣアバター２－２から出力される。 　E is the head position/direction information of the CG avatar. F is the head position information of the CG avatar. G is the head position information of the CG avatar. H is hand position information of the CG avatar. These are output from the NPC avatar 2-2.
　Ｉは、頭部方向制御情報である。Ｊは、アイコンタクト正否判定結果である。Ｋは、可否判定情報（２値）である。Ｌは、手部位置情報制御情報である。Ｍは、ハイタッチ完了情報である。 　I is the head direction control information. J is the eye contact correctness determination result. K is propriety determination information (binary). L is hand position information control information. M is high touch completion information.
　Ｎは、ＣＧアバターの頭部位置／方向、手部位置の制御情報である。Ｏは、ハイタッチ活性度の推定結果（２値）である。Ｐは、動画再生時刻ｔである。Ｑは、ＶＲ空間情報である。Ｒは、ＣＧアバターの頭部位置／方向情報である。 　N is control information for the head position/direction and hand position of the CG avatar. O is the estimation result (binary) of the high-touch activity. P is the video playback time t. Q is the VR spatial information. R is the head position/orientation information of the CG avatar.
　Ｓは、表示映像である。Ｔは、ＣＧアバターの頭部位置／方向、手部位置の制御情報である。Ｕは、頭部の実世界位置／方向情報である。Ｖは、手部の実世界位置情報である。 　S is the display image. T is control information for the head position/direction and hand position of the CG avatar. U is the real-world position/orientation information of the head. V is the real-world position information of the hand.
　ＶＲ空間映像提示部２１は、ＶＲ視聴空間１より入力される３Ｄポリゴン空間全体と、人アバター２－１から入力される人アバターのＶＲ空間上の頭部位置／頭部方向情報を用いて、頭部位置から頭部方向に向かってＶＲ視聴空間上に疑似的なカメラを置いた場合に見える映像を表示映像として常時出力する。 The VR space image presentation unit 21 uses the entire 3D polygon space input from the VR viewing space 1 and the head position/head direction information in the VR space of the human avatar input from the human avatar 2-1 to An image seen when a pseudo camera is placed in the VR viewing space from the head position toward the head direction is always output as a display image.
　人アバター制御部２２は、頭部位置方向制御装置１０２、手部位置制御装置１０３から送られてきた実世界の頭部位置、頭部方向、手部位置の座標をＶＲ視聴空間の座標に変換して、変換後の座標を常時人アバター２－１に出力する。 The human avatar control unit 22 converts the real-world head position, head direction, and hand position coordinates sent from the head position/direction control device 102 and hand position control device 103 into coordinates in the VR viewing space. Then, the converted coordinates are constantly output to the human avatar 2-1.
　人アバター制御部２２は、頭部位置方向制御装置１０２、手部位置制御装置１０３から与えられる頭部位置、頭部方向、両手の位置情報を元にＣＧアバター（人アバター２－１、ＮＰＣアバター２－２）のＣＧを制御するプログラムである。指定の頭・手の位置からＣＧアバターの肘・肩などの全体位置を決めるアルゴリズムは一般的な手法で実現できる。人アバター制御部２２で制御されるものは１体、ＮＰＣアバター制御部２７によって制御されるものは１体以上いるものとする。 The human avatar control unit 22 generates a CG avatar (a human avatar 2-1, an NPC avatar, and a 2-2) is a program for controlling the CG. Algorithms that determine the overall positions of the elbows, shoulders, etc. of the CG avatar from the designated head and hand positions can be realized by a general technique. It is assumed that there is one avatar controlled by the human avatar control unit 22 and one or more avatars controlled by the NPC avatar control unit 27 .
　ＶＲ空間映像提示部２１は、ＶＲ視聴空間１をＣＧアバター（人アバター２－１、ＮＰＣアバター２－２）の頭部位置、頭部方向から見たときの映像などのコンテンツを生成してＶＲ表示装置１０１に出力するプログラムである。 The VR space image presenting unit 21 generates contents such as images when the VR viewing space 1 is viewed from the head position and head direction of the CG avatar (human avatar 2-1, NPC avatar 2-2), and displays the VR space. It is a program to be output to the display device 101 .
　ＮＰＣアバター制御部２７は、頭部位置、頭部方向、両手の位置情報を指定してＮＰＣアバター２－２を制御するプログラムである。 The NPC avatar control unit 27 is a program that controls the NPC avatar 2-2 by designating the head position, head direction, and position information of both hands.
　ハイタッチ活性度計算処理部２８は、ＮＰＣアバター２－２と人アバター２－１のハイタッチ発生しやすさ（ハイタッチ活性度）を計算するプログラムである。 The high-touch activity calculation processing unit 28 is a program that calculates the likelihood of high-touch occurrence (high-touch activity) between the NPC avatar 2-2 and the human avatar 2-1.
　アイコンタクト行動生成・判定部２３は、ＮＰＣアバター２－２がアイコンタクトするために、ＮＰＣアバター２－２の頭部方向を計算するプログラムである。また、アイコンタクト行動生成・判定部２３は、アイコンタクトが成功したことを判定するプログラムである。 The eye contact action generation/judgment unit 23 is a program that calculates the head direction of the NPC avatar 2-2 so that the NPC avatar 2-2 makes eye contact. Also, the eye contact behavior generation/determination unit 23 is a program for determining that eye contact has been successful.
　ハイタッチ可能距離判定部２４は、人とＮＰＣアバター２－２がハイタッチ可能な距離にいるか判定するためのプログラムである。 The high-touchable distance determination unit 24 is a program for determining whether the person and the NPC avatar 2-2 are within a high-touchable distance.
　ハイタッチ要求動作生成部２５は、ＮＰＣアバター２－２がハイタッチを求める動作を生成するため、ＮＰＣアバター２－２の両手位置を制御するプログラムである。 The high touch request motion generation unit 25 is a program that controls the positions of both hands of the NPC avatar 2-2 so that the NPC avatar 2-2 generates a motion requesting a high touch.
　ハイタッチ結果判定部２６は、ハイタッチが成功したか否かを判定するプログラムである。 The high touch result determination unit 26 is a program that determines whether or not the high touch was successful.
　図３は、図１に示されるサーバ１０の一例を示す機能ブロック図である。図３に示されるサーバ１０は、例えば、ＣＰＵ等のプロセッサ２０と、メモリ３０、および通信インタフェース１１を備える。 FIG. 3 is a functional block diagram showing an example of the server 10 shown in FIG. The server 10 shown in FIG. 3 includes a processor 20 such as a CPU, a memory 30, and a communication interface 11, for example.
　通信インタフェース１１は、例えば１つ以上の有線または無線の通信インタフェースユニットを含み、ネットワークＮＷで規定される通信プロトコルに従ってＶＲ端末１００と通信する。有線インタフェースとしては、例えば有線ＬＡＮ（Local Area Network）が使用される。無線インタフェースとしては、例えば無線ＬＡＮやＢｌｕｅｔｏｏｔｈ（登録商標）などの小電力無線データ通信規格を採用したインタフェースが使用される。 The communication interface 11 includes, for example, one or more wired or wireless communication interface units, and communicates with the VR terminal 100 according to the communication protocol defined by the network NW. As a wired interface, for example, a wired LAN (Local Area Network) is used. As the wireless interface, an interface that adopts a low-power wireless data communication standard such as wireless LAN or Bluetooth (registered trademark) is used.
　メモリ３０は、記憶媒体として、例えば、ＨＤＤまたはＳＳＤ等の随時書込みおよび読出しが可能な不揮発性メモリと、ＲＡＭ等の揮発性メモリとを組み合わせて使用したもので、情報処理の過程で取得および作成された各種データを記憶するために用いられる。 The memory 30 uses, as a storage medium, a combination of, for example, a non-volatile memory such as an HDD or an SSD that can be written and read at any time, and a volatile memory such as a RAM. It is used to store various data that has been processed.
　プロセッサ２０は、実施形態に係わる処理機能として、図２に示すＶＲ空間映像提示部２１、人アバター制御部２２、アイコンタクト行動生成・判定部２３、ハイタッチ可能距離判定部２４、ハイタッチ要求動作生成部２５、ハイタッチ結果判定部２６、ＮＰＣアバター制御部２７、および、ハイタッチ活性度計算処理部２８を備える。 The processor 20 has, as processing functions related to the embodiment, a VR space image presentation unit 21, a human avatar control unit 22, an eye contact action generation/judgment unit 23, a high-touch possible distance judgment unit 24, and a high-touch request motion generation unit shown in FIG. 25 , a high touch result determination unit 26 , an NPC avatar control unit 27 , and a high touch activity calculation processing unit 28 .
　図４は、ハイタッチ制御に係わるプロセッサ２０の処理手順の一例を示すフローチャートである。図４において、プロセッサ２０は、ＶＲ視聴空間のハイタッチ活性度ｈを計算し、ハイタッチ活性度ｈが一定値よりも大きいかを判定する（ステップＳ１）。ステップＳ１でＹｅｓであれば、プロセッサ２０は、人アバターのハイタッチ可能距離内にＮＰＣアバターが存在するか否かを判定する（ステップＳ２）。 FIG. 4 is a flowchart showing an example of the processing procedure of the processor 20 related to high-touch control. In FIG. 4, the processor 20 calculates the high-touch activity level h of the VR viewing space and determines whether the high-touch activity level h is greater than a certain value (step S1). If Yes in step S1, the processor 20 determines whether or not the NPC avatar exists within the high touch possible distance of the human avatar (step S2).
　ステップＳ２でＹｅｓであれば、プロセッサ２０は、ＮＰＣアバターの頭部に対してアイコンタクト制御を実施する（ステップＳ３）。次に、プロセッサ２０は、ハイタッチ活性度ｈが既定の判定値Ｔより大きい間にアイコンタクトの判定結果がTrueになったかを判定する（ステップＳ４）。ステップＳ４であれば、プロセッサ２０は、ＮＰＣアバターに対してハイタッチ要求動作を実行させる（ステップＳ５）。次に、プロセッサ２０は、一定時間ｔの間にハイタッチ判定が真（True）になったか否かを判定する（ステップＳ６）。ステップＳ６でＹｅｓであれば、プロセッサ２０は、ハイタッチ時の演出（ＮＰＣアバターの制御・人アバターの操作ユーザコントローラの振動、タッチ音など）を行って、ハイタッチ制御を終了する（ステップＳ７）。 If Yes in step S2, the processor 20 performs eye contact control on the head of the NPC avatar (step S3). Next, the processor 20 determines whether the eye contact determination result is True while the high-touch activation level h is greater than the predetermined determination value T (step S4). If it is step S4, the processor 20 causes the NPC avatar to perform a high touch request action (step S5). Next, the processor 20 determines whether or not the high touch determination has become true for a certain period of time t (step S6). If Yes in step S6, the processor 20 performs high-touch effects (control of NPC avatars, operation of human avatars, vibration of user controllers, touch sounds, etc.), and ends high-touch control (step S7).
　図４のフローチャートを実施することで、ＮＰＣアバターからの非言語動作による自然なハイタッチを開始することができる。 By implementing the flowchart in Figure 4, it is possible to initiate a natural high-five from the NPC avatar through non-verbal actions.
　次に、ハイタッチ活性度計算処理部２８の実現例について説明する。ＶＲ視聴空間１のコンテンツの再生時刻ｔにおけるハイタッチ活性度ｈを計算する方法として、例えば以下の３つの方式を挙げることができる。 Next, an implementation example of the high-touch activity calculation processing unit 28 will be described. As a method of calculating the high-touch activity level h at the playback time t of the content in the VR viewing space 1, for example, the following three methods can be cited.
　［コンテンツ埋め込み方式］
　ＶＲ視聴空間１で再生される動画などのコンテンツの各再生時刻に応じて、事前にハイタッチの発生しやすさをハイタッチ活性度ｈとして記述しておく方式である。スポーツ映像であれば得点が入った時刻からΔｔ秒間は活性度ｈ＝１、音楽ライブであれば曲の終了後Δｔ秒間は活性度ｈ＝１、それ以外の時間帯はｈ＝０としておくなどが考えられる。 [Content embedding method]
This is a method in which the likelihood of a high touch occurring is described in advance as a high touch activity level h according to each playback time of content such as moving images played in the VR viewing space 1 . For example, in the case of a sports video, the activity level h=1 for Δt seconds from the time when a score is scored, in the case of a live music performance, the activity level h=1 for Δt seconds after the end of the song, and h=0 for other time periods. can be considered.
　［コンテンツ読み取り方式］
　ＶＲ視聴空間で再生される動画などのコンテンツデータをアルゴリズム処理して、各時刻におけるハイタッチ活性度ｈを計算する方式である。スポーツなどであれば観客音声の大きさがハイタッチの発生しやすさの判定に有用であるため、動画コンテンツにおける観客音声（の音声チャンネル）の各時刻ｔにおける音圧ＳＰｔに対して、係数αを乗算したα＊ＳＰｔを映像コンテンツ時刻ｔのハイタッチ活性度ｈとすることが考えられる。 [Content reading method]
This is a method of algorithmically processing content data such as moving images reproduced in a VR viewing space and calculating the high-touch activity level h at each time. In the case of sports, etc., the loudness of the audience voice is useful for determining the likelihood of occurrence of a high touch. Therefore , the coefficient α α*SP t , which is multiplied by , is considered to be the high-touch activity level h at video content time t.
　［コンテンツ視聴者行動計測方式］
　他にコンテンツを視聴している、または過去に同コンテンツを視聴していた人または人エージェントがいる場合、コンテンツ時刻ｔにおける人エージェント同士のハイタッチ数を、ハイタッチ数が多ければハイタッチ活性度が高いとして用いる方式である。例えばハイタッチ結果判定部２６のプログラムを用いて、コンテンツの時刻ｔからｔ＋ΔｔにおいてＶＲ視聴空間１で発生した人のＣＧアバター間のハイタッチ数をＮｔｏｕｃｈ＿ｔとすると、係数βを乗算した活性度ｈ＝β＊Ｎｔｏｕｃｈ＿ｔとすることが考えられる。 [Content viewer behavior measurement method]
If there are other people or human agents who are viewing the content or have viewed the same content in the past, the number of high touches between human agents at content time t is assumed to be high if the number of high touches is high. This is the method used. For example, using the program of the high touch result determination unit 26, if the number of high touches between human CG avatars that occurred in the VR viewing space 1 from time t to t+Δt of the content is N touch_t , the activity level h=β *N touch_t can be considered.
　ハイタッチ活性度計算処理部２８では、ハイタッチ活性度のスレッショルドＴを定義して、各方式によるハイタッチ活性度ｈがｈ＞Ｔとなるときは活性、それ以外のときは非活性として判定することができる。 The high-touch activity calculation processing unit 28 defines a threshold T for the high-touch activity, and determines that the high-touch activity is active when the high-touch activity h according to each method satisfies h>T, and is otherwise inactive. .
　アイコンタクト行動生成・判定部２３の実現例について説明する。アイコンタクト行動生成・判定部２３の実現手法の一つとして、人アバターの頭部位置／方向を決める頭部位置方向制御装置１０２、手部位置制御装置１０３と、ＮＰＣアバターの頭部位置／方向からアイコンタクト行動を生成する方法、およびアイコンタクトが成功したことを判定する方法の例を示す。ここで、アイコンタクト行動生成とは、何らかの処理に基づいてＮＰＣアバターの頭部方向を制御し、人アバターとＮＰＣアバターの頭部方向（顔正面方向）または視線方向が向かいあうようにする制御と定義する。 An implementation example of the eye contact behavior generation/determination unit 23 will be described. As one method for realizing the eye contact behavior generation/determining unit 23, a head position/direction control device 102 and a hand position control device 103 that determine the head position/direction of the human avatar, and the head position/direction of the NPC avatar. and how to determine that the eye contact was successful. Here, the generation of eye contact behavior is defined as control to control the head direction of the NPC avatar based on some processing so that the human avatar and the NPC avatar head direction (face front direction) or line-of-sight direction face each other. do.
　また、アイコンタクトが成功したことを判定する制御とは、人アバターとＮＰＣアバターの頭部方向または視線方向をもとに、アイコンタクトの正否を２値で返す制御と定義する。 Also, the control for determining successful eye contact is defined as the control for returning the success or failure of eye contact as a binary value based on the head direction or line-of-sight direction of the human avatar and the NPC avatar.
　（アイコンタクト行動生成方法について）
　図５は、ハイタッチ可能距離判定部２４の実現例について説明するための図である。図５に示されるように、人アバターの頭部中心を原点とする座標系において、現在の頭部方向（顔の正面方向）の単位ベクトルを（Ｅｘｕ，Ｅｙｕ，Ｅｚｕ）、人アバターの頭部中心からＮＰＣアバターの頭部中心方向への単位ベクトルを（Ｅｘ'ｕ，Ｅｙ'ｕ，Ｅｚ'ｕ）としたとき、(Ｅｘｕ，Ｅｙｕ，Ｅｚｕ)と（Ｅｘ'ｕ，Ｅｙ'ｕ，Ｅｚ'ｕ）の２つのベクトルがなす角度をθｕと定義する。 (Regarding eye contact behavior generation method)
FIG. 5 is a diagram for explaining an implementation example of the high touch possible distance determination unit 24. As shown in FIG. As shown in FIG. 5, in a coordinate system having the center of the head of the human avatar as the origin, the unit vector of the current head direction (the front direction of the face) is (Ex u , Ey u , Ez u ), and the human avatar Let the unit vector from the center of the head toward the center of the head of the NPC avatar be (Ex' u , Ey' u , Ez' u ), then (Ex u , Ey u , Ez u ) and (Ex' u , The angle formed by two vectors Ey′ u , Ez′ u ) is defined as θu.
　図６は、図６は、ハイタッチ要求動作生成部２５の実現例について説明するための図である図６において、ＮＰＣアバターの頭部中心座標の高さ成分０である（Ｘｎ，Ｙｎ，０）から人アバターの頭部中心座標の高さ成分０である（Ｘｕ，Ｙｕ，０）に向かう単位ベクトルＡを計算する。 6 is a diagram for explaining an implementation example of the high-touch request motion generation unit 25. In FIG. 6, the height component of the head center coordinates of the NPC avatar is 0 (X n , Y n 0) to (X u , Y u , 0), which is the height component 0 of the head center coordinates of the human avatar.
　次に、単位方向ベクトルＡをＮＰＣアバターからみて右手水平方向にθｒ回転させた単位ベクトルＡｒと左手水平方向にθｌ回転させた単位ベクトルＡｌに対して、事前にＣＧアバターの大きさなどに応じて定義された固定長の長さＲを乗算したＲＡｒとＲＡｌを計算する。ＲＡｒ＝（ＸＲＡｒ，ＹＲＡｒ，０）、ＲＡｌ＝（ＸＲＡｌ，ＹＲＡｌ，０）とする。 Next, the unit vector A is rotated by θr in the right-hand horizontal direction as viewed from the NPC avatar, and the unit vector Al is rotated by θl in the left-hand horizontal direction. Calculate RAr and RAl multiplied by the length R of the defined fixed length. Let RAr=( XRAr , YRAr , 0) and RAl=( XRAl , YRAl , 0).
　次にＮＰＣアバターの中心座標の高さ成分０の座標である（Ｘｎ，Ｙｎ，０）に対してＲＡｒとＲＡｌを加算し、さらに高さ方向の値を固定値のｚとして置いた座標を、それぞれ右手ターゲット座標ＴＲ、左手ターゲット座標ＴＬとしてＴＲ＝（Ｘｎ＋ＸＲＡｒ，Ｙｎ＋ＹＲＡｒ，ｚ）、ＴＬ＝（Ｘｎ＋ＸＲＡｌ，Ｙｎ＋ＹＲＡｌ，ｚ）とする。 Next, RAr and RAl are added to (X n , Y n , 0), which is the coordinate of the height component 0 of the center coordinates of the NPC avatar, and the value in the height direction is set as a fixed value of z. are set as right-hand target coordinates TR and left-hand target coordinates TL, respectively, as TR=( Xn + XRAr , Yn + YRAr , z) and TL=( Xn + XRAI , Yn + YRAI , z).
　図６に示されるように、ＮＰＣアバターの頭部中心を原点とする座標系において、現在の頭部方向の単位ベクトルを（Ｅｘｎ，Ｅｙｎ，Ｅｚｎ）とし、ＮＰＣアバターの頭部中心から人アバターの頭部中心方向への単位ベクトルを（Ｅｘ’ｎ，Ｅｙ’ｎ，Ｅｚ’ｎ）としたとき（Ｅｘｎ，Ｅｙｎ，Ｅｚｎ）と（Ｅｘ’ｎ，Ｅｙ’ｎ，Ｅｚ’ｎ）の２つのベクトルがなす角度をθｎと定義する。 As shown in FIG. 6, in a coordinate system with the center of the head of the NPC avatar as the origin, the unit vector of the current head direction is (Ex n , Eyn , Ez n ), and from the center of the NPC avatar's head When the unit vector toward the head center direction of the human avatar is ( Ex'n , Ey'n , Ez'n ), ( Exn , Eyn , Ezn ) and (Ex'n, Ey'n , Ez ' n ) is defined as θn.
　θｕ＜θｎのとき、ＮＰＣアバターの頭部方向ベクトルに対して変更を加える。（Ｅｘｎ，Ｅｙｎ，Ｅｚｎ）と（Ｅｘ’ｎ，Ｅｙ’ｎ，Ｅｚ’ｎ）を結んだ線上の点を（Ｅｘ’’ｎ，Ｅｙ’’ｎ，Ｅｚ’’ｎ）としたとき（Ｅｘ’’ｎ，Ｅｙ’’ｎ，Ｅｚ’’ｎ）と（Ｅｘ’ｎ，Ｅｙ’ｎ，Ｅｚ’ｎ）の２つのベクトルがなす角度θ’’ｎが、θ’’ｎ＝θｕとなるような（Ｅｘ’’ｎ，Ｅｙ’’ｎ，Ｅｚ’’ｎ）を計算する。 When θu<θn, change the head direction vector of the NPC avatar. When the point on the line connecting ( Exn , Eyn , Ezn ) and ( Ex'n , Ey'n , Ez'n ) is (Ex''n, Ey''n , Ez''n ) The angle θ''n formed by the two vectors (Ex''n, Ey''n , Ez''n ) and ( Ex'n, Ey''n , Ez''n ) is θ''n=θu Compute ( Ex''n , Ey''n , Ez''n ) such that
　そして、ＮＰＣアバターの頭部中心座標において、（Ｅｘ’’ｎ，Ｅｙ’’ｎ，Ｅｚ’’ｎ）を新しい頭部方向ベクトルなるようＮＰＣアバターの頭部方向を制御する。 Then, the head direction of the NPC avatar is controlled so that ( Ex''n , Ey''n , Ez''n ) becomes a new head direction vector in the head center coordinates of the NPC avatar.
　以上の処理により、人アバターがＮＰＣアバターの頭部に向かって頭部方向を変更したときにＮＰＣアバターもユーザアバターの頭部に対して視線を向け返すアイコンタクト行動を実装することができる。 With the above processing, when the human avatar changes its head direction toward the NPC avatar's head, the NPC avatar can also implement an eye contact behavior in which the user avatar's head is turned back.
　（アイコンタクト成功判定方法について）
　アイコンタクト行動生成方法におけるθ’’ｎが予め定義した角度θｅｙｅ＿ｃｏｎｔａｃｔより小さい状況が、指定時間ｔｅｙｅ＿ｃｏｎｔａｃｔより大きくなった場合に、アイコンタクトが成功したと定義する。 (Regarding eye contact success determination method)
A successful eye contact is defined when θ″ n in the eye contact behavior generation method is less than a predefined angle θ eye_contact and is greater than a specified time t eye_contact .
　ハイタッチ可能距離判定部２４の実現例について説明する。ここでは、人アバターの頭部位置とＮＰＣアバターの頭部位置から、ハイタッチ可能距離かを判定する方法の例を示す。人アバターの頭部中心座標を （Ｘｕ，Ｙｕ，Ｚｕ）、ＮＰＣキャラクタの頭部中心座標を（Ｘｎ，Ｙｎ，Ｚｎ）としたとき、両座標の距離Ｄを式（１）に定義する。　 An implementation example of the high touch possible distance determination unit 24 will be described. Here, an example of a method of determining whether a high touch is possible distance is shown from the head position of a human avatar and the head position of an NPC avatar. Assuming that the human avatar's head center coordinates are (X u , Yu , Z u ) and the NPC character's head center coordinates are (X n , Y n , Z n ), the distance D between the two coordinates is given by the formula (1) ).
Figure JPOXMLDOC01-appb-M000001
Figure JPOXMLDOC01-appb-M000001
　Ｄが予め定められた一定距離Ｄｔｏｕｃｈａｂｌｅ未満になれば、ハイタッチ可能と判定する。このように、ハイタッチ可能距離判定部２４を、人アバターに関する何らかの位置情報と、ＮＰＣアバターに関する何らかの位置情報から、両者がハイタッチ可能な距離にいるか否かの２値情報を返すプログラムとして実装する。 If D becomes less than a predetermined constant distance D touchable , it is determined that a high touch is possible. In this way, the high-touchable distance determining unit 24 is implemented as a program that returns binary information indicating whether or not the two are within a high-touchable distance from some kind of positional information about the human avatar and some kind of positional information about the NPC avatar.
　ハイタッチ要求動作生成部２５の実現例について説明する。ハイタッチ要求動作生成部２５は、人アバターの頭部位置・手位置とＮＰＣアバターの頭部位置・手位置とから、ＮＰＣのハイタッチ要求動作を生成する。ハイタッチ要求動作生成部２５ではＮＰＣアバターの両手位置の座標を出力するものと定義し、ハイタッチ要求動作生成部２５で出力された両手座標位置に応じて、ＮＰＣアバター制御部２７によってＮＰＣアバターの動作を生成することでハイタッチ要求動作をＮＰＣアバターに実行させる。 An implementation example of the high touch request motion generation unit 25 will be described. The high-touch request motion generation unit 25 generates a high-touch request motion of the NPC from the head position/hand position of the human avatar and the head position/hand position of the NPC avatar. The high-touch request motion generation unit 25 is defined to output the coordinates of the positions of both hands of the NPC avatar, and the NPC avatar control unit 27 controls the motion of the NPC avatar according to the coordinate positions of both hands output by the high-touch request motion generation unit 25. By generating it, the NPC avatar performs a high touch request action.
　上記の結果を受けて、ＮＰＣアバター制御部２７の処理では、上記で求めたＴＲとＴＬに向けて、現在のＮＰＣアバターの右手・左手の中心座標をそれぞれ移動させる。上記のにＮＰＣアバターの両手が、人アバターに向けて伸ばされ、一定の高さで静止する形となり、ハイタッチを要求するモーションを生成・実行することができる。 In response to the above results, the processing of the NPC avatar control unit 27 moves the center coordinates of the current right and left hands of the NPC avatar toward the TR and TL obtained above. Both hands of the above NPC avatar are stretched out toward the human avatar and stand still at a certain height, and motions requiring a high touch can be generated and executed.
　ハイタッチ結果判定部２６の実現例について説明する。ここでは、人アバターの手位置とＮＰＣアバターの手位置から、人アバター・ＮＰＣアバターの間でハイタッチが成功したか否かの判定を実現する方法を示す。ハイタッチ結果判定部２６は、人アバター・ＮＰＣアバターの間でハイタッチが成功したか否かを２値で出力するものと定義する。ハイタッチ結果判定部２６は、人アバターの右手中心座標とＮＰＣアバターの左手中心座標の距離をＲ１として、人アバターの左手中心座標とＮＰＣアバターの右手中心座標の距離をＲ２としたとき、Ｒ１、Ｒ２が両方とも一定距離Ｒｔｏｕｃｈｅｄ未満になれば、ハイタッチが成功したとみなして真を返す。 An implementation example of the high touch result determination unit 26 will be described. Here, a method for determining whether or not a high touch has been successful between a human avatar and an NPC avatar based on the hand position of the human avatar and the hand position of the NPC avatar will be described. The high touch result determination unit 26 is defined as outputting a binary value indicating whether or not the high touch was successful between the human avatar and the NPC avatar. When the distance between the right hand center coordinates of the human avatar and the NPC avatar's left hand center coordinates is R1 , and the distance between the left hand center coordinates of the human avatar and the NPC avatar's right hand center coordinates is R2 , R If both 1 and R2 are less than a fixed distance R touched , it is considered that the high touch was successful and returns true.
　図７は、ＮＰＣアバター制御部２７による処理手順の一例を示すフローチャートである。図７において、ＮＰＣアバター制御部２７は、ハイタッチ活性度計算処理部２８から入力されるハイタッチ活性度は活性状態か否かを判定する（ステップＳ１１）。ステップＳ１１でＹｅｓであれば、ＮＰＣアバター制御部２７は、ハイタッチ可能距離判定部２４による、ハイタッチ可能距離にあるか否かの判定結果が（True）であるかを判定する（ステップＳ１２）。 FIG. 7 is a flowchart showing an example of the processing procedure by the NPC avatar control unit 27. In FIG. 7, the NPC avatar control unit 27 determines whether or not the high touch activity input from the high touch activity calculation processing unit 28 is active (step S11). If Yes in step S11, the NPC avatar control unit 27 determines whether the determination result of whether or not the high-touchable distance is within the high-touchable distance by the high-touchable distance determination unit 24 is (True) (step S12).
　ステップＳ１２でＹｅｓであれば、ＮＰＣアバター制御部２７は、アイコンタクト行動生成・判定部２３から出力されるＮＰＣアバターの頭部方向データに基づき、人アバター２－１の頭部方向の制御（アイコンタクト制御）を実施する（ステップＳ１３）。 If Yes in step S12, the NPC avatar control unit 27 controls the head direction (eye direction) of the human avatar 2-1 based on the head direction data of the NPC avatar output from the eye contact behavior generating/determining unit 23. contact control) is performed (step S13).
　次に、ＮＰＣアバター制御部２７は、ハイタッチ活性度計算処理部２８から入力されるハイタッチ活性度が活性状態（True）であり、かつ、ハイタッチ可能距離判定部２４によるアイコンタクト判定結果がTrueになったかを判定する（ステップＳ１４）。ステップＳ１４でＹｅｓであれば、ＮＰＣアバター制御部２７は、ハイタッチ要求動作生成部２５からＮＰＣアバターの両手の位置座標情報を受け取る。そしてＮＰＣアバター制御部２７は、ＮＰＣアバター２－２の両手の座標を、受け取った座標方向に徐々に変化させ、座標到着後は停止させてハイタッチ要求動作を実行する（ステップＳ１５）。 Next, the NPC avatar control unit 27 confirms that the high touch activity input from the high touch activity calculation processing unit 28 is in an active state (True) and the eye contact determination result by the high touch possible distance determination unit 24 is True. (Step S14). If Yes in step S<b>14 , the NPC avatar control unit 27 receives the positional coordinate information of both hands of the NPC avatar from the high touch request motion generation unit 25 . Then, the NPC avatar control unit 27 gradually changes the coordinates of both hands of the NPC avatar 2-2 in the direction of the received coordinates, stops after reaching the coordinates, and executes the high touch request operation (step S15).
　次に、ＮＰＣアバター制御部２７は、ハイタッチ要求動作終了後、一定時間ｔの間にハイタッチ結果判定部２６から入力されるハイタッチ判定結果がTrueになったか否かを判定する（ステップＳ１６）。ステップＳ１６でＹｅｓであれば、ＮＰＣアバター制御部２７は、ハイタッチ制御を終了する。（ステップＳ１７）。 Next, the NPC avatar control unit 27 determines whether or not the high touch determination result input from the high touch result determination unit 26 is True within a certain period of time t after the high touch request operation is completed (step S16). If Yes in step S16, the NPC avatar control unit 27 ends the high-touch control. (Step S17).
　図８は、ハイタッチ活性度計算処理部２８のコンテンツ埋め込み方式における事前埋め込みデータの一例を示す図である。ハイタッチ活性度計算処理部２８については、［コンテンツ埋め込み方式］による例を示す。 FIG. 8 is a diagram showing an example of pre-embedded data in the content embedding method of the high-touch activity calculation processing unit 28. As shown in FIG. As for the high-touch activity calculation processing unit 28, an example based on the [content embedding method] is shown.
　ハイタッチ活性度計算処理部２８は、ＶＲ視聴空間１より常時動画再生時刻ｔを受け取る。またハイタッチ活性度計算処理部２８は、図８に示すように各動画時刻ｔにおいて活性度ｈが一意に定まるような事前埋め込みデータを持っているものとする。そして、ハイタッチ活性度計算処理部２８は、現在の動画時刻ｔにおける活性度ｈが１であればＴｒｕｅを、－１であればＦａｌｓｅをＮＰＣアバター制御部２７に出力する。 The high touch activity calculation processing unit 28 constantly receives the video playback time t from the VR viewing space 1 . It is also assumed that the high-touch activity level calculation processing unit 28 has pre-embedded data that uniquely determines the activity level h at each moving image time t, as shown in FIG. Then, the high-touch activity calculation processing unit 28 outputs True to the NPC avatar control unit 27 if the activity h at the current video time t is 1, and False if it is -1.
　アイコンタクト行動生成・判定部２３は、（アイコンタクト行動生成方法について）で示したように、人アバターの頭部位置／方向情報と、ＮＰＣアバターの頭部位置／方向情報を元に、アイコンタクト制御時のＮＰＣアバターの頭部方向情報をＮＰＣアバター制御部２７に出力する。 The eye contact behavior generation/determining unit 23, as described in (Method of generating eye contact behavior), based on the head position/direction information of the human avatar and the head position/direction information of the NPC avatar, Head direction information of the NPC avatar at the time of control is output to the NPC avatar control unit 27 .
　また、アイコンタクト行動生成・判定部２３は、ＮＰＣアバター制御部２７に情報を出力するタイミングで、（アイコンタクト成功判定方法について）で示したアイコンタクトの正否判定を実施し、判定結果のＴｒｕｅ／ＦａｌｓｅをＮＰＣアバター制御部２７に出力する。 In addition, the eye contact action generation/judgment unit 23 performs the eye contact correct/incorrect judgment shown in (regarding the eye contact success judgment method) at the timing of outputting information to the NPC avatar control unit 27, and the judgment result is True/ False is output to the NPC avatar control unit 27 .
　ハイタッチ可能距離判定部２４は、、人アバターの頭部位置情報とＮＰＣアバターの頭部位置情報を常時参照する。そしてハイタッチ可能距離かどうかの判定を常時行い、判定結果のＴｒｕｅ／ＦａｌｓｅをＮＰＣアバター制御部２７に出力する。 The high touch possible distance determination unit 24 always refers to the head position information of the human avatar and the head position information of the NPC avatar. Then, determination as to whether or not the high-touchable distance is reached is always performed, and the determination result True/False is output to the NPC avatar control unit 27 .
　ハイタッチ要求動作生成部２５は、図５に関する説明で述べたように、人アバターの頭部位置情報と、ＮＰＣアバターの頭部位置情報とを常時参照する。そしてハイタッチ要求動作生成部２５は、ハイタッチ要求時のＮＰＣアバターの両手の位置に該当する、両手の位置情報の制御情報をＮＰＣアバター制御部２７に常時出力する。 As described in the description of FIG. 5, the high touch request motion generation unit 25 always refers to the head position information of the human avatar and the head position information of the NPC avatar. Then, the high-touch request motion generation unit 25 constantly outputs control information of both-hands position information corresponding to the positions of the NPC avatar's hands at the time of the high-touch request to the NPC avatar control unit 27 .
　ハイタッチ結果判定部２６は、人アバターの手位置情報と、ＮＰＣアバターの手位置情報を常時参照する。そしてハイタッチが実施されているか否かの判定結果を、Ｔｒｕｅ／Ｆａｌｓｅで常時、ＮＰＣアバター制御部２７に出力する。 The high touch result determination unit 26 always refers to the hand position information of the human avatar and the hand position information of the NPC avatar. Then, the judgment result as to whether or not the high touch is performed is always output to the NPC avatar control unit 27 with True/False.
　以上述べたように、実施形態によればコンテンツ視聴中に、視聴者とコンピュータのアバター間の距離、アイコンタクト、及びハイタッチの発生しやすさを計算し、それらの値が一定の条件を満たした場合に、視聴者のアバターとコンピュータのアバターがハイタッチするように制御するようにした。これにより、ハイタッチを開始するための明示的な提示行為を行わなくても、人アバターとＮＰＣアバターとの間でハイタッチ行為ができるようになる。 As described above, according to the embodiment, the distance between the viewer and the computer avatar, the likelihood of eye contact, and high-touch occurrence are calculated while the content is being viewed, and these values satisfy certain conditions. In this case, the avatar of the viewer and the avatar of the computer are controlled so that they touch each other. This allows a high-five action between a human avatar and an NPC avatar without an explicit presentation action to initiate the high-five.
　コンテンツ視聴を伴うＶＲ（Virtual Reality）空間において、人エージェント同士のハイタッチが可能なシステムは知られていた。しかしながらハイタッチをＮＰＣエージェントと人エージェントでする場合、ハイタッチを自然に開始するためにＮＰＣエージェントを如何に制御するのかが定かでなかった。 A system that allows high-touch between human agents in a VR (Virtual Reality) space with content viewing is known. However, when high-touching is performed by an NPC agent and a human agent, it was not clear how to control the NPC agent in order to naturally initiate high-touching.
　ＮＰＣエージェントからハイタッチなどのコンテンツに対する感情表現かつ相互動作をすることを人エージェント側に促す場合、ＮＰＣエージェントから「ハイタッチしようよ」といった発話・テキスト、またはハイタッチを連想させるＣＧアイコンなどを提示する方法などが考えられる。しかし、現実空間でスポーツや音楽ライブを視聴する人同士が、発話・テキスト・アイコンなどでハイタッチ開始の意思を伝えてハイタッチを開始することは稀であり、通常は非言語動作を通じた自然な合意によってハイタッチが開始される。このため、稀な不自然なハイタッチの開始は、ハイタッチがもたらす共感できた感覚・体験を損なう可能性がある。 When the NPC agent urges the human agent to express emotions and interact with the content, such as high fives, the NPC agent presents utterances and texts such as "Let's give a high five" or CG icons associated with high fives. can be considered. However, it is rare for people watching live sports or music in the real world to start high-five by communicating their intention to start high-five through speech, text, icons, etc. Usually, it is a natural agreement through non-verbal actions. starts a high five. For this reason, a rare and unnatural initiation of a high touch may impair the sympathetic feeling/experience brought about by the high touch.
　ＮＰＣエージェントから非言語身体動作のみで自然にハイタッチなどのコンテンツに対する感情表現かつ相互動作を開始するための方法は明らかになっていない。ＮＰＣエージェントが自然にハイタッチ開始の意図を非言語身体動作で人に伝えられなければ、人にハイタッチ開始が無視されてしまうばかりか、人がＮＰＣエージェント動作に不自然さ・違和感を感じてコンテンツ視聴をＮＰＣエージェントと一緒に見るという体験自体を損なってしまう可能性がある。 It has not been clarified how NPC agents can naturally express emotions and initiate interactions with content such as high fives using only non-verbal body movements. If the NPC agent cannot naturally convey the intention of starting the high-five to the person through non-verbal body movements, not only will the person ignore the start of the high-five, but the person will feel unnatural and uncomfortable with the NPC agent's movement and watch the content. with an NPC agent.
　これに対し実施形態によれば、ＮＰＣエージェントから非言語身体動作のみでハイタッチなどのコンテンツに対する感情表現かつ相互動作の開始を人に自然に伝え、コンテンツに対する感情表現かつ相互動作を開始することが可能になる。 On the other hand, according to the embodiment, the NPC agent can naturally convey to the person the start of emotional expression and interaction with respect to the content, such as a high touch, using only non-verbal body movements, and can start the emotional expression and interaction with the content. become.
　なお、この発明は、上記実施形態そのままに限定されるものではない。例えば、人アバター２－１とＮＰＣアバター２－２は、コンピュータ（サーバ１０）により実現される仮想空間において相互作用するだけでなく、実空間において相互作用することもできる。さらに、人アバター２－１とＮＰＣアバター２－２は、実空間に拡張現実を重畳した環境において相互作用することもできる。また、人アバター２－１は人そのものであっても良いし、ＮＰＣアバター２－２は、実世界のロボット等であっても良い。要するに人アバター２－１は、人の意思を反映する第１キャラクターの一例であり、ＮＰＣアバター２－２は、第１キャラクターとは異なる第２キャラクターの一例である。　 It should be noted that the present invention is not limited to the above embodiments as they are. For example, the human avatar 2-1 and the NPC avatar 2-2 can interact not only in a virtual space implemented by a computer (server 10), but also in real space. Furthermore, the human avatar 2-1 and the NPC avatar 2-2 can also interact in an environment in which augmented reality is superimposed on the real space. Also, the human avatar 2-1 may be a person itself, and the NPC avatar 2-2 may be a real-world robot or the like. In short, the human avatar 2-1 is an example of a first character that reflects human intentions, and the NPC avatar 2-2 is an example of a second character different from the first character.　
　また、本発明は、実施段階ではその要旨を逸脱しない範囲で構成要素を変形して具体化できる。さらに、上記実施形態に開示されている複数の構成要素の適宜な組み合せにより種々の発明を形成できる。例えば、実施形態に示される全構成要素から幾つかの構成要素を削除してもよい。さらに、異なる実施形態に亘る構成要素を適宜組み合せてもよい。 Also, in the implementation stage, the present invention can be embodied by modifying the constituent elements without departing from the gist of the present invention. Furthermore, various inventions can be formed by appropriate combinations of the plurality of constituent elements disclosed in the above embodiments. For example, some components may be omitted from all components shown in the embodiments. Furthermore, constituent elements of different embodiments may be combined as appropriate.
　１…ＶＲ視聴空間
　２－１…人アバター
　２－２…ＮＰＣアバター
　１０…サーバ
　１１…通信インタフェース
　２０…プロセッサ
　２１…ＶＲ空間映像提示部
　２２…人アバター制御部
　２３…アイコンタクト行動生成・判定部
　２４…ハイタッチ可能距離判定部
　２５…ハイタッチ要求動作生成部
　２６…ハイタッチ結果判定部
　２７…ＮＰＣアバター制御部
　２８…ハイタッチ活性度計算処理部
　３０…メモリ
　１００…ＶＲ端末
　１００－１…仮想現実端末
　１００－２…仮想現実端末
　１０１…ＶＲ表示装置
　１０２…頭部位置方向制御装置
　１０３…手部位置制御装置。 Reference Signs List 1 VR viewing space 2-1 Human avatar 2-2 NPC avatar 10 Server 11 Communication interface 20 Processor 21 VR space video presentation unit 22 Human avatar control unit 23 Eye contact action generation/judgment unit 24 High touch possible distance determination unit 25 High touch request motion generation unit 26 High touch result determination unit 27 NPC avatar control unit 28 High touch activity calculation processing unit 30 Memory 100 VR terminal 100-1 Virtual reality terminal 100-2 Virtual reality terminal 101 VR display device 102 Head position/direction control device 103 Hand position control device.

Claims (8)

  1. 　人の意思を反映する第１キャラクターと、前記第１キャラクターとは異なる第２キャラクターとを相互作用させる情報処理システムであって、
    　情報処理端末と、
    　前記情報処理端末との間で通信が可能なサーバ装置とを具備し、
    　前記サーバ装置は、
    　前記第１キャラクターと前記第２キャラクターとの間の距離情報を算出し、この距離情報に基づいて、提示されたコンテンツに対する前記第１キャラクターと前記第２キャラクターとの間での感情表現に基づく相互動作の可能性を判定する判定部と、
    　前記相互動作が可能と判定された場合に、前記第２キャラクターの動作を制御して前記相互動作を実行させる要求動作制御部とを備える、情報処理システム。     An information processing system that interacts with a first character that reflects a person's intention and a second character that is different from the first character,
    an information processing terminal;
    A server device capable of communicating with the information processing terminal,
    The server device
    Distance information between the first character and the second character is calculated, and based on the distance information, interaction based on emotional expression between the first character and the second character with respect to the presented content. a determination unit that determines the possibility of motion;
    an information processing system, comprising: a requested action control unit that controls an action of the second character to perform the interaction when it is determined that the interaction is possible.
  2. 　前記第１キャラクターは、人により操作される第１アバターであり、
    　前記第２キャラクターは、コンピュータにより操作される第２アバターである、請求項１に記載の情報処理システム。     The first character is a first avatar operated by a person,
    The information processing system according to claim 1, wherein said second character is a second avatar operated by a computer.
  3. 　前記コンテンツは、前記コンピュータにより実現される仮想空間において提示され、
    　前記第１アバターおよび前記第２アバターは、前記仮想空間において相互作用する、請求項２に記載の情報処理システム。     the content is presented in a virtual space implemented by the computer;
    3. The information processing system according to claim 2, wherein said first avatar and said second avatar interact in said virtual space.
  4. 　前記コンテンツは、実空間に拡張現実を重畳した環境において提示され、
    　前記第１アバターおよび前記第２アバターは、前記実空間に拡張現実を重畳した環境において相互作用する、請求項２に記載の情報処理システム。     The content is presented in an environment in which augmented reality is superimposed on real space,
    3. The information processing system according to claim 2, wherein said first avatar and said second avatar interact in an environment in which augmented reality is superimposed on said real space.
  5. 　前記コンテンツは、実空間において提示され、
    　前記第１アバターおよび前記第２アバターは、前記実空間において相互作用する、請求項２に記載の情報処理システム。     the content is presented in real space;
    3. The information processing system according to claim 2, wherein said first avatar and said second avatar interact in said real space.
  6. 　人の意思を反映する第１キャラクターと、前記第１キャラクターとは異なる第２キャラクターとを相互作用させる情報処理システムに設けられるサーバ装置であって、
    　前記第１キャラクターと前記第２キャラクターとの間の距離情報を算出し、この距離情報に基づいて、提示されたコンテンツに対する前記第１キャラクターと前記第２キャラクターとの間での感情表現に基づく相互動作の可能性を判定する判定部と、
    　前記相互動作が可能と判定された場合に、前記第２キャラクターの動作を制御して前記相互動作を実行させる要求動作制御部とを備える、サーバ装置。     A server device provided in an information processing system that allows a first character that reflects a person's intention and a second character that is different from the first character to interact,
    Distance information between the first character and the second character is calculated, and based on the distance information, interaction based on emotional expression between the first character and the second character with respect to the presented content. a determination unit that determines the possibility of motion;
    A server apparatus comprising: a requested action control unit that controls an action of the second character to perform the interaction when it is determined that the interaction is possible.
  7. 　人の意思を反映する第１キャラクターと、前記第１キャラクターとは異なる第２キャラクターとを相互作用させる情報処理システムに設けられるサーバ装置により実行される情報処理方法であって、
    　前記サーバ装置が、前記第１キャラクターと前記第２キャラクターとの間の距離情報を算出し、この距離情報に基づいて、提示されたコンテンツに対する前記第１キャラクターと前記第２キャラクターとの間での感情表現に基づく相互動作の可能性を判定することと、
    　前記サーバ装置が、前記相互動作が可能と判定された場合に、前記第２キャラクターの動作を制御して前記相互動作を実行させることとを備える、情報処理方法。     An information processing method executed by a server device provided in an information processing system for causing interaction between a first character that reflects a person's intention and a second character that is different from the first character,
    The server device calculates distance information between the first character and the second character, and based on the distance information, the distance between the first character and the second character with respect to the presented content. Determining likelihood of interaction based on emotional expression;
    An information processing method, comprising: when the server apparatus determines that the interaction is possible, controlling the action of the second character to perform the interaction.
  8. 　人の意思を反映する第１キャラクターと、前記第１キャラクターとは異なる第２キャラクターとを相互作用させる情報処理システムに設けられるサーバ装置のプログラムであって、
    　前記サーバ装置を、前記第１キャラクターと前記第２キャラクターとの間の距離情報を算出し、この距離情報に基づいて、提示されたコンテンツに対する前記第１キャラクターと前記第２キャラクターとの間での感情表現に基づく相互動作の可能性を判定する判定部として機能させ、
    　前記サーバ装置を、前記相互動作が可能と判定された場合に、前記第２キャラクターの動作を制御して前記相互動作を実行させる要求動作制御部として機能させる、プログラム。

          A program for a server device provided in an information processing system that causes a first character that reflects human intentions and a second character that is different from the first character to interact,
    The server device calculates the distance information between the first character and the second character, and based on the distance information, the distance between the first character and the second character with respect to the presented content. Function as a judgment unit that judges the possibility of interaction based on emotional expression,
    A program that causes the server device to function as a request action control unit that controls the action of the second character to execute the interaction when it is determined that the interaction is possible.
PCT/JP2021/040270 2021-11-01 2021-11-01 Information processing system, server device, information processing method, and program WO2023073991A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/JP2021/040270 WO2023073991A1 (en) 2021-11-01 2021-11-01 Information processing system, server device, information processing method, and program

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2021/040270 WO2023073991A1 (en) 2021-11-01 2021-11-01 Information processing system, server device, information processing method, and program

Publications (1)

Publication Number Publication Date
WO2023073991A1 true WO2023073991A1 (en) 2023-05-04

Family

ID=86159670

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/040270 WO2023073991A1 (en) 2021-11-01 2021-11-01 Information processing system, server device, information processing method, and program

Country Status (1)

Country Link
WO (1) WO2023073991A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116668605A (en) * 2023-05-10 2023-08-29 北京国际云转播科技有限公司 Meta-universe studio system, playing method, storage medium and electronic device

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004141435A (en) * 2002-10-24 2004-05-20 Namco Ltd Game information, information storage medium and game device
JP2017217352A (en) * 2016-06-10 2017-12-14 任天堂株式会社 Information processing program, information processing device, information processing system, and information processing method
JP2020052993A (en) * 2018-09-25 2020-04-02 未來市股▲ふん▼有限公司 Artificial intelligence system and interactive response method
JP2020096768A (en) * 2018-12-19 2020-06-25 任天堂株式会社 Information processing system, information processor, information processing program, and information processing method
WO2021153367A1 (en) * 2020-01-30 2021-08-05 株式会社ドワンゴ Avatar display device, avatar display system, avatar display method, and avatar display program

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004141435A (en) * 2002-10-24 2004-05-20 Namco Ltd Game information, information storage medium and game device
JP2017217352A (en) * 2016-06-10 2017-12-14 任天堂株式会社 Information processing program, information processing device, information processing system, and information processing method
JP2020052993A (en) * 2018-09-25 2020-04-02 未來市股▲ふん▼有限公司 Artificial intelligence system and interactive response method
JP2020096768A (en) * 2018-12-19 2020-06-25 任天堂株式会社 Information processing system, information processor, information processing program, and information processing method
WO2021153367A1 (en) * 2020-01-30 2021-08-05 株式会社ドワンゴ Avatar display device, avatar display system, avatar display method, and avatar display program

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116668605A (en) * 2023-05-10 2023-08-29 北京国际云转播科技有限公司 Meta-universe studio system, playing method, storage medium and electronic device

Similar Documents

Publication Publication Date Title
US20180324229A1 (en) Systems and methods for providing expert assistance from a remote expert to a user operating an augmented reality device
RU2621644C2 (en) World of mass simultaneous remote digital presence
WO2020090786A1 (en) Avatar display system in virtual space, avatar display method in virtual space, and computer program
JP5208810B2 (en) Information processing apparatus, information processing method, information processing program, and network conference system
EP3691280B1 (en) Video transmission method, server, vr playback terminal and computer-readable storage medium
CN110996097B (en) VR multimedia experience quality determination method and device
US20220066541A1 (en) Virtual reality system with posture control
Jo et al. Chili: viewpoint control and on-video drawing for mobile video calls
US20210072947A1 (en) Remote interaction via bi-directional mixed-reality telepresence
WO2023073991A1 (en) Information processing system, server device, information processing method, and program
CN114125529A (en) Method, equipment and storage medium for generating and demonstrating video
JP7244450B2 (en) Computer program, server device, terminal device, and method
US20230336689A1 (en) Method and Device for Invoking Public or Private Interactions during a Multiuser Communication Session
CN114746796A (en) Dynamic browser stage
KR20190031805A (en) Coaching system for users participating in virtual reality contents
US11790588B2 (en) Display control device, display control method, and display system
CN114053694A (en) Application server, application service method thereof and computer readable storage medium
WO2023079987A1 (en) Distribution device, distribution method, and program
JP7062126B1 (en) Terminals, information processing methods, programs, and recording media
JP7250721B2 (en) Computer program, server device, terminal device, and method
JP7011746B1 (en) Content distribution system, content distribution method, and content distribution program
US20220383601A1 (en) Virtual avatars and physical object interaction in augmented reality applications
WO2023233849A1 (en) Information processing device, information processing method, and recording medium
Sun Remote Assistance for Repair Tasks Using Augmented Reality
KR20230103135A (en) Operation method for dome display in a metaverse environment

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21962529

Country of ref document: EP

Kind code of ref document: A1