JP2019220147A - Data replacement device, terminal and data replacement program - Google Patents

Abstract

To prevent deterioration in virtual experience of a user with an avatar image.SOLUTION: A data replacement device comprises an acquisition unit, a determination unit, and a replacement unit. The acquisition unit acquires first control data having a value based on the position of a first device that can be worn on the head of a user and the position of a second device that can be held by the user. The determining unit determines whether a predetermined condition is satisfied. The replacement unit replaces the first control data with second control data when it is determined that the condition is satisfied. The second control data has a value that is not based on the position of at least one of the first device and the second device.SELECTED DRAWING: Figure 1

Description

  The present invention relates to VR (Virtual Reality), AR (Augmented Reality), or MR (Mixed Reality) technology.

  2. Description of the Related Art Conventionally, it is known that a plurality of users wearing HMDs (Head Mounted Displays) share a virtual experience, that is, a VR experience via a network. Specifically, it is known to control the posture of an avatar, which is the user's alter ego in virtual space, according to the posture of the user (for example, gaze, head direction, etc.).

  Patent Document 1 discloses that when a first user removes an HMD, the second avatar (first user) does not respond because the second user does not know that the first user does not wear the HMD. It is intended to improve the problem that the user feels strange ([0004]), and to provide the user with a rich virtual experience ([0005]). Specifically, in Patent Document 1, it is determined whether the user A has removed the HMD 110 based on the inclination information of the HMD 110 [0095], and if it is determined that the user A has removed the HMD 110, non-wearing information is generated. [0100] It is described that the user terminal 1B sets the facial expression of the avatar 4A to the default facial expression after receiving the non-wearing information [0101]. It is also described that it is determined whether the user A has worn the HMD 110 again [0113], and when the user A has worn it, the facial expression of the avatar 4A is updated [0115].

Japanese Patent No. 622666

  When controlling the posture of the avatar of the user in the virtual space according to the posture of the user, the avatar may take a posture that the user does not intend, for example, an unnatural posture. In particular, when the HMD displays a first-person viewpoint virtual space image, the user cannot directly see his or her avatar, so it is difficult to detect the anomaly and try to correct it.

  If the user's avatar takes an unnatural posture, not only will the user themselves feel embarrassing or discomfort when they realize that fact, but also other users will be able to complete the virtual space including the avatar. It may be aroused as low. This can occur not only in the VR experience but also in an overall experience including a virtual element such as AR or MR (hereinafter referred to as a virtual experience).

  An object of the present invention is to prevent a user's virtual experience from deteriorating due to an avatar image.

  A data replacement device according to one aspect of the present invention includes an acquisition unit, a determination unit, and a replacement unit. The acquisition unit acquires first control data having a value based on the position of the first device that can be worn on the head of the user and the position of the second device that can be held by the user. The determining unit determines whether a predetermined condition is satisfied. The replacement unit replaces the first control data with the second control data when it is determined that the condition is satisfied. The second control data has a value that is not based on the position of at least one of the first device and the second device.

  According to the present invention, it is an object to prevent a virtual experience of a user from being deteriorated by an avatar image.

FIG. 2 is a block diagram illustrating a data replacement device according to the embodiment. FIG. 1 is an exemplary block diagram illustrating a virtual experience sharing system including a terminal incorporating a data replacement device according to an embodiment. FIG. 1 is an exemplary block diagram illustrating an example of a terminal in which a data replacement device according to an embodiment is incorporated. The figure which shows an example of a state of HMD, a controller, and a user. FIG. 5 is a diagram illustrating an avatar image of a user generated using first control data based on the position of the HMD and the controller in FIG. 4. The figure which shows another example of a state of HMD, a controller, and a user. FIG. 7 is a diagram illustrating an avatar image of a user generated using second control data obtained by replacing first control data based on the positions of the HMD and the controller in FIG. 6. The figure which shows another example of a state of HMD, a controller, and a user. 4 is a flowchart illustrating the operation of the terminal in FIG. 3. FIG. 4 is a block diagram showing a modification of FIG. 3.

  Hereinafter, an embodiment will be described with reference to the drawings. Hereinafter, the same or similar elements as those already described are denoted by the same or similar reference numerals, and duplicate description will be basically omitted. For example, when a plurality of identical or similar elements exist, a common reference numeral may be used to describe each element without distinction, and the common reference numeral may be used to distinguish and describe each element. In addition, branch numbers may be used.

(Embodiment)
The data replacement device according to the embodiment can be incorporated in a terminal configuring a virtual experience sharing system. Such a system is illustrated in FIG. In this system, the terminals 200 are connected to each other via a network such as the Internet, and can transmit and receive data.

  In the system shown in FIG. 2, a C / S (Client) in which terminals 200 transmit and receive data via a server for the purpose of reducing a delay associated with data transmission and sharing a virtual experience with high real-time properties. Instead of a / Server) network, a P2P (Peer to Peer) network in which terminals 200 directly transmit and receive data is adopted. However, even if a C / S type network is adopted, it is possible to construct a virtual experience sharing system using the terminal 200 incorporating the data replacement device according to the present embodiment.

  The terminal 200 is a computer configured to control a VR / AR / MR image displayed on an HMD (first device) 10 that can be worn on the head of the user, and that the HMD 10 and the user can hold. It is connected to a controller (second device) 20. Note that two (for two hands) or one (for one hand) controllers 20 may be prepared for each user. Further, the controller 20 may be mounted on a part other than the hand. Further, although not shown in FIG. 2, a base station which is one of the elements included in the position sensor system for detecting the position of the HMD 10 and / or the controller 20 may be further connected to the terminal 200.

  The terminal 200 may be connected to the HMD 10, the controller 20, and / or the base station by a wired communication means such as a USB (Universal Serial Bus) cable or an HDMI (registered trademark) (High-Definition Multimedia Interface) cable. The connection may be made by wireless communication means such as Bluetooth, WirelessHD, WHDI (Wireless Home Digital Interface), or the like.

  The terminal 200 acquires first control data having a value based on the position of the HMD 10 and the position of the controller 20 as described later, and replaces the first control data with the second control data conditionally. As described later, the second control data has a value that is not based on the position of at least one of the HMD 10 and the controller 20. Then, terminal 200 transmits the second control data to another terminal 200 when the replacement is performed, and transmits the first control data to another terminal 200 otherwise.

  The other terminal 200 that has received the first control data or the second control data generates an avatar image of the user based on the received control data. That is, the user can cause the avatar, which is his / her own, to take a free posture by moving the head holding the HMD 10 or the hand holding the controller 20.

  Further, the terminal 200 may transmit audio data generated by a microphone described later to another terminal 200. Other terminals 200 reproduce and output the audio data. This makes it possible to realize a realistic event such as a voice chat between users or a live event in the VR / AR / MR space.

  Specifically, the other terminal 200 determines the position of the head of the avatar image according to the position of the HMD 10 indicated by the received control data, and determines the position of the head of the avatar image according to the position of the controller 20 indicated by the control data. Determine the position of For example, in a state where the user wears the HMD 10 on the head and holds the controller 21 (for the right hand) and the controller 22 (for the left hand) (both of which may be the same as the controller 20 described above) in both hands, FIG. Assume that the user takes the posture illustrated in FIG. In this case, the user's avatar takes a posture illustrated in FIG. For controlling the posture of the avatar, for example, IK (Inverse Kinematics) technology can be used.

  The HMD 10 may include various sensors, speakers, microphones, and the like, in addition to a display device for displaying VR / AR / MR images including avatar images. The various sensors may include motion sensors, mounting sensors, or some of the elements included in the position sensor system (markers or cameras described below). The HMD 10 receives image data and audio data from the terminal 200, outputs them, and transmits sensor data of various sensors to the terminal 200.

  The display device may be a transmissive display or a non-transmissive display. For example, the size and arrangement of the display device are determined so as to cover at least a part of the field of view of the user wearing the HMD 10. The display device may be composed of a left-eye display device and a right-eye display device, or both may be integrated.

  The motion sensor can be, for example, an acceleration sensor, a gyroscope, a magnetic sensor, and the like. Sensor data detected by the motion sensor can be used for estimating the posture (eg, inclination) of the user's head. Specifically, based on the sensor data, a Yaw angle, a Roll angle, and a Pitch angle, which are three-dimensional rotation angles of the user's head, are estimated, and a virtual camera that determines the field of view of the user accordingly. Of the avatar and the attitude of the avatar's head can be controlled.

  The mounting sensor generates sensor data (event data) indicating that the user has mounted / removed the HMD 10. The mechanism of the wearing sensor is arbitrary, but for example, changes in the elastic force of a spring provided in the HMD 10, changes in the current flowing between pads that contact a part of the body such as the user's nose when the HMD 10 is worn, and the like. Event data can be generated accordingly.

  The speaker receives audio data from terminal 200 and outputs audio based on the audio data. The speaker is typically a headphone type, but may be configured as another speaker system. Further, the speaker may be separate from the HMD 10. The microphone collects voice (mainly a user's utterance). The microphone generates audio data based on the collected audio and sends the audio data to terminal 200.

  The controller 20 may include, for example, a motion sensor similar to the HMD 10, a part of elements included in the position sensor system, and the like, in addition to a button for receiving a user input. Further, the controller 20 may include a grip sensor that generates sensor data (event data) indicating that the user has gripped / released the controller 20. The mechanism of the grip sensor is optional, but may be determined according to, for example, a change in pressure applied to the controller 20 due to a user's grip force, a change in capacitance between a sensor electrode provided on the surface of the controller 20 and a human body, and the like. Event data. The controller 20 transmits user input data, sensor data of various sensors, and the like to the terminal 200.

  The position sensor system is realized by, for example, a combination of a camera (position tracking camera) and a marker (also called a tracker). The marker may be an emitter of infrared light or visible light (for example, an LED (Light Emitting Diode)) or a reflecting material for reflecting infrared light or visible light emitted from the emitter when photographing with a camera. May be. The camera can be an infrared sensor if the marker emits or reflects infrared light, or a visible light camera if the marker emits or reflects visible light.

  Typically, a plurality of markers are attached to the HMD 10 / controller 20, and the camera is attached to a device (base station) installed at a position remote from the HMD 10 / controller 20. The position of the HMD 10 / controller 20 can be estimated based on the image captured by the camera. Specifically, the base station can detect the position, inclination, emission intensity, and the like of the detection point based on the captured image. The base station may calculate the position (coordinate) data of the HMD 10 / controller 20 based on the data of the detection point, or the calculation may be left to the terminal 200. As a modification, it is also possible to provide a camera on the HMD 10 / controller 20 side and a marker on the base station side. Further, in addition to the markers attached to the HMD 10 and the controller 20, further markers may be attached to the user's joints, erasures, and the like. This makes it possible to more accurately estimate the posture of the user.

  Hereinafter, the hardware configuration of terminal 200 will be described. The terminal 200 is a variety of electronic devices that can be a control device of the HMD 10, for example, a PC (Personal Computer), a mobile terminal (for example, a tablet, a Fablet, a smartphone, a laptop, a feature phone, a wearable device, a portable game machine, and the like). , A stationary game machine, or the like, but is not limited thereto.

  Note that the terminal 200 does not necessarily need to be separate from the HMD 10 or another device (the controller 20, the base station, or the like). For example, the terminal 200 may be built in the HMD 10 or the base station, or the terminal 200 may be handled as the controller 20.

  The terminal 200 includes a processor that performs processing as a data replacement device according to the embodiment, input / output control, communication control, image / audio processing, and the like. Here, the processor is typically a CPU (Central Processing Unit) and / or a GPU (Graphics Processing Unit), but is a microcomputer, an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), or other general-purpose. Alternatively, a dedicated processor may be used.

  In addition, the terminal 200 includes a program executed by a processor to realize such processing and data used by the processor, such as an avatar, a background, and various objects for producing a VR / AR / MR experience. A memory for temporarily storing the base image data is included. The memory may include a RAM (Random Access Memory) having a work area in which the program / data is expanded.

  Note that the terminal 200 may handle all data in an on-memory state, or some data may be saved in the auxiliary storage device. The auxiliary storage device may be, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like built in or external to the terminal 200, or a database server accessible from the terminal 200. May be.

  The terminal 200 can further use a communication I / F (interface) for connecting to a network. The communication I / F may be built in the terminal 200 or may be externally attached to the terminal 200. The communication I / F is a module for communicating with another terminal 200 and / or an external device, for example, the HMD 10, the controller 20, a base station, and the like, and includes a signal processing circuit for transmission and reception, an antenna, It may include a LAN (Local Area Network) terminal or the like. The communication I / F may be, for example, a module for wide area communication such as mobile communication, a module for wireless / wired LAN, a module for Bluetooth (registered trademark), or the like.

  The terminal 200 can further use an input / output I / F for connecting a cable to an external device, for example, the HMD 10, the controller 20, the base station, or the like. The input / output I / F is a USB terminal, a DVI (Digital Visual Interface) terminal, an HDMI terminal, or a terminal for other data transfer cables.

  The terminal 200 may further include a bus for transferring data between each element, for example, a processor, a memory, an auxiliary storage device, a communication I / F, an input / output I / F, and the like.

  Subsequently, a data replacement device according to the embodiment will be described. FIG. 1 illustrates a functional configuration of a data replacement device 100 according to the embodiment. The data replacement device 100 includes a data acquisition unit 101, a state determination unit 102, and a data replacement unit 103.

  The data acquisition unit 101 can be realized by, for example, the above-described communication I / F, input / output I / F, or processor. The data acquisition unit 101 acquires the first control data and sends it to the state determination unit 102. Here, the first control data has a value based on the position of the HMD 10 and the position of the controller 20. As described above, the position of the HMD 10 / controller 20 can be estimated using a position sensor system.

  When an external device, for example, the HMD 10, the controller 20, and / or the base station estimates the position of the HMD 10 / controller 20 and generates the first control data, the data acquisition unit 101 performs the first control from the external device. You just need to get the data. On the other hand, the terminal 200 (a position estimating unit (not shown) of the terminal 200) estimates the position of the HMD 10 / controller 20 based on the output data of the position sensor system (for example, data indicating the position, inclination, emission intensity, etc. of the detection point). When generating the first control data, the data acquisition unit 101 may acquire the first control data from the memory of the terminal 200 or the auxiliary storage device.

  In the above description, it is assumed that the data replacement device 100 performs the replacement conditionally before transmitting the first control data to another terminal 200. However, as will be described in a modified example 1 described below, the terminal 200 transmits its own first control data to another terminal 200 without replacing it, and the data replacement device 100 incorporated in the other terminal 200 receives the first control data. It is also possible to replace the first control data with a condition. Assuming the latter configuration, the data acquisition unit 101 acquires the first control data received via the network.

  When the HMD 10 / controller 20 includes the above-described mounting sensor / gripping sensor, the data acquisition unit 101 may further acquire event data from the mounting sensor / gripping sensor and send the event data to the state determination unit 102.

  The state determination unit 102 can be realized by, for example, the processor described above. The state determination unit 102 receives the first control data (and the event data) from the data acquisition unit 101. The state determination unit 102 determines whether a predetermined replacement start condition is satisfied with respect to the first control data. Then, the state determination unit 102 sends the first control data to the data replacement unit 103 together with the data indicating the determination result.

  Here, the state determination unit 102 may determine whether the replacement start condition is satisfied based on the first control data, or may determine whether the replacement start condition is satisfied based on the event data. May be. The replacement start condition can be defined to be satisfied, for example, when the avatar is expected to take an unnatural posture when controlling the avatar faithfully according to the first control data.

  Specifically, the state determination unit 102 may determine whether the HMD 10 is worn on the head of the user as an example of the replacement start condition. When the HMD 10 is removed from the user's head, the position of the HMD 10 is no longer linked to the position of the user's head, and is extremely low or high, or extremely high from the position of the user's hand (more precisely, the controller 20). May be moved to remote locations. In such a case, if the position of the avatar's head is determined according to the position of the HMD 10, the avatar's posture may be unnatural.

  The state determination unit 102 may determine whether or not the HMD 10 is worn on the user's head, based on event data output from a wearing sensor when a mounting / removing event of the HMD 10 occurs, for example.

  In addition, the user may lift the HMD 10 to a position significantly higher than the head when removing the HMD 10 from the head. Therefore, as an example of the replacement start condition, the state determination unit 102 may determine whether the first control data has a value indicating that the height of the HMD 10 is greater than or equal to the predetermined first height. Good.

  The first height may mean an upper limit of the head height required for the avatar's posture not to be defined as unnatural. The first height may be determined for each user, or may be determined for each virtual experience sharing system. As an example, the first height may be the height of an individual user or an average user's head plus an offset, and the like.

  In addition, after removing the HMD 10 from the head, the user may place the HMD 10 on a lowland such as a floor. In such a case, if the position of the avatar's head is determined according to the position of the HMD 10, the avatar may take a posture such as falling down with the head attached to a lowland, for example.

  Therefore, the state determination unit 102 may determine whether the first control data has a value indicating that the height of the HMD 10 is less than the predetermined second height as an example of the replacement start condition. Good. In the example of FIG. 6, since the second height is set to H0 and the height of the HMD 10 is lower than H0, the state determination unit 102 determines that the replacement start condition is satisfied.

  The second height may mean a lower limit of the head height required for the avatar's posture not to be defined as unnatural. The second height may be determined for each user, or may be determined for each virtual experience sharing system. By way of example, the second height may be the waist height, knee height, head height with forward flexion, etc. of the individual or average user.

  In addition, the replacement start condition may be that the first control data has a value indicating that the height of the HMD 10 is equal to or greater than the first height or less than the second height. In the example of FIG. 8, the first height and the second height are set to H1 and H2, respectively, and the height of the HMD 10 is equal to or higher than H1, so that the state determination unit 102 satisfies the replacement start condition. Then, it is determined. Accordingly, when the height of the HMD 10 does not fall within the range from the lower limit value to the upper limit value, the data is replaced to prevent the avatar's posture from collapsing.

  Also, the user may have released the controller 20 to free his hands before removing the HMD 10 from the head. Therefore, as an example of the replacement start condition, the state determination unit 102 determines whether the first control data has a value indicating that the controller 20 is stationary, in addition to any of the above-described conditions regarding the height of the HMD 10. It may be determined whether or not. Whether or not the controller 20 is stationary can be determined based on, for example, a history of the position of the controller 20 in the first control data, output data of a motion sensor or a grip sensor of the controller 20, and the like.

  Furthermore, as long as the user wears the HMD 10 on the head and holds the controller 20, the distance between the two does not become extremely large. In other words, when the distance between the HMD 10 and the controller 20 is extremely large, there is a possibility that at least one of the HMD 10 and the controller 20 is not worn by the user. Therefore, in such a case, if the avatar is faithfully controlled based on the position data of the HMD 10 / controller 20, the avatar may take an unnatural posture. Therefore, the state determination unit 102 determines, as an example of the replacement start condition, whether the first control data has a value indicating that the distance between the HMD 10 and the controller 20 is greater than or equal to a predetermined distance. Is also good.

  The predetermined distance may mean an upper bound on the head-hand distance required for the avatar's posture not to be defined as unnatural. The predetermined distance may be determined for each user or for each virtual experience sharing system. As an example, the predetermined distance may be determined based on the distance from the user's personal or average user's head to hand. Specifically, the predetermined distance is a posture in which the distance from the head to the hand is the largest, for example, a distance obtained by adding an offset to the distance from the head to the hand of the user when a careful posture is taken, And so on.

  After once determining that the replacement start condition is satisfied, the state determination unit 102 determines whether a predetermined replacement end condition is satisfied. After once determining that the replacement end condition is satisfied, the state determination unit 102 determines again whether the replacement start condition is satisfied. The replacement end condition can be defined to be satisfied, for example, when the avatar is expected not to take an unnatural posture even if the avatar is controlled faithfully according to the first control data. By defining the replacement end condition in this way, after changing the avatar from a natural posture (for example, upright) to a dummy posture (for example, bending), the same or different natural posture is again performed without sandwiching an unnatural posture. (For example, stand up and wave your hand).

  The replacement end condition may typically correspond to a negative of the replacement start condition, that is, that the replacement start condition is not satisfied. For example, when the replacement start condition is “the HMD 10 is mounted on the user's head”, the replacement end condition may be determined as “the HMD 10 is not mounted on the user's head”.

  However, the replacement end condition may not correspond to the negation of the replacement start condition. Specifically, the replacement start condition is “the first control data has a value that indicates that the height of the HMD 10 is equal to or greater than the first height and that the controller 20 is stationary. , The replacement end condition indicates that “the first control data indicates that the height of the HMD 10 is less than the first height (and not less than the second height), and that the controller 20 is stationary. To have a value indicating that it has not been done. "

  The data replacement unit 103 can be realized by the above-described processor. When it is determined that the replacement start condition is satisfied, the data replacement unit 103 replaces the first control data with the second control data, and outputs the second control data to the outside of the data replacement device 100. On the other hand, when it is not determined that the replacement start condition is satisfied, the data replacement unit 103 outputs the first control data to the outside of the data replacement device 100 as it is. Note that, once the data replacement is started, the data replacement unit 103 continues the data replacement until the state determination unit 102 determines that the replacement end condition is satisfied. Here, the second control data is dummy data for preventing the avatar from taking an unnatural posture, and a specific example thereof will be described below.

  For example, if the replacement start condition is that the HMD 10 is worn on the head of the user or the height of the HMD 10, the second control data has a value that is not based on the height of the HMD 10. obtain. Specifically, the second control data indicates that the height of the HMD 10 is less than the above-described first height, greater than or equal to the above-mentioned second height, or less than the first height, and It may have a value indicating that the height is greater than or equal to the second height. Thereby, for example, even when the height of the HMD 10 is less than the second height H0 as in the example of FIG. 6, the head of the avatar does not lower unnaturally, but buckles as shown in FIG. 7, for example. You can make your avatar take such a posture. In addition to the bent posture, the avatar may be placed in various postures such as sitting, kneeling, standing upright, etc., depending on the algorithm for controlling the posture of the avatar and the height of the HMD 10 indicated by the second control data. .

  In addition, when the replacement start condition is related to the distance between the HMD 10 and the controller 20, the second control data may have a value that is not based on at least one of the positions of the HMD 10 and the controller 20. Specifically, the second control data may have a value indicating that the distance between the HMD 10 and the controller 20 is less than a predetermined distance.

  When the user does not wear the HMD 10, the HMD 10 can be expected to be stationary, and when the user does not hold the controller 20, the controller 20 can be expected to be stationary. Therefore, the second control data may have a value that is not based on the position of one or both of the HMD 10 and the controller 20 at rest. In other words, in the second control data, the position of the non-stationary device (non-stationary device) of the HMD 10 and the controller 20 in the first control data is maintained, and the position of the non-stationary device is the position of the non-stationary device. Can be rewritten to a position that is less than the predetermined distance from. Whether the HMD 10 / controller 20 is stationary or not is determined based on, for example, the history of the position of the HMD 10 / controller 20 in the first control data, output data of the motion sensor or the mounting sensor / grasping sensor of the HMD 10 / controller 20, and the like. Can be determined.

  Next, a terminal 200 incorporating the data replacement device 100 will be described. FIG. 3 illustrates a functional configuration of terminal 200 incorporating data replacement device 100. The terminal 200 includes the data replacement device 100, a transmission unit 201, a reception unit 211, and an image generation unit 212.

  The transmission unit 201 can be realized by the communication I / F described above. The transmission unit 201 receives output data of the data replacement device 100 and transmits the output data to another terminal 200 as a communication partner via a network. Here, as described above, the output data of the data replacement device 100 is the second control data when the data replacement is performed, and is the first control data otherwise.

  Note that the transmitting unit 201 may receive audio data generated by a microphone in addition to the first or second control data, and transmit this to another terminal 200 that is a communication partner via a network.

  The receiving unit 211 can be realized by the above-described communication I / F. The receiving unit 211 receives control data for controlling an avatar of a user of the other terminal 200 from another terminal 200 to be a communication partner via a network. The control data to be received may be the first control data or the second control data, but any of them does not affect the subsequent processing. The receiving unit 211 sends the control data to the image generating unit 212.

  The receiving unit 211 can receive audio data in addition to the control data. The receiving unit 211 may send the audio data to an audio processing unit (not shown). The audio processing unit reproduces audio data and causes the above-described speaker to output audio.

  The image generation unit 212 can be realized by the processor described above. The image generation unit 212 controls the avatar image of the user corresponding to the control data based on the image data of the object stored in the memory or the auxiliary storage device of the terminal 200 and the control data from the reception unit 211, for example. Is generated (updated). The control data here corresponds to the second control data when the data is replaced, and otherwise corresponds to the first control data. The image generation unit 212 sends the generated image data to the HMD 10.

  As described above, the image generation unit 212 determines the position of the head / hand of the avatar image according to the position of the HMD 10 / controller 20 indicated by the control data, and controls the avatar's posture using, for example, IK technology. May be.

  Hereinafter, the operation of terminal 200 will be described with reference to FIG. The operation of FIG. 9 is performed every time the first control data is generated.

  First, the data acquisition unit 101 in the data replacement device 100 acquires the first control data (Step S301). Then, the state determination unit 102 in the data replacement device 100 determines whether a predetermined condition is satisfied (step S302). If the predetermined condition is satisfied, the process proceeds to step S303; otherwise, the process skips step S303 and proceeds to step S304.

  Note that the predetermined condition in step S302 means a replacement start condition when the replacement start condition is never satisfied, and is a period from when the replacement start condition is satisfied to when the replacement end condition is satisfied. This means a condition equivalent to negation of the replacement end condition, and may mean the replacement start condition from when the replacement end condition is satisfied until the replacement start condition is satisfied again.

  In step S303, the data replacement unit 103 in the data replacement device 100 replaces the first control data with second control data having a value that is not based on at least one of the positions of the HMD 10 and the controller 20.

  Next, the transmitting unit 201 transmits the control data (step S304), and the operation in FIG. 9 ends. When step S303 is performed before step S304, transmitting section 201 transmits the second control data. On the other hand, when step S303 is skipped before step S304, transmitting section 201 transmits the first control data.

  As described above, the data replacement device according to the embodiment converts the first control data having a value based on the position of the HMD and the controller to at least one of the HMD and the controller when the predetermined replacement start condition is satisfied. Replace with the second control data having a value not based on the position. Therefore, according to this data replacement device, for example, if the avatar may take an unnatural posture when the avatar is faithfully controlled in accordance with the first control data, a dummy second data may be used instead of the first control data. The avatar can be controlled using the second control data. That is, irrespective of the actual positions of the HMD and the controller, the posture of the avatar is maintained so as not to be unnatural, so that it is possible to prevent at least deterioration of the user's virtual experience due to unnaturalness of the avatar image. .

(Modification 1)
In the above-described embodiment, the terminal that is the source of the first control data replaces the data for the first control data conditionally. However, data replacement for the first control data may be performed in a terminal that is a destination instead of a source of the first control data. A terminal 400 of such a modification is illustrated in FIG.

  The terminal 400 includes a data acquisition unit 402, a transmission unit 201, a reception unit 211, the data replacement device 100, and an image generation unit 212. Here, since the transmission unit 201, the reception unit 211, the data replacement device 100, and the image generation unit 212 are the same or similar to the elements having the same names in FIG. 3, in the following description, the difference between them and the data acquisition unit 402 will be described. I will explain mainly.

  The data acquisition unit 402 can be realized by, for example, the above-described communication I / F, input / output I / F, or processor. The data acquisition unit 402 acquires the first control data and sends it to the transmission unit 201. As described above, the position of the HMD 10 / controller 20 can be estimated using a position sensor system.

  When an external device, for example, the HMD 10, the controller 20, or the base station estimates the position of the HMD 10 / controller 20 and generates the first control data, the data acquisition unit 402 outputs the first control data from the external device. You only need to get it. On the other hand, the terminal 200 (a position estimating unit (not shown) of the terminal 200) estimates the position of the HMD 10 / controller 20 based on the output data of the position sensor system (for example, data indicating the position, inclination, emission intensity, etc. of the detection point). When generating the first control data, the data acquisition unit 402 may acquire the first control data from the memory of the terminal 200 or the auxiliary storage device.

  When the HMD 10 / controller 20 includes the above-described mounting sensor / gripping sensor, the data acquisition unit 402 may further acquire event data from the mounting sensor / gripping sensor and transmit the event data to the transmission unit 201. In this case, the transmitting unit 201 may transmit event data to the terminal 400 in addition to the first control data, in order to determine a predetermined condition in the terminal 400 as the destination. Further, receiving section 211 can also receive event data in addition to the first control data transmitted from another terminal 400 and send the data to data replacement apparatus 100.

  As described above, in this modified example, the data replacement process is performed in a distributed manner in the terminal that is the destination instead of the source of the first control data. Therefore, according to this modification, the load on the transmission source can be reduced.

(Modification 2)
In the above embodiment, each terminal transmits data to each other via a P2P type network. On the other hand, each terminal can also transmit data to each other via a C / S type network. Then, in this case, the data replacement device 100 may be incorporated in the server. Then, the server does not simply relay the first control data between the terminals, but conditionally replaces the first control data received from the terminal that is the transmission source, and then performs the first Alternatively, the second control data may be transmitted to the destination terminal.

  By adopting a C / S type network as in this modification, the transmission delay of control data may be larger than when a P2P type network is employed, but the data replacement processing is concentrated on the server. Therefore, the load on each terminal can be reduced.

  The embodiments described above merely show specific examples to help understand the concept of the present invention, and are not intended to limit the scope of the present invention. In the embodiment, various components can be added, deleted, or changed without departing from the gist of the present invention.

  In the above embodiments, some functional units have been described, but these are merely examples of implementation of each functional unit. For example, a plurality of functional units described as being mounted on one device may be mounted on a plurality of separate devices, and conversely, a description may be made on a plurality of separate devices. The function unit described above may be mounted on one device.

  The various functional units described in each of the above embodiments may be realized by using a circuit. The circuit may be a dedicated circuit for realizing a specific function or a general-purpose circuit such as a processor.

  At least a part of the processing of each of the above embodiments can also be realized by using, for example, a processor mounted on a general-purpose computer as basic hardware. A program for realizing the above processing may be provided by being stored in a computer-readable recording medium. The program is stored in a recording medium as a file in an installable format or a file in an executable format. The recording medium is a magnetic disk, an optical disk (CD-ROM, CD-R, DVD, etc.), a magneto-optical disk (MO, etc.), a semiconductor memory, or the like. The recording medium may be any as long as it can store a program and can be read by a computer. Further, a program for implementing the above processing may be stored on a computer (server) connected to a network such as the Internet, and downloaded to a computer (client) via the network.

10 ... HMD
20, 21, 22 ... controller 100 ... data replacement device 101, 402 ... data acquisition unit 102 ... state determination unit 103 ... data replacement unit 200 ... terminal 201 ... transmission unit 211: receiving unit 212: image generating unit

Claims (14)

An acquisition unit configured to acquire first control data having a value based on a position of a first device that can be mounted on a user's head and a position of a second device that can be gripped by the user;
A determining unit that determines whether a predetermined condition is satisfied;
And a replacement unit that replaces the first control data with the second control data when it is determined that the condition is satisfied,
The second control data has a value that is not based on a position of at least one of the first device and the second device;
Data replacement device. The determination unit determines whether the first device is worn on the head of the user,
The replacement unit is configured to, when it is determined that the first device is not mounted on the head of the user, change the first control data to a value that is not based on the height of the first device. Replace with 2 control data,
The data replacement device according to claim 1. The determination unit determines whether the first control data has a value indicating that the height of the first device is less than a predetermined height,
The replacement unit, when it is determined that the first control data has a value indicating that the height of the first device is less than the predetermined height, the first control data to the first control data Replacing the second control data with a value that is not based on the height of the first device;
The data replacement device according to claim 1.   The data replacement device according to claim 2, wherein the second control data has a value indicating that the height of the first device is equal to or greater than the predetermined height. The determination unit determines whether or not the first control data has a value indicating that the height of the first device is equal to or greater than a predetermined first height or less than a predetermined second height. Judge,
The replacement unit is configured to determine whether the first control data has a value indicating that the height of the first device is equal to or greater than the first height or less than the second height. Replacing the first control data with second control data having a value that is not based on the height of the first device;
The second height is lower than the first height;
The data replacement device according to claim 1. The determination unit indicates that the first control data indicates that the height of the first device is greater than or equal to a predetermined first height or less than a predetermined second height, and Determine whether the device has a value indicating that it is stationary,
The replacement unit indicates that the first control data indicates that the height of the first device is equal to or greater than the first height or less than the second height, and the second device When it is determined that the first control data has a value indicating that it is stationary, the first control data is replaced with second control data having a value that is not based on the height of the first device;
The second height is lower than the first height;
The data replacement device according to claim 1. The determination unit indicates that the first control data indicates that the height of the first device is greater than or equal to the first height or less than the second height, and the second device After determining that the first device has a value indicating that it is stationary, the first control data further indicates that the height of the first device is less than the first height and greater than or equal to the second height. And whether or not the second device has a value indicating that it is not stationary,
The replacement unit indicates that the first control data indicates that the height of the first device is equal to or greater than the first height or less than the second height, and the second device Is determined to have a value indicating that the device is stationary, the first control data indicates that the height of the first device is less than the first height and greater than or equal to the second height. And that the first control data is replaced with the second control data until it is determined that the second device has a value indicating that the second device is not stationary.
The data replacement device according to claim 6.   8. The method according to claim 5, wherein the second control data has a value indicating that the height of the first device is less than the first height and equal to or more than the second height. 9. 2. The data replacement device according to claim 1.   The data replacement device according to claim 5, wherein the first height is equal to or greater than a height of the head of the user. The determination unit determines whether the first control data has a value indicating that the distance between the first device and the second device is equal to or greater than a predetermined distance,
The replacement unit, when it is determined that the first control data has a value indicating that the distance between the first device and the second device is greater than or equal to a predetermined distance, the Replacing the first control data with second control data having a value that is not based on the position of at least one of the first device and the second device;
The data replacement device according to claim 1.   The second control data has a value that is not based on the position of the first device when the first control data has a value indicating that the first device is stationary, The data replacement device according to claim 10, wherein when the first control data has a value indicating that the second device is stationary, the data replacement device has a value that is not based on the position of the second device.   The data according to claim 10 or 11, wherein the second control data has a value indicating that a distance between the first device and the second device is less than the predetermined distance. Replacement device. A data replacement device according to any one of claims 1 to 12,
An image generation unit that generates an avatar image of the user based on the second control data when the first control data is replaced by the second control data. Computer
Means for acquiring first control data having a value based on a position of a first device that can be mounted on a user's head and a position of a second device that can be gripped by the user;
Means for determining whether a predetermined condition is satisfied,
Functioning as a means for replacing the first control data with second control data when it is determined that the condition is satisfied;
The second control data has a value that is not based on a position of at least one of the first device and the second device;
Data replacement program.