JP7111645B2 - control program - Google Patents

Description

The present invention relates to VR (Virtual Reality), AR (Augmented Reality), or MR (Mixed Reality) technology.

2. Description of the Related Art Conventionally, it is known to share a virtual experience, that is, a VR experience, via a network among a plurality of users wearing HMDs (Head Mounted Displays). Specifically, it is known to control the posture of an avatar, which is the alter ego of the user in a virtual space, according to the user's posture (eg, line of sight, head orientation, etc.).

According to Patent Document 1, when the first user removes the HMD, the second user does not know that the first user is not wearing the HMD, so the first avatar (first user) does not react. ([0004]), and provide the user with a rich virtual experience ([0005]). Specifically, in Patent Document 1, based on the tilt information of the HMD 110, it is determined whether or not the user A has removed the HMD 110 [0095], and when it is determined that the user A has removed the HMD 110, non-wearing information is generated. [0100], and after receiving the non-wearing information, the user terminal 1B sets the facial expression of the avatar 4A to the default facial expression [0101]. It also describes determining whether or not the user A has worn the HMD 110 again [0113], and updating the facial expression of the avatar 4A when the HMD 110 has been worn [0115].

Japanese Patent No. 6212666

When controlling the posture of the user's avatar in the virtual space according to the user's posture, the avatar may take a posture unintended by the user, such as an unnatural posture. In particular, when the HMD displays a virtual space image from a first-person viewpoint, the user cannot directly see the appearance of his/her own avatar, making it difficult to perceive anomalies and attempt corrections.

If a user's avatar takes an unnatural posture, not only will the user themselves feel embarrassed or uncomfortable when they realize this fact, but other users will also feel that the virtual space including the avatar is not perfect. There is a possibility that it will make you lose interest because it is low. This can occur not only in VR experiences but also in general experiences including virtual elements such as AR or MR (hereinafter referred to as virtual experiences).

An object of the present invention is to prevent deterioration of a user's virtual experience due to avatar images.

A data replacement device according to an aspect of the present invention includes an acquisition unit, a determination unit, and a replacement unit. The acquisition unit acquires first control data having a value based on the position of a first device that can be worn on the user's head and the position of a second device that can be held by the user. A determination unit determines whether or not a predetermined condition is satisfied. The replacement unit replaces the first control data with the second control data when it is determined that the condition is satisfied. The second control data has a value that is not based on the location of at least one of the first device and the second device.

SUMMARY OF THE INVENTION It is an object of the present invention to prevent deterioration of a user's virtual experience due to avatar images.

1 is a block diagram illustrating a data replacement device according to an embodiment; FIG. 1 is a block diagram illustrating a virtual experience sharing system including a terminal incorporating a data replacement device according to an embodiment; FIG. 1 is a block diagram showing an example of a terminal incorporating a data replacement device according to an embodiment; FIG. The figure which shows an example of the appearance of HMD, a controller, and a user. FIG. 5 is a diagram exemplifying a user's avatar image generated using first control data based on the positions of the HMD and controller of FIG. 4; The figure which shows another example of the appearance of HMD, a controller, and a user. FIG. 7 is a diagram exemplifying a user's avatar image generated using second control data that replaces first control data based on the positions of the HMD and controller of FIG. 6 ; The figure which shows another example of the appearance of HMD, a controller, and a user. 4 is a flowchart illustrating the operation of the terminal of FIG. 3; FIG. 4 is a block diagram showing a modification of FIG. 3;

An embodiment will be described below with reference to the drawings. Elements that are the same as or similar to elements that have already been explained are denoted by the same or similar reference numerals, and overlapping explanations are basically omitted. For example, when there are a plurality of identical or similar elements, common reference numerals may be used to describe each element without distinction, and the common reference numerals may be used to distinguish and describe each element. In addition, branch numbers are sometimes used.

(embodiment)
A data replacement device according to an embodiment can be incorporated in a terminal that constitutes a virtual experience sharing system. Such a system is illustrated in FIG. In this system, terminals 200 are connected to each other via a network such as the Internet, and can transmit and receive data.

In the system of FIG. 2, the terminals 200 communicate with each other via a server for the purpose of reducing the delay associated with data transmission and sharing a highly real-time virtual experience. A P2P (Peer to Peer) type network in which the terminals 200 directly transmit and receive data is adopted instead of a type network of the P2P (Peer to Peer) type. However, even if a C/S type network is adopted, it is possible to construct a virtual experience sharing system using the terminal 200 incorporating the data replacement device according to this embodiment.

The terminal 200 is a computer configured to control VR/AR/MR images displayed on an HMD (first device) 10 that can be worn on the user's head. are connected to a controller (second device) 20 respectively. Two controllers 20 (for both hands) or one controller 20 (for one hand) may be prepared for each user. Also, the controller 20 may be worn on a part other than the hand. Furthermore, although not shown in FIG. 2, a base station, which is one of the elements included in the position sensor system for detecting the position of HMD 10 and/or controller 20, may be further connected to terminal 200.

The terminal 200 may be connected to the HMD 10, the controller 20, and/or the base station by wired communication means such as a USB (Universal Serial Bus) cable, HDMI (registered trademark) (High-Definition Multimedia Interface) cable, or the like. Alternatively, they may be connected by wireless communication means such as Bluetooth, WirelessHD, and WHDI (Wireless Home Digital Interface).

The terminal 200 acquires first control data having a value based on the position of the HMD 10 and the position of the controller 20 as described later, and conditionally replaces the first control data with the second control data. As will be described later, the second control data has a value that is not based on the position of at least one of HMD 10 and controller 20 . Then, the terminal 200 transmits the second control data to the other terminal 200 when the replacement is performed, and transmits the first control data to the other terminal 200 otherwise.

Other terminals 200 that have received the first control data or the second control data generate user avatar images based on the received control data. That is, the user can make the avatar, which is the alter ego of the user, take a free posture by moving the head on which the HMD 10 is mounted or the hand holding the controller 20 .

Furthermore, the terminal 200 may transmit audio data generated by a microphone, which will be described later, to another terminal 200 . Other terminals 200 reproduce and output this audio data. As a result, it is possible to realize a realistic event such as a voice chat between users or a live event in the VR/AR/MR space.

Specifically, the other terminal 200 determines the position of the head of the avatar image according to the position of the HMD 10 indicated by the received control data, and moves the hands of the avatar image according to the position of the controller 20 indicated by the control data. determine the position of For example, when the user wears the HMD 10 on the head and holds the (right hand) controller 21 and the (left hand) controller 22 (both of which can be the same as the controller 20 described above) in both hands, FIG. Suppose that the posture illustrated in is taken. In this case, the user's avatar will take the pose illustrated in FIG. IK (Inverse Kinematics) technology, for example, can be used to control the posture of the avatar.

The HMD 10 may include various sensors, speakers, microphones, etc. in addition to a display device for displaying VR/AR/MR images including avatar images. The various sensors may include motion sensors, wearable sensors, or some of the elements included in the position sensor system (markers or cameras, discussed below). The HMD 10 receives image data and audio data from the terminal 200 and outputs them and transmits sensor data from various sensors to the terminal 200 .

The display device may be a transmissive display or a non-transmissive display. For example, the size and arrangement of the display device are determined so as to cover at least part of the field of view of the user wearing the HMD 10 . The display device may be composed of a left-eye display device and a right-eye display device, or both may be integrated.

Motion sensors can be, for example, accelerometers, gyroscopes, magnetic sensors, and the like. Sensor data detected by the motion sensor can be used to estimate the pose (eg, tilt) of the user's head. Specifically, based on this sensor data, the Yaw angle, Roll angle, and Pitch angle, which are the three-dimensional rotation angles of the user's head, are estimated, and the virtual camera determines the user's field of view accordingly. , and the pose of the avatar's head can be controlled.

The wearing sensor generates sensor data (event data) indicating that the user has worn/removed the HMD 10 . Although the mechanism of the wearing sensor is arbitrary, for example, changes in the elastic force of the springs provided in the HMD 10, changes in current flowing between pads that contact a part of the body such as the user's nose when the HMD 10 is worn, etc. Event data can be generated accordingly.

The speaker receives audio data from the terminal 200 and outputs audio based on this. The speaker is typically of the headphone type, but may be configured as a speaker system other than this. Also, the speaker may be separate from the HMD 10 . A microphone collects sound (mainly user speech). The microphone generates audio data based on the collected audio and sends it to terminal 200 .

The controller 20 may include, for example, a motion sensor similar to the HMD 10, some of the elements included in the position sensor system, etc., in addition to buttons for accepting user input. Further, the controller 20 may include a grip sensor that generates sensor data (event data) indicating that the user has gripped/released the controller 20 . The mechanism of the grip sensor is arbitrary. can generate event data. The controller 20 transmits user input data, sensor data from various sensors, and the like to the terminal 200 .

A position sensor system is realized, for example, by a combination of a camera (position tracking camera) and a marker (also called a tracker). The marker may be an emitter of infrared light or visible light (for example, an LED (Light Emitting Diode)), or a reflective material for reflecting the infrared light or visible light emitted from the emitter during photographing by a camera. may The camera can be an infrared sensor if the marker emits or reflects infrared light or a visible light camera if the marker emits or reflects visible light.

Typically, a plurality of markers are attached to the HMD 10/controller 20, and a camera is attached to a device (base station) installed at a position distant from the HMD 10/controller 20. FIG. The position of the HMD 10/controller 20 can be estimated based on the image captured by the camera. Specifically, the base station can detect the position, tilt, luminescence intensity, etc. of the detection point based on the captured image. The base station may calculate the position (coordinate) data of the HMD 10/controller 20 based on the data of such detection points, or such calculation may be entrusted to the terminal 200 . As a modification, it is also possible to provide the camera on the HMD 10/controller 20 side and the marker on the base station side. Also, in addition to the markers attached to the HMD 10 and controller 20, additional markers may be attached to the user's joints, peripheral portions, and the like. This makes it possible to more accurately estimate the posture of the user.

The hardware configuration of terminal 200 will be described below. The terminal 200 is various electronic devices that can serve as a control device for the HMD 10, such as PCs (Personal Computers), mobile terminals (eg tablets, phablets, smartphones, laptops, feature phones, wearable devices, portable game machines, etc.). , a stationary game machine, etc., but is not limited to these.

Note that the terminal 200 does not necessarily have to be separate from the HMD 10 or other devices (controller 20, base station, etc.). For example, the terminal 200 may be built in the HMD 10 or the base station, or the terminal 200 may be treated as the controller 20.

The terminal 200 includes a processor that performs processing, input/output control, communication control, image/audio processing, etc. as a data replacement device according to the embodiment. Here, the processor is typically a CPU (Central Processing Unit) and/or a GPU (Graphics Processing Unit), but may be a microcomputer, FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), or other general-purpose processor. Alternatively, a dedicated processor or the like may be used.

The terminal 200 also includes programs executed by a processor and data used by the processor to realize such processing, such as avatars, backgrounds, and other various objects for creating VR/AR/MR experiences. It includes memory for temporarily storing the underlying image data. The memory may include a RAM (Random Access Memory) having a work area in which such programs/data are deployed.

Note that the terminal 200 may handle all data in an on-memory state, or part of the data may be saved in an auxiliary storage device. The auxiliary storage device may be, for example, an internal or external HDD (Hard Disc Drive), SSD (Solid State Drive), flash memory, or the like, or a database server accessible from the terminal 200. may

Terminal 200 can also use a communication I/F (interface) for connecting to a network. The communication I/F may be built in terminal 200 or externally attached to terminal 200 . The communication I/F is a module for communicating with another terminal 200 and/or an external device such as the HMD 10, controller 20, base station, etc., and includes a signal processing circuit for transmission and reception, an antenna, A LAN (Local Area Network) terminal or the like may be included. The communication I/F can be, for example, a module for wide area communication such as mobile communication, a module for wireless/wired LAN, a module for Bluetooth (registered trademark), or the like.

The terminal 200 can also use an input/output I/F for cable connection to external devices such as the HMD 10, controller 20, base station, and the like. The input/output I/F is a USB terminal, a DVI (Digital Visual Interface) terminal, an HDMI terminal, or other terminals for data transfer cables.

Terminal 200 may further include buses for transferring data between elements such as processors, memories, auxiliary storage devices, communication I/Fs, input/output I/Fs, and the like.

Next, a data replacement device according to an embodiment will be described. FIG. 1 illustrates the functional configuration of a data replacement device 100 according to an embodiment. Data replacement device 100 includes data acquisition unit 101 , state determination unit 102 , and data replacement unit 103 .

The data acquisition unit 101 can be realized by, for example, the aforementioned communication I/F, input/output I/F, or processor. Data acquisition section 101 acquires the first control data and sends it to state determination section 102 . Here, the first control data has values based on the position of the HMD 10 and the position of the controller 20 . As previously mentioned, the position of HMD 10/controller 20 can be estimated using a position sensor system.

When an external device such as the HMD 10, the controller 20 and/or the base station estimates the position of the HMD 10/controller 20 and generates the first control data, the data acquisition unit 101 receives the first control data from the external device. Just get the data. On the other hand, terminal 200 (a position estimating unit, not shown) estimates the position of HMD 10/controller 20 based on the output data of the position sensor system (for example, the data indicating the position, tilt, light emission intensity, etc. of the detection point). When generating one piece of control data, the data acquisition unit 101 may acquire the first control data from the memory or auxiliary storage device of the terminal 200 .

In the above description, it is assumed that the data replacement device 100 conditionally replaces the first control data before transmitting the first control data to the other terminal 200 . However, as will be described in Modification 1 below, the terminal 200 transmits its own first control data to the other terminal 200 without replacing it, and the data replacement device 100 incorporated in the other terminal 200 receives it. It is also possible to conditionally replace the first control data. Assuming the latter configuration, the data acquisition unit 101 acquires the first control data received via the network.

If the HMD 10 /controller 20 includes the wear sensor/grip sensor described above, the data acquisition unit 101 may further acquire event data from this wear sensor/grip sensor and send it to the state determination unit 102 .

The state determination unit 102 can be implemented by, for example, the processor described above. State determination unit 102 receives first control data (and event data) from data acquisition unit 101 . The state determination unit 102 determines whether or not a predetermined replacement start condition is satisfied for the first control data. Then, state determination section 102 sends first control data to data replacement section 103 together with data indicating the determination result.

Here, the state determination unit 102 may determine whether or not the replacement start condition is satisfied based on the first control data, or may determine whether or not the replacement start condition is satisfied based on the event data. You may The replacement start condition can be defined to be satisfied, for example, when the avatar is expected to take an unnatural posture if the avatar is controlled faithfully to the first control data.

Specifically, the state determination unit 102 may determine whether or not the HMD 10 is worn on the user's head as an example of the replacement start condition. When the HMD 10 is removed from the user's head, the position of the HMD 10 is no longer tied to the position of the user's head, but rather from an extremely low or high position, or from the position of the user's hands (more precisely, the controller 20). can be moved away from each other. In such a case, if the position of the avatar's head is determined according to the position of the HMD 10, the posture of the avatar may become unnatural.

The state determination unit 102 can determine, for example, whether the HMD 10 is worn on the user's head based on event data output from the wear sensor when an event of wearing/removing the HMD 10 occurs.

Also, the user may lift the HMD 10 significantly higher than the head when removing the HMD 10 from the head. Therefore, the state determination unit 102 determines whether or not the first control data has a value indicating that the height of the HMD 10 is equal to or higher than the predetermined first height, as an example of the replacement start condition. good.

The first height may refer to the upper limit of head height required for the avatar's pose not to be defined as unnatural. The first height may be defined for each user or may be defined for each virtual experience sharing system. As an example, the first height may be the height of the user's individual or average user's head plus an offset, and so on.

Also, after removing the HMD 10 from the head, the user may place it on a low ground such as the floor. In such a case, if the position of the avatar's head is determined according to the position of the HMD 10, the avatar may, for example, take a posture as if it were lying down with its head on a low ground.

Therefore, the state determination unit 102 determines whether or not the first control data has a value indicating that the height of the HMD 10 is less than the predetermined second height, as an example of the replacement start condition. good. In the example of FIG. 6, the second height is set to H0, and the height of the HMD 10 is lower than H0, so the state determination unit 102 determines that the replacement start condition is satisfied.

The second height can mean the lower limit of the head height required for the avatar's pose not to be defined as unnatural. The second height may be defined for each user or may be defined for each virtual experience sharing system. As an example, the second height can be the height of the user's individual or average user's waist, knee height, head height when bending forward, and the like.

Also, the replacement start condition may be that the first control data has a value indicating that the height of the HMD 10 is equal to or greater than the first height or less than the second height. In the example of FIG. 8, the first height and the second height are set to H1 and H2, respectively, and the height of the HMD 10 is H1 or more. Then judge. Thereby, when the height of the HMD 10 does not fall within the range from the lower limit value to the upper limit value, it is possible to prevent the collapse of the posture of the avatar by replacing the data.

Also, the user may have let go of the controller 20 to free his hands before removing the HMD 10 from his head. Therefore, the state determination unit 102 determines whether the first control data has a value indicating that the controller 20 is stationary, in addition to any of the above-described conditions relating to the height of the HMD 10, as an example of replacement start conditions. It may be determined whether Whether or not the controller 20 is stationary can be determined based on, for example, the history of the position of the controller 20 in the first control data, the output data of the motion sensor or grip sensor of the controller 20, and the like.

Furthermore, as long as the user wears the HMD 10 on the head and holds the controller 20, the distance between the two will not be extremely large. In other words, if the distance between HMD 10 and controller 20 is extremely large, at least one of HMD 10 and controller 20 may not be worn by the user. Therefore, in such a case, if the avatar is controlled faithfully to the position data of the HMD 10/controller 20, the avatar may take an unnatural posture. Therefore, the state determination unit 102 determines whether or not the first control data has a value indicating that the distance between the HMD 10 and the controller 20 is equal to or greater than a predetermined distance, as an example of the replacement start condition. good too.

The default distance may represent an upper limit on the distance between the head and hands that is required so that the avatar's pose is not defined as unnatural. The default distance may be defined per user or per virtual experience sharing system. As an example, the default distance may be based on the user's individual or average user's head-to-hand distance. Specifically, the predetermined distance is the distance obtained by adding an offset to the distance from the user's head to the hand when the user is in a posture that maximizes the distance from the head to the hand, and so on.

After determining that the replacement start condition is once satisfied, the state determination unit 102 determines whether or not the predetermined replacement end condition is satisfied. After determining once that the replacement end condition is satisfied, the state determination unit 102 determines again whether or not the replacement start condition is satisfied. The replacement end condition can be defined so as to be satisfied, for example, when the avatar is expected not to take an unnatural posture even if the avatar is faithfully controlled according to the first control data. By defining the replacement end condition in this way, after the avatar transitions from a natural posture (e.g., standing upright) to a dummy posture (e.g., crouching), the same or different natural posture can be restored without inserting an unnatural posture. (for example, standing up and waving).

The replacement end condition may typically correspond to the negation of the replacement start condition, ie the replacement start condition is not satisfied. For example, if the replacement start condition is "the HMD 10 is worn on the user's head", the replacement end condition may be set to "the HMD 10 is not worn on the user's head".

However, the replacement end condition may not correspond to the negation of the replacement start condition. Specifically, the replacement start condition is that "the first control data has a value indicating that the height of the HMD 10 is equal to or greater than the first height and that the controller 20 is stationary." , the replacement end condition is "the first control data indicates that the height of the HMD 10 is less than the first height (and is equal to or greater than the second height), and the controller 20 is having a value indicating that it is not

The data replacement unit 103 can be implemented by the processor described above. The data replacement unit 103 replaces the first control data with the second control data when it is determined that the replacement start condition is satisfied, and outputs the second control data to the outside of the data replacement device 100 . On the other hand, the data replacement unit 103 outputs the first control data to the outside of the data replacement device 100 as it is when it is determined that the replacement start condition is not satisfied. Note that once the data replacement unit 103 starts replacing data, it continues to replace data until the state determination unit 102 determines that the replacement end condition is satisfied. Here, the second control data is dummy data for preventing the avatar from taking an unnatural posture, and a specific example thereof will be described below.

For example, if the replacement start condition relates to the fact that the HMD 10 is worn on the user's head or the height of the HMD 10, the second control data has a value that is not based on the height of the HMD 10. obtain. Specifically, the second control data indicates that the height of the HMD 10 is less than the first height described above, is greater than or equal to the second height described above, or is less than the first height and It can have a value indicating a height greater than or equal to a second height. As a result, for example, even when the height of the HMD 10 is less than the second height H0 as in the example of FIG. 6, the head of the avatar is not unnaturally lowered, and the avatar is bent as shown in FIG. You can make the avatar take such a posture. In addition to the crouching posture, depending on the algorithm for controlling the posture of the avatar and the height of the HMD 10 indicated by the second control data, the avatar can be made to take various postures such as sitting, kneeling, and standing. .

Also, if the replacement start condition relates to the distance between HMD 10 and controller 20 , the second control data may have a value that is not based on the position of at least one of HMD 10 and controller 20 . Specifically, the second control data may have a value indicating that the distance between HMD 10 and controller 20 is less than a predetermined distance.

When the user is not wearing the HMD 10, the HMD 10 can be expected to be stationary, and when the user is not holding the controller 20, the controller 20 can be expected to be stationary. Therefore, the second control data may have a value that is not based on the position of one or both of HMD 10 and controller 20 when stationary. In other words, in the second control data, among the HMD 10 and the controller 20 in the first control data, the position of the non-stationary device (non-stationary device) is maintained, and the position of the other device is the position of the non-stationary device. to a position that is less than the above predefined distance. Whether the HMD 10/controller 20 is stationary is determined based on, for example, the history of the position of the HMD 10/controller 20 in the first control data, or the output data of the movement sensor or the mounting sensor/grip sensor of the HMD 10/controller 20. can be determined by

Next, a terminal 200 incorporating this data replacement device 100 will be described. FIG. 3 illustrates the functional configuration of a terminal 200 incorporating the data replacement device 100. As shown in FIG. Terminal 200 includes data replacement device 100 , transmitter 201 , receiver 211 , and image generator 212 .

The transmission unit 201 can be realized by the communication I/F described above. The transmission unit 201 receives the output data of the data replacement device 100 and transmits it via the network to another terminal 200 as a communication partner. Here, as described above, the output data of the data replacement device 100 is the second control data when data replacement is performed, and is the first control data otherwise.

In addition to the first or second control data, the transmission unit 201 may receive voice data generated by the microphone and transmit the data to another terminal 200 as a communication partner via the network.

The receiving unit 211 can be realized by the communication I/F described above. The receiving unit 211 receives control data for controlling the avatar of the user of the other terminal 200 via the network from the other terminal 200 as a communication partner. The received control data may be the first control data or the second control data, but whichever it is, there is no effect on subsequent processing. The receiver 211 sends the control data to the image generator 212 .

Note that the receiving unit 211 can receive audio data in addition to control data. The receiving unit 211 may send the audio data to an audio processing unit (not shown). This audio processing unit reproduces audio data and causes the aforementioned speaker to output audio.

The image generation unit 212 can be realized by the processor described above. The image generating unit 212 controls the user's avatar image corresponding to the control data based on the image data of the object stored in the memory or auxiliary storage device of the terminal 200 and the control data from the receiving unit 211, for example. Generate (update) VR/AR/MR images, including The control data here corresponds to the second control data when data replacement is performed, and corresponds to the first control data otherwise. The image generator 212 sends the generated image data to the HMD 10 .

As described above, the image generation unit 212 determines the position of the head/hands of the avatar image according to the position of the HMD 10/controller 20 indicated by the control data, and controls the posture of the avatar using, for example, IK technology. may

The operation of terminal 200 will be described below with reference to FIG. The operation of FIG. 9 is performed each time the first control data is generated.

First, the data acquisition unit 101 in the data replacement device 100 acquires first control data (step S301). Then, the state determination unit 102 in the data replacement device 100 determines whether or not a predetermined condition is satisfied (step S302). If the predetermined condition is satisfied, the process proceeds to step S303; otherwise, the process skips step S303 and proceeds to step S304.

Note that the predetermined condition in step S302 means the replacement start condition if the replacement start condition is not satisfied even once. It means a condition corresponding to negation of the replacement end condition, and can mean the replacement start condition from when the replacement end condition is satisfied until when the replacement start condition is satisfied again.

In step S<b>303 , the data replacement unit 103 in the data replacement device 100 replaces the first control data with second control data having a value not based on the position of at least one of the HMD 10 and controller 20 .

Next, the transmission unit 201 transmits control data (step S304), and the operation of FIG. 9 ends. In addition, when step S303 is executed before step S304, the transmitting unit 201 transmits the second control data. On the other hand, if step S303 is skipped before step S304, the transmitter 201 transmits the first control data.

As described above, the data replacement device according to the embodiment transfers the first control data having a value based on the positions of the HMD and the controller to at least one of the HMD and the controller when the predetermined replacement start condition is satisfied. Replace with second control data having values that are not position based. Therefore, according to this data replacement device, for example, when the avatar is likely to take an unnatural posture if the avatar is faithfully controlled according to the first control data, a dummy first control data is used instead of the first control data. 2 control data can be used to control the avatar. That is, regardless of the actual positions of the HMD and the controller, the posture of the avatar is maintained so as not to be unnatural, so at least the deterioration of the user's virtual experience due to the unnaturalness of the avatar image can be prevented. .

(Modification 1)
In the above-described embodiment, data replacement for the first control data is conditionally performed in the terminal that is the transmission source of the first control data. However, data replacement for the first control data may be performed in the terminal that is the destination, not the source of the first control data. A terminal 400 of such a modification is illustrated in FIG.

Terminal 400 includes data acquisition section 402 , transmission section 201 , reception section 211 , data replacement device 100 , and image generation section 212 . Here, the transmitting unit 201, the receiving unit 211, the data replacement device 100, and the image generating unit 212 are the same as or similar to the elements with the same names in FIG. Description will be made in the center.

The data acquisition unit 402 can be realized by, for example, the aforementioned communication I/F, input/output I/F, or processor. The data acquisition unit 402 acquires first control data and sends it to the transmission unit 201 . As previously mentioned, the position of HMD 10/controller 20 can be estimated using a position sensor system.

When an external device such as the HMD 10, controller 20, or base station estimates the position of the HMD 10/controller 20 and generates the first control data, the data acquisition unit 402 obtains the first control data from the external device. You should get it. On the other hand, terminal 200 (a position estimating unit, not shown) estimates the position of HMD 10/controller 20 based on the output data of the position sensor system (for example, the data indicating the position, tilt, light emission intensity, etc. of the detection point). When generating one piece of control data, the data acquisition unit 402 may acquire the first control data from the memory or auxiliary storage device of the terminal 200 .

If the HMD 10 /controller 20 includes the wear sensor/grip sensor described above, the data acquisition unit 402 may further acquire event data from this wear sensor/grip sensor and send it to the transmission unit 201 . Also, in this case, the transmission unit 201 can transmit event data in addition to the first control data to the destination terminal 400 for determination of a predetermined condition at the terminal 400 . Furthermore, the receiving unit 211 can also receive event data in addition to the first control data transmitted from the other terminal 400 and transmit it to the data replacement device 100 .

As described above, in this modified example, data replacement processing is performed in a distributed manner not at the source of the first control data but at the destination terminal. Therefore, according to this modification, the load on the transmission source can be reduced.

(Modification 2)
In the above embodiments, each terminal transmits data to each other via a P2P type network. On the other hand, each terminal can also transmit data to each other via a C/S type network. In this case, the data replacement device 100 may be incorporated into the server. Then, instead of simply relaying the first control data between terminals, the server conditionally replaces the first control data received from the terminal that is the transmission source, and then replaces the first control data with the first control data. Alternatively, the second control data may be transmitted to the destination terminal.

By adopting a C/S type network as in this modification, there is a possibility that the transmission delay of control data will be longer than when a P2P type network is adopted, but data replacement processing is centralized in the server. Since this is performed systematically, the load on each terminal can be reduced.

The above-described embodiments merely show specific examples to aid understanding of the concept of the present invention, and are not intended to limit the scope of the present invention. Embodiments can add, delete, or convert various components without departing from the gist of the present invention.

Although some functional units have been described in the above-described embodiments, these are only examples of implementation of each functional unit. For example, a plurality of functional units described as being implemented in one device may be implemented across a plurality of separate devices, and conversely, may be implemented across a plurality of separate devices. It is also conceivable that the functional units described may be implemented in a single device.

Various functional units described in the above embodiments may be realized by using circuits. A circuit may be a dedicated circuit that implements a specific function, or it may be a general-purpose circuit such as a processor.

At least part of the processing of each of the above-described embodiments can also be realized by using a processor installed in a general-purpose computer as basic hardware. A program that implements the above process may be provided by being stored in a computer-readable recording medium. The program is stored in the recording medium as an installable format file or an executable format file. Recording media include magnetic disks, optical disks (CD-ROM, CD-R, DVD, etc.), magneto-optical disks (MO, etc.), semiconductor memories, and the like. Any recording medium may be used as long as it can store the program and is readable by a computer. Alternatively, the program that implements the above processing may be stored on a computer (server) connected to a network such as the Internet, and downloaded to the computer (client) via the network.

10 HMDs
20, 21, 22... Controller 100... Data replacement device 101, 402... Data acquisition unit 102... State determination unit 103... Data replacement unit 200... Terminal 201... Transmission unit 211... Receiving unit 212... Image generating unit

Claims (2)

A control program for controlling the posture of a user's avatar in a virtual space according to the user's posture,
the computer,
means for obtaining a first position of a first device wearable on the user's head and a second position of a second device grippable by the user;
means for determining whether the avatar takes an unnatural posture based on the first position and the second position ;
means for outputting data for preventing the avatar from taking an unnatural posture when it is determined that the avatar takes an unnatural posture;
control program to function as The first position and the second position are images captured by a camera installed at a position away from the first device and the second device , or images captured by the first device and the second device. Estimated based on the images captured by the provided camera ,
The control program according to claim 1.

Images