JP2019133673A - Method of controlling character in virtual space - Google Patents

Abstract

To provide a character control method which provides a user with a deeper sense of immersion in a virtual space.SOLUTION: A method of controlling a character in a predetermined image region provided to a plurality of user devices is disclosed, the method comprising steps of: placing a character played by a first user in the predetermined image region displayed on each user device; receiving an input of the first user from a first user device; controlling motion of the character according to the input from the first user; displaying a plurality of selectable objects on each user device; acknowledging selection of one object of the plurality of objects from a user device associated with a second user; and displaying the one object in the predetermined image region. When an input from the first user for the one object is detected, the character is controlled to interact with the object.SELECTED DRAWING: Figure 8

Description

  The present invention relates to a method for controlling a character in a virtual space. More specifically, the present invention relates to a character control method using a device mounted on a body such as a user's head, for example, a head mounted display (HMD).

  Motion capture is a technique for digitally capturing the movement of a performer in a real space, and the captured movement is used for computer animation such as a moving image and expression of character movement in a game or the like.

  Conventionally, as a motion capture technique, an optical type or a mechanical type has been adopted. First, as an optical example, the tracker is equipped with a plurality of trackers, such as digital cameras, in which a performer wears a full-body suit equipped with markers and tracks the markers in a certain space such as a room or a photography studio. By capturing the reflection of the marker and analyzing the change of the marker position for each frame, the time-series movement of the performer is constructed as a spatial expression. By applying this spatial expression to the control of the character in the virtual space, it is possible to reproduce the movement of the performer as the movement of the character. As an example of a technique for improving the tracking accuracy of the optical motion capture technique, for example, a technique disclosed in JP 2012-248233 A can be cited. Further, as an example of a mechanical type, there is a method in which an accelerometer, gyroscope, and geomagnetic sensor are attached to the entire body of the performer, and the motion of the performer detected by the sensor is applied to the control of the character in the virtual space. As an example of a motion capture technique using a sensor, for example, a technique disclosed in JP-A-2006-126500 can be cited.

JP 2012-248233 A JP-A-2006-126500

  The technique disclosed in the above-mentioned document requires that a dedicated motion capture system is employed for controlling the character in the virtual space. In particular, the performer needs to wear markers or sensors throughout the body, and it is necessary to increase the number of markers and sensors in order to improve accuracy.

  Accordingly, an object of the present invention is to provide a technique that allows a character and a user to share a sense of unity while realizing the movement of the performer in the real space as the movement of the character.

A character control method in a virtual space provided by an embodiment of the present invention is a character control method in a predetermined image area provided to a plurality of user devices, and each user device of a plurality of user devices. A step of arranging a character played by the first user in a predetermined image area displayed on the screen, a step of receiving an input of the first user from the first user device, and an input of the first user, A step of controlling the movement of the character; a step of displaying a plurality of selectable objects on each of the user devices; and a user device associated with a second user of any of the plurality of user devices. Receiving a selection of one object among the plurality of objects; and And displaying a constant image area, when detecting an input of said first user for said one object, the character is equal to or to control so as to interact with the object.

  According to the present invention, the movement of the user in the real space is reproduced as the movement of the character, and the character and the user can share a sense of unity.

The schematic diagram of the external appearance of the head mounted display 110 concerning 1st Embodiment is shown. The schematic diagram of the external appearance of the controller 210 concerning 1st Embodiment is shown. 1 is a configuration diagram of an HMD system 300 according to a first embodiment. The functional block diagram of HMD110 concerning 1st Embodiment is shown. The functional block diagram of the controller 210 concerning 1st Embodiment is shown. 1 is a functional configuration diagram of an image generation apparatus 310 according to a first embodiment. It is a figure which shows an example of the virtual space displayed on a user apparatus concerning 1st Embodiment. The flowchart explaining the control method of the character concerning 1st Embodiment is shown. The flowchart explaining the control method of the character concerning 1st Embodiment is shown. The functional block diagram of the image generation apparatus 1010 concerning 2nd Embodiment is shown. It is a figure which shows an example of the virtual space displayed on a user apparatus concerning 2nd Embodiment. It is a figure which shows another example of the virtual space displayed on a user apparatus concerning 2nd Embodiment. In 2nd Embodiment, the flowchart as an example of the process in which a character and an item interact on a virtual space is shown. The flowchart as an example of the process which a character and an item interact on virtual space concerning 3rd Embodiment is shown. An example of the item management table 1510 stored in the item data storage unit 1054 of the image generation apparatus 1010 is shown. The flowchart as an example of the process which a character and an item interact on virtual space concerning 4th Embodiment is shown. An example of the user ranking management table 1610 stored in the user data storage unit 1055 of the image generation apparatus 1010 is shown. An example of the virtual space user interface for performer users concerning a 5th embodiment is shown.

<Embodiment 1>
A specific example of a program for controlling a head mounted display system according to an embodiment of the present invention will be described below with reference to the drawings. In addition, this invention is not limited to these illustrations, is shown by the claim, and intends that all the changes within the meaning and range equivalent to the claim are included. In the following description, the same reference numerals are given to the same elements in the description of the drawings, and redundant descriptions are omitted.

FIG. 1 is a schematic view of the appearance of a head mounted display (hereinafter referred to as HMD) 110 according to the present embodiment. The HMD 110 is mounted on the user's head and includes a display panel 120 so as to be placed in front of the user's left and right eyes. As the display panel, an optical transmission type and a non-transmission type display are conceivable. In this embodiment, a non-transmission type display panel capable of providing a more immersive feeling is exemplified. The display panel 120 displays a left-eye image and a right-eye image, and can provide a user with a stereoscopic effect by using parallax between both eyes. If the image for the left eye and the image for the right eye can be displayed, the display for the left eye and the display for the right eye can be provided separately, or an integrated display for the left eye and the right eye can be provided. is there.

Furthermore, the housing part 130 of the HMD 110 includes a sensor 140. Although not shown in the figure, the sensor may include, for example, a magnetic sensor, an acceleration sensor, a gyro sensor, or a combination thereof, in order to detect movements such as the orientation and tilt of the user's head. The vertical direction of the user's head is the Y axis, and among the axes orthogonal to the Y axis, the axis corresponding to the user's front-rear direction connecting the center of the display panel 120 and the user is the Z axis, and the Y axis and the Z axis And the axis corresponding to the left and right direction of the user is the X axis, the sensor 140 has a rotation angle around the X axis (so-called pitch angle), a rotation angle around the Y axis (so-called yaw angle), Z A rotation angle around the axis (so-called roll angle) can be detected.

In addition, or instead of the sensor 140, the housing unit 130 of the HMD 110 can include a plurality of light sources 150 (for example, infrared light LEDs and visible light LEDs) and is installed outside the HMD 110 (for example, indoors). The detected camera (for example, an infrared light camera or a visible light camera) detects these light sources, so that the position, orientation, and inclination of the HMD 110 in a specific space can be detected. Alternatively, for the same purpose, the HMD 110 may be provided with a camera for detecting a light source installed in the housing part 130 of the HMD 110.

Further, the housing part 130 of the HMD 110 can include an eye tracking sensor. The eye tracking sensor is used to detect the gaze direction and the gazing point of the user's left eye and right eye. Various types of eye tracking sensors are conceivable. For example, the position of the reflected light on the cornea formed by irradiating the left eye and the right eye with weak infrared light is used as a reference point, and depending on the position of the pupil relative to the position of the reflected light. A method of detecting the line-of-sight direction and detecting the intersection of the left-eye and right-eye line-of-sight directions as a gazing point is conceivable.

  FIG. 2 is a schematic diagram of the appearance of the controller 210 according to the present embodiment. The controller 210 can support the user to perform a predetermined input in the virtual space. The controller 210 may be configured as a set of left hand 220 and right hand 230 controllers. The left hand controller 220 and the right hand controller 230 may each include an operation trigger button 240, an infrared LED 250, a sensor 260, a joystick 270, and a menu button 280.

  The operation trigger button 240 is arranged as 240a and 240b at positions that are assumed to perform an operation of pulling the trigger with the middle finger and the index finger when the grip 235 of the controller 210 is gripped. A frame 245 formed in a ring shape downward from both side surfaces of the controller 210 is provided with a plurality of infrared LEDs 250, and the position of these infrared LEDs is detected by a camera (not shown) provided outside the controller. Thus, the position, orientation and inclination of the controller 210 in a specific space can be detected.

In addition, the controller 210 can incorporate a sensor 260 in order to detect movements such as the orientation and inclination of the controller 210. Although not illustrated, the sensor 260 can include, for example, a magnetic sensor, an acceleration sensor, a gyro sensor, or a combination thereof. Furthermore, a joystick 270 and a menu button 280 can be provided on the upper surface of the controller 210. The joystick 270 can be moved 360 degrees around the reference point, and is assumed to be operated with the thumb when the grip 235 of the controller 210 is gripped. Similarly, it is assumed that the menu button 280 is operated with the thumb. Furthermore, the controller 210 can incorporate a vibrator (not shown) for applying vibration to the hand of the user who operates the controller 210. In order to output information such as the input contents of the user via a button or joystick, the position, orientation, and tilt of the controller 210 via a sensor or the like, and to receive information from the host computer, the controller 210 has an input / output Part and a communication part.

The system determines the movement and posture of the user's hand based on whether the user holds the controller 210 and operates various buttons and joysticks, and information detected by the infrared LED and sensor. The user's hand can be displayed and operated.

FIG. 3 is a configuration diagram of the HMD system 300 according to the present embodiment. The HMD system 300 can be configured by, for example, an HMD 110, a controller 210, and an image generation apparatus 310 that functions as a host computer. Furthermore, an infrared camera (not shown) for detecting the position, orientation, inclination, and the like of the HMD 110 and the controller 210 can be added. These devices can be connected to each other by wired or wireless means. For example, each device is equipped with a USB port, and communication can be established by connecting with a cable. Besides, wired such as HDMI, wired LAN, infrared, Bluetooth (registered trademark), WiFi (registered trademark), etc. Alternatively, communication can be established wirelessly. The image generation apparatus 310 may be an apparatus having a calculation processing function such as a PC, a game machine, or a mobile communication terminal. Further, the image generation apparatus 310 can connect to a plurality of user apparatuses such as 401A, 401B, and 401C via a network such as the Internet, and transmit the generated image in a streaming or download form. Each of the user devices 401A and the like can reproduce an image transmitted by being provided with an Internet browser or an appropriate viewer. Here, the image generation apparatus 310 can directly transmit an image to a plurality of user apparatuses, or can transmit an image via another content server.

FIG. 4 is a functional configuration diagram of the HMD 110 according to the present embodiment. As described in FIG. 1, the HMD 110 may include a sensor 140. As a sensor, although not shown in order to detect movements such as the orientation and inclination of the user's head, for example, a magnetic sensor, an acceleration sensor, a gyro sensor, or a combination thereof can be provided. Further, an eye tracking sensor can be provided. The eye tracking sensor is used to detect the gaze direction and the gazing point of the user's left eye and right eye. Furthermore, in order to detect movements such as the orientation and tilt of the user's head with high accuracy and to detect the position of the user's head, an LED 150 such as infrared light or ultraviolet light may be provided. Moreover, the camera 160 for image | photographing the outside scene of HMD can be provided. Moreover, the microphone 170 for collecting a user's utterance and the headphone 180 for outputting an audio | voice can also be provided. Note that the microphone and the headphone can be provided as an independent device from the HMD 110.

Furthermore, the HMD 110 can include, for example, an input / output unit 190 for establishing a wired connection with peripheral devices such as the controller 210 and the image generation device 310, and can include infrared, Bluetooth (registered trademark), and WiFi (registered trademark). The communication unit 115 for establishing a wireless connection can be provided. Information relating to movements such as the orientation and tilt of the user's head acquired by the sensor 140 is transmitted by the control unit 125 to the image generation apparatus 310 via the input / output unit 190 and / or the communication unit 115. Although details will be described later, the image generated by the image generation apparatus 310 based on the movement of the user's head is received via the input / output unit 190 and / or the communication unit 115, and the display unit 120 is displayed by the control unit 125. Is output.

FIG. 5 is a functional configuration diagram of the controller 210 according to the present embodiment. As described in FIG. 2, the controller 210 can be configured as a set of controllers for the left hand 220 and the right hand 230, but in either controller, the operation trigger button 240, the joystick 270, An operation unit 245 such as a menu button 280 can be provided. In addition, the controller 210 can incorporate a sensor 260 in order to detect movements such as the orientation and inclination of the controller 210. Although not illustrated, the sensor 260 can include, for example, a magnetic sensor, an acceleration sensor, a gyro sensor, or a combination thereof. Further, a plurality of infrared LEDs 250 are provided, and the position, orientation, and inclination of the controller 210 in a specific space are detected by detecting the positions of these infrared LEDs with a camera (not shown) provided outside the controller. be able to. The controller 210 can include, for example, an input / output unit 255 for establishing a wired connection with a peripheral device such as the HMD 110 or the image generation device 310, such as infrared rays, Bluetooth (registered trademark), WiFi (registered trademark), or the like. A communication unit 265 for establishing a wireless connection can be provided. Information input by the user via the operation unit 245 and information such as the orientation and inclination of the controller 210 acquired by the sensor 260 are transmitted to the image generation apparatus 310 via the input / output unit 255 and / or the communication unit 265. .

FIG. 6 is a functional configuration diagram of the image generation apparatus 310 according to the present embodiment. The image generation device 310 stores information about the user's head movement and controller movement and operation acquired from the user input information and sensors transmitted from the HMD 110 and the controller 210, and performs predetermined calculation processing. Devices such as a PC, a game machine, and a mobile communication terminal having a function for generating an image can be used. The image generation device 310 can include an input / output unit 320 for establishing a wired connection with peripheral devices such as the HMD 110 and the controller 210, for example, infrared, Bluetooth (registered trademark), WiFi (registered trademark), etc. A communication unit 330 for establishing a wireless connection can be provided. Information on the movement of the user's head and the movement and operation of the controller received from the HMD 110 and / or the controller 210 via the input / output unit 320 and / or the communication unit 330 is transmitted to the control unit 340 in the position of the user, Characters are controlled by executing a control program that is detected as input content including actions such as line of sight, posture, etc., utterances, operations, etc., and stored in the storage unit 350 according to the input content of the user. Is generated. The control unit 340 can be configured by a CPU, but by further providing a GPU specialized for image processing, information processing and image processing can be distributed, and overall processing efficiency can be improved. The image generation apparatus 310 can also communicate with other calculation processing apparatuses and share information processing and image processing with other calculation processing apparatuses.

Further, the control unit 340 of the image generation apparatus 310 is a user input detection unit that detects information about the movement of the user's head and the user's speech, and the movement and operation of the controller, received from the HMD 110 and / or the controller 210. 610, a character control unit 629 that executes a control program stored in the control program storage unit for a character stored in advance in the character data storage unit 650 of the storage unit 350, and an image based on the character control An image generation unit 630 is included. Here, for the movement control of the character, information such as the orientation and tilt of the user's head detected by the HMD 110 and the controller 210, and the movement of the hand is created in accordance with the movements and restrictions of the human body joints. This is realized by applying the motion of the bone structure by converting the motion of each part of the bone structure and associating the bone structure with the character data stored in advance.

The storage unit 350 stores, in the character data storage unit 650 described above, information related to the character, such as character attributes, in addition to the character image data. In addition, the control program storage unit 670 stores a program for controlling the movement and facial expression of the character in the virtual space. The streaming data storage unit 660 stores the image generated by the image generation unit 630.

FIG. 7 is a diagram illustrating an example of a virtual space displayed on the user device according to the present embodiment. As shown in FIG. 7, the user device can display an image of a virtual space including a character 720 on an image display unit 710 such as a viewer for displaying an image embedded in a built-in web browser. The character 720 arranged in the virtual space is a movement such as the tilt and orientation of the user's head, the user's utterance content, or the tilt of the controller 210 via the HMD 110 and / or the controller 210 worn by the user as a performer. It is possible to operate based on user input such as movement such as direction and user operation content via the controller 210.

8 and 9 are flowcharts for explaining a character control method according to this embodiment. First, the user input detection unit 610 of the image generation apparatus 310 receives the movement of the user's head, the user's speech, and the controller from the HMD 110 and / or the controller 210 via the input / output unit 320 and / or the communication unit 330. Information on the movement and operation of the user is received (S810). Subsequently, the user input detection unit 610 confirms whether the input content of the user is input from the HMD 110 or input from the controller 210 (S820). When the process of S820 is input from the HMD 110, the process proceeds to steps (S830 to S850) for confirming specific input contents. If the process of S820 is input from the controller 210, the process proceeds to the process shown in FIG.

When the process of S820 is input from the HMD 110, the user input detection unit 610 first confirms whether the content input from the HMD 110 is information regarding the motion of the user's head (S830). Specifically, when the information input from the sensor 140 of the HMD 110 relates to the orientation and inclination of the user's head (for example, the rotation angle around the XYZ axes with respect to the user's head), the character control unit 620 The movement of the character's head can be changed (S860). Here, for example, information related to the movement of the user's head, such as the orientation and inclination of the user, detected by the gyro sensor of the HMD 110 is converted into the movement of the head of the bone structure that defines the movement and restrictions of the human joint, The movement of the character can be controlled by applying the movement of the head of the character to the movement of the head of the character data stored in the character data storage unit 650. For example, when the gyro sensor receives information “roll angle: − (minus) 30 degrees” around the Y axis from the microphone, the character control unit 620 uses the information to move the head of the character 30 degrees to the left. The processing to change to shake. Furthermore, the facial expression of the character can be changed. Character facial expressions The elements that make up a character's facial expressions can be broken down into parts that make up the character's face, such as eyebrows, eyes, nose, mouth, etc. Can be stored as parameters. By combining the parameters of these parts, the character can have various expressions such as emotions. Then, by detecting the movement of the user's head, the parameter can be controlled to be changed randomly according to the movement. Thereby, the natural expression of the character can be reproduced. Or the user input detection part 610 confirms whether the input content is the information regarding a user's gaze point (S840). The gaze point can be obtained by calculating the line-of-sight direction based on information on the orientation and inclination of the user's head obtained through various sensors, or directly through the eye tracking sensor. The character control unit 620 can change the line of sight of the character according to the line-of-sight direction or the position indicated by the point of gaze (S870). Or the user input detection part 610 confirms whether a user input content is the information regarding a user's utterance (S850). Specifically, when the user input content is voice input from the microphone 170 of the HMD 110, the character control unit 620 controls the size of the character's mouth according to the size of the voice, The facial expression parameter can be controlled to be changed randomly (S880). Here, the user puts the HMD 110 on the head, for example, puts it on the head and changes the movement of the HMD 110 while viewing the character image displayed on the external monitor. It is also possible to reproduce the movement of the character according to the movement.

When the process of S820 is input from the controller 210, the process proceeds to steps (S910 to S930) for confirming specific input contents in FIG. First, the user input detection unit 610 confirms whether the input content from the controller 210 is information related to operation of the operation buttons of the controller 210 (S910). For example, if the input content relates to pressing of the operation trigger button 240a of the left-hand controller, the character control unit 620 causes the character to take an action corresponding to the operation of the button (for example, the character laughs). The facial expression parameters can be controlled. Alternatively, the user input detection unit 610 confirms whether the input content from the controller 210 relates to an operation such as the position, orientation, or tilt of the controller 210 detected via the sensor 260 provided in the controller 210 ( S920). If the input content relates to an operation such as the position, orientation, or tilt of the controller 210, the character control unit 620 can control the movement of the character's body (S950). For example, if the input content is related to a change in position detected by the acceleration sensor of the controller 210 (a movement to raise or lower the controller), the character control unit 620 controls to move the character's arm up and down. be able to. Further, when the input content is related to rotation detected by the gyro sensor of the controller 210, the character control unit 620 can control to open and close the character's finger. Here, if the movement of the HMD 110 and the controller 210 is combined, the character reproduces the movement of the head and hand corresponding to these, but the movement of the part connecting the head and the hand, such as the neck and shoulder. Uses computer graphics inverse kinematics (IK) to interpolate movements that cannot be obtained through sensors by back-calculating neck and shoulder movements based on head and hand movements. Can do. Or the user input detection part 610 confirms whether the input content from the controller 210 is the information regarding the combination of operation of the operation button of the controller 210 (S930). For example, when the input content is to simultaneously press the operation trigger button 240a of the left-hand controller and the operation trigger button 240a of the right-hand controller, the character takes an action corresponding to the combination (for example, the character is The facial expression parameter and the motion parameter can be controlled so as to laugh while holding the stomach (S960). Here, as with the facial expression parameters, the movement of the character's body is broken down into parts that make up the body, such as the head, both hands, and both feet, and the movement pattern and the degree of movement (the magnitude of the movement). , Speed) as a parameter. In order to express the natural movement of the character, the facial expression parameter and the motion parameter of the character can be randomly controlled according to some input to the user via the HMD 110 or the controller 210.

According to the present embodiment, by detecting the movement and operation content of each part of the user via the HMD 110 and the controller 210 instead of the marker or sensor directly attached to the user's body, the character corresponding to the user's movement is detected. While facilitating the reproduction of movement, it is possible to realize expressions rich in variations depending on the operation content.

<Second Embodiment>
FIG. 10 is a functional configuration diagram of an image generation apparatus 1010 according to the second embodiment. The HMD system according to this embodiment is the same as the system configuration in the first embodiment shown in FIG. 3, and the functional configuration of the image generation apparatus 1010 is basically the same as the configuration shown in FIG. It is. In the present embodiment, in the image generation device 1010, the control unit 1040 includes an item reception unit 1044 that receives selection of items to be arranged in the virtual space from other user devices, and the storage unit 1050 stores data regarding items. It has an item data storage unit 1054 for storing. Furthermore, the storage unit 1050 includes a user data storage unit 1055 that stores information about the user. The image generation apparatus according to the present embodiment is characterized by not only transmitting an image that is a virtual space to a plurality of user apparatuses, but also receiving items and comments from the user apparatuses. As in the first embodiment, the function of the image generation apparatus 1010 is specialized only to generate an image, a separate content service server is provided, and the content service server sends an image to the user apparatus. It is also possible to have a function of transmitting and receiving items and comments from the user device.

FIG. 11 is a diagram illustrating an example of a virtual space displayed on the user device according to the second embodiment. As in the first embodiment, the user device can display an image of the virtual space including the character 1120 in the image display unit 1110 such as a viewer for displaying an image embedded in the built-in web browser. The character 720 arranged in the virtual space is a movement such as the tilt or orientation of the user's head, the user's utterance, or the tilt of the controller 210 via the HMD 110 and / or the controller 210 worn by the user as a performer. It is possible to operate based on user input such as a movement such as a direction and a user operation content via the controller 210. In this embodiment, in addition to the image display unit 1110, a comment display unit 1130 that displays a comment received from each user device connected to the image generation device 310, a comment input unit 1140 that receives a comment input from the user, and a character An item display unit 1150 for selecting and displaying items to be given can be provided. In the present embodiment, a support comment is received from each user for a character displayed in the virtual space displayed on the image display unit 1110, and each user has a plurality of items each associated with a point. A service including so-called gifting, in which a purchase item is displayed in the virtual space by purchasing a desired item from the inside and performing an input for requesting transmission, is assumed. In such a service, it is conceivable that the character performs a predetermined action in accordance with the accumulated points of the items that have been given.

FIG. 12 is a diagram illustrating another example of the virtual space displayed on the user device according to the present embodiment. In this example, there is an effect as if the character 1120 lifts and rejoices the item 1210 placed on the image display unit 1110 through gifting from one user device. Thereby, a user and a character can share a further sense of unity in service.

FIG. 13 shows a flowchart as an example of processing in which a character and an item interact in a virtual space in the present embodiment. First, the item reception unit 1044 of the image generation device 1010 receives an item from any of a plurality of user devices connected to the service (S1310). The image generation unit 1043 of the image generation device 1010 generates an image in order to display the item 1210 together with the character 1120 on the image display unit 1110 of the user device (S1320). Thereafter, the user input detection unit 1141 of the image generation apparatus 1010 confirms whether the user input is detected from the HMD 110 or the controller 210 of the user who is a performer (S1330). For example, a user who is a performer performs an operation of pulling the trigger of the trigger button 240a for operating the left hand or the right hand while aligning the position of the controller 210 with respect to the item 1210 while viewing the virtual space displayed on the display of the HMD 110. Can do. The user input detection unit 1141 can accept and detect the operation through the input / output unit 1020 and / or the communication unit 1030 of the image generation apparatus 1010. If the user input detection unit 1041 does not detect the user input, the process returns to the original process. When the user input detection unit 1041 detects a user input, the character control unit 1042 performs a process of moving the position of the character's hand in accordance with the position of the controller 210, converts the position of the controller into coordinates in the virtual space, If this coordinate overlaps with the coordinate where the item is placed in the virtual space, next, a process of grasping the item is performed according to the operation via the controller 210. As a result, the image generation unit 1043 generates an image in which the character 1120 lifts the item 1210, that is, an image in which the character and the item interact (S1340).

<Third Embodiment>
FIG. 14 shows a flowchart as an example of processing in which a character and an item interact in a virtual space according to the third embodiment. In this embodiment, in addition to the viewpoint of image processing, a method for further enhancing the social nature of characters and users in a service is provided. The configuration of the system and the image generation apparatus of the present embodiment is basically the same as that of the second embodiment.

14, first, the item reception unit 1044 of the image generation device 1010 receives an item from any of a plurality of user devices connected to the service (S1410). The image generation unit 1043 of the image generation device 1010 generates an image in order to display the item 1210 together with the character 1120 on the image display unit 1110 of the user device (S1420). Thereafter, the user input detection unit 1141 of the image generation apparatus 1010 confirms whether the user input is detected from the HMD 110 or the controller 210 of the user who is a performer (S1430). For example, a user who is a performer performs an operation of pulling the trigger of the trigger button 240a for operating the left hand or the right hand while aligning the position of the controller 210 with respect to the item 1210 while viewing the virtual space displayed on the display of the HMD 110. Can do. The user input detection unit 1141 can accept and detect the operation through the input / output unit 1020 and / or the communication unit 1030 of the image generation apparatus 1010. If the user input detection unit 1041 does not detect the user input, the process returns to the original process. When the user input detection unit 1041 detects a user input, the character control unit 1042 confirms the number of points of the item 1210 arranged in the virtual space, and then confirms whether the number of points is equal to or greater than a predetermined point. (S1440).

FIG. 15 shows an example of the item management table 1510 stored in the item data storage unit 1054 of the image generation apparatus 1010. The item management table 1510 manages items that can be given by the user and the corresponding number of points. Here, the number of points refers to the number required to purchase an item in units when virtual currency that is valid in the service is provided in units of points. For example, an item “Dharma” has a point number “1”, whereas an item “rose” is assigned a point number “10”. The user purchases an item with points to support the character (or performer) and requests to display the purchased item in the virtual space. In the example listed in the item management table 1510, “ring” is the most expensive item.

Returning to S1440 in FIG. 14, for example, when the predetermined number of points is “10 pt” and the placed item is “Dharma”, the character control unit 1042 does not execute character control, Processing returns. On the other hand, when the arranged item is “rose”, the number of points of the rose is “10 pt”, which satisfies the condition. When it is confirmed that the condition is satisfied, the character control unit 1042 performs a process of moving the position of the character's hand according to the position of the controller 210, converts the position of the controller into coordinates in the virtual space, and the coordinates are virtual. If the item overlaps the coordinates where the item is placed, a process for grasping the item is performed in accordance with an operation via the controller 210. As a result, the image generation unit 1043 generates an image in which the character 1120 lifts the item 1210, that is, an image in which the character and the item interact (S1450).

<Fourth Embodiment>
FIG. 16 shows a flowchart as an example of processing in which a character and an item interact in a virtual space according to the fourth embodiment. In the present embodiment, in addition to the viewpoint of image processing, another method for further improving the social nature of characters and users in the service is provided. The configurations of the system and the image generation apparatus of this embodiment are basically the same as those of the second and third embodiments.

16, first, the item reception unit 1044 of the image generation device 1010 receives an item from any of a plurality of user devices connected to the service (S1610). The image generation unit 1043 of the image generation device 1010 generates an image in order to display the item 1210 together with the character 1120 on the image display unit 1110 of the user device (S1620). Thereafter, the user input detection unit 1141 of the image generation apparatus 1010 confirms whether the user input is detected from the HMD 110 or the controller 210 of the user who is the performer (S1630). For example, a user who is a performer performs an operation of pulling the trigger of the trigger button 240a for operating the left hand or the right hand while aligning the position of the controller 210 with respect to the item 1210 while viewing the virtual space displayed on the display of the HMD 110. Can do. The user input detection unit 1141 can accept and detect the operation through the input / output unit 1020 and / or the communication unit 1030 of the image generation apparatus 1010. If the user input detection unit 1041 does not detect the user input, the process returns to the original process. When the user input detection unit 1041 detects the user input, the character control unit 1042 confirms the ranking of the user who has given the item 1210 placed in the virtual space, and then the number of points is equal to or higher than the predetermined ranking. It is confirmed whether it exists (S1640).

FIG. 17 shows an example of a user ranking management table 1610 stored in the user data storage unit 1055 of the image generation apparatus 1010. The user ranking management table 1610 manages the total number of points corresponding to the items given by the user, and manages the rank based on the total number of points. For example, if user A purchases 5 items called “rings” and gives them a gift, the total number of points will be “500 pt”, and the total number of points will be first in comparison with other users. Become.

Returning to S1640 in FIG. 16, for example, when the predetermined ranking is “3rd or higher”, the character control unit 1042 is “user D” when the user who provided the arranged item is “user D”. Since the ranking of the user D is fourth, the character control is not executed and the processing returns to the original. On the other hand, if the user who has given the arranged item is “user C”, the ranking of user C is third, which satisfies the condition. When it is confirmed that the condition is satisfied, the character control unit 1042 performs a process of moving the position of the character's hand according to the position of the controller 210, converts the position of the controller into coordinates in the virtual space, and the coordinates are virtual. If the item overlaps the coordinates where the item is placed, a process for grasping the item is performed in accordance with an operation via the controller 210. As a result, the image generation unit 1043 generates an image in which the character 1120 lifts the item 1210, that is, an image in which the character and the item interact (S1650).

<Fifth Embodiment>
FIG. 18 shows an example of a virtual space user interface for a performer user according to the fifth embodiment. In the present embodiment, the performer user can check the character that moves based on the action performed by the performer user in the virtual space 1810 displayed on the display unit 120 by wearing the HMD 110 on the head. This virtual space 1810 has several operation panels, and the performer user can change the position of the virtual camera or perform other operations while performing by himself.

In FIG. 18, a virtual camera selection unit 1820 for operating the position of the virtual camera is provided. For example, the performer user can select a desired virtual camera by moving the position of the controller 210 to a position where a button of a specific camera is displayed and pressing an operation button of the controller 210. For example, the virtual camera can be disposed at a position such as 45 degrees to the left and 45 degrees to the right in front of the position where the character is disposed. Further, an icon 1830 indicating that the voice of the performer user is being recorded can be arranged. For example, the performer user can execute a recording operation by pressing a specific operation button of the controller 210, and can check the recording status through the mode of the icon 1730. In addition, a “STAND BY” icon 1840 can be placed in the virtual space, and the performer user can check the status of whether the distribution is being prepared or on the air through the icon 1840. In addition, although not shown, it is also possible to perform a process of displaying particles from the character's mouth in response to the voice and volume of the performer user, and intuition such as increasing the size of the particles according to the volume level. It is also possible to check the status.

110 Head Mount Display 115 Communication Unit 120 Display Panel 125 Control Unit 140 Sensor 150 Light Source 160 Camera 170 Microphone 180 Headphone 190 Input / Output Unit 210 Controller 220 Left Hand Controller 230 Right Hand Controller 240a, 240b Trigger Button 245 Operation Unit 250 Infrared LED
255 Input / output unit 260 Sensor 270 Joystick 280 Menu button 290 Frame 310 Image generation device 320 Input / output unit 330 Communication unit 340 Control unit 350 Storage unit 401A, 401B, 401C User device
610 User input detection unit 620 Character control unit 630 Image generation unit 650 Character data storage unit 660 Streaming data storage unit 6
670 Control program storage unit 710 Image display unit 720 Character 1110 Image display unit 1120 Character 1130 Comment display unit 1140 Comment input unit 1150 Item display unit 1210 Item 1810 Virtual space 1820 Virtual camera selection unit 1830 Icon 1840 Icon

Claims (3)

A method of controlling a character in a predetermined image area provided to a plurality of user devices,
Arranging a character played by the first user in a predetermined image area displayed on each user device of the plurality of user devices;
Receiving a first user input from a first user device;
Controlling the action of the character based on the input of the first user;
Displaying a plurality of selectable objects on each of the user devices;
Receiving a selection of one of the plurality of objects from a user device associated with a second user of any of the plurality of user devices;
Displaying the one object in the predetermined image area;
When receiving the input of the first user for the one object,
A control method, characterized in that the character is controlled to interact with the object. The control method according to claim 1, comprising:
A control method, comprising: controlling the character to interact with the object when the displayed object is a predetermined point or more. The control method according to claim 2,
Each of the plurality of objects is associated with a point, and includes the one object displayed in the virtual space, and based on the total number of points of the objects arranged in the virtual space, the second user's A control method, comprising: generating rankings for the plurality of users, and controlling the character to interact with the object when the ranking is equal to or higher than a predetermined order.