WO2019150469A1 - Method of character control in mixed reality space - Google Patents

Abstract

The purpose of the present invention is to realize a conventionally nonexistent character control method with which it is possible for a user to enjoy a greater sense of immersion in a mixed reality space that includes a virtual space. Provided is a method for controlling an object in a mixed reality space, the object control method including the steps of: arranging at least some of objects at a prescribed position apart by a prescribed distance from a user outside the visual-field area of the user in an initial state; detecting the user's input in the mixed reality space; and controlling the output of the object at a prescribed position in accordance with the user's input.

Description

Character control method in mixed reality space

The present invention relates to a method for controlling a character in a mixed reality space. More specifically, the present invention relates to a method for controlling a character displayed in a virtual reality space provided via a device worn on a body such as a user's head, for example, a head mounted display (HMD).

Mixed reality space means a technology that amplifies the real space by superimposing various information by computer processing on the real space (Mixed Reality: MR), and complements the real space information to the virtual space Technology that emphasizes the virtual space (Augusted Virtuality: AV) and technology that adds information to the real world (Augusted Reality: AR). It also includes a technology (Virtual Reality: VR) that enables interactive interaction with various objects in the space while being buried in the space.

Recent technical development and cost reduction of Head Mounted Display (HMD) has made it possible for general users to experience the mixed reality space more closely. As a common specification of HMDs generally provided in recent years, one or two display panels are arranged with respect to the left and right eyeball positions of the user, and the movement of the user's head, specifically, A sensor group is provided for detecting tilt, left / right swing, clockwise rotation, and counterclockwise rotation. The movement of the user's head detected by the sensor group is transmitted to a host computer such as a PC, and an image suitable for the movement of the user's head is generated in the host computer, and this image signal is transmitted to the HMD. An image is displayed on the display panel provided in the HMD. It is also possible to generate an image in which the background and the object are integrated by generating an object such as a character object as an image element in the host computer and superimposing the object image on the background image.

The character object displayed in the mixed reality space, particularly the virtual space, is usually arranged at the tip of the user's visual field direction in the initial state immediately after the user wears the HMD and turns on the power. Such a form is exemplified in a battle game disclosed in Patent Document 1 and an interactive game disclosed in Patent Document 2.

Since the character disclosed in the above document is a user's opponent or dialogue partner, it is important that the character is arranged at a position that can be easily identified by the user in the virtual space. In addition, this patent applicant presents the following patent documents as literature well-known inventions related to this invention.

JP 2017-606611 A JP 2017-0788893

However, although the method of arranging characters in the virtual space as exemplified in the above patent document can give the user a certain presence and immersion, the character arrangement applied to the conventional two-dimensional game space This method follows the method, and the emphasis is placed on how to realize the conventional character arrangement method in the virtual space.

Accordingly, an object of the present invention is to realize a character control method that allows a user to enjoy a more immersive feeling in a mixed reality space including a virtual space, which has not been provided in the past.

In one embodiment of the present invention, there is provided a method for controlling an object in a mixed reality space, and in an initial state, at least a part of an object is arranged at a predetermined position away from the user by a predetermined distance outside the user's view area. There is provided a method for controlling an object, comprising: a step; detecting a user input in the mixed reality space; and controlling an output of the object at a predetermined position in accordance with the user input.

According to the present invention, in the initial state, the character object is not placed at a position facing the user but at a position close to the side, so that it is not a pseudo setting like a game space, It is possible to provide a feeling as if a friend or lover is in the real space.

The schematic diagram of the external appearance of the head mounted display 110 concerning 1st Embodiment is shown. The schematic diagram of the external appearance of the controller 210 concerning 1st Embodiment is shown. 1 is a configuration diagram of an HMD system 300 according to a first embodiment. The functional block diagram of HMD110 concerning 1st Embodiment is shown. The functional block diagram of the controller 210 concerning 1st Embodiment is shown. 1 is a functional configuration diagram of an image generation apparatus 310 according to a first embodiment. 2 is a functional configuration diagram relating to a control unit of the image generation apparatus 310 according to the first embodiment. FIG. 2 is a functional configuration diagram relating to a storage unit of the image generation apparatus 310 according to the first embodiment. FIG. It is a figure explaining a visual field area. It is a figure which shows the visual field area | region in mixed reality space in 1st Embodiment. It is a figure which shows the visual field area | region in mixed reality space in 1st Embodiment. 3 shows a flowchart for object control processing in the first embodiment. 5 is a flowchart as an example of object control processing in the first embodiment. 7 is a flowchart as another example of the object control process in the first embodiment. 9 shows a flowchart as still another example of object control processing in the first embodiment. 6 shows a flowchart as still another example of object control processing in the first embodiment. The flowchart as an example of the output control of the character in 1st Embodiment is shown. The table which manages the information regarding the character object in 2nd Embodiment is shown. The block diagram of the HMD system 1800 in 2nd Embodiment is shown. The user interface which displayed the time bar 1901 on the screen which shows the mixed reality space in 3rd Embodiment is shown. The storage list 2001 of the moving image file in 3rd Embodiment is shown. The functional block diagram of the image generation apparatus 2100 in 4th Embodiment is shown. 10 shows a flowchart as an example of a method for preventing a user from accessing a mixed reality space in the fourth embodiment. An example of the table which stores the prevention reference | standard in 4th Embodiment is shown. 10 shows a flowchart as another example of a method for preventing a user from accessing a mixed reality space in the fourth embodiment. An example of the table which stores the event in 4th Embodiment is shown. 15 is a flowchart as still another example of a method for preventing a user from accessing a mixed reality space according to the fourth embodiment. The other example of the table which stores the prevention reference | standard in 4th Embodiment is shown. The function example of the image generation apparatus 2800 in 5th Embodiment is shown. The flowchart as an example of the method which provides the mixed reality space in 5th Embodiment is shown. An example of the relevant information of the virtual space object stored in the virtual space object storage unit 2842 in the fifth embodiment is shown. The figure which imaged that a virtual space object is substituted by a real space object and a mixed reality space is produced | generated is shown. The figure which imaged that a virtual space object is substituted by a real space object and a mixed reality space is produced | generated is shown.

<Embodiment 1>
A specific example of a program for controlling a head mounted display system according to an embodiment of the present invention will be described below with reference to the drawings. In addition, this invention is not limited to these illustrations, is shown by the claim, and intends that all the changes within the meaning and range equivalent to the claim are included. In the following description, the same reference numerals are given to the same elements in the description of the drawings, and redundant descriptions are omitted.

FIG. 1 is a schematic view of the appearance of a head mounted display (hereinafter referred to as HMD) 110 according to the present embodiment. The HMD 110 is mounted on the user's head and includes a display panel 120 so as to be placed in front of the user's left and right eyes. As the display panel, an optical transmission type and a non-transmission type display are conceivable. In this embodiment, a non-transmission type display panel capable of providing a more immersive feeling is exemplified. The display panel 120 displays a left-eye image and a right-eye image, and can provide a user with a stereoscopic effect by using parallax between both eyes. If the image for the left eye and the image for the right eye can be displayed, the display for the left eye and the display for the right eye can be provided separately, or an integrated display for the left eye and the right eye can be provided. is there.

Furthermore, the housing part 130 of the HMD 110 includes a sensor 140. Although not shown in the figure, the sensor may include, for example, a magnetic sensor, an acceleration sensor, a gyro sensor, or a combination thereof, in order to detect movements such as the orientation and tilt of the user's head. The direction perpendicular to the ground of the user's head is the Y axis, and the axis that connects the center of the display panel 120 and the user among the axes orthogonal to the Y axis is the Z axis, and the Y axis When the X axis is an axis that is orthogonal to the Z axis and corresponds to the left and right direction of the user, the sensor 140 has a rotation angle around the X axis (so-called pitch angle) and an angle around the Y axis (so-called yaw angle). The angle around the Z axis (so-called roll angle) can be detected.

In addition, or instead of the sensor 140, the housing unit 130 of the HMD 110 can include a plurality of light sources 150 (for example, infrared light LEDs and visible light LEDs) and is installed outside the HMD 110 (for example, indoors). The detected camera (for example, an infrared light camera or a visible light camera) detects these light sources, so that the position, orientation, and inclination of the HMD 110 in a specific space can be detected. Alternatively, for the same purpose, the HMD 110 may be provided with a camera for detecting a light source installed outside the housing unit 130 of the HMD 110 (for example, indoors).

Further, the housing part 130 of the HMD 110 can include an eye tracking sensor. The eye tracking sensor is used to detect the gaze direction and the gazing point of the user's left eye and right eye. Various types of eye tracking sensors are conceivable. For example, the position of the reflected light on the cornea formed by irradiating the left eye and the right eye with weak infrared light is used as a reference point, and depending on the position of the pupil relative to the position of the reflected light. A method of detecting the line-of-sight direction and detecting the intersection of the left-eye and right-eye line-of-sight directions as a gazing point is conceivable. An HMD 110 as shown in FIG. 1 is a representative example, but an HMD applicable to the present embodiment is not limited to this, and other types (for example, other head-mounted type, glasses-type type) are applicable. Such a display is also applicable.

FIG. 2 is a schematic diagram of the appearance of the controller 210 according to the present embodiment. The controller 210 can support the user to make a predetermined input in the mixed reality space. The controller 210 may be configured as a set of left hand 220 and right hand 230 controllers. The left hand controller 220 and the right hand controller 230 may each include an operation trigger button 240, an infrared LED 250, a sensor 260, a joystick 270, and a menu button 280.

The operation trigger button 240 is arranged as 240a and 240b at positions that are assumed to perform an operation of pulling the trigger with the middle finger and the index finger when the grip 235 of the controller 210 is gripped. A frame 290 formed in a ring shape downward from both sides of the controller 210 is provided with a plurality of infrared LEDs 250, and the position of these infrared LEDs is detected by a camera (not shown) provided outside the controller. Thus, the position, orientation and inclination of the controller 210 in a specific space can be detected.

In addition, the controller 210 can incorporate a sensor 260 in order to detect movements such as the orientation and inclination of the controller 210. Although not illustrated, the sensor 260 can include, for example, a magnetic sensor, an acceleration sensor, a gyro sensor, or a combination thereof. Furthermore, a joystick 270 and a menu button 280 can be provided on the upper surface of the controller 210. The joystick 270 can be moved 360 degrees around the reference point, and is assumed to be operated with the thumb when the grip 235 of the controller 210 is gripped. Similarly, it is assumed that the menu button 280 is operated with the thumb. Furthermore, the controller 210 can incorporate a vibrator (not shown) for applying vibration to the hand of the user who operates the controller 210. In order to output information such as the input contents of the user via a button or joystick, the position, orientation, and tilt of the controller 210 via a sensor or the like, and to receive information from the host computer, the controller 210 has an input / output Part and a communication part.

The system determines the movement and posture of the user's hand based on whether the user holds the controller 210 and operates various buttons and joysticks, and information detected by the sensor and the infrared camera, and simulates in the mixed reality space. The user's hand can be displayed and operated. A controller 210 as shown in FIG. 2 is a representative example, but a controller applicable to the present embodiment is not limited to this, and other types (for example, a general-purpose game controller, a keyboard, voice, gesture input) A controller such as a device that enables the above-mentioned is also applicable.

FIG. 3 is a configuration diagram of the HMD system 300 according to the present embodiment. The HMD system 300 can be configured by, for example, an HMD 110, a controller 210, and an image generation apparatus 310 that functions as a host computer. Furthermore, an infrared camera (not shown) for detecting the position, orientation, inclination, and the like of the HMD 110 and the controller 210 can be added. These devices can be connected to each other by wired or wireless means. For example, each device is equipped with a USB port, and communication can be established by connecting with a cable. In addition, HDMI, wired LAN, infrared, Bluetooth (registered trademark), the Internet, WiFi (registered trademark), etc. The connection can also be established by wired or wireless communication. The image generation apparatus 310 may be an apparatus having an information processing function such as a general computer, a PC, a game machine, or a mobile communication terminal.

FIG. 4 is a functional configuration diagram of the HMD 110 according to the present embodiment. As described in FIG. 1, the HMD 110 may include a sensor 140. As a sensor, although not shown in order to detect movements such as the orientation and inclination of the user's head, for example, a magnetic sensor, an acceleration sensor, a gyro sensor, or a combination thereof can be provided. Further, an eye tracking sensor can be provided. The eye tracking sensor is used to detect the gaze direction and the gazing point of the user's left eye and right eye. Furthermore, in order to detect movements such as the orientation and tilt of the user's head with high accuracy and to detect the position of the user's head, an LED 150 such as infrared light or ultraviolet light may be provided. Further, a camera 160 (for example, a 360-degree omnidirectional camera) for photographing the outside scene of the HMD can be provided. Moreover, the microphone 170 for collecting a user's utterance and the headphone 180 for outputting an audio | voice can also be provided. Moreover, although not shown in figure, the sensor for detecting an odor can also be provided.

Furthermore, the HMD 110 can include an input / output unit 190 for establishing a wired connection with peripheral devices such as the controller 210 and the image generation device 310, for example, and includes hardware, firmware, TCP / IP driver, PPP A communication unit 115 that is implemented as communication software such as a driver or a combination of these and configured to be able to communicate with the image generation apparatus 310 via a communication network such as the Internet, infrared, Bluetooth (registered trademark), WiFi (registered trademark), or the like. Can be provided. Information relating to the movement of the user's head, such as the orientation and tilt of the user, acquired by the sensor 140 is transmitted to the image generating apparatus 310 via the input / output unit 190 or the communication unit 115 by the control unit 125 such as a CPU. In the image generation device 310, an image generated based on the movement of the user's head is received via the input / output unit 190 or the communication unit 115 of the HMD 110 and output to the display unit 120 by the control unit 125. Each of these components is electrically connected to each other via a bus.

FIG. 5 is a functional configuration diagram of the controller 210 according to the present embodiment. As described in FIG. 2, the controller 210 can be configured as a set of controllers for the left hand 220 and the right hand 230, but in any of the controllers, the operation trigger button 240, the joystick 270, the menu button 280, etc. An operation unit 245 can be provided. In addition, the controller 210 can incorporate a sensor 260 in order to detect movements such as the orientation and inclination of the controller 210. Although not illustrated, the sensor 260 can include, for example, a magnetic sensor, an acceleration sensor, a gyro sensor, or a combination thereof. Further, a plurality of infrared LEDs 250 are provided, and the position, orientation, and inclination of the controller 210 in a specific space are detected by detecting the positions of these infrared LEDs with a camera (not shown) provided outside the controller. be able to. The controller 210 can include, for example, an input / output unit 255 for establishing a wired connection with peripheral devices such as the HMD 110 and the image generation device 310, and includes hardware, firmware, a TCP / IP driver, a PPP driver, or the like. And a communication unit 265 configured to be communicable with the terminal device 30 via a communication network such as the Internet, infrared, Bluetooth (registered trademark), WiFi (registered trademark), or the like. Can do. Information input by the user via the operation unit 245 and information such as the orientation and inclination of the controller 210 acquired by the sensor 260 are transmitted to the image generation apparatus 310 via the input / output unit 255 or the communication unit 265.

FIG. 6 is a functional configuration diagram of the image generation apparatus 310 according to the present embodiment. The image generation device 310 stores user input information transmitted from the HMD 110 or the controller 210, information on the user's head or controller movement acquired by a sensor, etc., and performs predetermined program processing to generate an image. It is possible to use a device for performing information processing, such as a general computer, a PC, a game machine, and a mobile communication terminal, which has a function for performing the above-described function. The image generation apparatus 310 can include, for example, an input / output unit 320 for establishing a wired connection with peripheral devices such as the HMD 110 and the controller 210, and includes hardware, firmware, a TCP / IP driver, a PPP driver, and the like. Implemented as communication software or a combination thereof, and configured to be able to communicate with other devices (for example, HMD 110 and controller 210) via a communication network such as the Internet, infrared, Bluetooth (registered trademark), WiFi (registered trademark), etc. The communication unit 330 can be provided. User input information received from the HMD 110 or the controller 210 via the input / output unit 320 or the communication unit 330 and information on the movement of the user's head or controller are transmitted to the control unit 340 such as the user's position, line of sight, and posture. Detected as input content including movement, speech, operation, facial expression, physical condition, etc., and the input content is stored as history information (log) in the storage unit 350 such as ROM, RAM, HDD, etc. Accordingly, based on the image data and parameters of the object stored in the storage unit 350, the control unit 340 determines control of the view field image and the object, and generates an image. The control unit 340 can be configured by a CPU, but by further providing a GPU specialized for image processing, information processing and image processing can be distributed, and overall processing efficiency can be improved. The image generation apparatus 310 can also communicate with other calculation processing apparatuses and share information processing and image processing with other calculation processing apparatuses.

FIG. 7 is a functional configuration diagram relating to the control unit of the image generation apparatus 310 according to the present embodiment. The control unit 340 of the image generation apparatus 310 includes a display control unit 610 that mainly generates a field-of-view image in the mixed reality space, and an object control unit 620 that generates an object image mainly arranged in the mixed reality space.

The display control unit 610 detects a user position based on information on the position, orientation, and tilt of the user's head transmitted from a sensor built in the HMD 110 or an infrared camera disposed outside the HMD 110. 611, a user gaze detection unit 612 that detects the user's gaze direction and gazing point, and a user posture detection unit 613 that detects the orientation and inclination of the user's head. When movements such as the user's position and the user's head orientation and inclination are detected, these pieces of information are output to the visual field region determination unit 614.

The field-of-view area determination unit 614 determines the user's field-of-view area in the mixed reality space based on information on the user's position, head movement, and the like. Here, the mixed reality space can be realized, for example, by displaying a celestial spherical panoramic space image generated in advance as a background image. The panorama space image generated in advance is prepared as background image data stored in the background image data storage unit 712 in the view image storage unit 710 or in the HMD 110 in the storage unit 350 of the present embodiment shown in FIG. Captured by the camera 160 and stored in the external image data storage unit 713 as real space image data.

As shown in FIG. 9, the field-of-view area 920 is determined based on the user's reference line of sight 940 starting from the reference point 930 in the stereoscopic mixed reality space 910 displayed on the display unit 120. The reference point 930 is arranged to be located on the same XZ plane as the center A of the celestial sphere forming the mixed reality space, functions as a starting point of a virtual camera that images the mixed reality space in the reference line-of-sight direction, and is detected by the sensor 140 It moves in the mixed reality space according to the position to be moved, and the direction and inclination of the reference line of sight 940 are changed according to the direction, inclination and line-of-sight direction or gaze point. As shown in FIG. 10A, a predetermined angle α, − (minus) α formed in the left-right direction of the XZ section around the reference line of sight 940 in the mixed reality space of the sensor, and YZ as shown in FIG. 10B. A range including predetermined angles β and − (minus) β formed in the vertical direction of the cross section is set as the visual field region 920. When the view area is set, an area corresponding to the view area of the panorama image is generated as a view area image by the view image generation unit 615. Object image data (for example, a chair or a table) stored in advance in the object image data storage unit 711 of the view field image storage unit 710 can be displayed as a view area image so as to be superimposed on the background image.

The object control unit 620 includes detection units 621 to 625 that detect user input from the HMD 110 or the controller 210, an object control determination unit 626 that determines the output content of the object based on the detected input content, and the determination. An object image generation unit 627 for generating an object image. As user input, for example, the user's speech via the microphone 170 provided in the HMD 110, the user's operation detected via the sensor provided in the HMD 110 or the controller 210, the user's operation via the controller 210, the face Changes in the user's body state acquired via facial expressions of the user acquired through a recognition sensor (not shown), heartbeats and pulses attached to the user's body, body temperature sensors (not shown), etc. Furthermore, the smell etc. which were acquired via the smell sensor (not shown) etc. are mentioned.

The user utterance detection unit 621 detects the user's utterance content transmitted from the microphone 170, and the user operation detection unit 622 is a user's transmission transmitted from a sensor built in the HMD 110, an infrared camera disposed outside the HMD 110, or the like. Based on information on the position, orientation, and tilt of the head and information on the position of the controller 210 transmitted from the controller 210, an infrared camera or the like disposed outside the controller, the user's head or hand movement is detected. Further, the user operation detection unit 623 detects the user operation content based on the information related to the user button operation transmitted from the controller 210. Further, the user facial expression detection unit 624 detects the user's facial expression based on information related to the user's face transmitted from the face recognition sensor (not shown) provided in the HMD 110, and the user physical condition detection unit 625 A change in the user's physical condition is detected based on information such as the heart rate, pulse rate, and body temperature of the user transmitted from the heart rate, pulse rate, body temperature sensor, and the like worn on the body. In addition, the user's odor can be detected based on information on the user's odor transmitted by the odor sensor, and the user's odor detection unit (not shown) can also detect a change in the user's odor.

The object control determination unit 626 determines the output content of the object based on the input content detected by each of the detection units 621 to 625. As the output of the object, for example, when the object is a character object, speech, movement, facial expression, or change in state of the character object can be cited. Based on the output contents of the character object determined by the object control determination unit 626, the object image generation unit 627 generates an image of the character object.

As shown in FIG. 10A, in the present embodiment, at least a part of the character object is placed outside the user's field of view in the mixed reality space at a certain distance from the user in the initial state. It is characterized by. More specifically, when the user's field of view is defined as α, − (minus) α around the user's reference line of sight 1010 in the XZ plane in the mixed reality space, a part of the character object 1020 is , And a certain distance from the reference point 930, which is arranged outside the field of view. Alternatively, in the initial state, the character object 1020 can be arranged at a position of 45 degrees and − (minus) 45 degrees with the user's reference line of sight 1010 as the center in the mixed reality space. . In an initial state such as after the user's HMD 110 is turned on or after an application is started, the character object is placed in a position close to the user, not in a position facing the user. Instead of a typical setting, the user can provide a feeling as if a friend or lover is in the real space. The character object 1020 is stored as object image data stored in the object image data storage unit 721 of the object information storage unit 720 shown in FIG.

In the present embodiment, the character object is characterized in that its movement and facial expression are controlled based on predetermined parameters stored in the parameter data storage unit 722 of the object control information storage unit 720. In this embodiment, a character object can be composed of a plurality of part objects like a human body, and motion parameters corresponding to each part object can be set. Each part object can correspond to a human body part, for example, head, chest, hand, foot, and the like. As the motion parameters, the motion pattern unique to each body part and its size and speed can be set with numerical values in 10 steps. As facial expression parameters, for parts that make up the head as a part object, such as eyebrows, eyes, nose, mouth, etc., the movement pattern specific to each part and its size and speed should be set with 10 numerical values. Can do. Also, a combination of facial expression parameters of each part for expressing emotion parameters (for example, “joy”, “anger”, “sorrow”, “easy”) for the character object is stored in advance, and the user input contents Accordingly, an appropriate combination of facial expression parameters can be determined as the output content. In this way, by combining the movements of a plurality of part objects to form the entire movement of the character object, it is possible to provide the movement of the character object close to the real world. Furthermore, by using random number control as the parameter of each part object, it is possible to provide a motion of a character object that is not a periodic motion and is closer to the real world.

FIG. 11 shows a flowchart for object control processing in the present embodiment. The image generation apparatus 310 receives user input from the HMD 110 or the controller 210 via the input / output unit 320 or the communication unit 320 (S1101). Next, the object control unit 620 of the image generation apparatus 310 detects user input content (S1102). The detection of the user input content is detected by the user utterance detection unit 621, the user action detection unit 622, the user operation detection unit 623, the user facial expression detection unit 624, the user physical condition detection unit 625, and the like. . In response to the detection of the user input content, the object control determination unit 626 determines the output content of the object (for example, a character object) (S1103). The output contents of the character object include utterances, actions, facial expressions or changes in state. When the output content of the object is determined, the object image generation unit 627 generates a display of the object image, and the generated content is displayed in an appropriate data format such as MPEG format together with the background image in the visual field of the user. The data is transmitted to a display device such as the HMD 110. A series of these processes is realized by executing an object control program stored in the object control program storage unit 723 of the storage unit 350 such as a ROM, RAM, or HDD of the image generation apparatus 310.

FIG. 12 shows a flowchart as an example of the object control process. First, the image generation apparatus 310 receives user input from the HMD 110 via the input / output unit 320 or the communication unit 320 (S1201). When the user input is information related to the orientation and inclination of the user's head from the HMD 110 or the line-of-sight direction, the user action detection unit 622 detects the orientation of the user's head (S1202). As an example, the user's line-of-sight direction in the mixed reality space is specified based on information on the orientation and inclination of the user's head transmitted from the HMD 110. Subsequently, the user motion detection unit 622 checks whether or not the orientation of the user's head is directed to the direction of the character / object arranged in the mixed reality space (S1203). As an example, based on the user's line-of-sight direction, the user's view area in the mixed reality space is specified by the xyz coordinates, and the coordinates are compared to determine whether or not this view area overlaps the area where the character object is placed. To confirm. If the orientation of the user's head in the mixed reality space is facing the character object, that is, if the user is looking at the character object in the mixed reality space, the object control decision is made The unit 626 can determine to perform an action based on an action parameter of directing the head of the character object in the direction in which the user is present as the output content of the character object (S1204). When the orientation of the user's head is not suitable for the character object, the user's head orientation is detected after receiving the user input again.

When the object control determining unit 626 determines the action of the character object, or in addition to this, the object control determining unit 626 sets the facial expression parameter in order for the character object to show the feeling of “joy”. (S1205). A combination of facial expression parameters indicating “joy”, for example, a combination of lifting the eyebrows part of the character object, meditating one eye, opening the mouth, and the like can be determined as output contents. When the object control determination unit 626 determines the motion and / or emotional expression of the character object, the object image generation unit 627 generates an image of the character object (S1206).

FIG. 13 shows a flowchart as another example of the object control process. First, the image generation apparatus 310 receives user input from the HMD 110 or an infrared camera disposed outside the HMD 110 via the input / output unit 320 or the communication unit 320 (S1301). For example, when the user input is information related to a change in the position of the user transmitted from the infrared camera, the user operation detection unit 622 detects the position of the user (S1302). As an example, based on the information transmitted from the infrared camera, the displacement (Δx, Δy, Δz) of the user's position (for example, with reference point 930 shown in FIG. 9 as a reference) in the mixed reality space is specified, and the user Is specified by the xyz coordinates. Subsequently, the user motion detection unit 622 detects the distance between the user's position and the character object placed in the mixed reality space (S1303). As an example, based on the latest position coordinates of the user in the mixed reality space and the position coordinates where the character object is placed (for example, the coordinates of the position closest to the user in the area where the character object is placed). Calculate the shortest distance. For example, when the user moves by Δx only in the x coordinate direction, the difference between the x coordinate of the latest position of the user and the x coordinate of the character object is calculated as the distance. When the distance is equal to or less than the predetermined distance, that is, when it is determined that the user is too close to the character object in the mixed reality space, the object control determination unit 626 outputs the character object output content. Then, it is possible to decide to adjust the motion parameter so that the character object is moved away from the position where the user is present (S1304). If the position where the user is located and the position where the character object is located are more than a predetermined distance, the user position detection process is performed after receiving the user input again.

When the object control determination unit 626 determines the action of the character object, or in addition to this, the object control determination unit 626 displays the facial expression in order for the character object to show the feeling of “anger” or “uncomfortable”. A parameter is set (S1305). A combination of facial expression parameters indicating “uncomfortable”, for example, a combination of the movement of the character object that moves the left and right eyebrows parts, the movement of narrowing the eyes, the movement of closing the mouth, and the like can be determined as the output content. When the object control determination unit 626 determines the motion and / or emotional expression of the character object, the object image generation unit 627 generates an image of the character object (S1306).

FIG. 14 shows a flowchart as still another example of the object control process. First, the image generating apparatus 310 receives user input from the HMD 110 via the input / output unit 320 or the communication unit 320 (S1401). For example, when the user input is information on a user gazing point from a sensor provided in the HMD 110, for example, an eye tracking sensor, the user action detection unit 622 detects the gazing point of the user (S1402). As an example, xyz coordinates indicating the user's point of interest in the mixed reality space are specified. Subsequently, the user motion detection unit 622 confirms whether or not the user's gaze point is directed toward the second object (for example, a painting canvas) arranged in the mixed reality space (S1403). . As an example, confirm whether the coordinates indicating the user's point of sight are within the range where the painting canvas is placed in the mixed reality space by comparing with the coordinates indicating the area where the painting canvas is placed To do. When the user's gaze point is directed to the second object, that is, when the user is performing an action of looking at the painting canvas in the mixed reality space, the object control determination unit 626 performs the character control. It is possible to determine to perform the action based on the action parameter of directing the line of sight of the character object to the second object as the output contents of the object (S1404). When the user's gaze point is not directed to the second object, the user's gaze point is detected after receiving the user input again.

When the object control determination unit 626 determines the action of the character object, the object image generation unit 627 generates an image of the character object (S1405). Furthermore, the image generation device continuously detects the user's gaze point in the user motion detection unit 622, and the object control determination unit 626 causes the character object to be displayed when the user turns away from the painting canvas. It is also possible to perform control such as turning the line of sight toward the direction.

FIG. 15 shows a flowchart as still another example of the object control process. This example assumes a scene in which a user draws a picture on a painting canvas arranged in the mixed reality space, and a character object close to the user serves as a teacher. First, the image generation apparatus 310 receives user input from the HMD 110 or the controller 210 via the input / output unit 320 or the communication unit 320 (S1501). When the user input is, for example, an operation performed on the second object (for example, a painting canvas) in the mixed reality space via the controller 210, the user operation detection unit 623 displays the operation content of the user. The object control determination unit 626 detects and updates the content of the second object, that is, determines the control for drawing the picture on the canvas based on the operation content of the user, and updates the object data updated on the painting canvas. Is stored in the object update data storage unit 733 (S1502). Subsequently, the object control determination unit 626 executes a program stored in the matching program storage unit 724 of the storage unit 350 when determining the action of the character object, and is stored in the object processing reference data storage unit 725. With reference to the reference data, it is confirmed whether or not the content of the updated object matches the predetermined correct data (image data) (similarity) (S1503). When the updated object content matches the correct data, the object control determination unit 626 sets the character object to make a predetermined utterance (S1504). For example, it is set to speak such as “You are good”. On the other hand, when the updated object content does not match the correct data, the object control determination unit 626 refers to the reference data, and the character object is determined to be a predetermined correct data ( The image data is set to be drawn on the painting canvas (S1505). The image generation unit 617 and the sound generation unit (not shown) generate an image and sound of the character object (S1506). Further, the object control determining unit 626, for example, moves the character object from outside the user's field of view to a certain position on the painting canvas to draw on the painting canvas, or after the drawing is finished, After the elapse of time, it can be controlled to return again to the same position as the initial state and outside the user's field of view (reference position). In other cases, even if the character object moves to a position different from the reference position in the mixed reality space, control is performed so that the character object returns to the reference position after a predetermined action is completed or after a predetermined time has elapsed. can do. As a result, the sense of solidarity between the user and the character object is enhanced, and the immersive feeling can be further enhanced.

FIG. 16 shows a flowchart as an example of character output control. As the object control determination unit 626 determines the output content of the character object (S1601), the user is inquired whether to accept the output content (S1602). This inquiry can be provided, for example, by performing a pop-up display in the mixed reality space. For example, assuming that the user normally accepts the output contents of the character object, each time the output contents are executed, only the “No” button is displayed for a predetermined time, and when the output contents are accepted, a pop-up display is displayed. The user may be allowed to select the “No” button only when the button is left unaccepted. This is because the conventional pop-up display in which “Yes” and “No” are selected may hinder the user's behavior in the mixed reality space and reduce the immersive effect. When the user accepts the output content of the character object by any of the methods, the output content and a record of acceptance are stored as history information in the object output data storage unit 732 of the storage unit 350 (S1603). . Even when the user does not accept the output content of the character object, the output content and a record of the rejection are stored as history information in the object output data storage unit 732 of the storage unit 350 (S1604). Here, when the user does not accept the output content of the character object, the output of the character object may or may not be executed. When the user accepts the output content of the character object, the object control determining unit 626 refers to the object output data storage unit 732 of the storage unit 350, and how many times the user outputs the output content of the character object including this time. If it is confirmed whether or not it has been accepted a predetermined number of times or more, the object control determination unit 626 refers to the user input data storage unit 731 and executes the user input content executed before and after the output content. Is confirmed (S1606). If the user input content has a certain regularity, the input content and the output content can be stored as a set in the event table or the like (S1607). Thereby, when the user executes the input content stored in the event table after the next time, the character object can be set so that the same output content can be executed again. If there are regularities in the input content before and after the output content, the user can set the input content and output content as a set and store them in the event table etc. Can be set so that the character object can be executed ahead of the output contents of several hands ahead. For example, the input contents of the user operate the PC in the mixed reality space (input 1), display certain data on the PC screen (input 2), request the character object to print the screen (input 3), In response to this, when an input / output pattern such as a character object performing printing in the mixed reality space (output 1) is repeated several times, by registering this pattern in the event table, the next time Thereafter, at the same time that the user starts operating the PC (input 1), the character object may execute output contents such as turning on the printer power (output 1) without waiting for input 2 or input 3. it can.

In this way, a method of controlling the output contents of the character object by machine learning based on the input / output pattern between the user and the character object stored as history information is also conceivable. Furthermore, in conjunction with an application program interface (API) for machine learning provided by a third party, the history information related to input / output necessary for machine learning is used as reference data, so that characters can be obtained through machine learning.・ It is also possible to autonomously control the movement of objects.

<Embodiment 2>
In order to further enhance the effect of the present embodiment, the character object in the first embodiment is at least partially arranged outside the user's view area in the mixed reality space in the initial state. It was devised. In the present embodiment, a plurality of character objects can be selected by the user, and the motion parameters and emotion parameters of each character object can be selected and set by the user.

FIG. 17 shows a table for managing information related to character objects. The character information management table 1700 is managed by the content server 1810 in the HMD 1800 system according to this embodiment shown in FIG. Character server image data and parameter data are stored in the content server 1810. For example, the content server 1810 receives a character / object acquisition request from the image generation apparatus 310 (for example, a user PC). The image data of the character object and the parameter data can be provided to the image generation apparatus 310. The provided related data of, for example, the downloaded character object is stored in the storage unit 350 of the image generation apparatus 310 (in the example of FIG. 8, the object image data storage unit 721 and the parameter data storage unit of the object control information storage unit 720). 722). In another example of this embodiment, character information and image data can be stored in advance in the image generation apparatus 310 without providing a content server.

In the character information management table shown in FIG. 17, a unique character ID is assigned to each character object, and an IP, that is, a character image, is associated with each character ID. In the character information management table, the characters IP associated with the same “A” have the same character image. Even if the character IP is the same, the movement of the character can be set differently depending on the difference of the motion parameters. The character information management table further stores information about motion parameters. For example, the character ID 10001 has IP “A”, but the head, limbs, and torso of the character are set not to move. Has been. On the other hand, in the character ID 10002, the IP is “A”, and the motion parameter is set to “5” for the head and the torso. As described above, when it is assumed that the motion pattern specific to each body part of the character and its size and speed can be set with 10-stage numerical values as the motion parameter, the motion parameter is “5”. The magnitude and strength of the movement is half the maximum value. On the other hand, the character ID 10003 has the same IP “A”, and the motion parameters of the head, limbs, and trunk are all set to “10”. Thus, by providing different motion parameters while having the same image, a wide variety of character objects can be provided.

In this way, by providing stepwise settings for each character object, a character object that is one rank higher when a user performs a predetermined action or satisfies a predetermined condition in the mixed reality space. (For example, the character ID “10002” is unlocked with respect to the character ID “10001” held by the user), the fee is set step by step, and the character object is provided to the user through the charging process You can also The charging process can be executed by a charging server 1820 shown in FIG. Although not shown, the facial expression parameters can be similarly set in a stepwise manner, and as described above, emotion parameters such as “joy”, “anger”, “sorrow”, and “easy” can be selectively selected for the user. It can also be provided.

Further, in the present embodiment, the content server 1810 is operated by the user input data stored in the user input data storage unit 731 of the image generation device 310 (for example, user PC), more specifically, by the user facial expression detection unit 624. Referencing the detected user's facial expression and the physical condition (heart rate, pulse rate, body temperature, etc.) detected by the user physical condition detection unit 625 may recommend a character / object according to the user's emotion and physical condition. it can. For example, when the content server 1810 receives information indicating that the user's facial expression detection unit 624 has detected that the user's feelings are sinking, the content server 1810 displays a character object (for example, soothing user) , A character object having an emotion parameter indicating “sorrow” can be selected and recommended. Alternatively, the content server 1810 can refer to the user's profile data and can recommend a character object that matches the user's profile (age, gender, hobbies, preference type, etc.). It is also possible to manage the object selection history and select a character object with a high probability of being selected by the user using a technique such as collaborative filtering.

<Embodiment 3>
In the present embodiment, it is possible to retrospectively view a moving image including a motion of a user and a character object in a mixed reality space on demand while in the mixed reality space. FIG. 19 shows a user interface in which a time bar 1901 is displayed on a screen showing a mixed reality space. Similar to the follow-up playback of a conventional television program, the user operates an indicator 1902 indicated as a playback position on the time bar 1901 via a controller or the like, thereby allowing the user to play an arbitrary playback position in the mixed reality space. The moving image in the mixed reality space can be reproduced. On the time bar 1901, an area corresponding to 1903 is a buffer area indicating that image data is accumulated in a memory on a reproduction apparatus such as the image generation apparatus 310 or the HMD 110 and can be used by the user. By aligning the indicator 1902 in the buffer area 1903, the user can reproduce the mixed reality space at an arbitrary reproduction position.

FIG. 20 shows a moving image file storage list 2001. The image generation device 310 stores the images generated by the image generation units 615 and 627 in the history information storage unit 730 as a moving image file. The user selects a desired file from the moving image file list 2001 through a predetermined user interface and reproduces it. In the present embodiment, an example in which a moving image file is stored in the image generation device 310 is illustrated. However, for example, a server such as the content server 1810 shown in FIG. 18 connected to the image generation device 310 via the Internet. The moving image file can be stored in the server, and if necessary, the moving image file stored in the server can be streamed or downloaded to a user device such as an HMD or a user PC.

<Embodiment 4>
The present embodiment provides a method for preventing the user from being over-immersed such as staying in the mixed reality space. FIG. 21 is a functional configuration diagram of the image generation apparatus 2100 in the present embodiment.

The image generation apparatus 2100 can include an input / output unit 2110 that receives signals from the HMD 110, the controller 210, and the like, and a communication unit 2120 that receives signals via communication such as the Internet. Further, the image generation apparatus 2100 can include a control unit 2130 that executes a program and a storage unit 2140 that stores programs and data. In the present embodiment, the HMD 110 and the controller 210 described in the first embodiment can be applied. However, the present invention is not limited to this as long as the apparatus can provide a function necessary for realizing the method provided by the present embodiment. However, the HMD and the controller may be in other forms.

First, the storage unit 2140 includes a blocking criterion storage unit 2141 that stores a criterion for blocking a user's access to the mixed reality space input or selected by the user via the HMD 110 or the controller 210, that is, a blocking criterion. Further, a blocking target user storage unit 2142 for storing a blocking target user ID and the like, an event storage unit 2143 for storing an event that triggers access blocking, and a user information storage for registering predetermined user information such as a user ID and an attribute A unit 2144, a program storage unit 2145 for storing a program related to the method provided by this embodiment, and the like can be provided. The control unit 2130 executes a program stored in the program storage unit 2145, and executes a method for preventing a user from accessing the mixed reality space. The blocking criterion determination unit 2131 determines whether or not the blocking criterion is satisfied based on the blocking criterion stored in the blocking criterion storage unit 2141 or the event stored in the event storage unit 2143. The blocking execution unit 2132 If the criteria are met, block access by the user. In addition, the event detection unit 2133 detects an event stored in the event storage unit 2143. The user information disclosure availability determination unit 2134 determines the range of information to be disclosed to the mixed reality space for the user information stored in the user information storage unit 2144. When determining the range of information, when there is a user to be blocked stored in the block to be blocked user storage unit 2142, the necessity of disclosure to the user is also determined.

FIG. 22 shows a flowchart as an example of a method for preventing the user from accessing the mixed reality space. First, when a user inputs or selects a blocking criterion through a blocking criterion setting screen or the like via a user device such as the HMD 110 or the controller 210, the image generation device 2100 performs the input or selection on the input / output unit 2110 or the communication unit 2120. And stored in the blocking criterion storage unit 2141 (S2201). As a blocking criterion, for example, as shown in FIG. 23, the user's stay time (for example, “1 hour”, “2 hours”, “3 hours”, etc.) in the mixed reality space provided via the HMD 110 or the like. , Time zone (for example, “18: 00-6 o'clock”, “22: 00-6 o'clock”, etc.), the number of times of input of commands, posture changes, etc. in applications such as games in the user's mixed reality space (for example, “25 times ”,“ 50 times ”,“ 100 times ”, etc.) Ratings set by providers or third parties for applications that provide mixed reality space (for example,“ PG ”,“ PG16 ”,“ R ”, etc.) Can be set through selection from choices or free input. Next, the image generation device 2100 provides the mixed reality space to the HMD 110 or the like by the image generation device or the like (S2202). Here, the timing before and after the setting of the blocking criterion (S2201) and the provision of the mixed reality space (S2202) is not limited, and the user can set the blocking criterion setting screen or the like at any timing after the presetting or the provision of the mixed reality space. You can set a blocking criterion through In this example, the entity that sets the organization standard is assumed to be a parent of a user (child) who is provided with the mixed reality space.

Subsequently, the blocking criterion determination unit 2131 compares the predetermined action of the user in the mixed reality space with the blocking criterion stored in the blocking criterion storage unit 2141 and determines whether the blocking criterion is satisfied (S2203). ). For example, the image generation device includes a timer (not shown), and the timer measures the user's stay time in the mixed reality space after the HMD 110 is turned on or after the mixed reality space is provided, and is set as a blocking criterion. When the waiting time (for example, 3 hours) elapses, the blocking criterion determination unit 2131 can determine that the blocking criterion is satisfied. Or, for example, when the user has previously set the rating as “R” as the blocking criterion, the blocking is performed when the user selects an application set to “R” by the distribution provider or a third party. It can be determined that the standard is satisfied. In this case, since the mixed reality space is not originally provided, S2202 is skipped. If it is determined in S2203 that the blocking criterion is satisfied, provision of the mixed reality space to the user is blocked (S2204). That is, the user cannot view or play all or part of the composite provision space. In this case, as an execution form of prevention, when an optical transmission type display is used as the display 120 of the HMD 110, the blending ratio between the real space and the virtual space is adjusted by performing control to adjust the transmittance, The user can be returned to the real space. It is also possible to associate a preset blocking criterion with the transmittance and set the transmittance according to the blocking criterion when performing blocking.

FIG. 24 shows a flowchart as another example of a method for preventing a user from accessing the mixed reality space. First, when a user inputs or selects a predetermined event via a user device such as the HMD 110 or the controller 210 through a blocking criterion setting screen or the like, the image generation device 2100 performs the input or selection by the input / output unit 2110 or The data is received by the communication unit 2120 and stored in the event storage unit 2143 (S2401). As the predetermined event, for example, as shown in FIG. 25, “emergency news bulletin” is associated with “notification”, and “earthquake bulletin”, “fire information”, etc. are classified as types. In addition, as another form of “notification”, reception of a call or a message such as a telephone or a chat application can be cited. In addition, detecting an approach of another person or object to the user can be set as an event. The user can register an event as a blocking criterion by activating each item such as notification and approach detection by selecting “YES” and “NO” on a predetermined setting screen. Next, the image generation device 2100 provides the mixed reality space to the HMD 110 or the like (S2402). Here, before and after the timing of the event setting (S2401) and the provision of the mixed reality space (S2402) is not limited, and the user can set through the blocking criterion setting screen or the like at any timing after the presetting or the provision of the mixed reality space. Blocking criteria can be set.

Subsequently, the event detection unit 2133 detects an event such as notification or approach detection (2403). The event is detected via a user's information communication terminal (not shown) connected to the HMD 110 or the like by a wired or wireless communication means or a sensor (not shown) for detecting approach, and the event of the image generation apparatus 2100 is detected. Received as information indicating an event by the input / output unit 2110 or the communication unit 2120. Information indicating an event is detected as a specific event by the event detection unit 2133, and subsequently, the prevention criterion determination unit 2131 determines whether the detected event matches an event stored in advance in the event storage unit 2143, that is, Then, it is determined whether or not the blocking criterion is satisfied (S2404). If it is determined in S2404 that the blocking criterion is satisfied, the provision of the mixed reality space to the user is blocked (S2405). That is, the user cannot view or play all or part of the composite provision space.

FIG. 26 shows a flowchart as yet another example of a method for preventing a user from accessing the mixed reality space. In this example, it is assumed that a large number of specific or unspecified users share the same mixed reality space via a network such as the Internet. First, when a user inputs or selects a blocking criterion through a blocking criterion setting screen or the like via a user device such as the HMD 110 or the controller 210, the image generation device 2100 performs the input or selection on the input / output unit 2110 or the communication unit. It is received by 2120 and stored in the block criterion storage unit 2141 (S2601). As the blocking criteria, for example, as shown in FIG. 27, as an item of “event”, in the mixed reality space provided via the HMD 110 or the like, another user approaches the user, or another user approaches the user. Utterances or chat messages. Furthermore, the user can set specific or all users in association with the event. For example, the user can limit the event “approaching” to an event of approaching by a specific user related to the user ID “30008” by predetermined input or selection. Furthermore, a predetermined condition can be associated with the event. For example, for the event “approaching”, for example, “10 m” is set as the approaching distance on the basis of the distance defined in association with the coordinate information in the mixed reality space. “Approaching within 10 m” can be set as a blocking criterion. Furthermore, steps can be provided as “blocking levels” indicating the degree of blocking performed against blocking criteria. For example, when there are three blocking levels, it is possible to set a criterion such as the strictest “3” (for example, the user cannot use the application that provides the mixed reality space). In the example shown in FIG. 27, when the content of “event” is “utterance”, the users who speak are all users, and “condition” of “utterance” is “Sexual (sexual expression)” in advance. Can be set to utter a word that matches a keyword registered as “”, and the “blocking level” is set to “2” (eg, the user who made the utterance blocks access to the uttered user, ie , Further communication and can not be taken). Some of the blocking criteria as shown in FIG. 27 can be set by the user, and others may be set in advance by the application provider. Next, the image generation device 2100 provides the mixed reality space to the HMD 110 or the like (S2602). Here, the timing before and after the setting of the blocking criterion (S2601) and provision of the mixed reality space (S2602) is not limited, and the user can set the blocking criterion setting screen or the like at any timing after the presetting or the provision of the mixed reality space. You can set a blocking criterion through

Subsequently, the blocking criterion determination unit 2131 compares a predetermined event in the mixed reality space with the blocking criterion stored in the blocking criterion storage unit 2141 and determines whether or not the blocking criterion is satisfied (S2603). For example, when the image generation apparatus 2100 receives that the user associated with the user ID 30008 has detected that the user has approached a distance within 10 m in the mixed reality space, the blocking criterion storage unit 2141 , It is determined that the blocking criterion “a specific user (user ID 30008) approaches the user within 10 m” is satisfied. Alternatively, when the image generation apparatus 2100 receives that a certain user has made an utterance using a pre-registered sexual word, the blocking criterion stored in the blocking criterion storage unit 2141 is used. It is determined that the blockage criterion “speaking a sexual expression to the user” is satisfied. When it is determined that the blocking criterion is viewed, the blocking criterion determination unit 2131 displays the user who is the target of executing the blocking criterion, the user associated with the user ID 30008 and the user who performed the utterance in the above example. Are stored in the blocked user storage unit 2142 as a so-called black list (S2604). Subsequently, the blocking execution unit 2132 executes blocking for the user stored in the blocking target user storage unit 2142 based on the blocking level stored in the blocking reference storage unit 2141 (S2605). For example, in the above example, the user associated with the user ID 30008 is stopped using the application that provides the mixed reality space, or depending on the blocking level, the user is prevented from approaching or approached the approaching user. It is possible to prevent the user from seeing. In addition, the user who has made an utterance can no longer communicate with the user who made the utterance, such as utterance or chat. Further, in addition to or instead of the processing of the blocking execution unit 2132 in S2605, the user information disclosure permission determination unit 2134 discloses the user information stored in the user information storage unit 2144 to the mixed reality space. The range of information to be determined can be determined. When determining the range of information, when there is a user to be blocked stored in the block to be blocked user storage unit 2142, it is also possible to determine whether disclosure is necessary for the user. As a result, in addition to execution of prevention for the user to be blocked, security can be further strengthened, and when replacing with execution of blocking for the user to be blocked, a load is imposed on the calculation processing of the network band and other devices. However, it is possible to obtain an effect substantially equivalent to the execution of the blocking for the target user. Furthermore, in the user information disclosure availability determination unit 2134, as a default setting, at least the real name of the user or the information that can identify the individual is forcibly disclosed, so that the approach in the mixed reality space can be It can also prevent harassment through remarks.

Note that, as described above, this example is based on the premise that a large number of specific or unspecified users share the same mixed reality space via a network such as the Internet. Therefore, the image generation apparatus 2100 or another server It is assumed that a network connection is established with another user device via the device. Therefore, all or a part of the blocking criteria and user information stored in the image generation device 2100 is shared by the other server device when there is another server device, and the blocking determination or blocking is performed. The function such as execution may be executed by the other server apparatus together with the image generation apparatus 2100 or alternatively.

<Embodiment 5>
In the present embodiment, there is provided means for realizing a user's immersive feeling in the mixed reality space by utilizing a blending function between the real space and the virtual space in the mixed reality space. FIG. 28 shows a functional configuration example of the image generation apparatus 2800 in the present embodiment. In the present embodiment, the HMD 110 and the controller 210 described in the first embodiment can be applied. However, the present invention is not limited to this as long as the apparatus can provide a function necessary for realizing the method provided by the present embodiment. However, the HMD and the controller may be in other forms.

The image generation apparatus 2800 can include an input / output unit 2810 that receives signals from the HMD 110, the controller 210, and the like, and a communication unit 2820 that receives signals from the apparatus and the like via communication such as the Internet. In addition, the image generation apparatus 2800 can include a control unit 2830 that executes a program and a storage unit 2840 that stores programs and data.

The control unit 2830 is further imaged by the image generation unit 2831 that generates an image for providing the mixed reality space, and the camera 160 provided in the HMD 110 that is stored in the object determination program storage unit 2843 of the storage unit 2840. An object determination unit 2832 for recognizing an object arranged in an image of the real space and searching for a similar image, executing an object determination program, and performing determination for object recognition. In addition, the storage unit 2840 stores a real space image captured by the camera 160 as image data in units of objects, and a real space object storage unit 2841 that superimposes the image on the real space image to superimpose the mixed reality space. The virtual space object storage unit 2842 can be provided with the image of the object in the virtual space used to generate the image as image data in units of objects. Note that the above-described object determination program can also be implemented by accessing an image recognition or image search program provided by a third party via an API.

FIG. 29 shows a flowchart as an example of a method for providing a mixed reality space in the present embodiment. First, the image generating apparatus 2800 receives image data in the real space outside the HMD 110 captured by the camera 160 provided in the HMD 110 via the input / output unit 2810 or the communication unit 2820 (S2901). Next, the object determination unit 2832 executes the object determination program stored in the object determination program storage unit 2835 and recognizes the shape and target of the object data constituting the real space based on the image data of the real space ( S2902). The image data of the object constituting the real space and the related information are stored in the real space object storage unit 2841. Subsequently, the object determination unit 2832 determines whether there is a virtual space object corresponding to the image data of the object constituting the real space stored in the real space object storage unit 2841 and related information, and the virtual space object storage unit 2842. A comparison is made with the image data and related information of the virtual space object stored in (S2903).

An example of the related information of the virtual space object stored in the virtual space object storage unit 2842 is shown in FIG. As virtual space objects, item IDs are assigned to the categories of each object, and different object IDs are associated with each item ID and managed. When a specific example of the object managed in the table of FIG. 30 is given, the item ID corresponding to the category “Coffee Table” is “70022”, and the virtual space object storage unit 2842 corresponds to the item ID “70022”. Two objects are stored, and they have object IDs “800001” and “800002”, respectively. This indicates that two different objects are stored for the item coffee table. For example, a round coffee table and a square coffee table are stored.
In addition, various objects may be stored according to the shape and color of the object. These objects are stored in the same virtual space object storage unit 2842 as image data, and the image data and related information stored in an external database are downloaded in advance to the image generation apparatus 2800, and the virtual data The object can be stored in the space object storage unit 2842, or an object stored in an external database can be downloaded and displayed in response to a user request.

Returning to FIG. 28, the object determination unit 2832 can search for information on the virtual space object stored in the virtual space object storage unit 2842 using, for example, the ID of the recognized real space object as a key as the search process. . When a virtual object that matches an object constituting the real space is stored in the virtual space object storage unit 2842 (or an external database), the image generation device 2800 can replace the real space object with the user. A list of virtual space objects can be displayed. If a virtual object that matches an object constituting the real space is not stored in the virtual space object storage unit 2842, the process returns to the first process in the flowchart. When the selection of the virtual space object by the user is received (S2904), the image generation unit 2831 superimposes the selected virtual space object on the position coordinates of the real space object to be replaced in the real space image. (S2905). As a result, it is possible to create a mixed reality space as if the object in the real space was replaced with the object in the virtual space.

FIG. 31A and FIG. 31B are diagrams illustrating a mixed reality space generated by replacing a virtual space object with a real space object. FIG. 31A is an image obtained by imaging the real space. There is a coffee table in the center of the room, with dead flowers near the background wall. In the room, paper scraps and fallen coffee cups are randomly placed. FIG. 31B shows that the user can enjoy a mixed reality space different from the real space through the display of the HMD 110 by replacing the virtual space object with the real space object using the object replacement method according to the present embodiment. Can do. For example, a square coffee table arranged in the center of the room is replaced with a virtual space object ID 800001 (circular coffee table) in FIG. 30, and a withered flower is replaced with a flower of object ID 890188. Further, by replacing the floor with a beautiful floor having the object ID 810020, the user can live in a clean room through the display of the HMD 110.

As mentioned above, although embodiment of this invention was described, this invention is not limited to above-described embodiment, The change of the conditions etc. which do not deviate from a summary are all the application scopes of this invention.

The control method of the present invention can be suitably used for all applications such as games and entertainment in which objects such as character objects are arranged in the mixed reality space. Further, since the user can stay in the mixed reality space and enjoy interaction with the object, it can be applied to a wide range of industries such as entertainment such as games, nursing care and welfare.

110 Head Mount Display 115 Communication Unit 120 Display Panel 125 Control Unit 140 Sensor 150 Light Source 160 Camera 170 Microphone 180 Headphone 190 Input / Output Unit 210 Controller 220 Left Hand Controller 230 Right Hand Controller 240a, 240b Trigger Button 245 Operation Unit 250 Infrared LED
255 Input / output unit 260 Sensor 270 Joystick 280 Menu button 290 Frame 310 Image generation device 320 Input / output unit 330 Communication unit 340 Control unit 350 Storage unit 610 Display control unit 611 User position detection unit 612 User gaze detection unit 613 User posture detection unit 614 Visual field region determination unit 615 Visual field image generation unit 620 Object control unit 621 User speech detection unit 622 User motion detection unit 623 User operation detection 624 User facial expression detection unit 625 User body state detection unit 626 Object control determination unit 627 Object image generation unit 710 Image storage unit 711 Object image data storage unit 712 Background image data storage unit 713 External image data storage unit 720 Object control information storage unit 721 Object image data storage unit 722 Meter data storage unit 723 Object control program storage unit 724 Matching program storage unit 725 Object processing reference data storage unit 730 History information storage unit 731 User input data storage unit 732 Object output data storage unit 733 Object update data storage unit 910 Mixed reality Space 920 Field of view 930 Reference point 940 Reference line of sight 1010 Reference line of sight 1020 Character object 1810 Content server 1820 Billing server 1901 Time bar 1902 Indicator 1903 Buffer area 2100 Image generation device 2110 Input / output unit 2120 Communication unit 2130 Control unit 2131 Blocking reference determination unit 2132 Blocking criterion 2133 Event detection unit 2134 User information disclosure availability determination unit 2140 Storage unit 2141 Blocking criterion storage unit 21 42 Blocking user storage unit 2143 Event storage unit 2144 User storage unit 2145 Program storage unit 2800 Image generation device 2810 Input / output unit 2820 Communication unit 2830 Control unit 2831 Image generation unit 2832 Object determination unit 2840 Storage unit 2841 Real space object storage unit 2842 Virtual space object storage unit 2843 Object determination program storage unit

Claims (53)

1. A method of controlling an object in a mixed reality space,
   In an initial state, disposing at least part of the object at a predetermined position away from the user by a predetermined distance outside the user's field of view;
   Detecting the user's input in the mixed reality space;
   Controlling the output of the object at the predetermined position in response to the input of the user;
   How to control the object.
2. The object control method according to claim 1, comprising:
   The object control method according to claim 1, wherein the user input is any one of the user's utterance, motion, operation, facial expression, or change in body condition or smell in the mixed reality space.
3. The object control method according to claim 1, comprising:
   The object control method according to claim 1, wherein the output of the object is any one of an utterance, an action, an operation, a facial expression or a state change of the object in the mixed reality space.
4. The object control method according to claim 1, comprising:
   Detecting the user's input includes detecting the user's line-of-sight direction in the mixed reality space;
   The step of controlling the output of the object includes controlling the line-of-sight direction of the object.
5. The object control method according to claim 1, comprising:
   The object is composed of a plurality of part objects,
   Controlling the output of the object includes controlling the movement of the part object.
6. 2. The object control method according to claim 1, wherein the predetermined position is set at a position of 45 degrees or −45 degrees with respect to a user's line-of-sight direction in an initial state.
7. The object control method according to claim 1, wherein the predetermined position is set as a reference position of the object.
8. The object control method according to claim 7, comprising:
   In the step of controlling the output of the object, the object is moved to a position different from the reference position and moved to the reference position after a predetermined time has elapsed or after the object has executed a predetermined output. A method for controlling an object comprising:
9. 3. The object control method according to claim 2, wherein the movement of the object is controlled based on a predetermined movement parameter.
10. 4. The method for controlling an object according to claim 3, wherein the expression of the object is controlled based on a predetermined emotion parameter.
11. 3. The object control method according to claim 2, wherein the operation of the object is controlled based on a random number parameter.
12. 3. The method of controlling an object according to claim 2, wherein the operation of the object is controlled based on a parameter acquired by a machine learning process.
13. 2. The object control method according to claim 1, wherein the step of detecting an input of the user includes detecting an input of the user to the object, and the step of controlling the output of the object includes: A control method, comprising: controlling an output of the object based on an input to the object.
14. The method for controlling an object according to claim 1, wherein the step of detecting the input of the user includes detecting the gazing point of the user in the mixed reality space, and controlling the output of the object. And controlling the output of the object based on the gaze point.
15. The object control method according to claim 1, wherein the output of the object is controlled based on a distance between the user and the object in the mixed reality space.
    
16. The object control method according to claim 1, wherein a second object is arranged in the mixed reality space,
    Detecting the user's input includes detecting a gaze point of the user with respect to the object in the mixed reality space, and controlling the operation of the object based on the gaze point. Characteristic control method.
17. The object control method according to claim 1, comprising:
    Placing a second object in the mixed reality space;
    Detecting the user's input in the mixed reality space, detecting the gazing point of the user with respect to the object, and controlling the output of the object in the step of controlling the output of the object; A control method comprising controlling
18. The object control method according to claim 17, comprising:
    A control method comprising: controlling the gaze point of the user and the gaze point of the object so as to be located within a predetermined area in the second object.
19. The object control method according to claim 16, comprising:
    Control that controls the object to execute a predetermined action in response to executing a predetermined process on the second object based on an input of the user in the mixed reality space Method.
20. The object control method according to claim 19, comprising:
    A control method further comprising: executing a predetermined process on the second object based on a predetermined operation of the object.
21. The object control method according to claim 1, comprising:
    A control method comprising providing the user with an option to select image data of the object from a plurality of image data.
22. The object control method according to claim 9, comprising:
    A control method characterized by providing the user with an option of selecting a motion parameter of the object from a plurality of motion parameters.
23. The object control method according to claim 10, comprising:
    A control method comprising providing the user with an option of selecting an emotion parameter of the object from a plurality of emotion parameters.
24. The object control method according to claim 9, comprising:
    A control method comprising recommending a predetermined operation parameter of the object based on a profile of the user.
25. The object control method according to claim 10, comprising:
    A control method comprising recommending a predetermined emotion parameter of the object based on a profile of the user.
26. The object control method according to claim 9, comprising:
    A control method characterized by detecting a change in the physical condition of the user and recommending a predetermined motion parameter of the object based on the physical condition.
27. The object control method according to claim 10, comprising:
    A control method characterized by detecting a change in the physical condition of the user and recommending a predetermined emotion parameter of the object based on the physical condition.
28. The object control method according to claim 5, comprising:
    Set operation parameters to control the movement of the plurality of part objects,
    A control method characterized by selectively providing an operation parameter set to each of the plurality of part objects to a user.
29. An object control method according to claim 9 or claim 10, wherein:
    A control method comprising providing an operation parameter or an emotion parameter to the user through a billing process.
30. The object control method according to claim 1, comprising:
    A control method for controlling output of the object based on a history of contents of the user input with respect to the output of the object.
31. The object control method according to claim 1, comprising:
    A control method for displaying whether or not the user accepts the output content of the object.
32. The object control method according to claim 1, comprising:
    A control method comprising: providing the user with an option to cancel the output contents of the object.
33. The object control method according to claim 1, wherein the moving image data constituting the mixed reality space is stored in a server in association with time information, and the moving image data corresponds to a time point associated with a user request. The control method characterized by delivering.
34. 34. The object control method according to claim 33, wherein a transfer bar for instructing reproduction of moving image data at an arbitrary point of time among moving image data stored in the server is arranged, and the transfer is performed. A control method of displaying moving image data at a time point associated with a position designated by a user on a bar.
35. The object control method according to claim 1, wherein the input content of the user is stored in the server as history information.
36. A method for controlling access to a mixed reality space provided via a display attached to a user's body,
    Entering blocking criteria to block access to the mixed reality space;
    Providing the user with a mixed reality space,
    When the blocking criteria are met,
    Prohibiting providing a mixed reality space to the user.
37. A method for performing access control according to claim 36, comprising:
    The blocking criterion includes a staying time for staying in the mixed reality space for the user.
38. A method for controlling access to a mixed reality space provided via a display attached to a user's body,
    Providing the user with a mixed reality space,
    Detecting certain events in the real world,
    Providing a mixed reality space to the user when the predetermined event is detected is prevented.
39. A method for performing access control according to claim 38, comprising:
    The predetermined event includes receiving an emergency news bulletin in the real world.
40. A method for performing access control according to claim 38, comprising:
    The method according to claim 1, wherein the predetermined event includes a predetermined action by the user in the real world.
41. A method for performing access control according to claim 38, comprising:
    The method, wherein the predetermined event includes an approach of another human body or object to the user in the real world.
42. A method for performing access control according to claim 36 or claim 38, comprising:
    Prohibiting providing the user with a mixed reality space includes changing the transparency of the real space to the virtual space in the mixed reality space.
43. A method for controlling access of a specific user to a mixed reality space via a display attached to a user's body,
    Setting deterrence criteria to deter specific users from accessing the mixed reality space;
    Providing the user with a mixed reality space,
    When the blocking criteria are met,
    Prohibiting providing a mixed reality space to the user.
44. A method for performing access control according to claim 43, comprising:
    The blocking criterion includes a user in the mixed reality space approaching a predetermined distance relative to other users in the mixed reality space.
45. A method for performing access control according to claim 43, comprising:
    The blocking criterion includes a user in the mixed reality space transmitting an utterance or message having a predetermined content to another user in the mixed reality space.
46. A method for performing access control according to claim 43, comprising:
    Prohibiting providing a mixed reality space to the user comprises the user controlling access to other users in the mixed reality space.
47. A method for performing access control according to claim 43, comprising:
    Prohibiting providing a mixed reality space to the user includes the user controlling access to other specific users in the mixed reality space.
48. A method for performing access control according to claim 43, comprising:
    Prohibiting providing the user with a mixed reality space includes preventing other users in the mixed reality space from viewing the user.
49. (Virtual space established based on personal information)
    A method for performing access control according to claim 44, comprising:
    Disclosing personal information of the specific user in the mixed reality space.
50. A method of providing a mixed reality space through a display attached to a user's body,
    Detect objects in real space,
    Identify the object in the mixed reality space corresponding to the object in the real space,
    A method characterized by displaying the identified object superimposed on the object in the real space.
51. 51. A method of providing a mixed reality space according to claim 50, comprising:
    A method of providing an option for a user to select an object in the mixed reality space from a plurality of objects.
52. 51. A method of providing a mixed reality space according to claim 50, comprising:
    Further, an identifier corresponding to the detected object in the real space is specified,
    Identifying an object having an identifier corresponding to the identified identifier as an object in the mixed reality space.
53. 51. A method of providing a mixed reality space according to claim 50, comprising:
    Recognizing the shape of the detected object in the real space and generating the identified shape of the object in the mixed reality space to match the shape of the object in the real space.

Images