WO2025083847A1 - 体感空間切替方法、体感空間切替システム、体感空間切替装置、並びに体感空間切替プログラム及びそれを記録した記録媒体 - Google Patents

体感空間切替方法、体感空間切替システム、体感空間切替装置、並びに体感空間切替プログラム及びそれを記録した記録媒体 Download PDF

Info

Publication number
WO2025083847A1
WO2025083847A1 PCT/JP2023/037877 JP2023037877W WO2025083847A1 WO 2025083847 A1 WO2025083847 A1 WO 2025083847A1 JP 2023037877 W JP2023037877 W JP 2023037877W WO 2025083847 A1 WO2025083847 A1 WO 2025083847A1
Authority
WO
WIPO (PCT)
Prior art keywords
space
user
virtual
environment
recognized
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/JP2023/037877
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
良哉 尾小山
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Abal Inc
Original Assignee
Abal Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Abal Inc filed Critical Abal Inc
Priority to US18/839,156 priority Critical patent/US20250363752A1/en
Priority to JP2024510536A priority patent/JP7539006B1/ja
Priority to PCT/JP2023/037877 priority patent/WO2025083847A1/ja
Publication of WO2025083847A1 publication Critical patent/WO2025083847A1/ja
Anticipated expiration legal-status Critical
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—Three-dimensional [3D] animation
    • G06T13/40—Three-dimensional [3D] animation of characters, e.g. humans, animals or virtual beings
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/20—Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/20—Indexing scheme for editing of 3D models
    • G06T2219/2004—Aligning objects, relative positioning of parts
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/07—Target detection

Definitions

  • the present invention relates to a sensory space switching method executed by a computer system for switching the space experienced by a user via an environmental output device, a sensory space switching system that is the computer system, a sensory space switching device that is the computer system, a sensory space switching program that causes a computer system to execute the sensory space switching method, and a recording medium on which the program is recorded.
  • HMD head-mounted display
  • One such type of virtual space experience system changes the virtual space environment experienced by the user (e.g., the position of the user's viewpoint in the virtual space) when a specific trigger event (e.g., a specific action by the user) is recognized (see, for example, Patent Document 1).
  • a specific trigger event e.g., a specific action by the user
  • the virtual space experience system described in Patent Document 1 merely enables the environment experienced by the user to be switched from a first-person perspective to a third-person perspective within a single virtual space.
  • the switching of the virtual space environment experienced by the user is also something that could also be achieved in real space.
  • the present invention has been made in consideration of the above points, and aims to provide a sensory space switching method, sensory space switching system, sensory space switching device, sensory space switching program, and a recording medium having the same recorded thereon, which allow a user to fully experience an environment that cannot be experienced in real space.
  • composite space refers to a so-called mixed reality space (MR space) in which virtual objects exist in real space and the user in the real space can affect the virtual objects (for example, by moving the virtual objects).
  • MR space mixed reality space
  • Virtual space refers to a virtual space that the user perceives in place of the real space in which the user exists, a so-called virtual reality space (VR space).
  • VR space virtual reality space
  • virtual objects and avatars that move in response to the user's movements may be placed.
  • the sensory space switching method of the present invention includes: A sensory space switching method for switching a space to be recognized by a user via an environment output device, the method being executed by a computer system, comprising: A real space recognition unit recognizes a real space in which the user exists; A virtual object generation unit generates a virtual object to be recognized by the user; A composite space generating unit generates a composite space in which the virtual object is present in the real space; a step of a user environment determination unit determining an environment of the composite space to be recognized by the user via the environment output device; A user action recognition unit recognizes an action of the user in the real space; A trigger event recognition unit recognizes a predetermined trigger event, which is an action performed by the user on the virtual object; A virtual space generating unit generates a virtual space when the predetermined trigger event is recognized; The method is characterized in that it includes a step in which the user environment determination unit determines, via the environment output device, the environment of the virtual space to be recognized by the user, instead of the environment of the composite space that the user was
  • this method of switching between immersive spaces allows the user to switch between different environments, without distinguishing between a composite space in which virtual objects exist in real space and a virtual space that is separate from real space.
  • the trigger event that triggers the switch is an action taken by the user on a virtual object that exists in the composite space. And because that virtual object is virtual, it is reminiscent of a virtual space even though it exists in the real space.
  • the environment experienced by the user is not limited to a composite space or a virtual space as in the past, but can be a mixture of a composite space and a virtual space. Furthermore, since switching from a composite space to a virtual space can be performed without giving the user a sense of discomfort, it is possible to prevent the switching from impairing the user's immersive feeling. Ultimately, according to this immersive space switching method, it is possible to allow the user to experience environments that cannot be experienced in real space in a wider range of forms than before.
  • a target person recognition unit recognizes, from a plurality of people present in the real space, the user who is a target person for whom the type of space to be experienced is to be switched, and a non-target person for whom the type of space to be experienced is not to be switched; an avatar generating unit generating a first avatar that moves in response to a motion of the user and a second avatar that moves in response to a motion of the non-target person; It is preferable that the virtual space environment that the user is made to recognize includes the first avatar and the second avatar.
  • the experiential space switching method of the present invention may be configured so that multiple users in the same area of real space experience one composite space, and further, only the space experienced by some of the multiple users (target persons) is switched to virtual space. It may also be configured so that people (target persons) who experience a composite space and a virtual space and people (non-target persons) who do not experience a virtual space or composite space can exist simultaneously in the same area of real space.
  • the switched virtual space include not only the first avatar corresponding to the target user, but also the second avatar corresponding to the non-target user.
  • the virtual object has an attribute corresponding to an attribute of the virtual space generated when the predetermined trigger event occurs with respect to the virtual object.
  • attributes include the shape, pattern, color, or combinations of these of the virtual space.
  • the attributes of the virtual object and the attributes of the virtual space may match, or may be mutually associated.
  • the pattern of the background of the virtual space and the pattern of the virtual object may match, or if the virtual space is a space that resembles an "office," the shape of the virtual object may be a "building.”
  • the user who causes the trigger event via that virtual object can unconsciously predict the attributes of the virtual space into which they will subsequently move. This makes it possible to switch the type of space experienced by the user without giving them any further sense of discomfort.
  • the shape of the virtual space is generated according to the shape of the real space at the time when the trigger event is recognized.
  • the shape of real spaces (for example, the types of furniture arranged there) varies widely and can also change over time, so if a virtual space of a fixed shape is generated regardless of the shape of the real space, the range of movement may differ depending on the shape of the real space, even though it is the same virtual space. Ultimately, this could cause the user to feel uncomfortable.
  • the range in which movement is possible in the generated virtual space can be made to match that real space. This makes it possible to prevent the user from feeling uncomfortable due to differences between the shapes of the virtual space and the real space.
  • the method includes a step of generating a space in which the composite space and the virtual space are mixed by the virtual space generating unit, It is preferable that the user environment determination unit includes a step of determining an environment of the space that mixes the composite space and the virtual space, to be recognized by the user via the environment output device, in place of the environment of the composite space that the user was made to recognize before the specified trigger event was recognized, and then determining the environment of the virtual space to be recognized by the user via the environment output device.
  • This configuration can reduce the abruptness that may be felt by the user when switching the space they are experiencing from a composite space to a virtual space. In turn, it can also switch the type of space they are experiencing without giving the user any sense of discomfort.
  • the sensory space switching system of the present invention comprises:
  • the present invention is characterized in that it is a computer system that executes any one of the sensory space switching methods described above.
  • the sensory space switching device of the present invention comprises:
  • the present invention is characterized in that it is a computer system that executes any one of the sensory space switching methods described above.
  • the present invention relates to a sensory space switching program.
  • the present invention is characterized in that a computer system is caused to execute any one of the sensory space switching methods described above.
  • the recording medium of the present invention comprises:
  • the present invention is characterized in that the above-mentioned sensory space switching program is recorded, and the sensory space switching program is readable by the computer system.
  • FIG. 1 is a schematic diagram showing a schematic configuration of a spatial experience system according to an embodiment.
  • FIG. 2 is a block diagram showing the configuration of a processing unit of the spatial experience system of FIG. 1 .
  • FIG. 2 is a schematic diagram showing an example of a composite space generated by the spatial experience system of FIG. 1 .
  • FIG. 2 is a schematic diagram showing an example of a virtual space generated by the spatial experience system of FIG. 1 .
  • 2 is a schematic diagram showing the shape of a virtual space generated by the spatial experience system of FIG. 1 and the shape of a corresponding real space;
  • 2 is a schematic diagram showing a state of a space that the user is made to recognize when the spatial experience system of FIG.
  • 4 is a flowchart showing a process executed by the spatial experience system of FIG. 1 in a state in which the user is experiencing a complex space.
  • 4 is a flowchart showing the first half of the processing executed by the spatial experience system of FIG. 1 when switching the space experienced by the user.
  • 10 is a flowchart showing the latter half of the processing executed by the spatial experience system of FIG. 1 when switching the space experienced by the user.
  • 4 is a flowchart showing a process executed by the space experiencing system of FIG. 1 when a user is experiencing a virtual space.
  • the system S is a computer system that allows a first user U1 and a second user U2 (hereinafter collectively referred to as "users U") who are both present in a specific area of the real space RS (e.g., a single room; see Figure 1) to experience a composite space MS or a virtual space VS.
  • a first user U1 and a second user U2 hereinafter collectively referred to as "users U” who are both present in a specific area of the real space RS (e.g., a single room; see Figure 1) to experience a composite space MS or a virtual space VS.
  • the system S allows the user U to experience the complex space MS (see FIG. 3) corresponding to the real space RS by having the user recognize the environment (e.g., images, sounds, etc.) of the complex space MS (see FIG. 3) (e.g., having the user recognize that the user U exists in the complex space MS).
  • the system S also uses a specified trigger event as a trigger to allow the target user U to experience the virtual space VS (see FIG. 4) corresponding to the real space RS by having the target user U recognize the environment of the virtual space VS.
  • composite space refers to a so-called mixed reality space (MR space) in which virtual objects exist in real space and users in the real space can affect those virtual objects (for example, by moving the virtual objects).
  • MR space mixed reality space
  • Virtual space refers to a virtual space that a user perceives in place of the real space in which the user exists, and is known as a virtual reality space (VR space). In this virtual space, virtual objects and avatars that move in response to the user's movements may be placed.
  • VR space virtual reality space
  • the number of users using the sensory space switching system of the present invention is not limited to this configuration, and may be one person, or three or more people.
  • the system S includes a number of signs 1 attached to a user U present in the real space RS, a camera 2 that photographs the user U (or, more precisely, the signs 1 attached to the user U), a server 3 (a sensory space switching system) that determines the environment of the composite space MS or the virtual space VS1 that the user U experiences, and a head-mounted display (hereinafter referred to as "HMD 4") that is an environmental output device that allows the user to recognize the determined environment.
  • HMD 4 head-mounted display
  • the camera 2, server 3, and HMD 4 can wirelessly transmit and receive information between each other via the Internet network, public lines, short-distance wireless communication, etc. However, any of them may be configured to transmit and receive information between each other via wires.
  • the multiple signs 1 are attached to the user U's head, both hands, and both feet via the HMD 4, gloves, and shoes worn by the user U.
  • the multiple signs 1 are used to recognize the user U's movements in the real space RS, as described below. Therefore, the positions at which the signs 1 are attached, the number of signs 1 attached, etc. may be changed as appropriate depending on the other devices that make up the system S.
  • Camera 2 is installed so that it can capture images of the user U's range of motion (i.e., the range in which user U can move and act) in the real space RS in which user U exists, from multiple directions.
  • range of motion i.e., the range in which user U can move and act
  • the server 3 (sensory space switching system) recognizes (e.g., detects or calculates) the sign 1 from the image captured by the camera 2, and recognizes the user U's movements (e.g., changes in posture and coordinates, etc.) based on the position of the recognized sign 1 in the real space RS.
  • the server 3 also determines the environment of the composite space MS or virtual space VS that the user U will recognize based on the movements.
  • the HMD 4 is an environment output device that outputs and allows the user to recognize the environment (e.g., images and sounds) of the complex space MS or virtual space VS.
  • the HMD 4 is worn on the head of the user U.
  • the HMD 4 has a monitor 40 that allows the user U to recognize the image of the complex space MS or virtual space VS determined by the server 3, and a speaker 41 that allows the user U to recognize the sound of the complex space MS or virtual space VS determined by the server 3 (see Figure 2).
  • the monitor 40 When the user U is experiencing the composite space MS, the monitor 40 allows the real space RS to be observed through the monitor 40.
  • the user U When the user U experiences the complex space MS using the system S, the user U is able to recognize images and sounds of a virtual object corresponding to the complex space MS being experienced, along with images and sounds of the real space RS.
  • the virtual object is configured to perform a predetermined action (e.g., moving coordinates) in response to the actions of the user U.
  • the system S When the system S is used to experience the virtual space VS, the user U is made to perceive only the images and sounds of the virtual space VS via the HMD 4, and is made to perceive that he or she is present in the virtual space VS.
  • the system S is configured as a so-called immersive system.
  • the immersive space switching system of the present invention is not limited to the configuration using signs and cameras as described above (so-called motion capture devices), but may be configured to recognize real space and the user's movements.
  • the number and arrangement of signs and cameras may be different from those shown in FIG. 1.
  • signs may be attached not only to the user but also to real objects that exist in the real space.
  • Feature points may be recognized from the image itself without using signs.
  • the HMD may be equipped with a sensor such as a GPS, and the user's movements may be recognized based on the output from the sensor.
  • a sensor such as a GPS
  • Such a sensor may also be used in combination with the motion capture device described above.
  • the immersive space switching system of the present invention is not limited to being configured with one server, but any of the devices that make up the immersive space switching system may be configured to include a processing unit as described below.
  • the entire sensory space switching system may be configured using multiple servers.
  • at least one of the processing units or at least part of the functions of the processing unit may be implemented in a camera, HMD, or other device, and the system may be configured with these devices working in cooperation with the server, or with only these devices.
  • the camera 2, server 3, and HMD 4 that make up the system S are composed of one or more electronic circuit units including a CPU, RAM, ROM, interface circuits, etc.
  • the server 3 includes, as functions (processing units) realized by at least one of the implemented hardware configuration and the program, a real space recognition unit 30, a composite environment generation unit 31, a user environment determination unit 32, a user state recognition unit 33 (user action recognition unit), a virtual object control unit 34, a trigger event recognition unit 35, a target user recognition unit 36 (target person recognition unit), a virtual environment generation unit 37, and an avatar state control unit 38.
  • the real space recognition unit 30 recognizes image data of the real space RS captured by the camera 2, and recognizes the situation of the real space RS based on the image data.
  • the situation of the real space RS is, for example, the posture and coordinates of the user U and real objects (in this embodiment, the table RO1, the bookshelf RO2, and the drawer RO3) that exist in the real space RS.
  • the composite environment generation unit 31 has a virtual object generation unit 31a and a composite space generation unit 31b.
  • the virtual object generating unit 31a generates virtual objects (in this embodiment, a model MO, which is a virtual object of a model castle, virtual book objects packed in a bookshelf RRO2, etc.) that exist in the composite space MS and are recognized by the user U (see FIG. 3, etc.).
  • the virtual objects include objects that move in response to the movements of the user U in the composite space MS (for example, objects whose posture and coordinates in the composite space MS can be changed).
  • the composite space generation unit 31b creates a composite space MS as shown in FIG. 3 by placing the virtual objects generated by the virtual object generation unit 31a in the real space RS recognized by the real space recognition unit 30.
  • the monitor 40 mounted on the HMD 4 allows the real space RS to be observed through the monitor 40. Therefore, when the user U experiences the determined composite space MS, the determined virtual object is displayed so as to be superimposed on the real space RS that is recognized through the monitor 40, and the determined sound is generated by the speaker 41 mounted on the HMD 4.
  • the composite space generating unit 31b indirectly generates a composite space by determining the posture and coordinates of a virtual object in the real space RS that the user U is observing through the monitor 40 of the HMD 4, as well as the sound to be generated based on that virtual object.
  • the process of generating a composite space may involve generating an image in which a virtual object is superimposed on an image of the real space acquired by a camera or the like provided on the HMD, rather than using the real space observed by the user through the HMD, and using that image as the composite space.
  • the user environment determination unit 32 determines the environment of the composite space MS or the virtual environment VS that the user U will recognize through the monitor 40 and speaker 41 of the HMD 4.
  • the "environment to be recognized" by the user refers to a complex or virtual space environment that the user experiences through the five senses.
  • the environment is an environment that is composed of images of virtual objects that exist in the complex or virtual space that the user is to recognize, sounds that are generated based on the virtual objects, etc.
  • the user state recognition unit 33 (user action recognition unit) recognizes image data of the user U captured by the camera 2, and recognizes the state of the user U in the real space RS based on that image data.
  • the state of the user U in the real space RS refers to the posture and coordinates of the user U, and furthermore, the action of the user U indicated by the amount of change therein.
  • the user state recognition unit 33 has a user posture recognition unit 33a and a user coordinate recognition unit 33b.
  • the user posture recognition unit 33a extracts feature points of the user U's body, etc. from the input image data of the user U, and recognizes the posture of the user U in the real space RS based on the extraction results.
  • the user coordinate recognition unit 33b recognizes the coordinates of the user U in the real space RS based on the results of extraction of feature points of the user U's body, etc., extracted from the input image data of the user U, and the situation of the real space RS recognized by the real space recognition unit 30 (e.g., the coordinates of real objects, etc.).
  • the virtual object control unit 34 controls the state of virtual objects that exist in the composite space MS that is being experienced by the user U, based on the user U's actions in the real space RS recognized by the user action recognition unit.
  • the virtual object control unit 34 changes the posture, coordinates, shape, etc. of the virtual object in the composite space MS in accordance with the action.
  • the trigger event recognition unit 35 recognizes that a specific trigger event has occurred when a condition predetermined by a system designer or the like is satisfied. In this embodiment, the action of any one of the users U touching a virtual object is set as the specific trigger event.
  • the specified trigger event of the present invention is not limited to such a configuration, but may be any action that the user performs on a virtual object.
  • the specified trigger event may be an action that does not touch a virtual object. Specifically, it may be an action such as pointing at a virtual object, or an action such as selecting a virtual object via a virtual or real tablet, etc.
  • a specific trigger event may be one that the user is not aware of its occurrence.
  • a trigger event may be one that is not caused by the user's intention, such as the passage of a specific amount of time (for example, stopping within a specific area from a virtual object for a specific period of time).
  • the target user recognition unit 36 recognizes a first user (target person) for whom the type of space experienced will be switched as described below, and a second user (non-target person) for whom the type of space experienced will not be switched, based on the type of trigger event recognized and the actions of the user U when the trigger event occurs.
  • a user U who performs an action of touching a virtual object that is preset to trigger a trigger event (in this embodiment, a model MO, which is a virtual object that is a model of a castle) is recognized as a first user U1, and other users are recognized as second users U2.
  • the target user recognition unit in the present invention is not limited to such a configuration, but may recognize, from among a plurality of users, a first user who switches the type of space experienced, and a second user who does not switch the type of space experienced. Therefore, the method of distinguishing between the first user and the second user may be set appropriately by the system designer.
  • a system may be configured such that, by setting up predetermined groups in advance, when one of the users belonging to that group generates a trigger event, the space experienced by all users belonging to that group is switched (i.e., all users belonging to that group are recognized as first users).
  • the system since there are multiple users, the system performs a process of recognizing a first user who is a target for whom the type of space to be experienced will be switched, and a second user who is a non-target for whom the type of space to be experienced will not be switched.
  • the target user recognition unit may be omitted.
  • the virtual environment generation unit 37 has an avatar generation unit 37a and a virtual space generation unit 37b.
  • the avatar generation unit 37a generates an avatar to be present in the virtual space VS when the aforementioned specific trigger event is recognized.
  • the avatars include a first avatar A1 corresponding to a first user U1, and a second avatar A2 corresponding to a second user U2 (see FIG. 4).
  • the first avatar A1 and the second avatar A2 (hereinafter collectively referred to as "avatar A") move in the virtual space VS in response to the movements of the corresponding user U in the real space RS.
  • the system S when the user U experiences the virtual space VS using the system S, the user U only recognizes the images and sounds of the virtual space VS and is made to recognize that the user U himself/herself exists in the virtual space VS.
  • the system S is configured as a so-called immersive system.
  • the virtual space generation unit 37b When the aforementioned predetermined trigger event is recognized, the virtual space generation unit 37b generates a virtual space VS corresponding to the real space RS (see Figs. 1 and 5) in which the user U exists, as shown in Fig. 4. Specifically, the virtual space generation unit 37b generates images of the background of the virtual space VS, virtual objects and avatar A existing in the virtual space VS, and sounds related to these images.
  • system S of this embodiment does not have such a feature
  • the space switching system has a feature for realizing a specific sensation (such as a cushion with variable hardness) or a feature for generating a specific smell
  • the virtual space generation unit may generate the virtual space using the sensation and smell in addition to the images and sounds.
  • the shape of the virtual space VS generated by the virtual space generation unit 37b is configured to be generated according to the shape of the real space RS at the time when the trigger event is recognized.
  • the "shape of the real space” is recognized based on the shape of the area in which the virtual space is generated, and the posture, coordinates, and shape of real objects that exist in that real space (for example, in this embodiment, table RO1, bookshelf RO2, and drawer RO3).
  • the "shape of the virtual space” refers to the area in which the avatar corresponding to the user can move. Specifically, it is the range defined by the background image of the virtual space VS and the virtual objects placed in the virtual space.
  • the range of operation in the generated virtual space VS can be made to match the real space RS. This makes it possible to prevent the user U from feeling uncomfortable due to differences between the shapes of the virtual space VS and the real space RS.
  • the size of the background image of the virtual space VS is set to match the area of the real space RS, and virtual objects, a rock VO1, a rock wall VO2, and a pillar VO3, are generated in the virtual space VS in accordance with the posture, coordinates, and shape of a table RO1, a bookshelf RO2, and a drawer RO3 that exist in the real space RS.
  • the space switching system of the present invention is not limited to this configuration, and the shape of the generated virtual space does not necessarily have to correspond to the shape of the real space.
  • the system for experiencing the virtual space in the space switching system is not an immersive system, the user does not move, so the shape of the virtual space may be made different from the shape of the real space.
  • the attributes of the virtual space VS generated by the virtual space generating unit 37b correspond to the attributes of the virtual object that triggered the generation of the virtual space VS (in this embodiment, the model MO, which is a virtual object that is a model of a castle).
  • attributes include the shape, pattern, color, or combinations of these of the virtual space.
  • the attributes of the virtual object and the attributes of the virtual space may match, or may be mutually associated.
  • the pattern of the background of the virtual space and the pattern of the virtual object may match, or if the virtual space is a space that resembles an "office," the shape of the virtual object may be a "building.”
  • the user U who causes a trigger event via the model MO can unconsciously predict the attributes of the virtual space VS that he or she will move into afterwards. This makes it possible to switch the type of space that the user U experiences without giving the user any more sense of discomfort.
  • model MO exists as the virtual object that triggers the trigger event, but there may be multiple virtual objects.
  • the user can select the virtual space to switch to by simply selecting a virtual object.
  • the space switching system of the present invention is not limited to this configuration, and the attributes of the virtual object that triggers the trigger event do not necessarily have to correspond to the attributes of the trigger event.
  • the type of space experienced changes at a time unintended by the user, it will give the user a sense of discomfort, so it is better to explicitly or implicitly indicate that a change in the type of space experienced will occur when some action is performed on the virtual object.
  • a virtual message board could be generated near the virtual object, and information about the virtual space that will change when the virtual object is touched could be written on that message board.
  • the space switching system of the present invention may be configured to generate multiple virtual spaces corresponding to the predicted state (shape, etc.) of the real space in advance for each virtual object that triggers a trigger event, and when a trigger event is recognized, select a virtual space according to the type of trigger event and the state of the real space, and allow the target user to experience the selected virtual space.
  • the avatar state control unit 38 controls the state of the avatar A corresponding to the user U in the virtual space VS based on the state (i.e., posture and coordinates) of the user U in the real space RS recognized by the user state recognition unit 33.
  • the environment that users U perceive is determined based on the composite space MS until a specific trigger event is recognized, so that all users U experience the composite space MS.
  • the environment experienced by user U is determined based on the virtual space, so that the first user U1 who caused the trigger event among users U will experience the virtual space VS instead of the composite space MS that he had been experiencing up until that point.
  • this system S is capable of switching the environment experienced by the user U between a composite space MS in which virtual objects exist in the real space RS, and a virtual space VS that is separate from the real space RS, without any distinction between these two.
  • the trigger event that triggers the switch is an action taken by the user U on a virtual object that exists in the composite space MS (in this embodiment, a model MO, which is a virtual object that is a model of a castle). And because the model MO is a virtual object, it is pronounced of the virtual space VS even though it exists in the real space RS.
  • the user U can easily and unconsciously recognize that the virtual object is the trigger and the space is switched. As a result, the user U can switch from the composite space MS to the virtual space VS without feeling any discomfort.
  • the environment that the user U experiences is not limited to the complex space MS or the virtual space VS as in the past, but can be a mixture of the complex space MS and the virtual space VS. Furthermore, since switching from the complex space MS to the virtual space VS can be performed without giving the user U a sense of discomfort, it is possible to prevent the switching from impairing the user U's sense of immersion. Ultimately, according to this system S, it is possible to allow the user U to experience environments that cannot be experienced in the real space RS in a wider variety of ways than before.
  • the switch from the composite space to the virtual space may be performed immediately after the trigger event is recognized and the process of generating the virtual space is completed, or it may be performed after the user (first user) who is to switch the type of space to be experienced has first experienced a space in which the composite space and the virtual space are mixed.
  • This configuration can reduce the abruptness that may be felt by the user when switching the space they are experiencing from a composite space to a virtual space. In turn, it can also switch the type of space they are experiencing without giving the user any sense of discomfort.
  • the complex space MS is repainted into the virtual space VS from a predetermined starting point (in this embodiment, the left side of the drawing of the complex space MS) and the first user U1 is allowed to experience a space in which the users U, including the first user U1, are replaced with avatars A, and then the first user U1 is allowed to experience the virtual space VS.
  • a predetermined starting point in this embodiment, the left side of the drawing of the complex space MS
  • the space where the composite space and the virtual space are mixed is not limited to this configuration. Therefore, for example, the virtual object that triggers the trigger event may expand to engulf the user and become the virtual space VS.
  • the space in which the composite space and the virtual space are mixed does not have to be of one type, but may be of multiple types.
  • the mixed space may be different depending on the content of the trigger event (the type of virtual object being targeted, the type of action being performed on the virtual object, etc.).
  • the system S is configured to recognize, from among multiple people present in the real space, a first user U1 who is a target user who will switch the type of space experienced by the user, and a second user U2 who is not a target user who will not switch the type of space experienced by the user, and to only switch the type of space experienced by the first user when a trigger event is recognized.
  • a configuration may be used in which a person experiencing a composite space and a virtual space (a target person) and a person not experiencing a virtual space or a composite space (a non-target person) can exist simultaneously in the same area of real space.
  • the target person will not be able to recognize the non-target people, and there is a risk that the target person may unintentionally come into contact with the non-target people after the switch.
  • the system S is configured so that even if only the space experienced by the first user U1 is switched, the switched virtual space VS includes not only the first avatar A1 corresponding to the first user U1 who is the target person, but also the second avatar A2 corresponding to a non-target person such as the second user U2.
  • the first user U1 grasp the position of a non-target person such as the second user U2 via the second avatar U2, making it possible to prevent the first user U1 from unintentionally coming into contact with a non-target person such as the second user U2. It also makes it possible to prevent the first user U1 from being distracted from the sense of immersion.
  • the experiential space switching system of the present invention is not limited to this configuration. For example, if the real space is sufficiently large or the movement route is fixed, even if only the space experienced by the target person is switched, the possibility of contact between users is low, so it is not necessary to include a second avatar in the virtual space that corresponds to a non-target person for whom the type of space experienced is not switched.
  • the first avatar A1 that operates in response to the first user U1 who is a target person, and the second avatar A2 that operates in response to the second user U2 who is a non-target person are both configured to be animal avatars.
  • the form of the avatar corresponding to the target person and the form of the avatar corresponding to the non-target person do not necessarily have to be the same form.
  • the avatar that moves in response to the target person could be a deformed human, while the avatar that moves in response to the non-target person could be an animal, or the avatar that moves in response to the target person could be opaque to the background, while the avatar that moves in response to the non-target person could be semi-transparent and allow the background to show through.
  • the target person who recognizes the avatars can intuitively distinguish between the avatars that move in response to the target person and the avatars that move in response to the non-target person.
  • the real space recognition unit 30 of the server 3 recognizes the situation of the real space RS in which the user U is present (see FIG. 1) (FIG. 7/STEP 100).
  • the virtual object generation unit 31a of the composite environment generation unit 31 of the server 3 generates a virtual object to be present in the composite space MS and recognized by the user U (FIG. 7/STEP 101).
  • the virtual object generation unit 31a generates a model MO (see Figure 3) that serves as a trigger for a trigger event as a virtual object to be present in the composite space MS.
  • the composite space generation unit 31b of the composite environment generation unit 31 places the virtual objects generated by the virtual object generation unit 31a in the real space RS recognized by the real space recognition unit 30, generating a composite space MS as shown in FIG. 3 (FIG. 7/STEP 102).
  • the user state recognition unit 33 of the server 3 recognizes the image data of the user U captured by the camera 2, and recognizes the state of the user U (posture and coordinates in the real space RS) based on the image data ( Figure 7/STEP 103).
  • the user environment determination unit 32 of the server 3 determines the environment of the complex space MS that the user U will recognize based on the state of the user U (FIG. 7/STEP 104).
  • the user environment determination unit 32 determines the state of virtual objects to be displayed superimposed on the real space RS and the accompanying sounds based on the posture and coordinates of the user U in the real space RS as the environment to be recognized by the user U.
  • the HMD 4 worn by the user U outputs the determined environment ( Figure 7/STEP 105).
  • the HMD 4 displays the determined virtual object so that it is superimposed on the real space RS that is observed through a monitor 40 mounted on the HMD 4, and generates the determined sound from a speaker 41 mounted on the HMD 4.
  • the user state recognition unit 33 repeats the determination at a predetermined control period.
  • the virtual object control unit 34 of the server 3 determines whether the action of user U is an action on a virtual object (FIG. 7/STEP 107).
  • the virtual object control unit 34 changes the coordinates of the virtual object according to the content of the action.
  • system S terminates this processing.
  • the composite space MS is generated based on the real space RS, the composite space MS recognized by one of the first user U1 and the second user U2 also contains the other of the first user U1 and the second user U2.
  • the trigger event recognition unit 35 of the server 3 determines whether or not a specific trigger event has been recognized (FIG. 8A/STEP 200).
  • the trigger event recognition unit 35 determines whether any of the users U has performed an action such as touching the model MO, which is a virtual object that triggers a trigger event.
  • the trigger event recognition unit 35 repeats the determination at a predetermined control period.
  • the virtual space generation unit 37b of the virtual environment generation unit 37 of the server 3 recognizes the attributes of the virtual space VS to be generated based on the attributes of the virtual object that triggered the trigger event ( Figure 8A/STEP 201).
  • the virtual object that triggered the trigger event is a model MO, which is a virtual object of a model castle, so the generated virtual space VS has a castle theme. Specifically, it is a castle-like space that makes use of nature, consisting of rock walls VO2, rock pillars VO3, etc.
  • the virtual space generation unit 37b recognizes the shape of the real space RS ( Figure 8A/STEP 202).
  • the virtual space generation unit 37b recognizes the size of the area of the real space RS from which the virtual space VS will be generated, and the attitude, coordinates, and shape of the real objects (for example, in this embodiment, the table RO1, the bookshelf RO2, and the drawer RO3) that exist in the real space RS, from the image of the real space RS captured by the camera 2 at the time the trigger event is recognized.
  • the real objects for example, in this embodiment, the table RO1, the bookshelf RO2, and the drawer RO3
  • the virtual space generation unit 37b recognizes the shape of the virtual space VS to be generated based on the shape of the real space RS ( Figure 8A/STEP 203).
  • the virtual space generation unit 37b recognizes the size of the background image of the virtual space VS based on the size of the recognized area of the real space RS, and recognizes the posture, coordinates and shape of the virtual objects (rock VO1, rock wall VO2, pillar VO3) corresponding to those recognized real objects based on the posture, coordinates and shape of the recognized real objects.
  • the user status recognition unit 33 of the server 3 recognizes the status of the user U (FIG. 8A/STEP 204).
  • the user state recognition unit 33 recognizes the posture and coordinates of the user U in the real space RS at the time when the trigger event is recognized.
  • the avatar generation unit 37a of the virtual environment generation unit 37 of the server 3 generates an avatar A to be present in the virtual space VS based on the state of the user U (FIG. 8A/STEP 205).
  • the virtual space generation unit 37b of the virtual environment generation unit 37 of the server 3 generates a virtual space VS including the avatar generated by the avatar generation unit 37a (FIG. 8A/STEP 206).
  • the virtual space generation unit 37b generates a background image and virtual objects of the virtual space VS based on the attributes of the virtual space VS recognized in STEP 201 and the shape of the virtual space VS recognized in STEP 203, and includes the avatar A generated in STEP 205 in the image to generate the virtual space VS as shown in FIG. 5.
  • the user environment determination unit 32 of the server 3 generates a space in which the composite space MS and the virtual space VS are mixed ( Figure 8A/STEP 207).
  • the user environment determination unit 32 generates a space that mixes the composite space MS that was generated by the composite environment generation unit 31 and is currently being experienced by the user U, and an image of the virtual space VS that was generated by the virtual environment generation unit 37 and will be experienced by the user U.
  • the mixed space is a space in which the composite space MS is repainted into the virtual space VS from a specific starting point (in this embodiment, the left side of the composite space MS in the drawing), as shown in Figure 6, and users U, including themselves, are replaced by avatars A.
  • the target user recognition unit 36 of the server 3 recognizes the user who is the target of the switching process based on the content of the trigger event ( Figure 8B/STEP 208).
  • the target user recognition unit 36 recognizes a user (first user) for whom the type of space experienced will be switched and a user (second user) for whom the type of space experienced will not be switched, based on the type of trigger event recognized and the action of the user U when the trigger event occurs.
  • the user U who performs an action of touching the model MO which is a virtual object that triggers a trigger event
  • the model MO which is a virtual object that triggers a trigger event
  • other users are recognized as the second user U2.
  • the user environment determination unit 32 determines the spatial environment, which is a mixture of the composite space MS and the virtual space VS, to be recognized by the first user U1, who is the target of switching the type of space to be experienced, based on the states of the first user U1 and the second user U2 (FIG. 8B/STEP 209).
  • the HMD 4 worn by the first user U1 outputs the determined environment (FIG. 8B/STEP 210).
  • the HMD 4 displays the determined image on a monitor 40 mounted on the HMD 4, and generates the determined sound from a speaker 41 mounted on the HMD 4.
  • the user environment determination unit 32 determines whether a predetermined time has elapsed ( Figure 8B/STEP 211).
  • the user environment determination unit 32 determines whether or not the time has elapsed until the switching from the composite space MS to the virtual space VS is completed via a space in which the composite space MS and the virtual space VS are mixed.
  • the user status recognition unit 33 of the server 3 recognizes the status of user U again (FIG. 8B/STEP 212).
  • the HMD 4 worn by the first user U1 outputs the determined environment (FIG. 8B/STEP 215), and the system S ends this processing.
  • the space experienced by the first user U1 is switched from the composite space MS to the virtual space VS via a space in which the composite space MS and the virtual space VS are mixed. Meanwhile, the space experienced by the second user U2 remains the composite space MS.
  • the user environment determination unit 32 determines images and sounds of the virtual space VS1 that represent the environment of the first avatar A1 (including the relative coordinates of the second avatar A2) as the environment to be recognized by the first user U1.
  • the HMD 4 worn by the first user U1 outputs the determined environment (Fig. 9/STEP 303).
  • the user state recognition unit 33 of the server 3 determines whether the first user U1 or the second user U2 has performed any action (FIG. 9/STEP 304).
  • the server 3 determines whether or not it has recognized a signal instructing the end of processing (FIG. 9/STEP 305).
  • system S terminates this processing.
  • the first user U1 is made to recognize that he or she exists in the virtual space VS as the first avatar A1.
  • the second user U2 will continue to experience the compound space MS.
  • the second avatar A2 corresponding to the second user U2 will be moving in accordance with the movements of the second user U2.
  • the system S which is the sensory space switching system
  • the present invention is not limited to computer systems, and may be anything that executes the sensory space switching method of the present invention.
  • the device may be a sensory space switching device configured by a single computer equipped with the processing unit described in the above embodiment.
  • the device may be a sensory space switching program for causing any one or more computers to execute the sensory space switching method described above, and a recording medium on which the program is recorded and which is readable by a computer used by a user, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Architecture (AREA)
  • User Interface Of Digital Computer (AREA)
  • Processing Or Creating Images (AREA)
PCT/JP2023/037877 2023-10-19 2023-10-19 体感空間切替方法、体感空間切替システム、体感空間切替装置、並びに体感空間切替プログラム及びそれを記録した記録媒体 Pending WO2025083847A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US18/839,156 US20250363752A1 (en) 2023-10-19 2023-10-19 Experience space switching method, experience space switching system, experience space switching apparatus, and experience space switching program and recording medium with the same recorded therein
JP2024510536A JP7539006B1 (ja) 2023-10-19 2023-10-19 体感空間切替方法、体感空間切替システム、体感空間切替装置、並びに体感空間切替プログラム及びそれを記録した記録媒体
PCT/JP2023/037877 WO2025083847A1 (ja) 2023-10-19 2023-10-19 体感空間切替方法、体感空間切替システム、体感空間切替装置、並びに体感空間切替プログラム及びそれを記録した記録媒体

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2023/037877 WO2025083847A1 (ja) 2023-10-19 2023-10-19 体感空間切替方法、体感空間切替システム、体感空間切替装置、並びに体感空間切替プログラム及びそれを記録した記録媒体

Publications (1)

Publication Number Publication Date
WO2025083847A1 true WO2025083847A1 (ja) 2025-04-24

Family

ID=92424441

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/037877 Pending WO2025083847A1 (ja) 2023-10-19 2023-10-19 体感空間切替方法、体感空間切替システム、体感空間切替装置、並びに体感空間切替プログラム及びそれを記録した記録媒体

Country Status (3)

Country Link
US (1) US20250363752A1 (https=)
JP (1) JP7539006B1 (https=)
WO (1) WO2025083847A1 (https=)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7722763B1 (ja) * 2024-11-06 2025-08-13 株式会社Abal 環境生成方法、環境生成システム、環境生成装置、プログラム及び記録媒体

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2022117770A (ja) * 2021-02-01 2022-08-12 株式会社スクウェア・エニックス 仮想店舗管理プログラム、仮想店舗管理システムおよび仮想店舗管理方法
JP7138392B1 (ja) * 2021-08-11 2022-09-16 株式会社Abal 仮想空間体感システム
JP2023517073A (ja) * 2020-03-16 2023-04-21 アップル インコーポレイテッド コンピュータ生成体験を提供するためのデバイス、方法、及びグラフィカルユーザインタフェース
WO2023176374A1 (ja) * 2022-03-15 2023-09-21 ソニーグループ株式会社 情報処理装置、および情報処理システム、並びに情報処理方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023517073A (ja) * 2020-03-16 2023-04-21 アップル インコーポレイテッド コンピュータ生成体験を提供するためのデバイス、方法、及びグラフィカルユーザインタフェース
JP2022117770A (ja) * 2021-02-01 2022-08-12 株式会社スクウェア・エニックス 仮想店舗管理プログラム、仮想店舗管理システムおよび仮想店舗管理方法
JP7138392B1 (ja) * 2021-08-11 2022-09-16 株式会社Abal 仮想空間体感システム
WO2023176374A1 (ja) * 2022-03-15 2023-09-21 ソニーグループ株式会社 情報処理装置、および情報処理システム、並びに情報処理方法

Also Published As

Publication number Publication date
JP7539006B1 (ja) 2024-08-23
JPWO2025083847A1 (https=) 2025-04-24
US20250363752A1 (en) 2025-11-27

Similar Documents

Publication Publication Date Title
Bowman et al. 3d user interfaces: New directions and perspectives
US10564730B2 (en) Non-collocated haptic cues in immersive environments
EP3425481B1 (en) Control device
KR102285180B1 (ko) 시야를 이용하는 햅틱 피드백
Yixian et al. Zoomwalls: Dynamic walls that simulate haptic infrastructure for room-scale vr world
US10275025B2 (en) Gloves that include haptic feedback for use with HMD systems
KR102105637B1 (ko) 가상 현실에서 오브젝트와 손의 상황 감지형 충돌을 통한 입력
US10317997B2 (en) Selection of optimally positioned sensors in a glove interface object
EP3364272A1 (en) Automatic localized haptics generation system
KR20220012990A (ko) 인공 현실 시스템들을 위한 팔 응시-구동 사용자 인터페이스 요소 게이팅
CN115668104B (zh) 广告显示系统
KR20220018562A (ko) 인공 현실 시스템을 위한 모서리-식별 제스처-구동 사용자 인터페이스 요소 게이팅
TW202105129A (zh) 具有用於閘控使用者介面元件的個人助理元件之人工實境系統
Stuerzlinger et al. The value of constraints for 3D user interfaces
CN113785262A (zh) 具有手指映射自触觉输入方法的人工现实系统
EP3549127A1 (en) A system for importing user interface devices into virtual/augmented reality
KR20110054376A (ko) 실감 인터랙션 시스템 및 방법
JP7539006B1 (ja) 体感空間切替方法、体感空間切替システム、体感空間切替装置、並びに体感空間切替プログラム及びそれを記録した記録媒体
CN117590931A (zh) 人工现实环境中的手势移动
CN119013641A (zh) 使用多个模态的人工现实输入
JP2023079161A (ja) 表示制御装置、表示制御方法、及びプログラム
WO2021193421A1 (ja) 情報処理装置、情報処理方法、プログラム及び情報処理システム
JP2019117634A (ja) 表示制御プログラム、表示制御方法、及び表示制御システム
JP7055527B1 (ja) 仮想空間体感システム
WO2021240601A1 (ja) 仮想空間体感システム

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2024510536

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 2024510536

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 18839156

Country of ref document: US

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23956095

Country of ref document: EP

Kind code of ref document: A1

WWP Wipo information: published in national office

Ref document number: 18839156

Country of ref document: US