WO2023188104A1 - 遠隔体験システム、情報処理装置、情報処理方法およびプログラム - Google Patents

遠隔体験システム、情報処理装置、情報処理方法およびプログラム Download PDF

Info

Publication number
WO2023188104A1
WO2023188104A1 PCT/JP2022/015971 JP2022015971W WO2023188104A1 WO 2023188104 A1 WO2023188104 A1 WO 2023188104A1 JP 2022015971 W JP2022015971 W JP 2022015971W WO 2023188104 A1 WO2023188104 A1 WO 2023188104A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
expression
user
real environment
detection
Prior art date
Application number
PCT/JP2022/015971
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
正樹 春名
Original Assignee
三菱電機株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 三菱電機株式会社 filed Critical 三菱電機株式会社
Priority to PCT/JP2022/015971 priority Critical patent/WO2023188104A1/ja
Priority to JP2024510887A priority patent/JPWO2023188104A1/ja
Publication of WO2023188104A1 publication Critical patent/WO2023188104A1/ja

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/15—Conference systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Definitions

  • the present disclosure relates to a remote experience system, information processing device, information processing method, and program that provide information about a remote location to a user.
  • Patent Document 1 discloses a technology related to a remote conference in which in-house workers who work in the company and remote workers who work remotely coexist.
  • a virtual room is displayed on the terminals of in-house workers and remote workers, and the in-house workers and remote workers are displayed as virtual participants in the virtual room.
  • the sense of realism is enhanced by reflecting the movements of virtual workers and remote workers in the movements of virtual workers.
  • Patent Document 1 discloses participation of virtual participants in a virtual space, it does not disclose a technique for allowing remote participants in an event using a real environment to experience the real environment.
  • the present disclosure has been made in view of the above, and aims to provide a remote experience system that allows remote participants in an event using a real environment to experience the real environment.
  • a remote experience system is installed in a real environment where an event is held, and includes a device equipped with a device-mounted sensor that detects information about objects in the real environment.
  • a real environment sensor that is installed in a real environment and detects information about objects in the real environment;
  • a participant motion detection device that detects the motion of a first user who participates in the event from a location away from the real environment;
  • a first expression device capable of expressions that can be recognized by a user with at least one of the five senses;
  • a second expression device capable of presenting images to a second user who participates in the event from a location remote from the real environment; and information processing.
  • the real environment sensor includes a photographing device that detects information about objects in the real environment by photographing the real environment.
  • the information processing device includes a motion information receiving unit that receives motion information indicating the motion acquired by the participant motion detection device from the participant motion detection device, and uses the motion information to cause the device to perform a motion according to the motion information.
  • a device control unit that generates control information to cause the device to perform the operation; a control information transmitting unit that sends the control information to the device; and a device-mounted sensor that receives first detection information detected by the device-mounted sensor from the real environment sensor. and a detection information receiving section that receives second detection information detected by the real environment sensor.
  • the information processing device further uses the first detection information to generate first expression information to be transmitted to the first user, and uses the second detection information to generate a video corresponding to the virtual viewpoint of the second user in the real environment. and an expression information transmitter that transmits the first expression information to the first expression device and the second expression information to the second expression device.
  • the remote experience system has the effect of allowing remote participants in an event using a real environment to experience the real environment.
  • a diagram schematically showing distance education using the remote experience system of the embodiment Flowchart illustrating an example of a device control procedure in the information processing apparatus according to the embodiment
  • Flowchart illustrating an example of a procedure for generating expression information in the information processing apparatus according to the embodiment A diagram showing an example in which real participants, real avatar users, and virtual avatar users participate in an event.
  • FIG. 1 is a diagram showing a configuration example of a remote experience system according to an embodiment.
  • the remote experience system 100 of this embodiment is used for an event using the real environment 10.
  • the real environment 10 is, for example, a place where an event is held.
  • Events include, for example, distance education, remote experience, remote skill transfer, remote maintenance, remote training, experiential travel, healing, astronomical observation, on-site games, on-site golf, local sightseeing, boat trips, car trips, exploration, etc. Yes, but not limited to.
  • the remote experience system 100 of this embodiment there are two ways to participate in an event: one is to participate as a real participant who actually exists in the real environment 10, and the other is to use the device 1 as a real avatar that is your alter ego. It is possible to provide three types of participation methods: a participation method using a virtual avatar that does not exist in the real environment 10, and a participation method using a virtual avatar that does not exist in the real environment 10.
  • a user who participates using a real avatar and a user who participates using a virtual avatar are both virtual participants that do not exist in the real environment 10, and are remote participants.
  • information obtained by the device 1 from the real environment 10 can be transmitted to the user through the actions of the device 1, such as when the device 1 moves in the real environment 10 or when the device 1 touches objects around it. I can do it.
  • real avatars it is necessary to prepare devices 1 according to the number of users participating, and costs may be incurred depending on the number of participants in the event. Therefore, in the remote experience system 100 of this embodiment, by allowing participation as a virtual avatar, it is possible to reduce costs and increase the number of participants in an event. Details of the real avatar and virtual avatar will be described later. The following will mainly explain examples in which real participants, real avatars, and virtual avatars can coexist, but the remote experience system is not limited to this. 100 may be constructed.
  • the remote experience system 100 includes a device 1, sensors 3-1 to 3-3, and expression devices 4-1 and 4-2 installed in a real environment 10, and users who participate in an event.
  • the information processing device 2 is provided with participant motion detection devices 5-1, 5-2 and expression devices 6-1, 6-2, which are installed at a participation location 7, which is a place where participants participate.
  • the participant motion detection devices 5-1 and 5-2 installed at the participation location 7 detect the motions of users participating in the event, and transmit the detected information to the information processing device 2 as motion information.
  • the participant motion detection devices 5-1 and 5-2 are set at a participation location 7 where a user (first user) who participates using at least a real avatar exists, and detects the motion of the user.
  • the participant motion detection devices 5-1 and 5-2 are set at a participation location 7 where a user (second user) who participates using a virtual avatar exists and detects the motion of the user. good.
  • the participant motion detection devices 5-1 and 5-2 are, for example, positioning sensors, cameras, microphones, etc.
  • the expression devices 6-1 and 6-2 receive expression information indicating the state of the real environment 10 received from the information processing device 2, and transmit the state of the real environment 10 to the user based on the expression information.
  • the expression devices 6-1 and 6-2 are devices capable of expressions that the user can recognize with at least one of the five senses, and can, for example, present images to the user.
  • the expression devices 6-1 and 6-2 are, for example, a display, a speaker, a haptic groove, etc.
  • the displays used as the expression devices 6-1 and 6-2 may be head-mounted displays, VR (Virtual Reality) goggles, etc., or may be terminals such as smartphones and personal computers.
  • the expression devices 6-1 and 6-2 may be, for example, hologram displays, aerial projection displays, or the like. Furthermore, the expression devices 6-1 and 6-2 may include an olfactory expression device, a blower, an air conditioner, and the like.
  • participant motion detection devices 5-1 and 5-2 when the participant motion detection devices 5-1 and 5-2 are referred to without being individually distinguished, they will be referred to as participant motion detection device 5, and the expression devices 6-1 and 6-2 will be referred to without being individually distinguished. It is sometimes referred to as an expression device 6.
  • an expression device 6 Although two participant motion detection devices 5 and two expression devices 6 are shown in FIG. 1, the number of participant motion detection devices 5 and expression devices 6 is not limited to the example shown in FIG. 1.
  • one participating location 7 is shown in FIG. 1, when users participate from a plurality of different locations, a similar device is provided at each participating location 7.
  • the participant motion detection device 5 and the expression device 6 are basically provided for each user, but some of them may be shared.
  • the expression device 6 may transmit information to a plurality of users.
  • the participant motion detection device 5 may detect the motions of a plurality of users.
  • the sensors 3-1 to 3-3 are real environment sensors that are installed in the real environment 10 and detect information regarding objects in the real environment 10.
  • the object includes, for example, at least one of animals, plants, and air that exist in the real environment 10.
  • the sensors 3-1 to 3-3 are sensors that detect the environment in the real environment 10, and detect, for example, objects, sounds, temperature, wind speed, air volume, humidity, illuminance, smell, type of gas, etc. in the real environment 10. It is a sensor that That is, the sensors 3-1 to 3-3 detect at least one of temperature, humidity, wind, and odor in the real environment 10, for example.
  • object detection may be performed, for example, by photographing an image or by collecting sound.
  • the sensors 3-1 to 3-3 may include a photographing device such as a camera, or may include a microphone. Detection information indicating the detection result is transmitted to the information processing device 2.
  • the sensors 3-1 to 3-3 are shown without being individually distinguished, they will be referred to as sensor 3.
  • the device 1 is an avatar that is an alter ego of a user who participates in an event, and more specifically, it is a real avatar that exists as an object.
  • the device 1 may be, for example, a humanoid robot, a movable robot having a manipulator and a head that simulates a human face, or a manipulator. good.
  • the device 1 operates based on control information received from the information processing device 2.
  • the device 1 includes a control information receiving section 11, a drive control section 12, a driving section 13, sensors 14-1 and 14-2, and a detection information transmitting section 15.
  • the control information receiving unit 11 receives control information from the information processing device 2 and outputs the received control information to the drive control unit 12.
  • the drive control section 12 controls the drive section 13 based on the control information.
  • the drive unit 13 includes one or more actuators.
  • the actuator is, for example, a physical effector such as a manipulator that operates each joint in the device 1, a moving device, a speaker, or the like.
  • the sensors 14-1 and 14-2 are device-mounted sensors that detect information regarding objects in the real environment 10. Similar to the sensor 3, the sensors 14-1 and 14-2 include a photographing device such as a camera, a speaker, etc., but, for example, they detect at least one of temperature, humidity, wind, and odor in the actual environment 10. May be detected. Further, the sensors 14-1 and 14-2 may detect force-tactile sensation. The sensors 14-1 and 14-2 output detection information indicating detection results to the detection information transmitter 15. The detection information transmitter 15 transmits the detection information to the information processing device 2. Since the sensors 14-1 and 14-2 are mounted on the device 1, they can detect information depending on the location of the device 1. Hereinafter, when the sensors 14-1 and 14-2 are shown without being individually distinguished, they will be referred to as the sensor 14.
  • the senor 14 acquires an image depending on the movement of the device 1 and the direction of the face. can do.
  • the number of sensors 14 is not limited to the example shown in FIG.
  • the device 1 may include a manipulator, and the sensor 14 may include a force tactile sensor that detects force tactile sensation at the hand of the manipulator. Since the expression device 6 corresponding to the user using the real avatar includes a haptic device that transmits the haptic sensation to the hand of the user using the real avatar, the haptic sensation detected by the device 1 can be transmitted to the hand of the user using the real avatar. can be communicated to users who use it.
  • the expression devices 4-1 and 4-2 are devices that perform expressions to convey the virtual environment superimposed on the real environment 10 to users who participate in an event in the real environment 10.
  • the expression devices 4-1 and 4-2 are, for example, displays, speakers, and the like.
  • the displays used as the expression devices 4-1 and 4-2 are the same as the display used as the expression device 6 described above, but it is sufficient that they can express a portion of the virtual environment as described later.
  • the expression devices 4-1 and 4-2 will be referred to as expression device 4 when shown without distinguishing them individually.
  • the expression device 4 may not be provided in the real environment 10 depending on the configuration of the device 1 and the participation method of the users who participate in the event.
  • the sensor 3 may not be provided in the real environment 10.
  • FIG. 1 three sensors 3 and two expression devices 4 are illustrated, but the respective numbers of sensors 3 and expression devices 4 are not limited to the example shown in FIG. 1.
  • the number of devices 1 is not limited to the example shown in FIG.
  • the information processing device 2 includes a control information transmitter 21 , a device controller 22 , an operation information receiver 23 , a detection information receiver 24 , an expression information generator 25 , and an expression information transmitter 26 .
  • the motion information receiving unit 23 receives motion information indicating the motion acquired by the participant motion detection device 5 from the participant motion detection device 5 at the participation location 7, and outputs the received motion information to the device control unit 22. .
  • Identification information that can identify the corresponding user is added to the operation information. This identification information may be user identification information determined when the user registers to participate in the event, or may be identification information of the participant motion detection device 5.
  • the identification information added to the movement information is the identification information of the participant movement detection device 5, the identification information of the participant movement detection device 5 and the user identification information are associated at the time of registration for participation in the event, etc. Suppose that
  • the device control unit 22 uses the operation information to generate control information for causing the device 1 to perform an operation according to the operation information. That is, the device control unit 22 uses the operation information to generate control information for controlling the device 1 so that the device 1, which is the real avatar of the user, performs an operation corresponding to the user's operation, The generated control information is output to the control information transmitter 21.
  • the control information is, for example, information indicating how to drive each actuator of the device 1.
  • the device control unit 22 holds correspondence information indicating the correspondence between the device 1 and the user who uses the device 1 as a real avatar, and uses the correspondence information and the identification information added to the operation information to The device 1 corresponding to the operation information is identified.
  • the correspondence information may be determined, for example, when the user registers to participate in the event, or may be selected by the user when the event starts.
  • the user may check the functions and positions of each device 1, select the device 1 that will become the actual avatar, and notify the information processing device 2 of the selected device 1 via the user's terminal (not shown) or the like.
  • the control information transmitter 21 transmits the control information received from the device controller 22 to the device 1.
  • the detection information receiving unit 24 receives detection information from at least one of the sensor 14 and the sensor 3, and outputs the received detection information to the expression information generating unit 25.
  • the expression information generation unit 25 uses the detection information to generate expression information indicating the content to be expressed by the expression device 6, and outputs the generated expression information to the expression information transmission unit 26.
  • the expression information is transmitted to the expression device 6. Note that, similar to the correspondence between the device 1 and the user described above, the expression information generation unit 25 holds correspondence information indicating the correspondence between the expression device 6 and the user. Further, the expression information generation unit 25 generates expression information to be transmitted to users participating in the real environment 10, and outputs the generated expression information to the expression information transmission unit 26. It is transmitted to the expression device 4.
  • the communication line between the information processing device 2 and each device may be a wireless line, a wired line, or a mixture of a wireless line and a wired line.
  • Communication between the information processing device 2 and each device may be performed using any communication method, but for example, the Beyond 5G (5th Generation) 5th generation mobile communication system, which achieves large-capacity, low-latency transmission, ), information can be transmitted to the user with low delay and the sense of realism can be enhanced.
  • FIG. 2 is a diagram schematically showing distance education using the remote experience system 100 of this embodiment.
  • real participants 8-1 to 8-4 actually exist in a forest, which is the real environment 10.
  • the forest which is the real environment 10
  • the real participant 8-1 is, for example, a host such as a teacher
  • the real participants 8-2 to 8-4, users who participate as virtual avatars 9, and users who participate using the device 1 as a real avatar are: , is a student.
  • real participants 8-1 to 8-4 without distinguishing them individually, they will be referred to as real participants 8.
  • the real environment 10 is a forest, so the presence of birds, animals, plants, etc. causes changes in smell, air flow, and sound due to the movement of birds and animals. .
  • participants will be able to experience something unique to the real environment 10.
  • By transmitting various states in addition to images and sounds to the user it is possible, for example, to promote the user's understanding of the surrounding world.
  • users who remotely participate in distance education are also provided with a more realistic experience of the real environment 10.
  • the movements of users who participate using the device 1 as real avatars are detected by the participant movement detection device 5 at the participation location 7 shown in FIG. It is transmitted to the processing device 2. Since the information processing device 2 controls the operation of the device 1 using the operation information, the device 1 performs an operation according to the operation of the real avatar user. In addition, since the information processing device 2 causes the expression device 6 to express information indicating the state of the real environment 10, the real avatar user can, for example, express plants, animals, etc. in the forest that is the real environment 10 using images, sounds, etc. It can be recognized by Therefore, when the real avatar user performs an action of touching a plant in the real environment 10, the device 1, which is the real avatar, can touch the plant in the real environment 10. The haptic sensation caused by touching the plant is detected by the sensor 14 in the device 1, and transmitted via the information processing device 2 to the expression device 6, such as a haptic glove. Thereby, the expression device 6 can transmit the haptic sensation to the user.
  • the real avatar user can detect not only images and sounds but also the real environment 10. You can also experience the state of the air.
  • the real environment 10 can be more appropriately used for remote participants in an event using the real environment 10. It is possible to experience the environment and realize interaction with the real environment. By merging the real environment 10 and virtual objects, the real avatar user can experience the real environment 10 with a sense of presence.
  • the air condition may be expressed as similar to the detected information by using a scent reproduction device, an air conditioner, a blower, etc. as the expression device 6 for expressing the air condition.
  • the information may be converted into at least one of visual information and auditory information.
  • haptic detection information may be converted into at least one of visual information and auditory information.
  • device 1 which is a real avatar
  • device 1 is directly transmitted as a video to real avatar users and users who participate using virtual avatars (hereinafter also referred to as virtual avatar users), and real participants directly view device 1.
  • the information processing device 2 may generate a video simulating a real avatar user and replace the part of the device 1 with the video.
  • the virtual avatar does not actually exist in the real environment 10, but by specifying the position of the virtual avatar in the real environment 10, information simulating the presence of the virtual avatar user at that position is provided to the user. provided.
  • information simulating the presence of the virtual avatar user at that position is provided to the user.
  • a 360-degree free viewpoint camera a microphone for realizing 360-degree stereophonic sound, etc. as the sensor 3
  • images and sounds can be provided to the virtual avatar user according to the virtual avatar's position and face orientation. can do.
  • the position, face direction, etc. of the virtual avatar may be determined by the participant motion detection device 5 detecting the motion of the virtual avatar user and the information processing device 2 based on the motion information, similarly to the real avatar user. , may be specified by the virtual avatar user using a terminal (not shown).
  • the virtual avatar user can also use the real environment 10 as well as the real avatar user. By transmitting the air condition in the real environment 10, the virtual avatar user can also experience the air condition in the real environment 10.
  • the information processing device 2 displays images of the real environment 10 around the virtual avatar.
  • the virtual avatar may be made to perform actions such as touching objects or moving objects in the virtual space.
  • what kind of object the object is from the video of the real environment 10 is determined, and the result of the determination is used to generate information indicating the tactile sensation obtained by touching the object, and the information is transmitted to the expression device. 6, the sensation of touching the object may be transmitted to the virtual avatar user.
  • the initial values of the position and orientation of the virtual avatar in the real environment 10 may be specified by the virtual avatar user or may be determined in advance.
  • the information processing device 2 generates a video that simulates the virtual avatar user or a three-dimensional video of an arbitrary shape based on the video shot of the virtual avatar user as the virtual avatar, and transfers the video to the real environment 10.
  • the information may be superimposed on the video and transmitted as expression information to the expression device 6 and the expression device 4.
  • the virtual avatar may also be provided as a video to the real avatar user and real participants, or the virtual avatar user may be able to select whether or not to display the virtual avatar.
  • an image simulating the real avatar user or a three-dimensional image of an arbitrary shape may be generated and displayed in the location corresponding to the device 1, and whether or not to display the image may be determined.
  • a real avatar user may be selectable.
  • the above-described virtual space generated for the virtual avatar user may be reflected in the expression information sent to the real avatar user so that the real avatar user can share information with the virtual avatar user.
  • the representation device 4 by allowing the representation device 4 to represent the virtual space so that the real participants participating in the real environment 10 can share information with the virtual avatar users, the virtual space is superimposed on the real environment 10 that the real participants directly see. You may also choose to do so.
  • FIG. 3 is a flowchart showing an example of a control procedure for the device 1 in the information processing device 2 of this embodiment.
  • the information processing device 2 acquires operation information (step S1). Specifically, the motion information receiving unit 23 acquires motion information by receiving motion information from the participant motion detection device 5 that detects the motion of the real avatar user, and sends the acquired motion information to the device control unit 22. Output.
  • the information processing device 2 generates control information for reflecting the motion information on the movement of the real avatar (step S2). Specifically, the information processing device 2 uses the motion information to generate control information for causing the device 1, which is the real avatar of the real avatar user corresponding to the motion information, to perform the motion corresponding to the motion information. , outputs the generated control information to the control information transmitter 21.
  • the information processing device 2 transmits control information (step S3).
  • the control information transmitter 21 transmits control information to the device 1.
  • the movements of the real avatar user are reflected in the device 1, which is the real avatar corresponding to the real avatar user.
  • FIG. 4 is a flowchart illustrating an example of a procedure for generating virtual information regarding a virtual avatar in the information processing device 2 of this embodiment.
  • the information processing device 2 performs step S1 similarly to the example shown in FIG.
  • the motion information is motion information regarding the virtual avatar user
  • the motion information receiving section 23 outputs the motion information to the expression information generating section 25.
  • the information processing device 2 generates virtual information corresponding to the virtual avatar that reflects the motion information (step S4).
  • the expression information generation unit 25 uses the motion information to generate virtual information that is expression information such as images and sounds corresponding to a virtual space including the virtual avatar of the virtual avatar user corresponding to the motion information. .
  • virtual information for expressing a virtual space in which the movements of the virtual avatar user are reflected is generated.
  • information regarding the generation of virtual avatars and the virtual space may be generated using techniques such as spatial reconstruction methods, MR (Mixed Reality), and AR (Augmented Reality).
  • the generation of virtual avatars and the generation of information regarding virtual spaces are not limited to these examples.
  • FIG. 5 is a flowchart illustrating an example of an expression information generation processing procedure in the information processing device 2 of this embodiment.
  • the information processing device 2 acquires information detected by the sensor (step S11).
  • the detection information receiving unit 24 acquires detection information by receiving detection information from at least one of the sensor 14 and the sensor 3, and outputs the acquired detection information to the expression information generation unit 25.
  • the information processing device 2 generates first combination information by combining virtual information with the detection information to be combined among the detection information (step S12).
  • the expression information generation unit 25 generates the first combination information by combining the virtual information generated in step S4 shown in FIG. 4 with the detection information to be combined among the detection information.
  • the detection information to be synthesized is the detection information to be synthesized with the virtual information. For example, if the virtual information is a video, the detection information to be synthesized is a video, and if the virtual information is video and sound, the detection information to be synthesized is the detection information to be synthesized with the virtual information.
  • the detected information is images and sounds.
  • the expression information generation unit 25 uses the detection information to generate information indicating images and sounds corresponding to the virtual avatar or the real avatar, and synthesizes the generated information with virtual information.
  • the virtual information is combined with the corresponding detection information or information generated using the detection information of the sensor 3 for each remote participant. Note that the virtual information does not need to be combined.
  • the information processing device 2 converts the detected information to be converted out of the detected information into another type of information, and generates second combined information by combining the converted information with the first combined information (step S13).
  • the expression information generation unit 25 converts the detection information to be converted out of the detection information into another type of information, and generates the second composite information by combining the converted information with the first composite information. , and outputs the second composite information to the expression information transmitter 26 as expression information.
  • the air condition detection information such as temperature and humidity into visual information
  • the air condition detection information is the detection information to be converted, and other types of information are characters, images, etc.
  • This conversion can be done by, for example, converting the content of another type of information that allows a person to feel a similar state in advance, using supervised machine learning using clinical trial results, depending on the value of the detected information to be converted. By learning and inputting the detection information to be converted into the trained model, the conversion result can be obtained.
  • the conversion method is not limited to this example.
  • Examples of expressing the state of the air using visual and auditory information include, for example, generating audio information that exaggerates the sounds of moving birds and insects, displaying shaky images like a mirage to indicate hot and humid weather, and expressing cold.
  • the edges may be displayed to show a clear feeling, but the present invention is not limited to these examples.
  • step S13 is not executed, and the first composite information is outputted to the expression information transmitter 26 as expression information. Note that if there are multiple remote participants, steps S12 and S13 are performed for each remote participant.
  • the information processing device 2 transmits the second synthesis information and the detection information not to be synthesized and not to be converted to the expression device 6 of the remote participant (step S14).
  • the expression information generation unit 25 converts the second information and the detection information that is neither a synthesis target nor a conversion target into the expression device 6 of the remote participant, that is, the expression device 6 of the virtual avatar user and the real avatar user. Send to.
  • the information processing device 2 transmits the virtual information to the expression device 4 of the real environment 10 (step S15).
  • the expression information generation unit 25 transmits the virtual information to the expression device 4 of the real environment 10, that is, the expression device 4 that transmits information to the real participants.
  • the information converted in step S13 and the virtual information may be combined and transmitted to the expression device 4 of the real participant. Since the real participants can directly feel the state of the air in the real environment 10, the information converted in step S13 does not need to be sent to the expression device 4, but the converted information may also be transmitted to the real participants. By being transmitted, the real participants can share the information with the virtual avatar user and the real avatar user.
  • processing for example, every control cycle, information indicating the state of the real environment 10 is transmitted to remote participants, and virtual information is transmitted to real participants.
  • the expression device 6 corresponding to the real avatar user is the first expression device and the expression device 6 corresponding to the virtual avatar user (second user) is the second expression device, then the first expression device is , it is sufficient that the first user is capable of expressing an image that can be recognized with at least one of the five senses, and the second expression device is sufficient as long as it is capable of presenting an image to at least the second user.
  • the sensor 3 includes a photographing device that detects information regarding objects in the real environment 10 by photographing at least the real environment 10 in order to generate an image to be provided to the virtual avatar user.
  • the expression information generation unit 25 generates first expression information to be transmitted to the first user (real avatar user) using the first detection information that is the detection information received from the sensor 14 , and uses the first detection information that is the detection information received from the sensor 3 2 detection information is used to generate second expression information including an image corresponding to the virtual viewpoint in the real environment 10 of the second user (virtual avatar user), and the expression information transmitter 26 transmits the first expression information to the first expression device.
  • the expression information is transmitted, and the second expression information is transmitted to the second expression device.
  • the expression information generation unit 25 recognizes the first detection information indicating the detection result of at least one of temperature, humidity, wind, and odor in the real environment 10 by at least one of the visual and auditory senses.
  • the information may be converted into first expression information that can be expressed.
  • the expression information generation unit 25 is capable of recognizing second detection information indicating a detection result of at least one of temperature, humidity, wind, and odor in the real environment 10 with at least one of visual and auditory information. It may also be converted into second expression information.
  • the sensor may include information related to taste, and the expression information generation unit 25 may generate information related to taste in the real environment 10.
  • the expression device 4 and the expression device 6 include a taste expression device that reproduces taste. Alternatively, information regarding taste may be converted into other types of methods such as visual or auditory information.
  • the expression information generation unit 25 may generate the first expression information using the second detection information. That is, the information acquired by the sensor 3 may be used to generate expression information to be transmitted to the real avatar user.
  • the expression information generation unit 25 may generate the second expression information so that the virtual avatar is displayed at the virtual viewpoint, and the participant movement detection device 5 detects the movement of the virtual avatar user and generates the expression information.
  • the unit 25 detects objects in the real environment 10 and the virtual avatar in the virtual space, which is a range including the virtual viewpoint, based on the motion information of the virtual avatar user and the second detection information (video data of the real environment 10). virtually generates a video that changes in accordance with the movements of the virtual avatar user, synthesizes virtual information indicating the generated video with second expression information, and transmits the second expression information into which the virtual information is synthesized.
  • the expression information may be sent to a second expression device.
  • the expression information generation section 25 may combine the virtual information with the first expression information, and the expression information transmission section 26 may transmit the first expression information with the combined virtual information to the first expression device.
  • the virtual avatar may have only a virtual viewpoint, or may be able to hear and speak in addition to the virtual viewpoint.
  • the expression information transmitting unit 26 may transmit the virtual information to the expression device 4, which is a third expression device that can present images to real participants (third users). Further, the information processing device 2 may be able to set whether information regarding the virtual avatar user and the real avatar user is made public or private. Information regarding virtual avatar users and real avatar users includes, for example, facial information, eyeball information, voice prints, and other frequently used security information, as well as personal information such as real names, ages, and dates of birth. but not limited to. Furthermore, if the user does not wish to make his or her actual face or voice public for privacy reasons, the user may be able to change the face or voice and make it public.
  • FIG. 6 is a diagram showing an example in which real participants, real avatar users, and virtual avatar users participate in an event.
  • real participants 8-1 to 8-3 participate in the event in the real environment 10
  • the real avatar user 70-1 is the real avatar user.
  • Users 70-2 and 70-3, who are virtual avatar users participate in the event using device 1 and participate in the event from participating locations 7-2 and 7-3 using virtual avatars 9-1 and 9-2, respectively.
  • Participating in each of the participating locations 7-1 to 7-3 is provided with a participant motion detection device 5 and an expression device 6. Note that when the virtual avatars 9-1 and 9-2 are fixed without moving, the participant motion detection device 5 does not need to be provided to the users 70-2 and 70-3 who are virtual avatar users.
  • the device 1 includes a sensor 14 that detects information regarding each of visual, auditory, olfactory, and tactile sensations, and includes a physical acting device, a moving device, and a speaker as actuators.
  • the sensors 3-1 and 3-2 installed in the real environment 10 are cameras that acquire images for generating images from the viewpoints of the virtual avatars 9-1 and 9-2, and This is a microphone that collects sound to generate sound at position -2.
  • two sensors 3 are shown for the sake of simplification, but in general, a camera that acquires images for generating images from the viewpoints of the virtual avatars 9-1 and 9-2.
  • a plurality of microphones each collect sound to generate sound at the positions of the virtual avatars 9-1 and 9-2.
  • a sensor that detects odor, temperature, wind speed, air volume, humidity, type of gas, etc. of the actual environment 10 may be provided.
  • all information to be transmitted to the user 70-1, who is a real avatar user is detected by the sensor 14 installed in the device 1.
  • the field of view of the sensor 14, such as a camera that obtains visual information, in the device 1 is changed, and the device 1 operates in accordance with the movement of the real avatar user.
  • the image in the real environment 10 is displayed by the expression device 6. The same applies to sounds, smells, etc., and as the device 1 moves, information corresponding to sounds and smells in the real environment 10 is transmitted to the real avatar user.
  • the device 1 touches an actual object in the real environment 10 in accordance with the movement of the real avatar user
  • information regarding haptic sensation is detected by the sensor 14 and transmitted to the real avatar user. This allows the real avatar user to recognize the real environment 10 with a sense of realism.
  • users 70-2 and 70-3 who are virtual avatar users, are provided with images from the viewpoints of virtual avatars 9-1 and 9-2, and images from virtual avatars 9-1 and 9-1, respectively, using the detection information acquired by the sensor 3. , 9-2 are transmitted. Further, when a sensor that detects a smell is provided as the sensor 3, the information processing device 2 uses the detection information by the sensor 3 to provide the users 70-2 and 70-3, who are virtual avatar users, with the expression device. 6 can convey information indicating odor.
  • the expression device 4 is not provided in the real environment 10. You don't have to. As described above, when projecting the image corresponding to the real avatar at the location where the device 1 is present, and projecting the images corresponding to the locations corresponding to the virtual avatars 9-1 and 9-2, these are projected. A representation device 4 is provided. Similarly, when the real participants 8-1 to 8-3 are allowed to recognize the sounds corresponding to the virtual avatars 9-1 and 9-2, the expression device 4 allows the user 70-2 who is the virtual avatar user to recognize the sound. , 70-3 are transmitted to the actual participants by the expression device 4.
  • FIG. 7 is a diagram showing an example in which a real avatar user and a virtual avatar user participate in an event.
  • users 70-1, 70-4, and 70-5 who are real avatar users, are connected to devices whose real avatars correspond to their respective locations.
  • Users 70-2 and 70-3, who are virtual avatar users participate in the event using virtual avatars 9-1 and 9-3 from participation locations 7-2 and 7-3, respectively, using virtual avatars 9-1 and 9-3.
  • each of the participating locations 7-1 to 7-5 is provided with a participant motion detection device 5 and an expression device 6.
  • Each of the devices 1-1 to 1-3 is the device 1 described above.
  • each of devices 1-1 to 1-3 is similar to device 1 shown in FIG. 6.
  • the method of transmitting information to the real avatar user and the virtual avatar user in the example shown in FIG. 7 is also the same as in the example shown in FIG. 6.
  • there is no real participant 8 there is no need to provide the expression device 4 in the real environment 10.
  • FIG. 8 is a diagram showing an example of generating a virtual space corresponding to a virtual avatar.
  • a user 70-1 participates in an event using device 1, which is a real avatar, and users 70-2 and 70-3, who are virtual avatar users, use virtual avatar 9 from participating locations 7-2 and 7-3, respectively. I am participating in the event using -1 and 9-2.
  • each of the participating locations 7-1 to 7-3 is provided with a participant motion detection device 5 and an expression device 6.
  • the information processing device 2 acquires motion information from the participant motion detection device 5 corresponding to the user 70-3, who is a virtual avatar user, and uses the acquired information to A virtual space 90 is generated around the .
  • the information processing device 2 then synthesizes virtual information indicating the virtual space 90 with images, sounds, etc. from the viewpoint of the virtual avatar 9-2 generated using the detection information of the sensor 3, and uses the synthesized information as expression information. , and is transmitted to the expression device 6 corresponding to the user 70-3, who is the virtual avatar user.
  • the combined information is also sent to the user 70-1 who is the real avatar user and the user 70-2 who is the virtual avatar user.
  • FIG. 9 is a diagram showing an example of a video in which virtual spaces are synthesized.
  • a video 201 of the virtual space corresponding to the virtual avatar 9-2 is synthesized with a video 200 of the real environment 10 from the viewpoint of the user 70-2, who is a virtual avatar user. It shows.
  • the image 200 is a real image based on information captured by the sensor 3
  • the image 201 is information indicating a virtual space generated based on the information captured by the sensor 3.
  • the video 201 reflects the movement of the virtual avatar 9-2 in the virtual space. Therefore, when the virtual avatar 9-2 touches an object in the virtual space, the object also changes in the virtual space.
  • the user 70-3 who is the virtual avatar user corresponding to the virtual avatar 9-2 is also transmitted the video in which the virtual space has been synthesized, so that the user 70-3 who is the virtual avatar user can You can experience the same sensation as when you touch an object.
  • the real participants 8-1 to 8-3 can express the images projected onto the actual real environment 10. It is possible to visually recognize the state in which the virtual spaces are superimposed.
  • FIG. 10 is a diagram showing the reduction of sensors in equipment.
  • a user 70-1 participates in an event using device 1, which is a real avatar, and users 70-2 and 70-3, who are virtual avatar users, use virtual avatar 9 from participating locations 7-2 and 7-3, respectively. I am participating in the event using -1 and 9-2.
  • each of the participating locations 7-1 to 7-3 is provided with a participant motion detection device 5 and an expression device 6.
  • the sensors 3-1 to 3-4 include a camera for acquiring a 360-degree free viewpoint image and a microphone for reproducing sound according to the position.
  • Sensors 3-1 to 3-4 detect images of the real environment 10 from the viewpoints of users 70-2 and 70-3 who are virtual avatar users, and sounds at the positions of users 70-2 and 70-3 who are virtual avatar users. It is used to generate information indicating the image of the real environment 10 from the viewpoint of the real avatar and information indicating the sound at the position of the real avatar.
  • the sensor 14 included in the device 1 does not need to include a sensor that detects information related to vision and hearing.
  • the configuration of the device 1 can be made simpler than the example shown in FIG.
  • the first information may be generated from the information, and the other may be generated from the detection information of the sensor 14. Further, by using the sensor 3 that detects odor, the number of sensors that detect information related to smell among the sensors 14 included in the device 1 may be reduced.
  • FIG. 11 is a diagram showing an example of the configuration of a computer system that implements the information processing device 2 of this embodiment.
  • this computer system includes a control section 101, an input section 102, a storage section 103, a display section 104, a communication section 105, and an output section 106, which are connected via a system bus 107.
  • the control section 101 and the storage section 103 constitute a processing circuit.
  • control unit 101 is, for example, a processor such as a CPU (Central Processing Unit), and executes a program in which processing in the information processing device 2 of this embodiment is described. Note that a part of the control unit 101 may be realized by dedicated hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array).
  • the input unit 102 includes, for example, a keyboard and a mouse, and is used by a user of the computer system to input various information.
  • the storage unit 103 includes various memories such as RAM (Random Access Memory) and ROM (Read Only Memory), and storage devices such as a hard disk, and stores programs to be executed by the control unit 101 and in the process of processing. Store the necessary data obtained.
  • the storage unit 103 is also used as a temporary storage area for programs.
  • the display unit 104 is composed of a display, an LCD (liquid crystal display panel), etc., and displays various screens to the user of the computer system.
  • the communication unit 105 is a receiver and a transmitter that perform communication processing.
  • the output unit 106 is a printer, a speaker, or the like. Note that FIG. 11 is an example, and the configuration of the computer system is not limited to the example shown in FIG.
  • a computer program can be loaded from a CD-ROM or DVD-ROM set in a CD (Compact Disc)-ROM drive or a DVD (Digital Versatile Disc)-ROM drive (not shown). is installed in the storage unit 103. Then, when the program is executed, the program read from the storage unit 103 is stored in the main storage area of the storage unit 103. In this state, the control unit 101 executes processing as the information processing apparatus 2 of this embodiment according to the program stored in the storage unit 103.
  • a CD-ROM or DVD-ROM is used as a recording medium to provide a program that describes processing in the information processing device 2; however, the present invention is not limited to this; Depending on the capacity of the program, for example, a program provided via a transmission medium such as the Internet via the communication unit 105 may be used.
  • the program of this embodiment includes, for example, the step of receiving motion information indicating the motion of a first user who participates in an event at a location remote from the real environment 10 where the event is held, and using the motion information to A step of generating control information for causing a device installed in the device to perform an operation according to the operation information, a step of transmitting the control information to the device, and a step of detecting information regarding an object mounted on the device in the real environment 10. receiving first detection information detected by the device-mounted sensor from the device-mounted sensor; and second detection information detected by the real-environment sensor from the real-environment sensor installed in the real environment 10 and detecting information about an object in the real environment 10. and causing the computer system to perform the step of receiving detection information.
  • the program of this embodiment further includes a step of generating first expression information to be transmitted to the first user using the first detection information, and a step of generating first expression information to be transmitted to the first user using the second detection information. a step of generating second expression information including an image corresponding to a virtual viewpoint in the real environment 10 of a second user participating in the event;
  • the computer system is caused to execute the steps of transmitting the first expression information to the expression device and transmitting the second expression information to the second expression device capable of presenting the video to the second user.
  • the device control section 22 and the expression information generation section 25 shown in FIG. 1 are realized by the computer program stored in the storage section 103 shown in FIG. 11 being executed by the control section 101 shown in FIG. Ru.
  • the storage unit 103 shown in FIG. 11 is also used to realize the device control unit 22 and the expression information generation unit 25 shown in FIG.
  • the control information transmitting section 21, motion information receiving section 23, detection information receiving section 24, and expression information transmitting section 26 shown in FIG. 1 are realized by the communication section 105 shown in FIG.
  • the information processing device 2 may be realized by a plurality of computer systems.
  • the information processing device 2 may be realized by a cloud computer system.
  • some of the functions of the information processing device 2 may be realized by another device provided separately from the information processing device 2.
  • Another device may be provided in the real environment 10, may be provided near the residence of the user participating in the event, or may be provided at another location.
  • a user can remotely participate in the event.
  • This allows remote participants in an event using the real environment 10 to experience the real environment 10 more appropriately.
  • both participation by a real avatar using the device 1 and participation by a virtual avatar even when the number of participants in an event increases, it is possible to respond flexibly while keeping costs down.
  • transmitting information indicating the air condition of the real environment 10 to the remote participants it is possible to make the remote participants experience a state closer to the real environment 10 .
  • information corresponding to the real avatar is generated using the detection information of the sensor 3 used to provide information corresponding to the virtual avatar to the virtual avatar user, the configuration of the device 1 can be simplified and the cost will be reduced. can be reduced.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Processing Or Creating Images (AREA)
PCT/JP2022/015971 2022-03-30 2022-03-30 遠隔体験システム、情報処理装置、情報処理方法およびプログラム WO2023188104A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/JP2022/015971 WO2023188104A1 (ja) 2022-03-30 2022-03-30 遠隔体験システム、情報処理装置、情報処理方法およびプログラム
JP2024510887A JPWO2023188104A1 (de) 2022-03-30 2022-03-30

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2022/015971 WO2023188104A1 (ja) 2022-03-30 2022-03-30 遠隔体験システム、情報処理装置、情報処理方法およびプログラム

Publications (1)

Publication Number Publication Date
WO2023188104A1 true WO2023188104A1 (ja) 2023-10-05

Family

ID=88200265

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/015971 WO2023188104A1 (ja) 2022-03-30 2022-03-30 遠隔体験システム、情報処理装置、情報処理方法およびプログラム

Country Status (2)

Country Link
JP (1) JPWO2023188104A1 (de)
WO (1) WO2023188104A1 (de)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2012152395A (ja) * 2011-01-26 2012-08-16 Sony Computer Entertainment Inc 情報処理システム、情報処理システムの制御方法、プログラム及び情報記憶媒体
JP2013020389A (ja) * 2011-07-08 2013-01-31 Dowango:Kk 会場設置ディスプレイシステム
JP2017033536A (ja) * 2015-07-29 2017-02-09 イマージョン コーポレーションＩｍｍｅｒｓｉｏｎ Ｃｏｒｐｏｒａｔｉｏｎ 観衆ベースのハプティック
JP2018008369A (ja) * 2016-06-10 2018-01-18 ザ・ボーイング・カンパニーＴｈｅ Ｂｏｅｉｎｇ Ｃｏｍｐａｎｙ マルチモーダル感覚データに基づくロボットプラットフォームの遠隔制御
US20180338163A1 (en) * 2017-05-18 2018-11-22 International Business Machines Corporation Proxies for live events
WO2019225548A1 (ja) * 2018-05-21 2019-11-28 Ｔｅｌｅｘｉｓｔｅｎｃｅ株式会社 遠隔操作システム、情報処理方法、及びプログラム
JP2021144522A (ja) * 2020-03-12 2021-09-24 キヤノン株式会社 画像処理装置、画像処理方法、プログラム、及び画像処理システム

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2012152395A (ja) * 2011-01-26 2012-08-16 Sony Computer Entertainment Inc 情報処理システム、情報処理システムの制御方法、プログラム及び情報記憶媒体
JP2013020389A (ja) * 2011-07-08 2013-01-31 Dowango:Kk 会場設置ディスプレイシステム
JP2017033536A (ja) * 2015-07-29 2017-02-09 イマージョン コーポレーションＩｍｍｅｒｓｉｏｎ Ｃｏｒｐｏｒａｔｉｏｎ 観衆ベースのハプティック
JP2018008369A (ja) * 2016-06-10 2018-01-18 ザ・ボーイング・カンパニーＴｈｅ Ｂｏｅｉｎｇ Ｃｏｍｐａｎｙ マルチモーダル感覚データに基づくロボットプラットフォームの遠隔制御
US20180338163A1 (en) * 2017-05-18 2018-11-22 International Business Machines Corporation Proxies for live events
WO2019225548A1 (ja) * 2018-05-21 2019-11-28 Ｔｅｌｅｘｉｓｔｅｎｃｅ株式会社 遠隔操作システム、情報処理方法、及びプログラム
JP2021144522A (ja) * 2020-03-12 2021-09-24 キヤノン株式会社 画像処理装置、画像処理方法、プログラム、及び画像処理システム

Also Published As

Publication number Publication date
JPWO2023188104A1 (de) 2023-10-05

Similar Documents

Publication Publication Date Title
LaValle Virtual reality
US9654734B1 (en) Virtual conference room
National Research Council Virtual reality: scientific and technological challenges
Stanney et al. Extended reality (XR) environments
JP2022549853A (ja) 共有空間内の個々の視認
WO2018086224A1 (zh) 一种虚拟现实场景的生成方法、设备及虚拟现实系统
Larsson et al. The actor-observer effect in virtual reality presentations
JP6298130B2 (ja) シミュレーションシステム及びプログラム
JP3715219B2 (ja) バーチャル現場訓練装置
JP6683864B1 (ja) コンテンツ制御システム、コンテンツ制御方法、およびコンテンツ制御プログラム
JP2007151647A (ja) 画像処理装置、画像処理方法及びプログラム
JP6739611B1 (ja) 授業システム、視聴端末、情報処理方法及びプログラム
JP2017215577A (ja) バーチャルロボットを用いた教育システム
Vafadar Virtual reality: opportunities and challenges
JP2020080154A (ja) 情報処理システム
US10582190B2 (en) Virtual training system
McMenemy et al. A hitchhiker's guide to virtual reality
JP2018136944A (ja) シミュレーションシステム及びプログラム
WO2023188104A1 (ja) 遠隔体験システム、情報処理装置、情報処理方法およびプログラム
US20240096227A1 (en) Content provision system, content provision method, and content provision program
WO2022107294A1 (ja) Ｖｒ映像空間生成システム
JP7465737B2 (ja) 授業システム、視聴端末、情報処理方法及びプログラム
JP6892478B2 (ja) コンテンツ制御システム、コンテンツ制御方法、およびコンテンツ制御プログラム
Nesamalar et al. An introduction to virtual reality techniques and its applications
JP2021009351A (ja) コンテンツ制御システム、コンテンツ制御方法、およびコンテンツ制御プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22935250

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2024510887

Country of ref document: JP