WO2023188104A1

WO2023188104A1 - Remote experience system, information processing device, information processing method, and program

Info

Publication number: WO2023188104A1
Application number: PCT/JP2022/015971
Authority: WO
Inventors: 正樹春名
Original assignee: 三菱電機株式会社
Priority date: 2022-03-30
Filing date: 2022-03-30
Publication date: 2023-10-05

Abstract

A remote experience system (100) according to the present disclosure comprises: equipment (1) that has sensors (14-1, 14-2); sensors (3-1 to 3-3) that are placed in a real environment; participant-action detecting devices (5-1, 5-2) that detect the action of a first user participating in an event from a place that is away from the real environment; representation devices (6-1, 6-2) that can provide representations to the first user; representation devices (6-1, 6-2) that can present images to a second user; and an information processing device (2). The information processing device (2) comprises: an equipment control unit (22) that generates equipment control information using action information received from the participant-action detecting devices (5-1, 5-2); and a representation information generating unit (25) that uses first detection information received from the sensors (14-1, 14-2) to generate first representation information to be transmitted to the first user, and that uses second detection information received from the sensors (3-1 to 3-3) to generate second representation information including an image corresponding to a virtual view point of the second user in the real environment.

Description

Remote experience system, information processing device, information processing method and program

The present disclosure relates to a remote experience system, information processing device, information processing method, and program that provide information about a remote location to a user.

Systems that allow users to participate in events in remote locations, such as distance education and remote conferences, are being introduced. For example, a technique is known in which a teacher is present at an educational site and transmits information in one direction to a user (student) located at a location remote from the educational site.

Additionally, a technique is known that simulates a face-to-face conversation between distant participants by creating a virtual space and displaying virtual participants representing the participants together with the virtual space. For example, Patent Document 1 discloses a technology related to a remote conference in which in-house workers who work in the company and remote workers who work remotely coexist. In the technology described in Patent Document 1, a virtual room is displayed on the terminals of in-house workers and remote workers, and the in-house workers and remote workers are displayed as virtual participants in the virtual room. The sense of realism is enhanced by reflecting the movements of virtual workers and remote workers in the movements of virtual workers.

Patent No. 6888854

On the other hand, depending on the content of the event, it may be desirable to experience various states of the real environment rather than the virtual space. Although the above Patent Document 1 discloses participation of virtual participants in a virtual space, it does not disclose a technique for allowing remote participants in an event using a real environment to experience the real environment.

The present disclosure has been made in view of the above, and aims to provide a remote experience system that allows remote participants in an event using a real environment to experience the real environment.

In order to solve the above-mentioned problems and achieve the purpose, a remote experience system according to the present disclosure is installed in a real environment where an event is held, and includes a device equipped with a device-mounted sensor that detects information about objects in the real environment. , a real environment sensor that is installed in a real environment and detects information about objects in the real environment; a participant motion detection device that detects the motion of a first user who participates in the event from a location away from the real environment; a first expression device capable of expressions that can be recognized by a user with at least one of the five senses; a second expression device capable of presenting images to a second user who participates in the event from a location remote from the real environment; and information processing. A device. The real environment sensor includes a photographing device that detects information about objects in the real environment by photographing the real environment. The information processing device includes a motion information receiving unit that receives motion information indicating the motion acquired by the participant motion detection device from the participant motion detection device, and uses the motion information to cause the device to perform a motion according to the motion information. a device control unit that generates control information to cause the device to perform the operation; a control information transmitting unit that sends the control information to the device; and a device-mounted sensor that receives first detection information detected by the device-mounted sensor from the real environment sensor. and a detection information receiving section that receives second detection information detected by the real environment sensor. The information processing device further uses the first detection information to generate first expression information to be transmitted to the first user, and uses the second detection information to generate a video corresponding to the virtual viewpoint of the second user in the real environment. and an expression information transmitter that transmits the first expression information to the first expression device and the second expression information to the second expression device.

The remote experience system according to the present disclosure has the effect of allowing remote participants in an event using a real environment to experience the real environment.

A diagram showing a configuration example of a remote experience system according to an embodiment. A diagram schematically showing distance education using the remote experience system of the embodiment Flowchart illustrating an example of a device control procedure in the information processing apparatus according to the embodiment A flowchart illustrating an example of a procedure for generating virtual information regarding a virtual avatar in an information processing apparatus according to an embodiment Flowchart illustrating an example of a procedure for generating expression information in the information processing apparatus according to the embodiment A diagram showing an example in which real participants, real avatar users, and virtual avatar users participate in an event. Diagram showing an example of real avatar users and virtual avatar users participating in an event Diagram showing an example of generating a virtual space corresponding to a virtual avatar Diagram showing an example of a video in which virtual space is synthesized Diagram showing the reduction of sensors in equipment A diagram showing a configuration example of a computer system that implements an information processing device according to an embodiment.

Below, a remote experience system, an information processing device, an information processing method, and a program according to an embodiment will be described in detail based on the drawings.

Embodiment.
FIG. 1 is a diagram showing a configuration example of a remote experience system according to an embodiment. The remote experience system 100 of this embodiment is used for an event using the real environment 10. The real environment 10 is, for example, a place where an event is held. Events include, for example, distance education, remote experience, remote skill transfer, remote maintenance, remote training, experiential travel, healing, astronomical observation, on-site games, on-site golf, local sightseeing, boat trips, car trips, exploration, etc. Yes, but not limited to.

When the remote experience system 100 of this embodiment is used, there are two ways to participate in an event: one is to participate as a real participant who actually exists in the real environment 10, and the other is to use the device 1 as a real avatar that is your alter ego. It is possible to provide three types of participation methods: a participation method using a virtual avatar that does not exist in the real environment 10, and a participation method using a virtual avatar that does not exist in the real environment 10. A user who participates using a real avatar and a user who participates using a virtual avatar are both virtual participants that do not exist in the real environment 10, and are remote participants.

When using a real avatar, information obtained by the device 1 from the real environment 10 can be transmitted to the user through the actions of the device 1, such as when the device 1 moves in the real environment 10 or when the device 1 touches objects around it. I can do it. On the other hand, when real avatars are used, it is necessary to prepare devices 1 according to the number of users participating, and costs may be incurred depending on the number of participants in the event. Therefore, in the remote experience system 100 of this embodiment, by allowing participation as a virtual avatar, it is possible to reduce costs and increase the number of participants in an event. Details of the real avatar and virtual avatar will be described later. The following will mainly explain examples in which real participants, real avatars, and virtual avatars can coexist, but the remote experience system is not limited to this. 100 may be constructed.

As shown in FIG. 1, the remote experience system 100 includes a device 1, sensors 3-1 to 3-3, and expression devices 4-1 and 4-2 installed in a real environment 10, and users who participate in an event. The information processing device 2 is provided with participant motion detection devices 5-1, 5-2 and expression devices 6-1, 6-2, which are installed at a participation location 7, which is a place where participants participate.

The participant motion detection devices 5-1 and 5-2 installed at the participation location 7 detect the motions of users participating in the event, and transmit the detected information to the information processing device 2 as motion information. The participant motion detection devices 5-1 and 5-2 are set at a participation location 7 where a user (first user) who participates using at least a real avatar exists, and detects the motion of the user. As will be described later, the participant motion detection devices 5-1 and 5-2 are set at a participation location 7 where a user (second user) who participates using a virtual avatar exists and detects the motion of the user. good.

The participant motion detection devices 5-1 and 5-2 are, for example, positioning sensors, cameras, microphones, etc. The expression devices 6-1 and 6-2 receive expression information indicating the state of the real environment 10 received from the information processing device 2, and transmit the state of the real environment 10 to the user based on the expression information. The expression devices 6-1 and 6-2 are devices capable of expressions that the user can recognize with at least one of the five senses, and can, for example, present images to the user. The expression devices 6-1 and 6-2 are, for example, a display, a speaker, a haptic groove, etc. The displays used as the expression devices 6-1 and 6-2 may be head-mounted displays, VR (Virtual Reality) goggles, etc., or may be terminals such as smartphones and personal computers. , or a cockpit display that allows the user to move the display with the user when the user gets on board. Furthermore, the expression devices 6-1 and 6-2 may be, for example, hologram displays, aerial projection displays, or the like. Furthermore, the expression devices 6-1 and 6-2 may include an olfactory expression device, a blower, an air conditioner, and the like.

Hereinafter, when the participant motion detection devices 5-1 and 5-2 are referred to without being individually distinguished, they will be referred to as participant motion detection device 5, and the expression devices 6-1 and 6-2 will be referred to without being individually distinguished. It is sometimes referred to as an expression device 6. Although two participant motion detection devices 5 and two expression devices 6 are shown in FIG. 1, the number of participant motion detection devices 5 and expression devices 6 is not limited to the example shown in FIG. 1. Furthermore, although one participating location 7 is shown in FIG. 1, when users participate from a plurality of different locations, a similar device is provided at each participating location 7. Note that when a plurality of users participate in an event from the same participation location 7, the participant motion detection device 5 and the expression device 6 are basically provided for each user, but some of them may be shared. For example, when a hologram display, an aerial projection display, or the like is used as the expression device 6, the expression device 6 may transmit information to a plurality of users. Furthermore, the participant motion detection device 5 may detect the motions of a plurality of users.

The sensors 3-1 to 3-3 are real environment sensors that are installed in the real environment 10 and detect information regarding objects in the real environment 10. The object includes, for example, at least one of animals, plants, and air that exist in the real environment 10. The sensors 3-1 to 3-3 are sensors that detect the environment in the real environment 10, and detect, for example, objects, sounds, temperature, wind speed, air volume, humidity, illuminance, smell, type of gas, etc. in the real environment 10. It is a sensor that That is, the sensors 3-1 to 3-3 detect at least one of temperature, humidity, wind, and odor in the real environment 10, for example. Furthermore, object detection may be performed, for example, by photographing an image or by collecting sound. That is, the sensors 3-1 to 3-3 may include a photographing device such as a camera, or may include a microphone. Detection information indicating the detection result is transmitted to the information processing device 2. Hereinafter, when the sensors 3-1 to 3-3 are shown without being individually distinguished, they will be referred to as sensor 3.

The device 1 is an avatar that is an alter ego of a user who participates in an event, and more specifically, it is a real avatar that exists as an object. The device 1 may be, for example, a humanoid robot, a movable robot having a manipulator and a head that simulates a human face, or a manipulator. good. As will be described later, the device 1 operates based on control information received from the information processing device 2. The device 1 includes a control information receiving section 11, a drive control section 12, a driving section 13, sensors 14-1 and 14-2, and a detection information transmitting section 15.

The control information receiving unit 11 receives control information from the information processing device 2 and outputs the received control information to the drive control unit 12. The drive control section 12 controls the drive section 13 based on the control information. The drive unit 13 includes one or more actuators. The actuator is, for example, a physical effector such as a manipulator that operates each joint in the device 1, a moving device, a speaker, or the like.

The sensors 14-1 and 14-2 are device-mounted sensors that detect information regarding objects in the real environment 10. Similar to the sensor 3, the sensors 14-1 and 14-2 include a photographing device such as a camera, a speaker, etc., but, for example, they detect at least one of temperature, humidity, wind, and odor in the actual environment 10. May be detected. Further, the sensors 14-1 and 14-2 may detect force-tactile sensation. The sensors 14-1 and 14-2 output detection information indicating detection results to the detection information transmitter 15. The detection information transmitter 15 transmits the detection information to the information processing device 2. Since the sensors 14-1 and 14-2 are mounted on the device 1, they can detect information depending on the location of the device 1. Hereinafter, when the sensors 14-1 and 14-2 are shown without being individually distinguished, they will be referred to as the sensor 14. For example, if the device 1 is provided with a part that simulates a face and a camera is provided as the sensor 14 in the part corresponding to the eyes, the sensor 14 acquires an image depending on the movement of the device 1 and the direction of the face. can do. Although two sensors 14 are shown in FIG. 1, the number of sensors 14 is not limited to the example shown in FIG.

For example, the device 1 may include a manipulator, and the sensor 14 may include a force tactile sensor that detects force tactile sensation at the hand of the manipulator. Since the expression device 6 corresponding to the user using the real avatar includes a haptic device that transmits the haptic sensation to the hand of the user using the real avatar, the haptic sensation detected by the device 1 can be transmitted to the hand of the user using the real avatar. can be communicated to users who use it.

The expression devices 4-1 and 4-2 are devices that perform expressions to convey the virtual environment superimposed on the real environment 10 to users who participate in an event in the real environment 10. The expression devices 4-1 and 4-2 are, for example, displays, speakers, and the like. The displays used as the expression devices 4-1 and 4-2 are the same as the display used as the expression device 6 described above, but it is sufficient that they can express a portion of the virtual environment as described later. Hereinafter, the expression devices 4-1 and 4-2 will be referred to as expression device 4 when shown without distinguishing them individually.

Note that, as will be described later, the expression device 4 may not be provided in the real environment 10 depending on the configuration of the device 1 and the participation method of the users who participate in the event. Furthermore, depending on the configuration of the device 1 and the participation method of users participating in the event, the sensor 3 may not be provided in the real environment 10. Further, in FIG. 1, three sensors 3 and two expression devices 4 are illustrated, but the respective numbers of sensors 3 and expression devices 4 are not limited to the example shown in FIG. 1. Further, although one device 1 is shown in FIG. 1, the number of devices 1 is not limited to the example shown in FIG.

The information processing device 2 includes a control information transmitter 21 , a device controller 22 , an operation information receiver 23 , a detection information receiver 24 , an expression information generator 25 , and an expression information transmitter 26 .

The motion information receiving unit 23 receives motion information indicating the motion acquired by the participant motion detection device 5 from the participant motion detection device 5 at the participation location 7, and outputs the received motion information to the device control unit 22. . Identification information that can identify the corresponding user is added to the operation information. This identification information may be user identification information determined when the user registers to participate in the event, or may be identification information of the participant motion detection device 5. When the identification information added to the movement information is the identification information of the participant movement detection device 5, the identification information of the participant movement detection device 5 and the user identification information are associated at the time of registration for participation in the event, etc. Suppose that

The device control unit 22 uses the operation information to generate control information for causing the device 1 to perform an operation according to the operation information. That is, the device control unit 22 uses the operation information to generate control information for controlling the device 1 so that the device 1, which is the real avatar of the user, performs an operation corresponding to the user's operation, The generated control information is output to the control information transmitter 21. The control information is, for example, information indicating how to drive each actuator of the device 1. Note that the device control unit 22 holds correspondence information indicating the correspondence between the device 1 and the user who uses the device 1 as a real avatar, and uses the correspondence information and the identification information added to the operation information to The device 1 corresponding to the operation information is identified. The correspondence information may be determined, for example, when the user registers to participate in the event, or may be selected by the user when the event starts. The user may check the functions and positions of each device 1, select the device 1 that will become the actual avatar, and notify the information processing device 2 of the selected device 1 via the user's terminal (not shown) or the like. The control information transmitter 21 transmits the control information received from the device controller 22 to the device 1.

The detection information receiving unit 24 receives detection information from at least one of the sensor 14 and the sensor 3, and outputs the received detection information to the expression information generating unit 25. The expression information generation unit 25 uses the detection information to generate expression information indicating the content to be expressed by the expression device 6, and outputs the generated expression information to the expression information transmission unit 26. The expression information is transmitted to the expression device 6. Note that, similar to the correspondence between the device 1 and the user described above, the expression information generation unit 25 holds correspondence information indicating the correspondence between the expression device 6 and the user. Further, the expression information generation unit 25 generates expression information to be transmitted to users participating in the real environment 10, and outputs the generated expression information to the expression information transmission unit 26. It is transmitted to the expression device 4.

Note that the communication line between the information processing device 2 and each device may be a wireless line, a wired line, or a mixture of a wireless line and a wired line. Communication between the information processing device 2 and each device may be performed using any communication method, but for example, the Beyond 5G (5th Generation) 5th generation mobile communication system, which achieves large-capacity, low-latency transmission, ), information can be transmitted to the user with low delay and the sense of realism can be enhanced. That is, the communication line between the information processing device 2 and the participant motion detection device 5, the communication line between the information processing device 2 and the sensor 14, the communication line between the information processing device 2 and the sensor 3, and the information processing At least one of the communication line between the device 2 and the device 1, the communication line between the information processing device 2 and the expression device 6, and the communication line between the information processing device 2 and the expression device 4 is Beyond 5G line may be used.

Next, an application example of the remote experience system 100 of this embodiment will be described. FIG. 2 is a diagram schematically showing distance education using the remote experience system 100 of this embodiment. In the example shown in FIG. 2, for example, real participants 8-1 to 8-4 actually exist in a forest, which is the real environment 10. Furthermore, in the forest, which is the real environment 10, there is a device 1 used as a real avatar. Furthermore, there are also users who participate in the real environment 10 as virtual avatars 9 who do not have a physical presence. The real participant 8-1 is, for example, a host such as a teacher, and the real participants 8-2 to 8-4, users who participate as virtual avatars 9, and users who participate using the device 1 as a real avatar are: , is a student. Hereinafter, when referring to the real participants 8-1 to 8-4 without distinguishing them individually, they will be referred to as real participants 8.

In the example shown in FIG. 2, the real environment 10 is a forest, so the presence of birds, animals, plants, etc. causes changes in smell, air flow, and sound due to the movement of birds and animals. . In addition, by touching the animals and plants that exist in the forest, participants will be able to experience something unique to the real environment 10. By transmitting various states in addition to images and sounds to the user, it is possible, for example, to promote the user's understanding of the surrounding world. In this embodiment, users who remotely participate in distance education are also provided with a more realistic experience of the real environment 10.

The movements of users who participate using the device 1 as real avatars (hereinafter also referred to as real avatar users) are detected by the participant movement detection device 5 at the participation location 7 shown in FIG. It is transmitted to the processing device 2. Since the information processing device 2 controls the operation of the device 1 using the operation information, the device 1 performs an operation according to the operation of the real avatar user. In addition, since the information processing device 2 causes the expression device 6 to express information indicating the state of the real environment 10, the real avatar user can, for example, express plants, animals, etc. in the forest that is the real environment 10 using images, sounds, etc. It can be recognized by Therefore, when the real avatar user performs an action of touching a plant in the real environment 10, the device 1, which is the real avatar, can touch the plant in the real environment 10. The haptic sensation caused by touching the plant is detected by the sensor 14 in the device 1, and transmitted via the information processing device 2 to the expression device 6, such as a haptic glove. Thereby, the expression device 6 can transmit the haptic sensation to the user.

In addition, by using a sensor that detects air conditions such as temperature, wind speed, air volume, humidity, odor, and gas type as the sensor 3, the real avatar user can detect not only images and sounds but also the real environment 10. You can also experience the state of the air. In this way, in this embodiment, since a user can participate using the device 1 existing in the real environment 10 as a real avatar, the real environment 10 can be more appropriately used for remote participants in an event using the real environment 10. It is possible to experience the environment and realize interaction with the real environment. By merging the real environment 10 and virtual objects, the real avatar user can experience the real environment 10 with a sense of presence.

Note that the air condition may be expressed as similar to the detected information by using a scent reproduction device, an air conditioner, a blower, etc. as the expression device 6 for expressing the air condition. The information may be converted into at least one of visual information and auditory information. Similarly, haptic detection information may be converted into at least one of visual information and auditory information.

Regarding device 1, which is a real avatar, device 1 is directly transmitted as a video to real avatar users and users who participate using virtual avatars (hereinafter also referred to as virtual avatar users), and real participants directly view device 1. Although it may be visually observed, the information processing device 2 may generate a video simulating a real avatar user and replace the part of the device 1 with the video.

The virtual avatar does not actually exist in the real environment 10, but by specifying the position of the virtual avatar in the real environment 10, information simulating the presence of the virtual avatar user at that position is provided to the user. provided. For example, by using a 360-degree free viewpoint camera, a microphone for realizing 360-degree stereophonic sound, etc. as the sensor 3, images and sounds can be provided to the virtual avatar user according to the virtual avatar's position and face orientation. can do. The position, face direction, etc. of the virtual avatar may be determined by the participant motion detection device 5 detecting the motion of the virtual avatar user and the information processing device 2 based on the motion information, similarly to the real avatar user. , may be specified by the virtual avatar user using a terminal (not shown). In addition, when using a sensor that detects air conditions such as temperature, wind speed, air volume, humidity, odor, and gas type as the sensor 3, the virtual avatar user can also use the real environment 10 as well as the real avatar user. By transmitting the air condition in the real environment 10, the virtual avatar user can also experience the air condition in the real environment 10.

Furthermore, although the virtual avatar cannot interact with real objects such as touching or moving objects that exist in the real environment 10, the information processing device 2 displays images of the real environment 10 around the virtual avatar. By generating a virtual space based on the virtual space, the virtual avatar may be made to perform actions such as touching objects or moving objects in the virtual space. At this time, what kind of object the object is from the video of the real environment 10 is determined, and the result of the determination is used to generate information indicating the tactile sensation obtained by touching the object, and the information is transmitted to the expression device. 6, the sensation of touching the object may be transmitted to the virtual avatar user. The initial values of the position and orientation of the virtual avatar in the real environment 10 may be specified by the virtual avatar user or may be determined in advance.

In addition, the information processing device 2 generates a video that simulates the virtual avatar user or a three-dimensional video of an arbitrary shape based on the video shot of the virtual avatar user as the virtual avatar, and transfers the video to the real environment 10. The information may be superimposed on the video and transmitted as expression information to the expression device 6 and the expression device 4. The virtual avatar may also be provided as a video to the real avatar user and real participants, or the virtual avatar user may be able to select whether or not to display the virtual avatar. Similarly, for the real avatar, an image simulating the real avatar user or a three-dimensional image of an arbitrary shape may be generated and displayed in the location corresponding to the device 1, and whether or not to display the image may be determined. A real avatar user may be selectable.

Furthermore, the above-described virtual space generated for the virtual avatar user may be reflected in the expression information sent to the real avatar user so that the real avatar user can share information with the virtual avatar user. In addition, by allowing the representation device 4 to represent the virtual space so that the real participants participating in the real environment 10 can share information with the virtual avatar users, the virtual space is superimposed on the real environment 10 that the real participants directly see. You may also choose to do so.

Next, the operation of the information processing device 2 of this embodiment will be explained. FIG. 3 is a flowchart showing an example of a control procedure for the device 1 in the information processing device 2 of this embodiment. The information processing device 2 acquires operation information (step S1). Specifically, the motion information receiving unit 23 acquires motion information by receiving motion information from the participant motion detection device 5 that detects the motion of the real avatar user, and sends the acquired motion information to the device control unit 22. Output.

Next, the information processing device 2 generates control information for reflecting the motion information on the movement of the real avatar (step S2). Specifically, the information processing device 2 uses the motion information to generate control information for causing the device 1, which is the real avatar of the real avatar user corresponding to the motion information, to perform the motion corresponding to the motion information. , outputs the generated control information to the control information transmitter 21.

Next, the information processing device 2 transmits control information (step S3). Specifically, the control information transmitter 21 transmits control information to the device 1. By performing the above-described processing, for example, every control cycle, the movements of the real avatar user are reflected in the device 1, which is the real avatar corresponding to the real avatar user.

FIG. 4 is a flowchart illustrating an example of a procedure for generating virtual information regarding a virtual avatar in the information processing device 2 of this embodiment. The information processing device 2 performs step S1 similarly to the example shown in FIG. However, the motion information is motion information regarding the virtual avatar user, and the motion information receiving section 23 outputs the motion information to the expression information generating section 25.

Next, the information processing device 2 generates virtual information corresponding to the virtual avatar that reflects the motion information (step S4). Specifically, the expression information generation unit 25 uses the motion information to generate virtual information that is expression information such as images and sounds corresponding to a virtual space including the virtual avatar of the virtual avatar user corresponding to the motion information. . By performing the above-described processing, for example, every control cycle, virtual information for expressing a virtual space in which the movements of the virtual avatar user are reflected is generated. Note that information regarding the generation of virtual avatars and the virtual space may be generated using techniques such as spatial reconstruction methods, MR (Mixed Reality), and AR (Augmented Reality). However, the generation of virtual avatars and the generation of information regarding virtual spaces are not limited to these examples.

FIG. 5 is a flowchart illustrating an example of an expression information generation processing procedure in the information processing device 2 of this embodiment. The information processing device 2 acquires information detected by the sensor (step S11). Specifically, the detection information receiving unit 24 acquires detection information by receiving detection information from at least one of the sensor 14 and the sensor 3, and outputs the acquired detection information to the expression information generation unit 25.

Next, the information processing device 2 generates first combination information by combining virtual information with the detection information to be combined among the detection information (step S12). Specifically, the expression information generation unit 25 generates the first combination information by combining the virtual information generated in step S4 shown in FIG. 4 with the detection information to be combined among the detection information. The detection information to be synthesized is the detection information to be synthesized with the virtual information. For example, if the virtual information is a video, the detection information to be synthesized is a video, and if the virtual information is video and sound, the detection information to be synthesized is the detection information to be synthesized with the virtual information. The detected information is images and sounds. Note that here, an example will be described in which the video and sound are acquired by the sensor 14 installed in the device 1, but in the case where the video and sound are acquired by the sensor 3, which is a 360-degree free viewpoint camera, etc. The expression information generation unit 25 uses the detection information to generate information indicating images and sounds corresponding to the virtual avatar or the real avatar, and synthesizes the generated information with virtual information. When there are multiple remote participants, the virtual information is combined with the corresponding detection information or information generated using the detection information of the sensor 3 for each remote participant. Note that the virtual information does not need to be combined.

Next, the information processing device 2 converts the detected information to be converted out of the detected information into another type of information, and generates second combined information by combining the converted information with the first combined information (step S13). Specifically, the expression information generation unit 25 converts the detection information to be converted out of the detection information into another type of information, and generates the second composite information by combining the converted information with the first composite information. , and outputs the second composite information to the expression information transmitter 26 as expression information. For example, when converting air condition detection information such as temperature and humidity into visual information, the air condition detection information is the detection information to be converted, and other types of information are characters, images, etc. This conversion can be done by, for example, converting the content of another type of information that allows a person to feel a similar state in advance, using supervised machine learning using clinical trial results, depending on the value of the detected information to be converted. By learning and inputting the detection information to be converted into the trained model, the conversion result can be obtained. The conversion method is not limited to this example. Examples of expressing the state of the air using visual and auditory information include, for example, generating audio information that exaggerates the sounds of moving birds and insects, displaying shaky images like a mirage to indicate hot and humid weather, and expressing cold. For example, the edges may be displayed to show a clear feeling, but the present invention is not limited to these examples. Similarly, information regarding taste may be converted into visual information, auditory information, or the like. Note that if there is no detection information to be converted, step S13 is not executed, and the first composite information is outputted to the expression information transmitter 26 as expression information. Note that if there are multiple remote participants, steps S12 and S13 are performed for each remote participant.

Next, the information processing device 2 transmits the second synthesis information and the detection information not to be synthesized and not to be converted to the expression device 6 of the remote participant (step S14). Specifically, the expression information generation unit 25 converts the second information and the detection information that is neither a synthesis target nor a conversion target into the expression device 6 of the remote participant, that is, the expression device 6 of the virtual avatar user and the real avatar user. Send to.

Next, the information processing device 2 transmits the virtual information to the expression device 4 of the real environment 10 (step S15). Specifically, the expression information generation unit 25 transmits the virtual information to the expression device 4 of the real environment 10, that is, the expression device 4 that transmits information to the real participants. Note that the information converted in step S13 and the virtual information may be combined and transmitted to the expression device 4 of the real participant. Since the real participants can directly feel the state of the air in the real environment 10, the information converted in step S13 does not need to be sent to the expression device 4, but the converted information may also be transmitted to the real participants. By being transmitted, the real participants can share the information with the virtual avatar user and the real avatar user. By performing the above-described processing, for example, every control cycle, information indicating the state of the real environment 10 is transmitted to remote participants, and virtual information is transmitted to real participants.

Note that if the expression device 6 corresponding to the real avatar user (first user) is the first expression device and the expression device 6 corresponding to the virtual avatar user (second user) is the second expression device, then the first expression device is , it is sufficient that the first user is capable of expressing an image that can be recognized with at least one of the five senses, and the second expression device is sufficient as long as it is capable of presenting an image to at least the second user. Further, the sensor 3 includes a photographing device that detects information regarding objects in the real environment 10 by photographing at least the real environment 10 in order to generate an image to be provided to the virtual avatar user. Then, the expression information generation unit 25 generates first expression information to be transmitted to the first user (real avatar user) using the first detection information that is the detection information received from the sensor 14 , and uses the first detection information that is the detection information received from the sensor 3 2 detection information is used to generate second expression information including an image corresponding to the virtual viewpoint in the real environment 10 of the second user (virtual avatar user), and the expression information transmitter 26 transmits the first expression information to the first expression device. The expression information is transmitted, and the second expression information is transmitted to the second expression device.

Further, as described above, the expression information generation unit 25 recognizes the first detection information indicating the detection result of at least one of temperature, humidity, wind, and odor in the real environment 10 by at least one of the visual and auditory senses. The information may be converted into first expression information that can be expressed. Furthermore, the expression information generation unit 25 is capable of recognizing second detection information indicating a detection result of at least one of temperature, humidity, wind, and odor in the real environment 10 with at least one of visual and auditory information. It may also be converted into second expression information. Further, the sensor may include information related to taste, and the expression information generation unit 25 may generate information related to taste in the real environment 10. In this case, the expression device 4 and the expression device 6 include a taste expression device that reproduces taste. Alternatively, information regarding taste may be converted into other types of methods such as visual or auditory information.

Furthermore, the expression information generation unit 25 may generate the first expression information using the second detection information. That is, the information acquired by the sensor 3 may be used to generate expression information to be transmitted to the real avatar user.

Further, the expression information generation unit 25 may generate the second expression information so that the virtual avatar is displayed at the virtual viewpoint, and the participant movement detection device 5 detects the movement of the virtual avatar user and generates the expression information. The unit 25 detects objects in the real environment 10 and the virtual avatar in the virtual space, which is a range including the virtual viewpoint, based on the motion information of the virtual avatar user and the second detection information (video data of the real environment 10). virtually generates a video that changes in accordance with the movements of the virtual avatar user, synthesizes virtual information indicating the generated video with second expression information, and transmits the second expression information into which the virtual information is synthesized. The expression information may be sent to a second expression device. Further, the expression information generation section 25 may combine the virtual information with the first expression information, and the expression information transmission section 26 may transmit the first expression information with the combined virtual information to the first expression device. Note that the virtual avatar may have only a virtual viewpoint, or may be able to hear and speak in addition to the virtual viewpoint.

Furthermore, the expression information transmitting unit 26 may transmit the virtual information to the expression device 4, which is a third expression device that can present images to real participants (third users). Further, the information processing device 2 may be able to set whether information regarding the virtual avatar user and the real avatar user is made public or private. Information regarding virtual avatar users and real avatar users includes, for example, facial information, eyeball information, voice prints, and other frequently used security information, as well as personal information such as real names, ages, and dates of birth. but not limited to. Furthermore, if the user does not wish to make his or her actual face or voice public for privacy reasons, the user may be able to change the face or voice and make it public.

Next, the participation form of participants will be explained using an example. FIG. 6 is a diagram showing an example in which real participants, real avatar users, and virtual avatar users participate in an event. In the example shown in FIG. 6, real participants 8-1 to 8-3 participate in the event in the real environment 10, and from the participation location 7-1, the real avatar user 70-1 is the real avatar user. Users 70-2 and 70-3, who are virtual avatar users, participate in the event using device 1 and participate in the event from participating locations 7-2 and 7-3 using virtual avatars 9-1 and 9-2, respectively. Participating in Although not shown, each of the participating locations 7-1 to 7-3 is provided with a participant motion detection device 5 and an expression device 6. Note that when the virtual avatars 9-1 and 9-2 are fixed without moving, the participant motion detection device 5 does not need to be provided to the users 70-2 and 70-3 who are virtual avatar users.

The device 1 includes a sensor 14 that detects information regarding each of visual, auditory, olfactory, and tactile sensations, and includes a physical acting device, a moving device, and a speaker as actuators. In addition, the sensors 3-1 and 3-2 installed in the real environment 10 are cameras that acquire images for generating images from the viewpoints of the virtual avatars 9-1 and 9-2, and This is a microphone that collects sound to generate sound at position -2. In addition, in FIG. 6, two sensors 3 are shown for the sake of simplification, but in general, a camera that acquires images for generating images from the viewpoints of the virtual avatars 9-1 and 9-2. , a plurality of microphones each collect sound to generate sound at the positions of the virtual avatars 9-1 and 9-2. Furthermore, as at least one of the sensor 3 and the sensor 14, a sensor that detects odor, temperature, wind speed, air volume, humidity, type of gas, etc. of the actual environment 10 may be provided.

In the example shown in FIG. 6, all information to be transmitted to the user 70-1, who is a real avatar user, is detected by the sensor 14 installed in the device 1. As described above, when the device 1 operates in accordance with the movement of the real avatar user, the field of view of the sensor 14, such as a camera that obtains visual information, in the device 1 is changed, and the device 1 operates in accordance with the movement of the real avatar user. The image in the real environment 10 is displayed by the expression device 6. The same applies to sounds, smells, etc., and as the device 1 moves, information corresponding to sounds and smells in the real environment 10 is transmitted to the real avatar user. Further, when the device 1 touches an actual object in the real environment 10 in accordance with the movement of the real avatar user, information regarding haptic sensation is detected by the sensor 14 and transmitted to the real avatar user. This allows the real avatar user to recognize the real environment 10 with a sense of realism.

Furthermore, users 70-2 and 70-3, who are virtual avatar users, are provided with images from the viewpoints of virtual avatars 9-1 and 9-2, and images from virtual avatars 9-1 and 9-1, respectively, using the detection information acquired by the sensor 3. , 9-2 are transmitted. Further, when a sensor that detects a smell is provided as the sensor 3, the information processing device 2 uses the detection information by the sensor 3 to provide the users 70-2 and 70-3, who are virtual avatar users, with the expression device. 6 can convey information indicating odor.

In the case where the real participants 8-1 to 8-3 visually recognize the device 1 as it is and do not need to visually recognize the virtual avatars 9-1 and 9-2, the expression device 4 is not provided in the real environment 10. You don't have to. As described above, when projecting the image corresponding to the real avatar at the location where the device 1 is present, and projecting the images corresponding to the locations corresponding to the virtual avatars 9-1 and 9-2, these are projected. A representation device 4 is provided. Similarly, when the real participants 8-1 to 8-3 are allowed to recognize the sounds corresponding to the virtual avatars 9-1 and 9-2, the expression device 4 allows the user 70-2 who is the virtual avatar user to recognize the sound. , 70-3 are transmitted to the actual participants by the expression device 4.

FIG. 7 is a diagram showing an example in which a real avatar user and a virtual avatar user participate in an event. In the example shown in FIG. 7, from participating locations 7-1, 7-4, and 7-5, users 70-1, 70-4, and 70-5, who are real avatar users, are connected to devices whose real avatars correspond to their respective locations. Users 70-2 and 70-3, who are virtual avatar users, participate in the event using virtual avatars 9-1 and 9-3 from participation locations 7-2 and 7-3, respectively, using virtual avatars 9-1 and 9-3. I am participating in an event using 2. Although not shown, each of the participating locations 7-1 to 7-5 is provided with a participant motion detection device 5 and an expression device 6. Each of the devices 1-1 to 1-3 is the device 1 described above.

The configuration of each of devices 1-1 to 1-3 is similar to device 1 shown in FIG. 6. The method of transmitting information to the real avatar user and the virtual avatar user in the example shown in FIG. 7 is also the same as in the example shown in FIG. 6. In the example shown in FIG. 7, since there is no real participant 8, there is no need to provide the expression device 4 in the real environment 10.

FIG. 8 is a diagram showing an example of generating a virtual space corresponding to a virtual avatar. In the example shown in FIG. 8, similar to the example shown in FIG. A user 70-1 participates in an event using device 1, which is a real avatar, and users 70-2 and 70-3, who are virtual avatar users, use virtual avatar 9 from participating locations 7-2 and 7-3, respectively. I am participating in the event using -1 and 9-2. Although not shown, each of the participating locations 7-1 to 7-3 is provided with a participant motion detection device 5 and an expression device 6.

In the example shown in FIG. 8, the information processing device 2 acquires motion information from the participant motion detection device 5 corresponding to the user 70-3, who is a virtual avatar user, and uses the acquired information to A virtual space 90 is generated around the . The information processing device 2 then synthesizes virtual information indicating the virtual space 90 with images, sounds, etc. from the viewpoint of the virtual avatar 9-2 generated using the detection information of the sensor 3, and uses the synthesized information as expression information. , and is transmitted to the expression device 6 corresponding to the user 70-3, who is the virtual avatar user. The combined information is also sent to the user 70-1 who is the real avatar user and the user 70-2 who is the virtual avatar user.

FIG. 9 is a diagram showing an example of a video in which virtual spaces are synthesized. In the example shown in FIG. 9, a video 201 of the virtual space corresponding to the virtual avatar 9-2 is synthesized with a video 200 of the real environment 10 from the viewpoint of the user 70-2, who is a virtual avatar user. It shows. The image 200 is a real image based on information captured by the sensor 3, whereas the image 201 is information indicating a virtual space generated based on the information captured by the sensor 3. The video 201 reflects the movement of the virtual avatar 9-2 in the virtual space. Therefore, when the virtual avatar 9-2 touches an object in the virtual space, the object also changes in the virtual space. Similarly, the user 70-3 who is the virtual avatar user corresponding to the virtual avatar 9-2 is also transmitted the video in which the virtual space has been synthesized, so that the user 70-3 who is the virtual avatar user can You can experience the same sensation as when you touch an object.

In addition, by transmitting the virtual information to the expression devices 4 corresponding to the real participants 8-1 to 8-3, the real participants 8-1 to 8-3 can express the images projected onto the actual real environment 10. It is possible to visually recognize the state in which the virtual spaces are superimposed.

FIG. 10 is a diagram showing the reduction of sensors in equipment. In the example shown in FIG. 10, similar to the example shown in FIG. A user 70-1 participates in an event using device 1, which is a real avatar, and users 70-2 and 70-3, who are virtual avatar users, use virtual avatar 9 from participating locations 7-2 and 7-3, respectively. I am participating in the event using -1 and 9-2. Although not shown, each of the participating locations 7-1 to 7-3 is provided with a participant motion detection device 5 and an expression device 6.

In the example shown in FIG. 10, the sensors 3-1 to 3-4 include a camera for acquiring a 360-degree free viewpoint image and a microphone for reproducing sound according to the position. Sensors 3-1 to 3-4 detect images of the real environment 10 from the viewpoints of users 70-2 and 70-3 who are virtual avatar users, and sounds at the positions of users 70-2 and 70-3 who are virtual avatar users. It is used to generate information indicating the image of the real environment 10 from the viewpoint of the real avatar and information indicating the sound at the position of the real avatar. Thereby, the sensor 14 included in the device 1 does not need to include a sensor that detects information related to vision and hearing. As shown in FIG. 10, by generating information regarding the real avatar using the sensor 3, the configuration of the device 1 can be made simpler than the example shown in FIG. In the example shown in FIG. 10, regarding the real avatar, visual and auditory information is generated from the detection information of the sensor 3, but the present invention is not limited to this. The first information may be generated from the information, and the other may be generated from the detection information of the sensor 14. Further, by using the sensor 3 that detects odor, the number of sensors that detect information related to smell among the sensors 14 included in the device 1 may be reduced.

Next, the hardware configuration of the information processing device 2 of this embodiment will be explained. The information processing device 2 of this embodiment is configured so that the computer system can be used as the information processing device 2 by executing a program, which is a computer program in which processing in the information processing device 2 is described, on the computer system. Function. FIG. 11 is a diagram showing an example of the configuration of a computer system that implements the information processing device 2 of this embodiment. As shown in FIG. 11, this computer system includes a control section 101, an input section 102, a storage section 103, a display section 104, a communication section 105, and an output section 106, which are connected via a system bus 107. ing. The control section 101 and the storage section 103 constitute a processing circuit.

In FIG. 11, the control unit 101 is, for example, a processor such as a CPU (Central Processing Unit), and executes a program in which processing in the information processing device 2 of this embodiment is described. Note that a part of the control unit 101 may be realized by dedicated hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). The input unit 102 includes, for example, a keyboard and a mouse, and is used by a user of the computer system to input various information. The storage unit 103 includes various memories such as RAM (Random Access Memory) and ROM (Read Only Memory), and storage devices such as a hard disk, and stores programs to be executed by the control unit 101 and in the process of processing. Store the necessary data obtained. The storage unit 103 is also used as a temporary storage area for programs. The display unit 104 is composed of a display, an LCD (liquid crystal display panel), etc., and displays various screens to the user of the computer system. The communication unit 105 is a receiver and a transmitter that perform communication processing. The output unit 106 is a printer, a speaker, or the like. Note that FIG. 11 is an example, and the configuration of the computer system is not limited to the example shown in FIG.

Here, an example of the operation of the computer system until the program of this embodiment becomes executable will be described. In a computer system having the above-mentioned configuration, for example, a computer program can be loaded from a CD-ROM or DVD-ROM set in a CD (Compact Disc)-ROM drive or a DVD (Digital Versatile Disc)-ROM drive (not shown). is installed in the storage unit 103. Then, when the program is executed, the program read from the storage unit 103 is stored in the main storage area of the storage unit 103. In this state, the control unit 101 executes processing as the information processing apparatus 2 of this embodiment according to the program stored in the storage unit 103.

Note that in the above description, a CD-ROM or DVD-ROM is used as a recording medium to provide a program that describes processing in the information processing device 2; however, the present invention is not limited to this; Depending on the capacity of the program, for example, a program provided via a transmission medium such as the Internet via the communication unit 105 may be used.

The program of this embodiment includes, for example, the step of receiving motion information indicating the motion of a first user who participates in an event at a location remote from the real environment 10 where the event is held, and using the motion information to A step of generating control information for causing a device installed in the device to perform an operation according to the operation information, a step of transmitting the control information to the device, and a step of detecting information regarding an object mounted on the device in the real environment 10. receiving first detection information detected by the device-mounted sensor from the device-mounted sensor; and second detection information detected by the real-environment sensor from the real-environment sensor installed in the real environment 10 and detecting information about an object in the real environment 10. and causing the computer system to perform the step of receiving detection information. The program of this embodiment further includes a step of generating first expression information to be transmitted to the first user using the first detection information, and a step of generating first expression information to be transmitted to the first user using the second detection information. a step of generating second expression information including an image corresponding to a virtual viewpoint in the real environment 10 of a second user participating in the event; The computer system is caused to execute the steps of transmitting the first expression information to the expression device and transmitting the second expression information to the second expression device capable of presenting the video to the second user.

The device control section 22 and the expression information generation section 25 shown in FIG. 1 are realized by the computer program stored in the storage section 103 shown in FIG. 11 being executed by the control section 101 shown in FIG. Ru. The storage unit 103 shown in FIG. 11 is also used to realize the device control unit 22 and the expression information generation unit 25 shown in FIG. The control information transmitting section 21, motion information receiving section 23, detection information receiving section 24, and expression information transmitting section 26 shown in FIG. 1 are realized by the communication section 105 shown in FIG. Further, the information processing device 2 may be realized by a plurality of computer systems. For example, the information processing device 2 may be realized by a cloud computer system. Furthermore, some of the functions of the information processing device 2 may be realized by another device provided separately from the information processing device 2. Another device may be provided in the real environment 10, may be provided near the residence of the user participating in the event, or may be provided at another location.

As described above, in this embodiment, by using the physical device 1 as a real avatar in an event using the real environment 10, a user can remotely participate in the event. This allows remote participants in an event using the real environment 10 to experience the real environment 10 more appropriately. Further, by using both participation by a real avatar using the device 1 and participation by a virtual avatar, even when the number of participants in an event increases, it is possible to respond flexibly while keeping costs down. Further, by transmitting information indicating the air condition of the real environment 10 to the remote participants, it is possible to make the remote participants experience a state closer to the real environment 10 . Furthermore, if information corresponding to the real avatar is generated using the detection information of the sensor 3 used to provide information corresponding to the virtual avatar to the virtual avatar user, the configuration of the device 1 can be simplified and the cost will be reduced. can be reduced.

The configurations shown in the embodiments above are merely examples, and can be combined with other known techniques, or can be combined with other embodiments, within the scope of the gist. It is also possible to omit or change part of the configuration.

1, 1-1 to 1-3 equipment, 2 information processing device, 3-1 to 3-4, 14-1, 14-2 sensor, 4-1, 4-2, 6-1, 6-2 expression device , 5-1, 5-2 Participant motion detection device, 7, 7-1 to 7-5 Participation location, 8-1 to 8-4 Real participants, 9, 9-1, 9-2 Virtual avatar, 10 Real environment, 11 Control information receiving section, 12 Drive control section, 13 Driving section, 15 Detection information transmitting section, 21 Control information transmitting section, 22 Device control section, 23 Operation information receiving section, 24 Detection information receiving section, 25 Expression information Generation unit, 26 Expression information transmission unit, 100 Remote experience system.

Claims

A device installed in a real environment where an event is held, and equipped with a device-mounted sensor that detects information about objects in the real environment;
a real environment sensor that is installed in the real environment and detects information about objects in the real environment;
a participant motion detection device that detects the motion of a first user who participates in the event from a location away from the actual environment;
a first expression device capable of expressing expressions that the first user can recognize with at least one of the five senses;
a second expression device capable of presenting an image to a second user who participates in the event from a location remote from the actual environment;
an information processing device;
Equipped with
The real environment sensor includes a photographing device that detects information about objects in the real environment by photographing the real environment,
The information processing device includes:
a motion information receiving unit that receives, from the participant motion detection device, motion information indicating the motion acquired by the participant motion detection device;
a device control unit that uses the operation information to generate control information for causing the device to perform an operation according to the operation information;
a control information transmitter that transmits the control information to the device;
a detection information receiving unit that receives first detection information detected by the device-mounted sensor from the device-mounted sensor, and receives second detection information detected by the real-environment sensor from the real-environment sensor;
The first detection information is used to generate first expression information to be transmitted to the first user, and the second detection information is used to generate a first representation information that includes an image corresponding to a virtual viewpoint of the second user in the real environment. 2. An expression information generation unit that generates expression information;
an expression information transmitter that transmits the first expression information to the first expression device and transmits the second expression information to the second expression device;
A remote experience system characterized by comprising:
The remote experience system according to claim 1, wherein the object includes at least one of animals, plants, and air existing in the real environment.
The remote experience system according to claim 2, wherein the device-mounted sensor detects at least one of temperature, humidity, wind, and odor in the real environment.
The expression information generation unit is configured to detect the first detection information indicating a detection result of at least one of temperature, humidity, wind, and odor in the actual environment, and the expression information generating unit may be configured to detect the first detection information indicating the detection result of at least one of temperature, humidity, wind, and odor in the actual environment. 4. The remote experience system according to claim 3, wherein the remote experience system converts into first expression information.
The remote experience system according to claim 2 or 3, wherein the real environment sensor detects at least one of temperature, humidity, wind, and odor in the real environment.
The expression information generation unit is configured to detect the second detection information indicating a detection result of at least one of temperature, humidity, wind, and odor in the actual environment, and the expression information generation unit may be configured to detect the second detection information indicating the detection result of at least one of temperature, humidity, wind, and odor in the actual environment. 4. The remote experience system according to claim 3, wherein the remote experience system converts into second expression information.
The remote experience system according to any one of claims 1 to 5, wherein the expression information generation unit generates the first expression information using the second detection information.
The device includes a manipulator;
The device-mounted sensor includes a force tactile sensor that detects force tactile sensation of the hand of the manipulator,
The remote experience system according to any one of claims 1 to 3, wherein the first expression device includes a haptic device that transmits a haptic sensation to the first user's hand.
4. The expression information generation unit generates the second expression information so that a virtual avatar corresponding to the second user is displayed at the virtual viewpoint. The remote experience system described in 1.
a participant motion detection device that detects the second user's motion;
Equipped with
The motion information receiving unit receives motion information indicating the second user's motion acquired by the participant motion detecting device from the participant motion detecting device that detects the second user's motion;
The expression information generation unit is configured to determine, based on the motion information of the second user and the second detection information, the object in the real environment and the virtual avatar in a virtual space that is a range including the virtual viewpoint. virtually generating a video that changes in accordance with the actions of the second user, and combining virtual information indicating the generated video with the second expression information;
The remote experience system according to claim 9, wherein the expression information transmitting unit transmits the second expression information in which the virtual information is combined to the second expression device.
The expression information generation unit combines the virtual information with the first expression information,
11. The remote experience system according to claim 10, wherein the expression information transmitting unit transmits the first expression information in which the virtual information is combined to the first expression device.
a third expression device capable of presenting a video to a third user participating in the event in the real environment;
Equipped with
The remote experience system according to claim 10 or 11, wherein the expression information transmitter transmits the virtual information to the third expression device.
13. The information processing apparatus according to claim 1, wherein the information processing apparatus is capable of setting whether to make information about the first user and the second user public or private. Remote experience system.
A communication line between the information processing device and the participant motion detection device, a communication line between the information processing device and the equipment-mounted sensor, a communication line between the information processing device and the real environment sensor, A communication line between the information processing device and the device, a communication line between the information processing device and the first expression device, and a communication line between the information processing device and the second expression device. The remote experience system according to any one of claims 1 to 12, wherein at least one of them is a Beyond 5G line.
a behavior information receiving unit that receives behavior information indicating the behavior of a first user who participates in the event at a location away from the actual environment where the event is held;
a device control unit that uses the operation information to generate control information for causing a device installed in the actual environment to perform an operation according to the operation information;
a control information transmitter that transmits the control information to the device;
Receive first detection information detected by the device-mounted sensor from a device-mounted sensor that is mounted on the device and detects information about an object in the real environment, and detect information about the object installed in the real environment in the real environment. a detection information receiving unit that receives second detection information detected by the real environment sensor from the real environment sensor;
The first detection information is used to generate first expression information to be transmitted to the first user, and the second detection information is used to generate a second user who participates in the event from a location away from the actual environment. an expression information generation unit that generates second expression information including an image corresponding to a virtual viewpoint in the real environment;
The first user transmits the first expression information to a first expression device capable of making an expression that can be recognized with at least one of the five senses, and sends the first expression information to a second expression device capable of presenting an image to the second user. 2. An expression information transmitter that transmits expression information;
An information processing device comprising:
An information processing method in an information processing device, the method comprising:
receiving behavior information indicating the behavior of a first user who participates in the event at a location remote from the actual environment where the event is held;
Using the operation information, generate control information for causing a device installed in the actual environment to perform an operation according to the operation information,
transmitting the control information to the device;
receiving first detection information detected by the device-mounted sensor from a device-mounted sensor that is mounted on the device and detects information regarding an object in a real environment;
receiving second detection information detected by the real environment sensor from a real environment sensor installed in the real environment and detecting information about objects in the real environment;
generating first expression information to be transmitted to the first user using the first detection information;
using the second detection information to generate second expression information including an image corresponding to a virtual viewpoint in the real environment of a second user who participates in the event from a location away from the real environment;
The first user transmits the first expression information to a first expression device capable of making an expression that can be recognized with at least one of the five senses, and sends the first expression information to a second expression device capable of presenting an image to the second user. An information processing method characterized by transmitting 2-expression information.
to the computer system,
receiving behavior information indicating the behavior of a first user participating in the event at a location remote from the actual environment in which the event is held;
using the operation information to generate control information for causing a device installed in the actual environment to perform an operation according to the operation information;
transmitting the control information to the device;
receiving first detection information detected by the device-mounted sensor from a device-mounted sensor that is mounted on the device and detects information about an object in a real environment;
receiving second detection information detected by the real environment sensor from a real environment sensor installed in the real environment and detecting information about objects in the real environment;
generating first expression information to be communicated to the first user using the first detection information;
using the second detection information to generate second expression information including an image corresponding to a virtual viewpoint in the real environment of a second user who participates in the event from a location away from the real environment;
The first user transmits the first expression information to a first expression device capable of making an expression that can be recognized with at least one of the five senses, and sends the first expression information to a second expression device capable of presenting an image to the second user. 2. A step of transmitting expression information;
A program characterized by executing.