WO2023276215A1 - 情報処理装置、情報処理方法及びプログラム - Google Patents

情報処理装置、情報処理方法及びプログラム Download PDF

Info

Publication number
WO2023276215A1
WO2023276215A1 PCT/JP2022/003201 JP2022003201W WO2023276215A1 WO 2023276215 A1 WO2023276215 A1 WO 2023276215A1 JP 2022003201 W JP2022003201 W JP 2022003201W WO 2023276215 A1 WO2023276215 A1 WO 2023276215A1
Authority
WO
WIPO (PCT)
Prior art keywords
avatar
terminal device
user
information
information processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/003201
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
大輔 田島
大夢 弓場
美和 市川
智裕 石井
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Priority to JP2023531357A priority Critical patent/JPWO2023276215A1/ja
Priority to US18/572,293 priority patent/US20240296642A1/en
Publication of WO2023276215A1 publication Critical patent/WO2023276215A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/20—Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/04815—Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—Three-dimensional [3D] animation
    • G06T13/40—Three-dimensional [3D] animation of characters, e.g. humans, animals or virtual beings
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B9/00—Simulators for teaching or training purposes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/20—Indexing scheme for editing of 3D models
    • G06T2219/2004—Aligning objects, relative positioning of parts
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/20—Indexing scheme for editing of 3D models
    • G06T2219/2016—Rotation, translation, scaling

Definitions

  • the present invention relates to an information processing device, an information processing method, and a program.
  • An information processing system disclosed in Patent Document 1 is known as an information processing system that allows a user to simultaneously visually recognize a real object and a virtual object that exist in a real space, using a motion of a user's hand as a model for a virtual object. ing.
  • a virtual object is displayed by overlaying it on a real space, for example, on a transmissive head-mounted display.
  • an instructor can assist a worker in a remote location with a virtual object.
  • a model virtual object is displayed from a first-person viewpoint, and since the position of the viewpoint cannot be moved with respect to the virtual object, it may be difficult to grasp the movement of the virtual object. . If you try to explain movements that are difficult to understand through conversation, the instructor and the worker are not in the same place, so specific and detailed explanations are required, which reduces the efficiency of support.
  • the present disclosure proposes an information processing device, an information processing method, and a program capable of suppressing a decrease in support efficiency for workers.
  • first posture information representing positions and postures of one or more first users wearing first terminal devices and positions of one or more second users wearing second terminal devices and a second posture information representing a posture
  • a storage unit storing motion data representing a change in the posture of the user
  • one or more first avatars of the first user based on the first posture information. is arranged in the virtual space, one or more second avatars of the second user are arranged in the virtual space based on the second posture information, and a reproduction avatar capable of changing the position and orientation and reproducing the action data is provided.
  • a processing unit arranged in the virtual space, causing the first terminal device to display the second avatar and the playback avatar according to positions in the virtual space, and displaying the first avatar and the playback avatar in the virtual space.
  • an information processing method in which the information processing of the information processing device is executed by a computer, and a program for causing the computer to implement the information processing of the information processing device.
  • FIG. 1 is a diagram showing devices that constitute an information processing system according to an embodiment.
  • FIG. 2 is a diagram showing an example of a farm field and an avatar visually recognized by a worker.
  • FIG. 3 is a diagram showing an example of a 3D map and an avatar visually recognized by an instructor.
  • FIG. 4 is a diagram for explaining a tap operation that triggers the start and end of recording of the movement of the instructor.
  • FIG. 5 is a diagram for explaining an operation for changing the position and orientation of the playback avatar.
  • FIG. 6 is a block diagram showing the functional configuration of the information processing device 10. As shown in FIG. FIG. 7 is a block diagram showing the hardware configuration and functional configuration of the first terminal device.
  • FIG. 8 is a block diagram showing the hardware configuration and functional configuration of the second terminal device.
  • FIG. 7 is a block diagram showing the hardware configuration and functional configuration of the first terminal device.
  • FIG. 9 is a state transition diagram of the first terminal device when recording user's actions.
  • FIG. 10 is a flow chart showing the flow of processing for detecting a tap action.
  • FIG. 11 is a flow chart showing the flow of processing for recording user's actions.
  • FIG. 12 is a state transition diagram relating to movement of the playback avatar.
  • FIG. 13 is a flow chart showing the flow of processing when the access right to the playback avatar is changed.
  • FIG. 14 is a flow chart showing the flow of processing for moving the playback avatar AV3.
  • FIG. 15 is a hardware configuration diagram of an example of a computer that implements the functions of the information processing apparatus.
  • the movement of the avatar can only be seen from a predetermined direction, which may hinder work support.
  • the present disclosure proposes an information processing device, an information processing method, and a program that enable efficient support to a remote location.
  • a person who works in a field is appropriately called a "worker”, and a person who supports a worker in a remote location or field is appropriately called a "instructor”.
  • the worker visually recognizes information displayed in AR (Augmented Reality) or MR (Mixed Reality) in the field.
  • the instructor visually recognizes the information displayed by AR or MR when providing support in the field, and visually recognizes the field and information displayed by VR (Virtual Reality) when providing support from a remote location. do.
  • FIG. 1 is a diagram showing devices constituting an information processing system 1.
  • the information processing system 1 includes an information processing device 10 , a first terminal device 20 and a second terminal device 30 .
  • the information processing device 10 is connected to the communication line N by wire, but may be connected wirelessly.
  • Various devices can be connected to the information processing device 10 .
  • a first terminal device 20 and a second terminal device 30 are connected to the information processing device 10 via a communication line N, and information is linked between the devices.
  • the first terminal device 20 and the second terminal device 30 are also connected to the communication line N by wire or wirelessly.
  • the wireless connection of the first terminal device 20 and the second terminal device 30 to the communication line N is, for example, a connection via a wireless LAN, but is not limited to the wireless LAN. ) may be used.
  • the first terminal device 20 is, for example, an optical see-through type head-mounted display capable of AR display such as HoloLens (registered trademark) or HoloLens2. Also, the first terminal device 20 may be a terminal device such as a smart phone capable of AR display using ARCore (registered trademark), ARKit (registered trademark), or the like. Also, the first terminal device 20 may be a video see-through type AR device or XR device such as Varjo (registered trademark) XR-1. The first terminal device 20 is worn by an operator or an instructor. For example, the first terminal device 20 receives information about an avatar, which will be described later, from the information processing device 10, and performs AR display of the avatar based on the received information.
  • AR display such as HoloLens (registered trademark) or HoloLens2.
  • the first terminal device 20 may be a terminal device such as a smart phone capable of AR display using ARCore (registered trademark), ARKit (registered trademark), or the
  • the first terminal device 20 is equipped with a sensor, and information representing the position and tilt of the user's head, the direction in which the user is facing, and the position and posture of the user's hand sensed by the sensor is used as first posture information in real time. to the information processing apparatus 10 .
  • the first terminal device 20 senses and records the position and posture of the user's hands and the position and posture of the user's head in accordance with the user's operation. Motion data representing the position and posture of the head is transmitted to the information processing device 10 .
  • the second terminal device 30 is a head-mounted display capable of VR display. Also, the second terminal device 30 may be a terminal device such as a smart phone capable of VR display. The second terminal device 30 is worn by the instructor. The second terminal device 30 receives, for example, map information and avatar-related information, which will be described later, from the information processing device 10, and based on the received information, performs VR display of the farm field and VR display of the avatar. Further, the second terminal device 30 includes a sensor, and information representing the position and tilt of the user's head sensed by the sensor, the direction the user is facing, and the position and posture of the user's hand is used as second posture information. is transmitted to the information processing apparatus 10 in real time.
  • the number of the first terminal device 20 and the second terminal device 30 connected to the communication line N and provided with information from the information processing device 10 is not limited to one each, and may be a plurality of first terminals.
  • AR display and VR display can be performed based on information provided from the information processing device 10, respectively.
  • the information processing device 10 is an information processing device that performs processing for providing information for AR display to the first terminal device 20 and processing for providing information for VR display to the second terminal device 30. be.
  • the information processing device 10 stores, for example, a 3D map of a field generated by sensing the field in advance with LiDAR (Light Detection and Ranging), and sends map information representing the stored 3D map to the second terminal device 30. Send.
  • the information processing device 10 receives and stores motion data transmitted from the first terminal device 20 .
  • the information processing device 10 transmits to the first terminal device 20 and the second terminal device 30 reproduction avatar information representing an avatar that reproduces the stored action data.
  • the information processing device 10 receives first attitude information transmitted in real time from the first terminal device 20 and second attitude information transmitted in real time from the second terminal device 30 .
  • the information processing device 10 transmits first avatar information representing the avatar of the user of the first terminal device 20 to the second terminal device 30 based on the first posture information transmitted from the first terminal device 20 in real time.
  • the information processing device 10 transmits second avatar information representing the avatar of the user of the second terminal device 30 to the first terminal device 20 based on the second posture information transmitted from the second terminal device 30 in real time.
  • the information processing apparatus 10 is implemented by a PC, WS, or the like. Note that the information processing device 10 is not limited to a PC, WS, or the like.
  • the information processing device 10 may be an information processing device such as a PC, WS, or the like that implements the functions of the information processing device 10 as an application.
  • FIG. 2 shows a farm field viewed through the first terminal device 20 by the worker wearing the first terminal device 20, and an AR display on the first terminal device 20 viewed by the worker. It is a figure which shows an example of an avatar.
  • a worker is an example of a first user who wears the first terminal device 20 .
  • various plants are planted in the field where the worker works.
  • the AR display visually recognized by the worker includes a second avatar AV2 that is an avatar of an instructor who is in a remote location and wears the second terminal device 30, and a reproduction avatar AV3 that reproduces motion data. The worker can see the instructor's real-time movements from the displayed second avatar AV2.
  • the AR display visually recognized by the worker wearing the first terminal device 20 includes a virtual hand HL1, which is a virtual object that visualizes the left hand of the worker sensed by the first terminal device 20, and the first terminal device 20. and a virtual hand HR1, which is a virtual object that visualizes the operator's right hand sensed by .
  • the second avatar AV2 is displayed at a position corresponding to the position of the instructor in the virtual space represented by the 3D map of the field.
  • the reproduction avatar AV3 is displayed at a position corresponding to the position specified by the instructor wearing the second terminal device 30 in the virtual space represented by the 3D map of the field.
  • FIG. 3 shows a 3D map of a farm field viewed by an instructor wearing the second terminal device 30 in a VR display on the second terminal device 30, and a 3D map of a farm field visually displayed in a VR display on the second terminal device 30 by an instructor wearing the second terminal device 30.
  • the instructor is an example of a second user wearing the second terminal device 30 .
  • the origin of the virtual space represented by the 3D map displayed on the second terminal device 30 coincides with the origin defined in the field where the worker is located, and the instructor shares the space where the worker is located by the 3D map. be able to.
  • the VR display visually recognized by the instructor includes a first avatar AV1 that is an avatar of a worker in the field and wearing the first terminal device 20, and a reproduction avatar AV3 that reproduces motion data.
  • the instructor can see the real-time movement of the worker through the displayed first avatar AV1.
  • the VR display viewed by the instructor wearing the second terminal device 30 includes a virtual hand HL2, which is a virtual object that visualizes the left hand of the instructor sensed by the second terminal device 30, and the second terminal device 30. and a virtual hand HR2, which is a virtual object that visualizes the right hand of the pointer sensed by .
  • the first avatar AV1 is displayed at a position corresponding to the position of the worker in the virtual space represented by the 3D map of the farm field.
  • the reproduction avatar AV3 is displayed at a position corresponding to the position specified by the instructor wearing the second terminal device 30 in the virtual space represented by the 3D map of the field.
  • FIG. 4 is a diagram for explaining a tap operation that triggers the start and end of recording of the movement of the instructor.
  • FIG. 4(a) shows a state in which the instructor raises his/her thumb
  • FIG. 4(b) shows a state in which the distance from the tip of the thumb to the second joint of the index finger is equal to or less than the threshold.
  • the first terminal device 20 senses the orientation of the hand of the instructor, changes the orientation of the hand from the state shown in FIG. 4(a) to the state shown in FIG. 4(b), and returns to the state shown in FIG. 4(a). Detect tap actions.
  • the first terminal device 20 detects that the tap motion is performed a predetermined number of times within a predetermined period
  • the first terminal device 20 senses the position and posture of the hands of the indicator, the position and posture of the head of the indicator, and the like. Starts recording position and orientation data.
  • the tap operation is performed a predetermined number of times within a predetermined period.
  • the recording of the data of the sensed position and orientation is terminated, and the recorded data of the position and orientation are stored as action data representing the work of the instructor.
  • the first terminal device 20 transmits the stored motion data to the information processing device 10 , and the information processing device 10 stores the motion data transmitted from the first terminal device 20 .
  • the second terminal device 30 senses the position and orientation of the hands of the instructor and the position and orientation of the head of the instructor when the instructor in the remote location is wearing the second terminal device 30, and the sensing
  • the data of the determined position and orientation may be transmitted to the information processing apparatus 10 as motion data, and the information processing apparatus 10 may store the data.
  • the reproduction avatar AV3 displayed on the first terminal device 20 and the second terminal device 30 reproduces the action data.
  • the first terminal device 20 and the second terminal device 30 receive motion data transmitted from the information processing device 10 .
  • the first terminal device 20 and the second terminal device 30 operate the reproduction avatar AV3 on the basis of the received data, and display the position and posture of the head and the positions and postures of the hands from the start to the end of recording the motion data.
  • the posture is reproduced by the reproduction avatar AV3. For example, when the motion data of a plant in a field is recorded, the reproduction avatar V3 reproduces the motion of bud drop.
  • FIG. 5 is a diagram for explaining an operation for changing the position and orientation of the playback avatar AV3.
  • the position of the playback avatar AV3 can be changed by operating the playback avatar V3 with the virtual hand HL1, the virtual hand HR1, the virtual hand HL2, or the virtual hand HR2.
  • the instructor wearing the second terminal device 30 changes the position of the playback avatar V3 in the 3D map viewed by the second terminal device 30, as shown in FIG.
  • the position of the reproduction avatar AV3 is changed to the position of the changed box BX3. Also, by changing the direction of the box BX by operating it with the virtual hand HR2, the direction of the playback avatar AV3 is changed.
  • FIG. 6 is a block diagram showing the functional configuration of the information processing device 10. As shown in FIG. As shown in FIG. 6 , the information processing device 10 includes a control section 100 , a communication section 110 and a storage section 120 .
  • the communication unit 110 has a function of communicating with an external device.
  • the communication unit 110 supplies information received from the external device to the control unit 100 in communication with the external device.
  • the communication unit 110 supplies information transmitted from the first terminal device 20 and information transmitted from the second terminal device 30 to the control unit 100 .
  • the communication unit 110 transmits information supplied from the control unit 100 to an external device.
  • the communication unit 110 acquires first avatar information representing the first avatar AV1 generated by the control unit 100 based on the first posture information transmitted from the first terminal device 20, and acquires the acquired first avatar AV1.
  • Avatar information is transmitted to the second terminal device 30 .
  • the communication unit 110 acquires second avatar information representing the second avatar AV2 generated by the control unit 100 based on the second posture information transmitted from the second terminal device 30, and transmits the acquired second avatar information. It transmits to the first terminal device 20 . Further, the communication unit 110 obtains the reproduction avatar information representing the reproduction avatar AV3 generated by the control unit 100 based on the action data transmitted from the second terminal device 30, and transmits the obtained reproduction avatar information to the first terminal device 20. and to the second terminal device 30 . Also, the communication unit 110 transmits the map information to the second terminal device 30 .
  • the storage unit 120 is implemented by, for example, a semiconductor memory device such as a RAM or flash memory, or a storage device such as a hard disk or optical disk.
  • the storage unit 120 has a function of storing information regarding processing in the information processing apparatus 10 .
  • the storage unit 120 stores, for example, the 3D map described above and the motion data described above. Note that the 3D map and motion data stored in the storage unit 120 may be stored in an external storage device connected to the information processing device 10 .
  • the control unit 100 executes processing for controlling the operation of the information processing device 10 .
  • the control unit 100 performs processing for providing first avatar information, second avatar information, playback avatar information, and map information.
  • the control unit 100 has an acquisition unit 101, a processing unit 102, and an output unit 103, as shown in FIG.
  • the acquisition unit 101 has a function of acquiring information for generating first avatar information, second avatar information, playback avatar information, and map information.
  • the acquisition unit 101 acquires, for example, the first posture information transmitted from the first terminal device 20 via the communication unit 110 . Also, the acquisition unit 101 acquires the second posture information transmitted from the second terminal device 30 via the communication unit 110 . Also, the acquisition unit 101 acquires the motion data and the 3D map stored in the storage unit 120 .
  • the processing unit 102 has a function of generating first avatar information, second avatar information, playback avatar information, and map information.
  • the processing unit 102 has a position/orientation identification unit 1021 and a generation unit 1022, as shown in FIG.
  • the position/orientation identification unit 1021 determines the position and inclination of the head of the user wearing the first terminal device 20 in the field, the direction the user is facing, the position and orientation of the hand of the user, and the orientation of the hand. Identify Posture. Based on the second posture information, the position/posture identifying unit 1021 determines the position and tilt of the head of the user wearing the second terminal device 30 in the 3D map, the direction the user is facing, the position and orientation of the user's hands, and so on. Identify hand posture. The position/orientation identification unit 1021 identifies the stored position and inclination of the user's head, and the position and orientation of the hand, based on the motion data. Also, the position/orientation identification unit 1021 detects the action of operating the box BX based on the first orientation information or the second orientation information, and identifies the position and orientation of the reproduction avatar AV3.
  • the generation unit 1022 generates the position and inclination of the head of the user wearing the first terminal device 20, the direction in which the user is facing, and the position and posture of the hand of the user, which are specified by the position/orientation specifying unit 1021. Based on this, the first avatar information representing the first avatar AV1 is generated. The generation unit 1022 determines the position and inclination of the head of the user wearing the second terminal device 30, the direction in which the user is facing, and the position and posture of the hand of the user, which are specified by the position/orientation specifying unit 1021. Based on this, the second avatar information representing the second avatar AV2 is generated. The generation unit 1022 generates reproduction avatar information representing the reproduction avatar AV3 based on the action data.
  • the generation unit 1022 generates map information of the field based on the 3D map.
  • the generation unit 1022 arranges the first avatar AV1, the second avatar AV2, and the playback avatar AV3 in the virtual space represented by the 3D map.
  • the placement position of the first avatar AV1 in the virtual space is a position corresponding to the position where the user of the first terminal device 20 is in the field.
  • the placement position of the second avatar AV2 in the virtual space is a position corresponding to the position of the user of the second terminal device 30 in the virtual space.
  • the placement position of the playback avatar AV3 in the virtual space is the position designated by the operator or the instructor by operating the box BX.
  • the output unit 103 has a function of outputting information generated by the generation unit 1022 .
  • Output unit 103 outputs the first avatar information, the second avatar information, the reproduction avatar information, and the map information generated by generation unit 1022 to communication unit 110 .
  • FIG. 7 is a block diagram showing the hardware configuration and functional configuration of the first terminal device 20.
  • the first terminal device 20 has a control section 200 , a storage section 210 , a video output section 220 , an audio output section 230 , an external communication section 240 and a sensor section 250 .
  • the sensor unit 250 has a head position measurement unit 251, a hand posture measurement unit 252, and a voice acquisition unit 253.
  • the head position measurement unit 251 has an acceleration sensor 251a, an orientation sensor 251b, a depth sensor 251c, a gyro sensor 251d, a SLAM 251e, and a GPS module 251f.
  • the acceleration sensor 251a is, for example, a triaxial acceleration sensor.
  • the acceleration sensor 251a outputs acceleration information representing the measured acceleration.
  • the azimuth sensor 251b is a sensor that measures geomagnetism and detects the direction in which the first terminal device 20 is facing.
  • the orientation sensor 251b outputs orientation information representing the detected orientation.
  • the depth sensor 251c is a sensor that measures the distance from the first terminal device 20 to a person or object existing within the target range.
  • the depth sensor 251c outputs depth information representing the measured distance.
  • the gyro sensor 251 d is a sensor that measures the angular velocity of the first terminal device 20 .
  • the gyro sensor 251d outputs angular velocity information representing the measured angular velocity.
  • the SLAM 251e is, for example, a Lidar (Light Detection And Ranging) SLAM (Simultaneous Localization and Mapping) equipped with a laser scanner, or a Visual SLAM equipped with a camera.
  • the SLAM 251e senses the surroundings and outputs map information representing a map of the surroundings.
  • the GPS module 251f receives radio waves measured from satellites in the satellite positioning system and measures the position of the first terminal device 20 .
  • the GPS module 251f outputs position information representing the measured position.
  • the head position measurement unit 251 outputs head information including acceleration information, direction information, depth information, angular velocity information, map information, and position information.
  • the hand posture measurement unit 252 has a depth sensor 252a and an infrared camera 252b.
  • the infrared camera 252b outputs infrared light, receives infrared light reflected by the user's hand, and photographs the user's hand.
  • the depth sensor 252a measures the distance to the user's hand based on the image of the user's hand generated by the infrared camera 252b.
  • the hand posture measurement unit 252 outputs hand posture information including the measured distance to the user's hand and an image of the user's hand.
  • the voice acquisition unit 253 has a microphone 253a.
  • the microphone 253a picks up sounds around the first terminal device 20 and outputs audio information representing the picked-up sounds.
  • the storage unit 210 is realized by semiconductor memory devices such as RAM and flash memory, for example.
  • the storage unit 210 has a function of storing information about processing in the first terminal device 20 .
  • the storage unit 210 also stores application programs executed by the first terminal device 20 .
  • the application program stored in the storage unit 210 is, for example, a program that allows the user to visually recognize the second avatar AV2, the playback avatar AV3, etc. by AR display.
  • the control unit 200 is implemented by executing an application program stored in the storage unit 210.
  • the control unit 200 has a position processing unit 201, a hand posture processing unit 202, a motion recording unit 203, a display processing unit 204, a display control unit 205, and a communication control unit 206, as shown in FIG.
  • the position processing unit 201 identifies the position and inclination of the user's head, the direction in which the user is facing, etc. based on the head information output from the head position measurement unit 251 .
  • the hand posture processing unit 202 identifies the position and posture of the user's hands based on the hand posture information output from the hand posture measurement unit 252 .
  • the position specified by the position processing unit 201 is specified by a relative position from the origin, for example, with the position of an AR marker installed at a predetermined position in the field as the origin.
  • the motion recording unit 203 causes the storage unit 210 to store the head information output from the head position measurement unit 251 and the hand posture information output from the hand posture measurement unit 252 as motion data.
  • the display processing unit 204 generates images of the virtual hand HL1 and the virtual hand HR1 based on the position and posture specified by the hand posture processing unit 202. Also, the display processing unit 204 generates an image for AR display based on information provided from the information processing device 10 . Specifically, the display processing unit 204 generates an image of the second avatar AV2 based on the second avatar information provided from the information processing device 10, and generates an image of the second avatar AV2 based on the reproduced avatar information provided from the information processing device 10. to generate an image of the playback avatar AV3.
  • the display control unit 205 controls the video output unit 220 so that the image of the second avatar AV2 generated by the display processing unit 204 is AR-displayed, and the image of the playback avatar AV3 generated by the display processing unit 204 is AR-displayed.
  • the video output unit 220 is controlled so that The display position of the second avatar AV2 is a position corresponding to the position of the user of the second terminal device 30 in the virtual space represented by the 3D map.
  • the display position of the playback avatar AV3 is the position designated by the operator or the instructor by operating the box BX.
  • the display control unit 205 also controls the video output unit 220 so that the images of the virtual hand HL1 and the virtual hand HR1 are AR-displayed at the hand positions specified by the hand posture processing unit 202 .
  • the communication control unit 206 controls the external communication unit 240 to transmit information to the information processing device 10 and receive information from the information processing device 10 .
  • the video output unit 220 displays the AR image output from the control unit 200 and viewed by the user on the half mirror.
  • the audio output unit 230 includes a speaker and outputs sounds represented by audio signals supplied from an external device.
  • the external communication unit 240 has a function of communicating with an external device. For example, the external communication unit 240 supplies information received from the external device to the control unit 200 in communication with the external device. Specifically, the external communication unit 240 supplies the second avatar information and the reproduced avatar information received from the information processing device 10 to the control unit 200 . In addition, the external communication unit 240 transmits information supplied from the control unit 200 to the external device in communication with the external device. Specifically, the external communication unit 240 transmits first posture information including head information and hand posture information to the information processing device 10 . The external communication unit 240 also transmits the motion data stored in the storage unit 210 to the information processing device 10 .
  • FIG. 8 is a block diagram showing the hardware configuration and functional configuration of the second terminal device 30.
  • the second terminal device 30 has a control section 300 , a storage section 310 , a video output section 320 , an audio output section 330 , an external communication section 340 and a sensor section 350 .
  • the sensor unit 350 has a head position measurement unit 351, a hand posture measurement unit 352, and a voice acquisition unit 353.
  • the head position measurement unit 351 has an acceleration sensor 351a, an orientation sensor 351b, and a gyro sensor 351d.
  • the acceleration sensor 351a is, for example, a triaxial acceleration sensor, and outputs acceleration information representing the measured acceleration.
  • the azimuth sensor 351b is a sensor that measures geomagnetism and detects the direction in which the second terminal device 30 is facing.
  • the azimuth sensor 351b outputs azimuth information representing the detected azimuth.
  • the gyro sensor 351d is a sensor that measures the angular velocity of the second terminal device 30 .
  • the gyro sensor 351d outputs angular velocity information representing the measured angular velocity.
  • the hand posture measurement unit 352 has a depth sensor 352a and an infrared camera 352b.
  • the infrared camera 352b outputs infrared light, receives infrared light reflected by the user's hand, and photographs the user's hand.
  • the depth sensor 352a measures the distance to the user's hand based on the image of the user's hand generated by the infrared camera 352b.
  • the hand posture measurement unit 252 outputs hand posture information including the measured distance to the user's hand and an image of the user's hand.
  • the voice acquisition unit 253 has a microphone 253a.
  • the microphone 253a picks up sounds around the second terminal device 30 and outputs audio information representing the picked-up sounds.
  • the storage unit 310 is realized by semiconductor memory devices such as RAM and flash memory, for example.
  • the storage unit 310 has a function of storing information about processing in the second terminal device 30 .
  • the storage unit 310 also stores application programs executed by the second terminal device 30 .
  • the application program stored in the storage unit 310 is, for example, a program that allows the worker to visually recognize the first avatar AV1, the playback avatar AV3, a 3D map, and the like through VR display.
  • the control unit 300 is implemented by executing an application program stored in the storage unit 310.
  • the control unit 300 has a position processing unit 301, a hand posture processing unit 302, a display processing unit 304, a display control unit 305, and a communication control unit 306, as shown in FIG.
  • the position processing unit 301 identifies the position and inclination of the user's head, the direction in which the user is facing, etc. based on the head information output from the head position measurement unit 351 .
  • the hand posture processing unit 302 identifies the hand position and hand posture of the user based on the hand posture information output from the hand posture measurement unit 352 .
  • the position specified by the position processing unit 301 is specified by a relative position from the origin, for example, with the position corresponding to the position of the AR marker in the field on the 3D map of the field as the origin.
  • the display processing unit 304 generates images of the virtual hand HL2 and the virtual hand HR2 based on the position and posture specified by the hand posture processing unit 302. Also, the display processing unit 304 generates an image for VR display based on information provided from the information processing device 10 . Specifically, the display processing unit 304 generates an image of the first avatar AV1 based on the first avatar information provided from the information processing device 10, and generates an image of the first avatar AV1 based on the reproduced avatar information provided from the information processing device 10. to generate an image of the playback avatar AV3. The display processing unit 304 also generates a 3D map image of the field based on the map information provided from the information processing device 10 .
  • the display control unit 305 controls the video output unit 320 so that the image of the first avatar AV1 generated by the display processing unit 304 is displayed in VR, and the image of the playback avatar AV3 generated by the display processing unit 304 is displayed in VR.
  • the video output unit 320 is controlled so that The display control unit 305 also displays the 3D map image generated by the display processing unit 304 in VR.
  • the display position of the first avatar AV1 in the displayed 3D map is a position corresponding to the position of the user of the first terminal device 20 in the field.
  • the display position of the reproduction avatar AV3 in the displayed 3D map is the position designated by the operator or the instructor by operating the box BX.
  • the display control unit 305 controls the video output unit 320 so that the images of the virtual hand HL2 and the virtual hand HR2 are VR-displayed at the hand positions specified by the hand posture processing unit 202 .
  • the communication control unit 306 controls the external communication unit 340 to transmit information to the information processing device 10 and receive information from the information processing device 10 .
  • the video output unit 320 displays a VR image that is output from the control unit 300 and viewed by the worker.
  • the audio output unit 330 includes a speaker and outputs sounds represented by audio signals supplied from an external device.
  • the external communication unit 340 has a function of communicating with an external device. For example, the external communication unit 340 supplies information received from the external device to the control unit 300 in communication with the external device. Specifically, the external communication unit 340 supplies the first avatar information, the reproduced avatar information, and the map information received from the information processing device 10 to the control unit 300 . In addition, the external communication unit 340 transmits information supplied from the control unit 300 to the external device in communication with the external device. Specifically, the external communication unit 340 transmits second posture information including head information and hand posture information to the information processing device 10 .
  • FIG. 9 is a state transition diagram of the first terminal device 20 when recording user's actions.
  • the first terminal device 20 transitions to any one of a standby mode, a recording mode, and a recording end mode when recording user's actions.
  • Standby mode is a state in which user actions can be recorded.
  • the first terminal device 20 detects that the tap operation is performed a predetermined number of times within a predetermined period in the standby mode, the first terminal device 20 transitions to the recording mode.
  • the recording mode is a mode that records data on the position and posture of the user's hands and the position and posture of the user's head.
  • the recording end mode is a mode in which the recording of data on the position and posture of the instructor's hands and the position and posture of the instructor's head is finished, and the motion data is stored.
  • the first terminal device 20 deletes the data related to the tapping motion among the recorded data, and stores the data related to the motion other than the tapping motion in the storage unit 210 as motion data. After the operation data is stored in the storage unit 210, the first terminal device 20 transitions to the standby mode.
  • FIG. 10 is a flow chart showing the flow of processing for the first terminal device 20 to detect a tap action.
  • the first terminal device 20 first initializes the tap count, tap time, and contact time (step S101). Specifically, the tap count is set to 0, and the tap time and contact time are recorded as the time at this point.
  • the first terminal device 20 determines whether the distance from the tip of the user's thumb to the second joint of the index finger is equal to or less than a threshold based on the hand posture information output from the hand posture measurement unit 252 (step S102). When the distance from the tip of the user's thumb to the second joint of the index finger is equal to or less than the threshold (Yes in step S102), the first terminal device 20 determines that the elapsed time from the recorded contact time is 0.1 seconds or more. (step S103). When the elapsed time from the recorded contact time is 0.1 seconds or more (Yes in step S103), the first terminal device 20 turns on the tap flag (step S104), and records the recorded contact time. The current time is updated (step S105). If the elapsed time from the recorded contact time is less than 0.1 seconds (No in step S103), or if the process of step S105 is completed, the first terminal apparatus 20 shifts the process flow to step S102. return.
  • the first terminal device 20 determines whether the tap flag is on (step S106). When the tap flag is ON (Yes in step S106), the first terminal device 20 adds 1 to the recorded number of taps (step S107). Next, the first terminal device 20 turns off the tap flag (step S108), and updates the recorded tap time to the current time (step S109).
  • the first terminal device 20 shifts the flow of processing to step S110.
  • the first terminal device 20 determines whether the elapsed time from the recorded tap time is 0.5 seconds or more (step S110). When the elapsed time from the recorded tap time is 0.5 seconds or more (Yes in step S110), the first terminal device 20 determines whether the recorded number of taps exceeds 1 (step S111 ). When the number of taps being recorded exceeds 1 (Yes in step S111), the first terminal device 20 generates a tap event including the number of taps being recorded (step S112). The number of times is initialized to 0 (step S113).
  • the first terminal device 20 When the elapsed time from the recorded tap time is less than 0.5 seconds (No in step S11), the first terminal device 20 records when the number of taps is 1 or less (No in step S111). ), or if the processing of step S113 is completed, the flow of processing is returned to step S102. According to the process of FIG. 10, in the first terminal device 20, a tap event occurs when the tap operation is performed twice or more at intervals of less than 0.5 seconds.
  • FIG. 11 is a flow chart showing the flow of processing in which the first terminal device 20 records the user's actions.
  • the first terminal device 20 first transitions to the standby mode (step S201).
  • the first terminal device 20 determines whether a tap event has occurred (step S202). If no tap event has occurred in the process of FIG. 10 (No in step S202), the first terminal device 20 repeats the process of step S202 until a tap event occurs.
  • the first terminal device 20 acquires the number of taps included in the tap event in the process of FIG. 10 (step S203). The first terminal device 20 determines whether the acquired number of taps is 3 or more (step S204). When the acquired number of taps is 3 or more (Yes in step S204), the first terminal device 20 determines the current operation mode (step S205). Note that the number of taps determined here is not limited to three, and may be two or four or more.
  • the first terminal device 20 transitions to the recording mode (step S209), and the position and posture of the instructor's hand and the head of the instructor. Recording of position and orientation data is started (step S210).
  • the first terminal device 20 transitions to the recording end mode (step S206), and the position and posture of the hand of the instructor, the head of the instructor, and so on. Recording of position and orientation data is stopped (step S207).
  • the first terminal device 20 deletes the data related to the tap operation from the recorded data (step S208). Data after the tap action is deleted is stored in the storage unit 210 as action data. The motion data stored in the storage unit 210 is transmitted to the information processing device 10 .
  • the first terminal device 20 determines No in step S204, when the process of step S208 is completed, or when the process of step S210 is completed, it determines whether the recording end mode is set (step S211). If the first terminal device 20 is in the recording end mode (Yes in step S211), the process flow returns to step S201, and if it is not in the recording end mode (No in step S211), the process flow returns to step S202. .
  • FIG. 12 is a state transition diagram relating to movement of the playback avatar AV3.
  • the information processing device 10 transitions to any one of a fixed location mode, a move standby mode, a moveable mode, and a move mode.
  • the movement standby mode is a mode in which the reproduction avatar AV3 reproduces motion data at a position designated by the user.
  • the reproduction avatar AV3 reproduces the movement of the head and the movement of the hands represented by the action data.
  • the fixed location mode is a mode in which the playback avatar AV3 reproduces the action data at the user's position when the action data was recorded.
  • the position where the motion data is recorded is, for example, the position of the AR marker installed at a predetermined position in the field as the origin, and the position relative to this origin when the motion data is recorded.
  • the movable mode is a mode in which the reproduction of action data is stopped when any of the virtual hand HL1, the virtual hand HR1, the virtual hand HL2, or the virtual hand HR2 approaches the box BX of the reproduction avatar AV3.
  • the move mode is a mode in which the user can operate the box BX to move the position of the playback avatar AV3.
  • the reproduction of the action data by the reproduction avatar AV3 may be stopped and only the head and body of the avatar may be displayed.
  • moving image data may be reproduced in the movable mode.
  • FIG. 13 is a flow chart showing the flow of processing when the access right to the reproduction avatar AV3 for the user of the first terminal device 20 and the user of the second terminal device 30 is changed.
  • the information processing apparatus 10 first determines whether the user has the right to access the reproduction avatar AV3 (step S301). When the user does not have access rights to the reproduction avatar AV3 (No in step S301), the information processing apparatus 10 determines whether the reproduction avatar AV3 is reproducing motion data (step S302).
  • step S302 When the reproduction avatar AV3 is reproducing the action data (Yes in step S302), the information processing apparatus 10 stops the reproduction of the action data by the reproduction avatar AV3 (step S303), and sets the display flag of the reproduction avatar AV3 to the user. It is turned off (step S304).
  • the display flag for the worker is turned off by the processing of steps S301 to S304.
  • the information processing device 10 stops transmission of the reproduction avatar information to the first terminal device 20 because the worker's display flag is off. By stopping the transmission of the reproduction avatar information to the first terminal device 20, the reproduction avatar AV3 is no longer displayed in AR on the first terminal device 20.
  • step S301 If the user has access rights to the playback avatar AV3 (Yes in step S301), the information processing device 10 turns on the display flag of the playback avatar AV3 for the user (step S305).
  • the information processing apparatus 10 determines whether or not the reproduction avatar AV3 has already reproduced the action data (step S306). If the reproduction avatar AV3 has not reproduced the action data (No in step S306), the information processing apparatus 10 starts the reproduction of the action data by the reproduction avatar AV3 (step S307).
  • the display flag for the worker is turned on by the processing of steps S305 to S307.
  • the information processing device 10 transmits the reproduction avatar information to the first terminal device 20 because the worker's display flag is on.
  • the reproduction avatar AV3 is AR-displayed on the first terminal device 20 .
  • FIG. 14 is a flow chart showing the flow of processing for moving the playback avatar AV3 when the access right to the playback avatar AV3 is set for the user.
  • the information processing device 10 first transitions to the movement standby mode when moving the display position of the reproduction avatar AV3 (step S401).
  • the information processing device 10 determines whether the distance between the playback avatar AV3 and the user's hand is equal to or less than a threshold (step S402). If the distance between the playback avatar AV3 and the user's hand is not equal to or less than the threshold value (No in step S402), the information processing apparatus 10 determines whether there is a recording position reference flag (step S403).
  • the recorded position reference flag is a flag that indicates whether the user's position when motion data was recorded is stored.
  • the information processing apparatus 10 determines whether or not the fixed location mode is set when there is a position reference flag during recording (step S404). If the information processing apparatus 10 is in the fixed location mode (Yes in step S404), the processing flow returns to step S403, and if it is not in the fixed location mode (No in step S404), the information processing apparatus 10 transitions to the fixed location mode. Then, the flow of processing is returned to step S403.
  • step S406 determines whether the current mode is the movement standby mode. If the current mode is the movement standby mode (Yes in step S406), the information processing apparatus 10 returns the flow of processing to step S401, and if the current mode is not the movement standby mode (No in step S406), the process returns to step S402.
  • step S402 When the distance between the reproduction avatar AV3 and the user's hand is equal to or less than the threshold (Yes in step S402), the information processing apparatus 10 grants the user whose hand distance from the reproduction avatar AV3 is equal to or less than the threshold the access right to the reproduction avatar AV3. (step S407). If the user does not have the access right (No in step S407), the information processing apparatus 10 shifts the flow of processing to step S406.
  • step S408 the information processing device 10 transitions to the movable mode (step S408). After transitioning to the movable mode, the information processing apparatus 10 determines whether the user is pinching the box BX (step S409). If the user is pinching the box BX (Yes in step S409), the information processing apparatus 10 determines whether the current mode is the move mode (step S411). If the current mode is not the move mode (No in step S411), the information processing apparatus 10 transitions to the move mode (step S412). When the current mode is the move mode (Yes in step S411), or when the process of step S412 is finished, the information processing apparatus 10 updates the position of the reproduction avatar AV3 according to the position of the user's hand (step S413).
  • step S410 determines whether the distance between the playback avatar AV3 and the user's hand is equal to or less than a threshold (step S410). If the distance between the reproduced avatar AV3 and the user's hand is not equal to or less than the threshold (No in step S410), the information processing apparatus 10 shifts the flow of processing to step S406, and if the distance between the reproduced avatar AV3 and the user's hand is equal to or less than the threshold, the process proceeds to step S406. If there is (Yes in step S402), the flow of processing moves to step S408.
  • the above-described embodiment is configured to perform AR display and VR display corresponding to synecoculture, but the AR display and VR display displayed by the information processing system 1 are limited to those corresponding to synecoculture. Instead, for example, AR display and VR display corresponding to work in a factory or work at a construction site may be performed.
  • the photograph of the user of the first terminal device 20 may be synthesized with the first avatar AV1
  • the photograph of the user of the second terminal device 30 may be synthesized with the second avatar AV2. good.
  • the configuration when a plurality of first terminal devices 20 are connected to the information processing device 10, the configuration may be such that the first avatars AV1 of the plurality of users wearing the first terminal devices 20 are displayed. Further, when a plurality of second terminal devices 30 are connected to the information processing device 10, the configuration may be such that the second avatars AV2 of the plurality of users wearing the second terminal devices 30 are displayed. Further, in the above-described embodiment, the information processing apparatus 10 may store a plurality of motion data, and may display the reproduced avatar AV3 for each of the plurality of motion data selected from the stored motion data.
  • the second terminal device 30 may be configured to include the depth sensor, SLAM, and GPS modules that the first terminal device 20 includes. Also, the second terminal device 30 may be configured to include a motion recording unit, like the first terminal device 20 . According to this configuration, for example, the instructor can record the motion at a remote location, and the worker can view the motion recorded at the remote location on the field using the playback avatar AV3. In this case, the instructor can record a model action without going to the field.
  • the user of the first terminal device 20 is the operator and the user of the second terminal device 30 is the instructor.
  • the user of the second terminal device 30 in the room may be a worker.
  • the instructor records the motion in the field, and the worker can remotely view the recorded motion using the playback avatar AV3.
  • the worker can see the motions that serve as a model, and based on the motions she has seen, she can work in the actual field. It can be performed.
  • FIG. 15 is a block diagram illustrating an example hardware configuration of a computer that implements the functions of the information processing apparatus according to the embodiment.
  • the information processing device 900 shown in FIG. 15 can implement, for example, the information processing device 10, the first terminal device 20, and the second terminal device 30 shown in FIG.
  • Information processing by the information processing device 10, the first terminal device 20, and the second terminal device 30 according to the embodiment is realized by cooperation between software and hardware described below.
  • the information processing apparatus 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903.
  • the information processing device 900 also includes a host bus 904 a , a bridge 904 , an external bus 904 b , an interface 905 , an input device 906 , an output device 907 , a storage device 908 , a drive 909 , a connection port 910 and a communication device 911 .
  • the hardware configuration shown here is an example, and some of the components may be omitted. Also, the hardware configuration may further include components other than those shown here.
  • the CPU 901 functions, for example, as an arithmetic processing device or a control device, and controls all or part of the operation of each component based on various programs recorded in the ROM 902, RAM 903, or storage device 908.
  • the ROM 902 is means for storing programs to be read by the CPU 901, data used for calculation, and the like.
  • the RAM 903 temporarily or permanently stores, for example, programs to be read by the CPU 901 and various parameters that appropriately change when the programs are executed. These are interconnected by a host bus 904a composed of a CPU bus or the like.
  • the CPU 901, ROM 902, and RAM 903 can realize the functions of the control unit 100, the control unit 200, and the control unit 300 described with reference to FIGS. 6 to 8, for example, in cooperation with software.
  • the CPU 901, ROM 902, and RAM 903 are interconnected, for example, via a host bus 904a capable of high-speed data transmission.
  • the host bus 904a is connected via a bridge 904, for example, to an external bus 904b having a relatively low data transmission speed.
  • the external bus 904b is connected to various components via an interface 905. FIG.
  • the input device 906 is implemented by a device through which information is input, such as a mouse, keyboard, touch panel, button, microphone, switch, and lever.
  • the input device 906 may be, for example, a remote control device using infrared rays or other radio waves, or may be an external connection device such as a mobile phone or PDA corresponding to the operation of the information processing device 900.
  • the input device 906 may include, for example, an input control circuit that generates an input signal based on information input using the above input means and outputs the signal to the CPU 901 .
  • a user of the information processing apparatus 900 can input various data to the information processing apparatus 900 and instruct processing operations by operating the input device 906 .
  • the input device 906 may be formed by a device that detects the user's position.
  • the input device 906 includes an image sensor (eg, camera), depth sensor (eg, stereo camera), acceleration sensor, gyro sensor, geomagnetic sensor, optical sensor, sound sensor, ranging sensor (eg, ToF (Time of Flight ) sensors), force sensors, and the like.
  • the input device 906 also receives information about the state of the information processing device 900 itself, such as the attitude and movement speed of the information processing device 900, and information about the space around the information processing device 900, such as brightness and noise around the information processing device 900. may be obtained.
  • the input device 906 receives GNSS signals from GNSS (Global Navigation Satellite System) satellites (for example, GPS signals from GPS (Global Positioning System) satellites) and provides position information including the latitude, longitude and altitude of the device.
  • GNSS Global Navigation Satellite System
  • GPS Global Positioning System
  • a measuring GNSS module may be included.
  • the input device 906 may detect the position by Wi-Fi (registered trademark), transmission/reception with a mobile phone/PHS/smartphone, or short-distance communication.
  • the input device 906 can realize, for example, the functions of the sensor unit 250 described with reference to FIG. 7 and the functions of the sensor unit 350 described with reference to FIG.
  • the output device 907 is formed by a device capable of visually or audibly notifying the user of the acquired information.
  • Such devices include display devices such as CRT display devices, liquid crystal display devices, plasma display devices, EL display devices, laser projectors, LED projectors and lamps, sound output devices such as speakers and headphones, and printer devices. .
  • the output device 907 outputs, for example, results obtained by various processes performed by the information processing device 900 .
  • the display device visually displays the results obtained by various processes performed by the information processing device 900 in various formats such as text, image, table, and graph.
  • an audio output device converts an audio signal, which is composed of reproduced audio data, acoustic data, etc., into an analog signal and aurally outputs the analog signal.
  • the output device 907 can realize, for example, the functions of the video output unit 220 and the audio output unit 230 described with reference to FIG. 7, and the functions of the video output unit 320 and the audio output unit 330 described with reference to FIG. .
  • the storage device 908 is a data storage device formed as an example of the storage unit of the information processing device 900 .
  • the storage device 908 is implemented by, for example, a magnetic storage device such as an HDD, a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like.
  • the storage device 908 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, a deletion device that deletes data recorded on the storage medium, and the like.
  • the storage device 908 stores programs executed by the CPU 901, various data, and various data acquired from the outside.
  • the storage device 908 can realize the functions of the storage unit 120, the storage unit 210, and the storage unit 310 described with reference to FIGS. 6 to 8, for example.
  • the drive 909 is a reader/writer for storage media, and is built in or externally attached to the information processing apparatus 900 .
  • the drive 909 reads out information recorded on a removable storage medium such as a mounted magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 903 .
  • Drive 909 can also write information to a removable storage medium.
  • connection port 910 is, for example, a USB (Universal Serial Bus) port, an IEEE1394 port, a SCSI (Small Computer System Interface), an RS-232C port, or a port for connecting an external connection device such as an optical audio terminal. .
  • USB Universal Serial Bus
  • IEEE1394 Serial Bus
  • SCSI Serial Computer System Interface
  • RS-232C Serial Bus
  • a port for connecting an external connection device such as an optical audio terminal.
  • the communication device 911 is, for example, a communication interface formed by a communication device or the like for connecting to the communication network 920 .
  • the communication device 911 is, for example, a communication card for wired or wireless LAN (Local Area Network), LTE (Long Term Evolution), Bluetooth (registered trademark), or WUSB (Wireless USB).
  • the communication device 911 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), a modem for various types of communication, or the like.
  • This communication device 911 can, for example, transmit and receive signals to and from the Internet or other communication devices in accordance with a predetermined protocol such as TCP/IP.
  • the communication device 911 can implement the functions of the communication unit 110, the external communication unit 240, and the external communication unit 340 described with reference to FIGS. 6 to 8, for example.
  • the communication network 920 is a wired or wireless transmission path for information transmitted from devices connected to the communication network 920 .
  • the communication network 920 may include a public line network such as the Internet, a telephone line network, a satellite communication network, various LANs (Local Area Networks) including Ethernet (registered trademark), WANs (Wide Area Networks), and the like.
  • Communication network 920 may also include a dedicated line network such as IP-VPN (Internet Protocol-Virtual Private Network).
  • First posture information representing the position and posture of one or more first users wearing the first terminal devices
  • Second posture information representing the positions and postures of one or more second users wearing the second terminal devices
  • an acquisition unit that acquires posture information
  • a storage unit that stores motion data representing changes in the user's posture
  • a processing unit that arranges, in the virtual space, a reproduction avatar that can change its position and orientation and reproduces the action data; displaying the second avatar and the reproduction avatar on the first terminal device according to the position in the virtual space, and displaying the first avatar and the reproduction avatar according to the position in the virtual space on the second terminal device; an output to display on Information processing device.
  • the first terminal device performs AR display
  • the first terminal device is worn by a worker
  • the first avatar is synthesized with a photograph of the first user

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Architecture (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Business, Economics & Management (AREA)
  • Educational Administration (AREA)
  • Educational Technology (AREA)
  • User Interface Of Digital Computer (AREA)
PCT/JP2022/003201 2021-06-29 2022-01-28 情報処理装置、情報処理方法及びプログラム Ceased WO2023276215A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2023531357A JPWO2023276215A1 (https=) 2021-06-29 2022-01-28
US18/572,293 US20240296642A1 (en) 2021-06-29 2022-01-28 Information processing apparatus, information processing method, and program

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021-108085 2021-06-29
JP2021108085 2021-06-29

Publications (1)

Publication Number Publication Date
WO2023276215A1 true WO2023276215A1 (ja) 2023-01-05

Family

ID=84691052

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/003201 Ceased WO2023276215A1 (ja) 2021-06-29 2022-01-28 情報処理装置、情報処理方法及びプログラム

Country Status (3)

Country Link
US (1) US20240296642A1 (https=)
JP (1) JPWO2023276215A1 (https=)
WO (1) WO2023276215A1 (https=)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009145883A (ja) * 2007-11-20 2009-07-02 Rissho Univ 学習システム、記憶媒体及び学習方法
JP2014210213A (ja) * 2014-08-21 2014-11-13 株式会社スクウェア・エニックス ビデオゲーム処理装置、及びビデオゲーム処理プログラム
US20200249747A1 (en) * 2017-05-23 2020-08-06 Visionary Vr, Inc. System and method for generating a virtual reality scene based on individual asynchronous motion capture recordings
JP2020195551A (ja) * 2019-05-31 2020-12-10 イマクリエイト株式会社 身体活動支援システム、方法、及び、プログラム

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2018106392A (ja) * 2016-12-26 2018-07-05 株式会社コロプラ 仮想空間を介して通信するためにコンピュータによって実行される方法、当該方法をコンピュータに実行させるためのプログラム、およびコンピュータ装置
KR102946236B1 (ko) * 2020-02-28 2026-03-31 삼성전자주식회사 제스처 인식 기반의 영상 편집 방법 및 이를 지원하는 전자 장치

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009145883A (ja) * 2007-11-20 2009-07-02 Rissho Univ 学習システム、記憶媒体及び学習方法
JP2014210213A (ja) * 2014-08-21 2014-11-13 株式会社スクウェア・エニックス ビデオゲーム処理装置、及びビデオゲーム処理プログラム
US20200249747A1 (en) * 2017-05-23 2020-08-06 Visionary Vr, Inc. System and method for generating a virtual reality scene based on individual asynchronous motion capture recordings
JP2020195551A (ja) * 2019-05-31 2020-12-10 イマクリエイト株式会社 身体活動支援システム、方法、及び、プログラム

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
""For VRChat" The one to set the action recording avatar gimmick", BOOTH, pages 1 - 4, XP009543233, Retrieved from the Internet <URL:https://booth.pm/ja/items/2970849> [retrieved on 20230317] *
ANONYMOUS: "HoloLens realizes synchronous content distribution for many people ", DWANGO, 31 March 2017 (2017-03-31), XP093019242, [retrieved on 20230131] *

Also Published As

Publication number Publication date
US20240296642A1 (en) 2024-09-05
JPWO2023276215A1 (https=) 2023-01-05

Similar Documents

Publication Publication Date Title
US10701509B2 (en) Emulating spatial perception using virtual echolocation
RU2670784C2 (ru) Ориентация и визуализация виртуального объекта
CN111373347B (zh) 用于虚拟现实内容的提供的装置、方法和计算机程序
JP7095602B2 (ja) 情報処理装置、情報処理方法及び記録媒体
JP6102944B2 (ja) 表示制御装置、表示制御方法およびプログラム
JP2022544116A (ja) 複合現実における空間命令およびガイド
JP2020120336A (ja) プログラム、方法、および情報処理装置
US11151804B2 (en) Information processing device, information processing method, and program
JP6822410B2 (ja) 情報処理システム及び情報処理方法
CN111512370A (zh) 在录制的同时对视频作语音标记
CN108170153A (zh) 无人机飞行控制系统及其方法
EP3528024B1 (en) Information processing device, information processing method, and program
CN113678171A (zh) 信息处理装置、信息处理方法和记录介质
JP7400810B2 (ja) 情報処理装置、情報処理方法、及び記録媒体
WO2023276215A1 (ja) 情報処理装置、情報処理方法及びプログラム
US20250371827A1 (en) Information processing device, program, and information processing system
WO2021177186A1 (ja) 情報処理装置、情報処理方法および情報処理プログラム
JP7196856B2 (ja) 情報処理装置、情報処理方法、およびプログラム
JP7729381B2 (ja) 情報処理装置、情報処理方法およびプログラム
EP2746726A1 (en) System and method for tagging an audio signal to an object or a location; system and method of playing back a tagged audio signal
WO2019054037A1 (ja) 情報処理装置、情報処理方法、およびプログラム
WO2022269887A1 (ja) ウェアラブル端末装置、プログラムおよび画像処理方法
CN116048241B (zh) 一种提示方法、扩展现实设备及介质
WO2021075161A1 (ja) 情報処理装置、情報処理方法、及び情報処理プログラム
KR102807988B1 (ko) 웨어러블 장치 및 컨텐츠 제공 시스템

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22832383

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023531357

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22832383

Country of ref document: EP

Kind code of ref document: A1