WO2021153413A1 - 情報処理装置、情報処理システム及び情報処理方法 - Google Patents

情報処理装置、情報処理システム及び情報処理方法 Download PDF

Info

Publication number
WO2021153413A1
WO2021153413A1 PCT/JP2021/002042 JP2021002042W WO2021153413A1 WO 2021153413 A1 WO2021153413 A1 WO 2021153413A1 JP 2021002042 W JP2021002042 W JP 2021002042W WO 2021153413 A1 WO2021153413 A1 WO 2021153413A1
Authority
WO
WIPO (PCT)
Prior art keywords
viewer
information processing
artist
angle
processing device
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/002042
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
孝悌 清水
一真 高橋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Priority to EP21747621.7A priority Critical patent/EP4099275A4/en
Priority to US17/790,413 priority patent/US11907434B2/en
Priority to CN202180010474.2A priority patent/CN115004132A/zh
Priority to JP2021574692A priority patent/JP7544071B2/ja
Publication of WO2021153413A1 publication Critical patent/WO2021153413A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/0482—Interaction with lists of selectable items, e.g. menus
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/016—Input arrangements with force or tactile feedback as computer generated output to the user
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/243—Image signal generators using stereoscopic image cameras using three or more two-dimensional [2D] image sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H04N13/344—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays

Definitions

  • This disclosure relates to an information processing device, an information processing system, and an information processing method.
  • a person (user) such as an artist is imaged from multiple viewpoints, and a plurality of captured images are three-dimensionally combined to generate a live-action volumetric (stereoscopic image), which is projected or displayed at a remote location away from the person.
  • a live-action volumetric stereo image
  • an interactive experience there is a dialogue between an artist and a viewer (selected object) selected by the artist via a system (information processing system).
  • the system senses the selection operation of the artist to select the viewer with whom the artist interacts with the sensor, and detects the selected viewer based on the sensing data by the sensor. Then, the system communicates and connects the selected viewer-side device and the artist-side device to form a channel for dialogue.
  • OutlinAR an assisted interactive model building system with reduced computation information, Priced Bunnun, Walterio W. et al. Mayol-Cuevas, 2008 7th IEEE / ACM International Symposium on Mixed and Augmented Reality, IEEE
  • the present disclosure proposes an information processing device, an information processing system, and an information processing method capable of accurately selecting an object selected by a user's natural and intuitive selection operation.
  • the first detection unit includes a detection unit that detects an object selected by the user by a pointing gesture as a selection object, and the detecting unit indicates the position of the wrist of the user's hand corresponding to the pointing gesture. And the second reference point corresponding to the height of the wrist on the first vertical line perpendicular to the floor surface from the base of the user's arm corresponding to the pointing gesture.
  • An information processing apparatus is provided that detects the selected object based on the first extension line extending from one line segment toward the wrist side.
  • the detection unit includes a detection unit that detects an object selected by the user by the pointing gesture as a selection object, and the detection unit indicates the position of the wrist of the user's hand corresponding to the pointing gesture.
  • the first reference point is connected to the second reference point corresponding to the height of the wrist on the first vertical line perpendicular to the floor surface from the base of the user's arm corresponding to the pointing gesture.
  • An information processing system is provided that detects the selected object based on the first extension line extending from the first line segment toward the wrist side.
  • an information processing method for detecting an object selected by a user by a pointing gesture as a selected object, and the wrist of the user's hand corresponding to the pointing gesture by an information processing device.
  • a first reference point indicating the position of the wrist and a second reference point corresponding to the height of the wrist on the first vertical line perpendicular to the floor surface from the base of the user's arm corresponding to the pointing gesture.
  • An information processing method is provided that includes detecting the selected object based on a first extension line extending from the first line segment connecting the points toward the wrist side.
  • a plurality of components having substantially the same or similar functional configurations may be distinguished by adding different numbers after the same reference numerals. However, if it is not necessary to distinguish each of the plurality of components having substantially the same or similar functional configurations, only the same reference numerals are given. Further, similar components of different embodiments may be distinguished by adding different alphabets after the same reference numerals. However, if it is not necessary to distinguish each of the similar components, only the same reference numerals are given.
  • the virtual object means a virtual object (Virtual Object) that can be perceived by an artist (user) or a viewer as if it were a real object existing in the real space.
  • virtual objects can have various forms such as text, icons, animations, and the like.
  • the artist may be a virtual object including a CG (Computer Graphics) image representing the artist, a live-action volumetric image, or the like, or AR (AR). In Augmented Reality), it may be the artist himself who exists in the real space.
  • the viewer may be a virtual object including a live-action volumetric or the like in addition to the CG image representing the viewer, or may be the viewer himself / herself existing in the real space in AR.
  • FIG. 1 is a system diagram showing a schematic configuration of the interactive viewing system 1 according to the embodiment of the present disclosure.
  • the artist distributedor, user
  • the viewer may exist in different real spaces or in the same real space.
  • the viewer can see, for example, a live-action volumetric (three-dimensional image) displayed in the virtual space on the viewer side or a projection on the real space (in the following description, the "display” is "display”.
  • the interactive viewing system 1 senses the selection operation of the artist selecting a viewer (for example, a dialogue target) with a sensor, and the viewing selected by the artist based on the sensing data by the sensor. Detect a person. Then, the two-way viewing system 1 communicates and connects the selected viewer-side device and the artist-side device to form a channel for dialogue between the selected viewer and the artist.
  • the interactive viewing system 1 described here is an example of the information processing system used in the embodiment of the present disclosure, and the embodiment of the present disclosure is not limited to the use of such a system. ..
  • the content may be provided to the viewer when the artist and the viewer are physically separated from each other in a real space (for example, a studio and a room at home).
  • a real space for example, a studio and a room at home.
  • the artist and the viewer exist in the same real space (for example, a concert, an event, etc.)
  • the content may be provided to the viewer.
  • the information processing system is holding or wearing a device (for example, HMD (Head Mounted Display), smartphone, etc.) held or worn by the viewer selected by the artist, or a nearby device (for example, lighting, projector, speaker). Etc.) are expected to be controlled.
  • a device for example, HMD (Head Mounted Display), smartphone, etc.
  • a nearby device for example, lighting, projector, speaker. Etc.
  • VR Virtual Reality space sharing
  • the present invention is not limited to this, and when an artist and a viewer exist in the same real space and a CG image or the like is displayed in this real space (hereinafter, referred to as). It is possible to apply the technique according to the present disclosure to various systems capable of linking a virtual space and a real space in some way, such as AR space sharing).
  • VR space sharing it is not essential that the real space to which the artist and the viewer belong is physically separated, and the virtual space (hereinafter, also referred to as VR space) is created while both exist in the same real space. It may be shared.
  • AR space sharing it is not essential that the real space to which the artist and the viewer belong is the same, and an augmented space (hereinafter, also referred to as AR space) while each exists in a physically separated real space. May be shared.
  • the bidirectional viewing system 1 includes, for example, a distributor side server (information processing device) 100, a multi-camera system 130a, a distributor side display unit 160a, and a viewer side display.
  • a unit 260b, a viewer-side client 200, an HMD 260a, and a haptics device 260c can be included.
  • the distributor side server 100 and the viewer side client 200 can communicate with each other via various communication networks such as the Internet 300 and the like.
  • the number of each device included in the two-way viewing system 1 is not limited to the number shown in FIG. 1, and may be further increased. Further, the two-way viewing system 1 may include a device (not shown in FIG. 1).
  • the two-way viewing system 1 includes a general-purpose PC (Personal Computer), a tablet terminal, a game machine, a mobile phone such as a smartphone, a portable media player, a speaker, a projector, a display (digital signage, etc.), headphones, smart glasses, and the like. It can include a wearable device such as a smart watch, a robot (for example, a humanoid type robot or an automatic driving vehicle, etc.) and the like.
  • a wearable device such as a smart watch, a robot (for example, a humanoid type robot or an automatic driving vehicle, etc.) and the like.
  • the distributor side server 100 can acquire at least one sensing data (captured image, motion data, voice, position information, etc.) relating to one or a plurality of artists (users). Further, the distributor side server 100 processes the acquired sensing data to generate distribution content data (for example, live-action volumetric, CG image, audio data, etc.) and distributes it to the viewer side client 200 described later. be able to. Further, the distributor side server 100 distributes distribution data (for example, location information, etc.) based on at least one sensing data (for example, position information, captured image, motion data, voice, etc.) regarding one or a plurality of viewers (objects). Attitude information, audio data, etc.) can be acquired from the viewer side client 200. The detailed configuration of the distributor side server 100 will be described later.
  • the multi-camera system 130a includes a plurality of visible light cameras 132 (see FIG. 3) that capture an artist from multiple viewpoints, and obtains sensing data (for example, captured images) by imaging or the like on the distributor side server 100 described above. Can be output to.
  • the camera 132 may take an image of a real object or the like around or existing around the artist.
  • the multi-camera system 130a includes not only a plurality of visible light cameras 132 that image an artist, but also an infrared (IR) camera 134a that images a marker 134b (see FIG. 3) attached to a part of the artist's body. It may include a depth sensor 136 (see FIG. 3) that senses the posture of the artist's body.
  • IR infrared
  • the multi-camera system 130a may include a plurality of inertial sensors (IMUs) (not shown) mounted on a part of the artist's body.
  • IMUs inertial sensors
  • Each IMU includes an acceleration sensor (not shown) and an angular velocity sensor (not shown), and can sense the posture and movement of the artist from the obtained sensing data.
  • the multi-camera system 130a may include a geomagnetic sensor (not shown), a barometric pressure sensor (not shown), and the like as motion sensors in addition to the IMU.
  • the multi-camera system 130a includes a position sensor (not shown) that senses the user's position, a sound sensor (not shown) that senses the artist's voice and the ambient sound around the artist, and a line-of-sight sensor that senses the artist's line of sight.
  • a position sensor (not shown) that senses the user's position
  • a sound sensor (not shown) that senses the artist's voice and the ambient sound around the artist
  • a line-of-sight sensor that senses the artist's line of sight.
  • Biometric information sensor that senses the artist's biometric information (not shown) (For example, heartbeat, pulse, brain wave, respiration, sweating, myoelectric potential, skin temperature, skin electrical resistance, eye movement, or pupil diameter, etc. It may include an input device (not shown) that receives a text or the like input by an artist, and is not particularly limited.
  • the above-mentioned position sensor is a sensor that senses the position of an artist, and can be a GNSS (Global Navigation Satellite System) receiver or the like.
  • the position sensor acquires sensing data indicating the latitude / longitude information (position information in the global coordinate system) of the artist's current location based on the signal from the GNSS satellite.
  • RFID Radio Frequency Identification
  • Wi-Fi access points, etc. can sense the relative positional relationship of users from information of radio base stations, etc., and therefore, in the present embodiment, these are used.
  • a communication device such as the above may be used as a position sensor.
  • the detailed configuration of the multi-camera system 130a will be described later.
  • the distributor side display unit 160a is, for example, a tablet (not shown) or a smartphone (not shown) held by the artist, or a projector installed in the real space where the artist exists, which is communicably connected to the distributor side server 100 (not shown). It is realized by a display (not shown), a display (not shown), and the like.
  • the distributor side display unit 160a can display text, an icon, an animation, or the like in the real space where the artist exists.
  • the distributor side display unit 160a can display images of a plurality of viewers existing in a space different from the artist in the form of virtual objects such as avatars (CG images) in the real space where the artist exists. ..
  • the relative positional relationship between the artist and the viewer's image in the virtual space expressed on the artist's side (which may be substantially equivalent to the real space; the same applies hereinafter) is the virtual space on the viewer's side.
  • the image of the viewer will be displayed so as to correspond to the relative positional relationship between the image of the artist displayed in (for example, live-action volumetric) and each viewer.
  • the viewer-side client 200 acquires the distribution content data acquired from the distributor-side server 100 described above, and delivers the acquired distribution content data to the viewer via the HMD 260a, the viewer-side display unit 260b, and the haptics device 260c.
  • the HMD260a is basically a device intended for VR space sharing, but can also be used for AR space sharing.
  • the viewer side display unit 260b is basically a device intended for AR space sharing, but can also be used for VR space sharing.
  • the viewer side client 200 may acquire at least one sensing data (captured image, motion data, sound, position information, etc.) regarding one or a plurality of viewers and transmit it to the distributor side server 100. can.
  • the detailed configuration of the viewer side client 200 will be described later.
  • the HMD 260a can be communicatively connected to the viewer side client 200 and can display an image in the virtual space based on the distributed content data. For example, the HMD 260a can display the artist's live-action volumetric in virtual space. At this time, the relative positional relationship between the image of the artist displayed in the virtual space on the listener side and the viewer is the relative positional relationship between the image of the viewer and the artist in the virtual space expressed on the artist side. It is preferable to correspond.
  • the HMD 260a can be realized by, for example, a spectacle-shaped HMD worn on the viewer's head as shown in FIG.
  • the display unit (not shown) corresponding to the spectacle lens portion located in front of the viewer's eyes at the time of wearing is a transmissive type capable of visually recognizing the outside of the spectacle lens portion, or visually recognizes the outside of the spectacle lens portion. It is a non-transparent display that cannot display virtual objects.
  • the HMD 260a is attached to the head of the viewer and is communicably connected to the client 200 on the viewer side, but the present embodiment is limited to this. It's not a thing.
  • the HMD 260a may be attached to the head of the artist and connected to the distributor side server 100 in a communicable manner.
  • the HMD260a can also be a display device for realizing AR space sharing.
  • the HMD260a is a non-transparent display
  • the non-transparent display captures an image of the real space in which the viewer exists, which is captured by a camera (not shown) that captures the facial expression of the viewer and the surroundings of the viewer. It can also be displayed on (video see-through).
  • the HMD 260a can display the live-action volumetric of the artist, for example, in the real space where the viewer exists.
  • the HMD260a includes a speaker (not shown), an IMU (not shown), a geomagnetic sensor (not shown), a barometric pressure sensor (not shown), a position sensor (not shown), a sound sensor (not shown), and a line-of-sight sensor (not shown). ), A biometric information sensor (not shown), and the like may be included. An example of the appearance of the HMD 260a will be described later.
  • the interactive viewing system 1 may include a wearable device of another form (not shown) in addition to the HMD260a, which is a glasses-type wearable device worn on the head.
  • a wearable device include various types of wearable devices such as ear device (headphone) type, anklet type, bracelet type, collar type, glove type, pad type, batch type, and clothing type.
  • the viewer-side display unit 260b is connected to the viewer-side client 200 in a communicable manner, and is connected to a tablet (not shown) or a smartphone (not shown) held by the viewer, or a projector installed in a real space where the viewer exists (not shown). It is realized by a display (not shown), a display (not shown), and the like.
  • the viewer side display unit 260b can display text, an icon, an animation, or the like in the real space in which the viewer exists.
  • the viewer side display unit 260b can display the image of the artist in the real space where the viewer exists.
  • the relative positional relationship between the artist's image and the viewer in the virtual space expressed by the listener is the relative positional relationship between the viewer's image and the artist in the virtual space expressed by the artist. It is preferable to correspond.
  • the haptics device 260c is communicably connected to the viewer side client 200, is held by the viewer or is attached to a part of the viewer's body, and vibrates according to the distributed content data to give the viewer a tactile stimulus. Can be given.
  • the haptics device 260c can include a vibration device (not shown) that vibrates the viewer, an electrical stimulator (not shown) that electrically stimulates the viewer, and the like.
  • the haptics device 260c includes an IMU (not shown), a geomagnetic sensor (not shown), a barometric pressure sensor (not shown), a position sensor (not shown), a sound sensor (not shown), a biometric information sensor (not shown), and the like. May include.
  • Such a device for giving a tactile stimulus to the viewer may be included in the above-mentioned HMD260a, various wearable devices, a sart phone held by the viewer, and the like.
  • An example of the appearance of the haptics device 260c will be described later.
  • FIG. 2 is a diagram showing a functional configuration of the distributor side server 100 according to the embodiment of the present disclosure.
  • the distributor side server 100 acquires at least one sensing data related to the artist, processes the acquired sensing data to generate distribution content data, and is on the viewer side. It can be delivered to the client 200. As shown in FIG.
  • the distributor side server 100 includes a detection unit 100a, a distributor side data acquisition unit 102, an image data generation unit 116, a video streaming processing unit 118, a body position information generation unit 120, and the like. It mainly has a 3D model position information generation unit 122, a content data generation unit 124, and a distribution unit 126.
  • the detection unit 100a will be described later because it is a main part of the embodiment of the present disclosure.
  • the distributor side data acquisition unit 102 acquires sensing data (for example, captured image, motion data, voice, position information, etc.) about the artist from the multi-camera system 130a or the like, and has a detection unit 100a and an image data generation unit 116 which will be described later. And can be output to the body position information generation unit 120.
  • the distributor side data acquisition unit 102 acquires an image captured by an artist from multiple viewpoints from, for example, the multi-camera system 130a, and outputs the captured image to the image data generation unit 116.
  • the distributor side data acquisition unit 102 acquires a captured image showing the artist's posture and movement from the multi-camera system 130a and sensing data on the artist's posture and movement from the IMU (not shown), and generates body position information. Output to unit 120.
  • the distributor side server 100 includes at least one sensing data (for example, captured image, motion data, voice, position information, etc.) relating to one or a plurality of viewers, and the like. Including the viewer side data acquisition unit that acquires the distribution data generated based on the above from the viewer side client 200, the viewer side data acquisition unit outputs the acquired data to the image data generation unit 116 described later. Can be done.
  • the image data generation unit 116 generates the image acquired from the multi-view camera of the multi-camera system 130a as display data. Further, the image data generation unit 116 can output the generated display data to the video streaming processing unit 118, which will be described later.
  • the performer image for example, live-action volumetric
  • CG image for example, avatar
  • AR space sharing the performer image + CG image
  • the CG image is displayed from the distributor side server 100 to the viewer side client 200. Is transmitted to.
  • the video streaming processing unit 118 can perform processing for streaming distribution of the generated display data and output it to the content data generation unit 124 described later.
  • the body position information generation unit 120 is based on an captured image that shows the posture and movement of the artist acquired from the multi-camera system 130a and sensing data related to the posture and movement of the artist acquired from a plurality of IMUs (not shown). Information and posture information can be generated. Further, the body position information generation unit 120 can output the generated position information and posture information of the artist to the 3D model position information generation unit 122, which will be described later.
  • the 3D model position information generation unit 122 can generate 3D model position information including skeleton information expressing the position and posture of the artist's body by a skeleton structure based on the position information and the posture information. Then, the 3D model position information generation unit 122 can output the generated 3D model position information to the content data generation unit 124, which will be described later.
  • the skeleton structure contains information about each part of the artist's body and bones, which are line segments connecting the parts.
  • the part in the skeleton structure corresponds to, for example, a terminal part of the body, a joint part, or the like.
  • bones in a skeleton structure can correspond to, for example, human bones, but the positions and numbers of bones do not necessarily have to be consistent with the actual human skeleton.
  • the content data generation unit 124 generates distribution content data (for example, combining the artist image of the stereoscopic image data with skeleton information) based on the stereoscopic image data and the 3D model position information, and outputs the distribution content data to the distribution unit 126 described later. can do.
  • the distribution unit 126 can distribute the distribution content data to the viewer side client 200.
  • each functional block included in the distributor side server 100 is not limited to the functional block shown in FIG.
  • FIG. 3 is a diagram showing a configuration of the multi-camera system 130a according to the embodiment of the present disclosure.
  • the multi-camera system 130a according to the present embodiment includes a plurality of visible light cameras 132 and the like that capture the artist 600 from multiple viewpoints, and the image captured by the camera 132 is the distributor described above. It can be output to the side server 100.
  • a plurality of visible light cameras 132 are installed so as to surround the artist 600, and these cameras 132 image the artist 600 from different angles. Therefore, a three-dimensional image of the artist 600 can be obtained by synthesizing the captured images acquired by each camera 132 in consideration of the relative positional relationship of the plurality of cameras 132. Further, in the present embodiment, images of each part of the body of the artist 600 are extracted from each captured image, and processing is performed based on the position of each camera 132 to perform position information and posture information of each part of the artist 600. Can be obtained.
  • the multi-camera system 130a may include not only a plurality of visible light cameras 132 but also an IR camera 134a that captures a marker 134b attached to a part of the body of the artist 600. Often, a depth sensor 136 that senses the posture of the artist 600 may be included.
  • the artist 600 is placed in the real space.
  • a local coordinate system is set based on the origin virtually provided in.
  • the position of each camera 132, the position of each part of the body of the artist 600, and the floor surface on which the artist 600 stands can also be expressed by the local coordinate system.
  • the multi-camera system 130a may include a plurality of IMUs (not shown) attached to a part of the body of the artist 600, and the IMU may include an acceleration sensor (not shown) or an acceleration sensor (not shown). , Angular velocity sensor (not shown) can be included.
  • the position information and the posture information of the position of the mounting part of each IMU are generated based on the sensing data regarding the posture and movement of the artist acquired from the plurality of IMUs, and further, the position of the body of the artist 600 and the body position and the posture information of the artist 600 are generated. It is possible to generate 3D model position information including skeleton information that expresses the posture by a skeleton structure.
  • the position of the IMU mounting site can be calculated from the sensing data from the IMU, and the forward movement is performed based on the position information of the IMU and the like.
  • Skeleton information can be obtained by scholarly (FK: Forward Kinetics) calculation.
  • the forward kinematics calculation is a method of calculating the position of the terminal portion based on the posture of each joint portion.
  • skeleton information may be acquired by using inverse kinematics (IK) calculation instead of the forward kinematics calculation described above.
  • IK inverse kinematics
  • Inverse kinematics calculation is a method of calculating the posture of each joint part based on the position of the terminal part.
  • each device included in the multi-camera system 130a is not limited to the device shown in FIG.
  • FIG. 4 is a diagram showing a functional configuration of the viewer side client 200 according to the embodiment of the present disclosure.
  • the viewer-side client 200 according to the present embodiment acquires the distribution content data acquired from the distributor-side server 100, and the acquired distribution content data is used in the HMD 260a, the viewer-side display unit 260b, and the viewer-side display unit 260b. It can be provided to the viewer via a haptics device 260c or the like. As shown in FIG.
  • the viewer-side client 200 includes a distribution data acquisition unit 202, a content processing unit 204, a timing synchronization unit 206 and 212, an output control unit 208, and a viewer-side data acquisition unit 210. It mainly has an output data processing unit 214 and a distribution unit 216.
  • each functional block of the viewer-side client 200 will be sequentially described.
  • the distribution data acquisition unit 202 can acquire the distribution content data from the distributor side server 100 and output it to the content processing unit 204 described later.
  • the content processing unit 204 performs processing for providing the acquired distribution content data to the viewer via the HMD 260a, the viewer side display unit 260b, the haptics device 260c, etc., and outputs the acquired distribution content data to the timing synchronization unit 206 described later. be able to.
  • Timing synchronization units 206, 212 The timing synchronization unit 206 synchronizes the output timing of each distribution content data that has been processed to be provided to the viewer via the HMD 260a, the viewer side display unit 260b, the haptics device 260c, and the like, and is an output control unit described later. It can be output to 208. Further, the timing synchronization unit 212 can synchronize a plurality of sensing data from the viewer side data acquisition unit 210, which will be described later, and output the sensing data to the output data processing unit 214, which will be described later.
  • the output control unit 208 can control the HMD 260a, the viewer side display unit 260b, the haptics device 260c, and the like based on each distribution content data processed by the timing synchronization unit 206.
  • the viewer side data acquisition unit 210 acquires at least one sensing data (for example, captured image, motion data, sound, position information, etc.) regarding one or a plurality of viewers and outputs the data to the timing synchronization unit 212. Can be done.
  • sensing data for example, captured image, motion data, sound, position information, etc.
  • the output data processing unit 214 can perform processing for distributing the sensing data from the timing synchronization unit 212 to the distributor side server 100, and output the sensing data to the distribution unit 216, which will be described later.
  • the distribution unit 216 can distribute the sensing data processed by the output data processing unit 214 to the distributor side server 100.
  • each functional block included in the viewer side client 200 is not limited to the functional block shown in FIG.
  • FIG. 5 is a diagram showing an external configuration of the HMD 260a according to the embodiment of the present disclosure.
  • the HMD 260a according to the present embodiment can display an image in the virtual space according to the distribution content data or the like.
  • the HMD 260a is realized by, for example, a spectacle-shaped HMD worn on the viewer's head.
  • the display unit (not shown) corresponding to the spectacle lens portion located in front of the viewer's eyes when worn is a transmissive type capable of visually recognizing the outside of the spectacle lens portion, or visually recognizes the outside of the spectacle lens portion. It may be a non-transparent display that cannot be used.
  • the display unit can display an image such as a text or a figure in the through state, that is, even if a virtual object is displayed in the real space as Augmented Reality (AR). good.
  • AR Augmented Reality
  • a transmissive display for example, a half mirror or a transparent light guide plate is used to hold a virtual image optical system composed of a transparent light guide portion or the like in front of the viewer, and a virtual object is formed inside the virtual image optical system. Is displayed.
  • the HMD 260a is not limited to the form of displaying the virtual object to both eyes of the viewer, but the form of displaying the virtual object only to one eye of the viewer. You may have it.
  • the display unit may be realized as an LED (Light Emitting Diode) light source or the like that directly projects an image onto the retina of the viewer. That is, the HMD 260a may be realized as a projection type HMD.
  • the form of the HMD 260a is not limited to the example shown in FIG. 5, and may be, for example, a helmet type HMD (for example, the visor portion of the helmet corresponds to the display). ..
  • the HMD 260a can include a camera (not shown), a speaker (not shown), a position sensor (not shown), and the like that capture the facial expression of the viewer and the surroundings of the viewer.
  • the local coordinate system is based on the origin virtually provided in the real space on the viewer side. May be set. Furthermore, the position of the image of the artist 600 (for example, live-action volumetric) displayed in the virtual space on the viewer side can also be expressed by the local coordinate system. Further, the relative positional relationship between the viewer and the image of the artist 600 in the local coordinate system on the viewer side in the real space is the image of the artist 600 and the viewer in the local coordinate system on the artist 600 side described above. Corresponds to the relative positional relationship with (for example, match, similarity, etc.).
  • the local coordinate system set in the real space on the artist side, the local coordinate system set in the real space on the viewer side, and the coordinate system in the virtual space shared by the artist 600 and the viewer 610 are matched.
  • the positional relationship between the artist 600 and the viewer 610 on the artist side can be reproduced as it is on the viewer side.
  • FIG. 6 is a diagram showing an external configuration of the haptics device 260c according to the embodiment of the present disclosure.
  • the haptics device 260c is a device that gives a tactile stimulus to the viewer by being held by the viewer or attached to a part of the viewer's body and vibrating or the like according to the distributed content data.
  • the haptics device 260c can be a controller-type haptics device as shown in FIG. Specifically, a vibrating device (not shown) is provided in the cylindrical grip portion of the haptics device 260c, and the vibrating device vibrates according to the distributed content data or the like to grip the grip portion for viewing. Can give vibration to a person.
  • the haptics device 260c can include an electrical stimulator (not shown) that electrically stimulates the viewer.
  • the haptics device 260c may include an irradiation device that emits light according to the distribution content data or the like.
  • the embodiment of the haptics device 260c is not limited to the example shown in FIG. 6, and may be, for example, the best-shaped haptics device worn on the body of the viewer. ..
  • a method of providing content in which a live-action volumetric of an artist 600 is generated and a large number of viewers can view the live-action volumetric displayed at a remote location away from the artist 600.
  • An example of an interactive experience is a dialogue between the artist 600 and a viewer selected by the artist 600.
  • the interactive viewing system 1 that provides the above content senses the selection operation of the artist 600 to select a viewer to interact with with a sensor, and the selected viewer is selected based on the sensing data by the sensor. Is detected. Then, the two-way viewing system 1 communicates and connects the selected device on the viewer side and the device on the artist 600 side to form a channel for dialogue.
  • the present inventors have been diligently studying such an interactive viewing system 1, but in the process, in order for the interactive viewing system 1 to be less burdensome and comfortable for the artist 600.
  • the selection operation is preferably a natural and intuitive operation (Natural User Interface: NUI) for the artist 600.
  • NUI Natural User Interface
  • the interactive viewing system 1 that has been studied so far detects, for example, a selection operation in which the artist 600 turns his face or chest toward the selected viewer.
  • the two-way viewing system 1 senses the face orientation and chest orientation of the artist 600 with a sensor, and detects the viewer selected by the artist 600 based on the sensed orientation.
  • the detection surface is wide, and it is easy to sense these directions with a sensor.
  • the interactive viewing system 1 is an artist. For the 600, it was not a light and comfortable one.
  • a pointing gesture pointing at the selected viewer can be mentioned. Since such an operation is a daily operation, it is possible to avoid putting a burden on the arm of the artist 600.
  • the present inventors sense the direction indicated by the fingertip in the pointing gesture of the artist 600 with a sensor, and detect the viewer selected by the artist 600 based on the sensed direction.
  • the pointing gesture is a natural and intuitive operation, the artist 600 is not burdened.
  • the detection surface is narrow, and it is difficult for the sensor to accurately sense the direction.
  • the pointing gesture is a human movement, it is difficult to fix the posture of the fingertip during a predetermined time, and therefore it is difficult to accurately sense the direction of the fingertip by the sensor.
  • the present inventors can accurately select the viewer selected by the natural and intuitive selection operation by the artist 600, as described below in the present disclosure. It came to create an embodiment.
  • FIG. 7 is an explanatory diagram for explaining the outline of the embodiment of the present disclosure.
  • a pointing gesture of pointing at the selected viewer is adopted as the selection operation of the artist 600.
  • a pointing gesture of pointing at the selected viewer is adopted as the selection operation of the artist 600.
  • such an operation is an operation performed to point to an object or the like selected in daily life, and is a natural and intuitive operation, which imposes a burden on the artist 600. It does not take. Further, such an operation can be said to be a preferable operation because the meaning can be intuitively grasped from the viewer and the like.
  • the gesture is not limited to the above-mentioned pointing gesture, and may be another gesture, such as a predetermined movement by the body of the artist 600 (for example, dance, body language, etc.). ), Or an aerial painting operation or the like that virtually draws a trajectory in the air.
  • the distributor side server 100 of the interactive viewing system 1 can detect the viewer selected by the artist 600 as follows.
  • the distributor-side server 100 has a detection unit 100a (see FIG. 8) that detects a viewer selected by the artist 600 by a pointing gesture.
  • the detection unit 100a has a reference point (first reference point) A indicating the position of the wrist of the hand 602 of the artist (user) 600 corresponding to the pointing gesture, and the pointing gesture.
  • Reference point (second reference point) B corresponding to the height of the wrist on the vertical line (first vertical line) 500 perpendicular to the floor surface 700 from the base (reference point C) of the arm 604 of the corresponding artist 600.
  • the viewer (object) is detected as a selected object based on the extension line (first extension line) 520 extending from the line segment (first line segment) 510 connecting the above to the wrist side. More specifically, the detection unit 100a detects a viewer (object) corresponding to a point on the extension line (first extension line) 520 as a selection object.
  • the relative positional relationship between the artist 600 and the image of the viewer in the virtual space represented by the artist 600 is the image of the viewer and the artist 600 in the virtual space displayed on the viewer side. It is assumed that it corresponds to the relative positional relationship with.
  • the viewer 600 selected by the pointing gesture can be selected.
  • the reference point A indicating the position of the wrist of the hand 602 of the artist 600 and the reference point C indicating the position of the base of the arm 604 of the artist 600 corresponding to the pointing gesture are sensors. Since it can be easily detected by the artist 600, the viewer selected by the artist 600 can be easily selected by the method described above. Further, in the embodiment of the present disclosure, as the selection operation of the artist 600, a pointing gesture that points to the selected viewer is adopted, and the pointing gesture is a daily routine as described above.
  • the artist 600 is not burdened. That is, according to the embodiment of the present disclosure, the viewer selected by the artist 600 by the natural and intuitive selection operation can be selected with high accuracy.
  • the interactive viewing system 1 makes it easy for the viewer to have an interactive communication experience in which the artist 600 and a specific viewer feel as if they are in the same space and are directly interacting with each other. It becomes possible to provide, for example, the viewer can feel a stronger sense of unity with the artist 600.
  • the selected object is a viewer (human) existing around the live-action volumetric of the displayed artist 600, but in the embodiment of the present disclosure, the object is limited to this. It is not something that is done.
  • the selected object may be a viewer (human) that exists around the artist 600, or an animal or still life (book, stationery, painting, photograph) that exists around the artist 600 or the image of the artist 600.
  • the object selected selected.
  • the artist 600 and the selected object may be located in the same real space on the premise that they are located in the shared virtual space, and may be located in different real spaces. It may be located in.
  • the positions of the reference point A and the reference point C may be detected from the captured image obtained by the multi-camera system 130a, and is attached to a part of the body of the artist 600. It may be detected from the image captured by the IR camera 134a that captures the marker 134b, or may be detected from the sensing data by the depth sensor 136 that detects the posture of the artist's body. Further, in the present embodiment, the positions of the reference point A and the reference point C are a plurality of IMUs (not shown), a geomagnetic sensor (not shown), and a barometric pressure sensor (not shown) attached to a part of the body of the artist 600. ) Etc. may be detected from the sensing data obtained. The details of each of the embodiments of the present disclosure will be sequentially described below.
  • FIG. 8 is a diagram showing a functional configuration of the detection unit 100a according to the first embodiment
  • FIGS. 9 to 11 are explanatory views for explaining the operation of the determination unit 112 according to the first embodiment. be.
  • the detection unit 100a determines that the distributor side data acquisition unit 102, the gesture recognition unit 104, the laser point output control unit m, the viewer side data acquisition unit 108, and the detection unit 110. It mainly has a unit 112 and an object output control unit 114. Note that FIG. 8 also shows a distributor-side data acquisition device 130, a distributor-side output device 160, a viewer-side data acquisition device 230, and a viewer-side output device 260.
  • the data acquisition device 130 includes the above-mentioned multi-camera system 130a and the like
  • the distributor side output device 160 includes the above-mentioned distributor side display unit 160a and the like.
  • the viewer side data acquisition device 230 and the viewer side output device 260 include the above-mentioned HMD 260a, haptics device 260c, and the like.
  • each functional block of the detection unit 100a will be sequentially described.
  • the distributor side data acquisition unit 102 acquires at least one sensing data (for example, captured image, motion data, voice, position information, etc.) regarding one or a plurality of artists 600 from the multi-camera system 130a or the like, and will be described later. It can be output to the gesture recognition unit 104.
  • sensing data for example, captured image, motion data, voice, position information, etc.
  • the gesture recognition unit 104 can recognize the pointing action in which the artist 600 selects a viewer by a pointing gesture.
  • the gesture recognition unit 104 has, for example, a reference point (third reference point) C indicating the position of the base of the arm 604 of the artist 600 corresponding to the pointing gesture, and the position of the wrist of the hand 602 of the artist 600.
  • a line segment (second line segment) 512 (see FIG. 9) connecting the reference point (first reference point) A indicating the above, and a perpendicular line drawn vertically from the reference point C to the floor surface 700 (first).
  • the pointing action is recognized when the angle between the line segment 512 and the perpendicular line 500 is within a predetermined range (for example, 20 ° or more) on the plane including the perpendicular line (500). Then, the gesture recognition unit 104 outputs the recognition result to the laser point output control unit 106 and the detection unit 110, which will be described later.
  • a predetermined range for example, 20 ° or more
  • the laser point output control unit 106 sets the laser point and the beam arrow so that the artist 600 can easily see the destination pointed by the pointing gesture and the selection destination can be clearly indicated by the pointing gesture more stably. It can be displayed. Specifically, the laser point output control unit 106 can display a beam arrow extending from the fingertip to a predetermined distance along the fingertip of the pointing gesture in the real space where the artist 600 exists. Similarly, the laser point output control unit 106 sets the laser point in the real space where the artist 600 exists, at a position separated by a predetermined distance from the fingertip of the pointing gesture along the fingertip of the pointing gesture. Can be displayed.
  • the laser point output control unit 106 is installed in a tablet (not shown) or a smartphone (not shown) held by the artist 600, an HMD (not shown) worn by the artist 600, or in a real space where the artist 600 exists.
  • the display is performed by controlling the projector (not shown) or the like.
  • the viewer-side data acquisition unit 108 processes at least one sensing data (for example, captured image, motion data, sound, position information, etc.) or these related to one or a plurality of viewers from the viewer-side client 200.
  • the obtained distribution data can be acquired and output to the detection unit 110 described later.
  • the detection unit 110 can detect the viewer selected by the artist 600 by the pointing gesture as a selection object.
  • the detection unit 110 includes a reference point (first reference point) A indicating the position of the wrist of the hand 602 of the artist 600 corresponding to the pointing gesture, and an arm 604 of the artist 600 corresponding to the pointing gesture.
  • the selected viewer can be detected based on the extension line (first extension line) 520 extending from the line segment) 510 toward the wrist side. More specifically, the detection unit 110 can detect the viewer corresponding to the point on the extension line 520 as a selection object. Then, the detection unit 110 can output the detection result to the determination unit 112, which will be described later.
  • the determination unit 112 can determine whether the pointing action of the artist 600 selecting a viewer by a pointing gesture satisfies a predetermined stability condition. Then, the determination unit 112 can output the determination result to the anti-object output control unit 114, which will be described later.
  • the determination unit 112 has a reference point (third reference point) C indicating the position of the base of the arm 604 of the artist 600 corresponding to the pointing gesture, and the wrist of the hand 602 of the artist 600.
  • the determination unit 112 acquires the angle ⁇ of the angle a predetermined number of times within a predetermined time (for example, 1 second), calculates the variance ⁇ 2 based on the acquired value, and further calculates the standard deviation. If ⁇ is within a predetermined value, it is determined that the above pointing action satisfies the stability condition.
  • the variance ⁇ 2 and the standard deviation ⁇ are calculated according to the following mathematical formula (1).
  • the determination unit 112 of the artist 600 corresponding to the pointing gesture from the space origin O (virtually set in the real space where the artist 600 exists).
  • the determination unit 112 acquires the distance D a predetermined number of times within a predetermined time, calculates the variance ⁇ 2 based on the acquired value, and further calculates the standard deviation ⁇ within the predetermined value. If so, it is determined that the pointing action satisfies the stability condition.
  • the variance ⁇ 2 and the standard deviation ⁇ can be calculated according to the following mathematical formula (1).
  • the pointing gesture of the artist 600 stably indicates the selected viewer as intended at a predetermined time.
  • the pointing gesture of the artist 600 by statistically determining whether or not the pointing gesture of the artist 600 is stable, it is possible to avoid erroneously detecting the pointing gesture and the destination pointed to by the pointing gesture, and the detection robustness. You can improve your sex.
  • the determination is not limited to determining whether the stability condition is satisfied by using the variance ⁇ 2 and the standard deviation ⁇ , and the determination is made using other indexes or other statistical values. May be good.
  • the object output control unit 114 communicates with, for example, a plurality of viewer-side devices (for example, a smartphone, a wearable device, etc.) and an artist 600-side device (for example, a smartphone, a wearable device, etc.) and is selected.
  • a device on the viewer side and a device on the artist 600 side can be interactively connected.
  • the object output control unit 114 blinks, outputs audio, etc. to the viewer side device, a nearby device (for example, lighting, a projector, a speaker, etc.), etc., depending on the selected viewer. It can be controlled to perform a predetermined operation.
  • the anti-object output control unit 114 may display a virtual object (predetermined image) having various forms such as text, an icon, an animation, etc. in the virtual space displayed on the HMD 260a of the selected viewer. (VR space sharing).
  • the anti-object output control unit 114 controls a projector (not shown) installed in the real space where the selected viewer exists, and gives the viewer various forms such as text, icons, and animation.
  • the virtual object (predetermined image) to be held may be superimposed and displayed (AR space sharing).
  • the anti-object output control unit 114 erases, enlarges, reduces, changes the image displayed on the display (not shown) as the selected object, or changes the content such as text or image. Controls such as playback and stop may be performed. Further, the object output control unit 114 may, for example, play back and stop the sound of the speaker (not shown), or change the direction of the sound output of the speaker. Further, the anti-object output control unit 114 may, for example, operate a robot (not shown) as a selection object.
  • each functional block included in the detection unit 100a is not limited to the functional block shown in FIG.
  • FIG. 12 is a flowchart of an information processing method according to the present embodiment
  • FIG. 13 is an explanatory diagram for explaining the present embodiment.
  • the information processing method according to the present embodiment includes a plurality of steps from step S101 to step S107. The details of each step included in the information processing method according to the present embodiment will be described below.
  • the detection unit 100a determines whether or not the artist 600 is performing a pointing action of selecting a viewer by a pointing gesture based on at least one sensing data regarding one or a plurality of artists 600 (step S101). .. The detection unit 100a proceeds to step S102 when the artist 600 is performing the pointing gesture (step S101: Yes), and when the artist 600 is not performing the pointing gesture (step S101: No), the detection unit 100a proceeds to step S102. The process returns to the process of step S101 described above.
  • the detection unit 100a displays a laser point and a beam arrow so that the artist 600 can easily see the destination pointed by the pointing gesture and the selection destination can be more stably specified by the pointing gesture ( Step S102).
  • the detection unit 100a determines whether or not the viewer (object) 610 corresponding to the point on the extension line 520 exists (step S103). For example, in the present embodiment, the detection unit 100a has a relative positional relationship between the artist 600 and the image of the viewer 610 in the virtual space represented by the artist 600 based on the position information of the artist 600 and the viewer 610. To get. Then, the detection unit 100a determines whether or not the viewer (or the coordinates corresponding to the viewer) 610 exists on the extension line 520, as shown in FIG. 13, based on the relative positional relationship. can do.
  • step S103: Yes when the viewer 610b corresponding to the point on the extension line 520 exists (step S103: Yes), the detection unit 100a proceeds to step S104, and the viewer 610 corresponding to the point on the extension line 520 If it does not exist (step S103: No), the process returns to the process of step S101 described above.
  • the detection unit 100a controls the HMD260a or the like worn by the selected viewer (object) 610 and continues to display the laser point (not shown) or the like (step S104).
  • the detection unit 100a determines whether the pointing action of the artist 600 selecting the viewer 610 by the pointing gesture satisfies a predetermined stability condition (step S105). Then, when the detection unit 100a satisfies the predetermined stabilization condition (step S105: Yes), the process proceeds to step S106, and when the predetermined stabilization condition is not satisfied (step S105: No), the detection unit 100a is described above. Return to the process of step S101.
  • the detection unit 100a detects the viewer (object) 610 selected by the artist 600 by the pointing gesture as the selected object (step S106).
  • the detection unit 100a interactively connects the selected device on the viewer 610 side and the device on the artist 600 side. Further, the detection unit 100a controls the HMD 260a or the like worn by each viewer 610, superimposes it on the selected viewer 610, and displays a virtual object (step S107). Then, the detection unit 100a returns to the process of step S101 described above.
  • the viewer 610 selected by the artist 600 by the pointing gesture is detected.
  • the reference point A indicating the position of the wrist of the hand 602 of the artist 600 and the reference point C indicating the position of the base of the arm 604 of the artist 600 corresponding to the pointing gesture are Since it can be easily detected, the viewer 610 selected by the artist 600 can be easily selected by the method described above.
  • a pointing gesture that points to the selected viewer 610 is adopted, and the pointing gesture is used in daily life as described above.
  • the viewer 610 selected by the artist 600 is detected based on whether or not there is a viewer 610 corresponding to the point on the extension line 520.
  • the detection is not limited to such a method, and other methods can be used. Therefore, in the following, with reference to FIGS. 14 and 15, a method for detecting the selected viewer 610 used in another method will be described as a modification 1 of the present embodiment.
  • FIG. 14 is an explanatory diagram for explaining a modification 1 of the present embodiment
  • FIG. 15 is a flowchart of an information processing method according to the modification 1 of the present embodiment.
  • the detection unit 100a has a reference point (first reference point) A indicating the position of the wrist of the hand 602 of the artist (user) 600 corresponding to the pointing gesture. And the reference point (first) corresponding to the height of the wrist on the vertical line (first vertical line) 500 perpendicular to the floor surface 700 from the base (reference point C) of the arm 604 of the artist 600 corresponding to the pointing gesture.
  • the extension line (first extension line) 520 extending toward the wrist side from the line segment (first line segment) 510 connecting the reference point (2) B is acquired.
  • the above-mentioned collider is virtually set in the real space where the viewer 610 exists, and the viewer 610 located in the collider is detected, thereby being accurate and accurate.
  • the viewer 610 selected by the artist 600 can be easily selected without omission.
  • the information processing method according to this modification includes a plurality of steps from step S201 to step S208.
  • Steps S202 and S203 of this modification perform the same processing as the processing in steps S101 and S102 in the first embodiment shown in FIG.
  • step S205 to step S208 of this modification the same processing as the processing in steps S104 to S107 in the first embodiment shown in FIG. 12 is performed. Therefore, in the following description, only steps S201 and S204 different from the above-described first embodiment will be described, and the description of other steps common to the first embodiment will be omitted.
  • the detection unit 100a is a collider in which the detection mode of the selected object (viewer) 610 is virtually installed on the perpendicular line 502 drawn vertically from the point located on the extension line 520 to the floor surface 700 as described above. Is set to the mode using (step S201).
  • the detection unit 100a determines whether or not the viewer (object) 610 exists in the collider (step S204). Then, the detection unit 100a proceeds to step S205 when the viewer 610c exists in the collider (step S204: Yes), and when the viewer 610 does not exist in the collider (step S204: No), the detection unit 100a proceeds to step S205. The process returns to step S202 described above.
  • FIG. 16 is an explanatory diagram for explaining a modification 2 of the present embodiment.
  • the detection unit 100a includes a reference point (first reference point) A indicating the position of the wrist of the artist 600's hand 602 corresponding to the pointing gesture, and a finger.
  • a reference point (second reference) corresponding to the height of the wrist on a perpendicular line (first perpendicular line) 500 perpendicular to the floor surface 700 from the base (reference point C) of the arm 604 of the artist 600 corresponding to the difference gesture.
  • Point An extension line (first extension line) 520 extending from the line segment (first line segment) 510 connecting with B toward the wrist side is acquired.
  • the viewer 610c included in the conical space 530 with the extension line 520 as the axis and the reference point A as the apex is detected as an object.
  • the length of the conical shaft and the size of the diameter of the bottom surface may be dynamically changed according to the detection status of the selected viewer 610 and the like.
  • the conical space 530 as described above is virtually set in the real space where the viewer 610 exists, and the viewer 610 included in the conical space 530 is detected.
  • the viewer 610 selected by the artist 600 can be easily selected with high accuracy and without omission.
  • the detection unit 100a detects the first viewer 610b corresponding to the point on the extension line 520 as the selection object. Then, the detection unit 100a labels the detected viewer 610b with the selected one, and acquires the information of the viewer 610b. Next, the detection unit 100a detects another viewer 610c corresponding to a point on the extension line 520, which is located behind the viewer 610b and is hidden when viewed from the image of the artist 600. Then, the detection unit 100a detects, labels, and acquires information of the viewer 610c in the same manner as the viewer 610b. By repeating such an operation, a plurality of viewers 610 can be selected in this modification.
  • Second embodiment >> ⁇ 5.1 Detection mode>
  • the detection mode for detecting the selected viewer 610 may be dynamically changed according to the state of the pointing gesture of the artist 600. Therefore, as a second embodiment of the present disclosure, an embodiment of switching the detection mode will be described with reference to FIGS. 9, 18 and 19. 18 and 19 are explanatory views for explaining the second embodiment of the present disclosure.
  • the detection mode is switched between the three detection modes.
  • a reference point (third reference point) C indicating the position of the base of the arm 604 of the artist (user) 600 corresponding to the pointing gesture, and a finger.
  • the detection mode is switched according to the angle ⁇ of the angle formed by the line segment 512 and the perpendicular line 500 on the plane including the perpendicular line (first perpendicular line) 500 drawn perpendicular to.
  • the angle ⁇ is 20 ° (first angle) or more and 90 ° (second angle) or less
  • the first embodiment has been described.
  • the viewer 610 corresponding to the point on the extension line (first extension line) 520 is detected as the selected object (first detection mode).
  • the second detection mode when the angle ⁇ is larger than 90 ° (second angle), detection is performed in the second detection mode.
  • the second detection mode as shown in FIG. 18, viewing corresponding to a point on the extension line (second extension line) 522 extending from the line segment (second line segment) 512 toward the wrist side. 610 is detected as a selected object.
  • the angle ⁇ is greater than 90 °, the artist 600 wants to point to the viewer 610 located above him with a pointing gesture. Therefore, in the present embodiment, in order to reflect the intention of the artist 600 and detect the viewer 610 selected by the artist 600, the detection is performed in a second detection mode different from the first detection mode. ..
  • a third detection mode in which the selected object is not detected is performed. For example, if the angle ⁇ is less than 20 °, it is considered that the artist 600 is not trying to point to the viewer 610 selected by the pointing gesture. Therefore, in the present embodiment, the detection of the selected object is not performed in order to reflect the intention of the artist 600.
  • the artist 600 is selected by switching the detection mode according to the angle ⁇ of the angle formed by the line segment 512 and the perpendicular line 500, reflecting the intention of the artist 600. It is possible to select or not select the viewer 610.
  • the specific angle values shown in the above description are merely examples, and in the present embodiment, the values are not limited to the above values, and the threshold value is based on the intention of the artist 600. It is preferably selected to reflect.
  • FIG. 20 is a flowchart of an information processing method according to the present embodiment.
  • the information processing method according to the present embodiment includes a plurality of steps from step S301 to step S308. From step S301 to step S305 of this embodiment, the same processing as the processing in step S101 and step S105 in the first embodiment shown in FIG. 12 is performed. Further, steps S307 and S308 of the present embodiment perform the same processing as the processing in step S106 and step S107 in the first embodiment shown in FIG. Therefore, in the following description, only step S306 different from the above-described first embodiment will be described, and the description of other steps common to the first embodiment will be omitted.
  • the detection unit 100a determines the detection mode according to the angle ⁇ of the angle formed by the line segment 512 and the perpendicular line 500 (step S306).
  • the artist 600 is selected by switching the detection mode according to the angle ⁇ of the angle formed by the line segment 512 and the perpendicular line 500, reflecting the intention of the artist 600. It is possible to select or not select the viewer 610.
  • Example 1 is an example of VR space sharing
  • FIG. 21 is an explanatory diagram for explaining Example 1 of the embodiment of the present disclosure.
  • the artist 600 and the viewer 610 exist in different real spaces
  • FIG. 21 shows the real space in which the artist 600 exists.
  • the avatar (three-dimensional CG) corresponding to the viewer 610 is maintained while maintaining the relative positional relationship between the live-action volumetric of the artist 600 displayed in the virtual space on the viewer 610 side and each viewer 610.
  • Image It is assumed that 612a to 612e are displayed. Further, it is assumed that the sound sung by the artist 600 is delivered to the virtual space on the viewer side 610 as well as the live-action volumetric of the artist 600 performing the performance.
  • the artist 600 selects the avatar 612a corresponding to the viewer 610 with whom the dialogue is performed by the pointing gesture, so that the artist 600 and the viewer 610 corresponding to the avatar 612a can have a dialogue. It becomes.
  • a laser point display 620 (virtual object) clearly indicating that the avatar 612a has been selected may be displayed on the selected avatar 612a, and the shape, color, pattern, etc. of the avatar 612a clearly indicates that the selection has been made. It may change as follows.
  • a robot (real object) (not shown) arranged in the real space or virtual space where the artist 600 exists may be used.
  • the robot is selected.
  • the robot can move and perform a predetermined operation. For example, a plurality of robots may move according to an artist's fingertip gesture or form a predetermined shape such as a mass game in a virtual space.
  • Example 2 is an example of AR space sharing
  • FIG. 22 is an explanatory diagram for explaining Example 2 of the embodiment of the present disclosure.
  • the artist 600 performing the performance for example, dancing, etc.
  • the viewer 610 exist in the same real space
  • FIG. 21 shows the real space.
  • the laser point display 620 is displayed by the projector 260d on the wall or the like pointed by the artist 600 by the pointing gesture.
  • the speaker 260e outputs a predetermined sound toward the place where the laser point display 620 is displayed.
  • the directivity of the sound output of the speaker 260e may be controlled to output a predetermined sound toward the place where the laser point display 620 is displayed, and the position of the speaker 260e having the directivity may be moved.
  • a predetermined sound may be output toward the place where the laser point display 620 is displayed.
  • the laser point display 620 changes, or the voice output destination is changed. May move or the output audio may change.
  • the viewer can not only directly watch the performance of the artist 600, but also enjoy the content (display, sound) played by the pointing gesture of the artist 600 at the same time.
  • the synchronization of the movements of the plurality of viewers 610 (the height component of the arm swinging motion).
  • the display form (for example, particle display, etc.) of the laser point display 620 may be changed according to the level of synchronization by detecting the vector fluctuation timing, etc.).
  • a predetermined display an image in which the artist 600 performs a predetermined operation
  • captured images, motion data, etc. of the artist 600 and the viewer 610 can be stored, and the content can be re-edited (replacement of the background, etc.) using the stored data.
  • Example 3 is an example of VR space sharing
  • FIG. 23 is an explanatory diagram for explaining Example 3 of the embodiment of the present disclosure.
  • the artist 600 and the viewer 610 exist in different real spaces.
  • FIG. 23 (specifically, the right side of FIG. 23) shows a VR image that the viewer 610 is viewing using the HMD 260a or the smartphone 260f.
  • the live-action volumetric 600a of the artist 600 displayed in the virtual space on the viewer 610 side and the laser point display 620 displayed on the wall on the virtual space pointed by the artist 600 by the pointing gesture are shown. included.
  • the laser point display 620 may be displayed on a virtual space shared between the artist 600 and the viewer 610. Further, the laser point display 620 is an animation in which the pattern, color, etc. may change according to the change of the pointing gesture of the artist 600, and the ripples change according to the change of the pointing gesture. May be good. Further, in the present embodiment, when the artist 600 selects the viewer 610 with which the dialogue is to be performed, only the selected viewer 610 may be able to visually recognize the live-action volumetric 600a of the artist 600, or may be selected. Only the live-action volumetric 600a visually recognized by the viewer 610 may be enlarged, reduced, or changed.
  • the sound content by the artist 600 may be provided to the viewer 610 via the HMD 260a or the smartphone 260f.
  • images and sounds of the performance of the artist 600 existing in a real space different from the viewer 610 can be provided to a plurality of viewers 610 at the same time.
  • Example 4 is an example of AR space sharing
  • FIG. 24 is an explanatory diagram for explaining Example 4 of the embodiment of the present disclosure.
  • the artist 600 and the viewer 610 exist in different real spaces, and the viewer 610 is displayed on the wall on the viewer 610 side in the real space by the projector 260d (for example, a multi-axis drive projector or the like). It is assumed that the live-action volumetric 600a of the artist 600 is visually recognized. Further, in this embodiment, it is assumed that a plurality of large displays 260 g and 260 h are installed in the real space where the viewer 610 exists. Then, in this embodiment, the laser point display 620 can be displayed on the large display 260 g selected by the artist 600 by the pointing gesture.
  • Example 5 will be described with reference to FIG.
  • FIG. 25 is an explanatory diagram for explaining Example 5 of the embodiment of the present disclosure.
  • the artist 600 and the viewer 610 exist in the same real space, and FIG. 25 shows the real space.
  • the psyllium 260i that can communicate with the two-way viewing system 1 held by the viewer 610 selected by the artist 600 by the pointing gesture is turned on.
  • the viewer 610 selected by the artist 600 may be illuminated (spotlight irradiation) (in this case, the lighting device that illuminates the viewer 610 is the bidirectional viewing system 1). Communication is possible).
  • the present invention is not limited to the psyllium 260i, but may be a haptics device 260c, a seat in which the viewer 610 is seated (not shown), and is particularly limited. is not it.
  • Example 6 will be described with reference to FIG.
  • FIG. 26 is an explanatory diagram for explaining Example 6 of the embodiment of the present disclosure.
  • the artist 600 and the viewer 610 exist in the same real space, and FIG. 26 shows the real space.
  • a plurality of lights 260j capable of communicating with the bidirectional viewing system 1 are installed in the real space.
  • the illumination 260j selected by the artist 600 by the pointing gesture is turned on.
  • the lighting is not limited to the lighting 260j, and the color and brightness of the light of the selected lighting 260j may change.
  • the pointing gesture is not limited to the artist 600, and for example, an athlete (sports event), a teacher (education), an operator (remote control), and a doctor (telemedicine). Support) etc. may be used.
  • Example 7 is an example common to VR space sharing and AR space sharing
  • FIG. 27 is an explanatory diagram for explaining Example 7 of the embodiment of the present disclosure.
  • the above-described embodiment of the present disclosure can be used, for example, when selecting and operating a display in the display 800 (in the case of VR space sharing, a display in a display arranged in a virtual space).
  • the display 802 in the display 800 can be erased by selecting the display 802 by gesture (shown on the left side of FIG. 27) and performing a different gesture (on the right side of FIG. 27). show).
  • the detection unit 100a can recognize the gesture by the hand 602 from the captured image of the hand 602, the motion data acquired by the IMU, or the like.
  • Example 8 is an example common to VR space sharing and AR space sharing, and FIG. 28 is an explanatory diagram for explaining Example 8 of the embodiment of the present disclosure.
  • the window 804 in the display 800 (in the case of VR space sharing, the window in the display arranged in the virtual space) is selected by the pointing gesture (shown on the left side of FIG. 28), and further.
  • the position of the window 804 can be moved by moving the direction pointed by the pointing gesture (shown on the right side of FIG. 28).
  • the gesture is not limited to the pointing gesture, and various gestures such as a gesture that holds the palm of the hand can be applied.
  • Example 9 is an example common to VR space sharing and AR space sharing
  • FIG. 29 is an explanatory diagram for explaining Example 9 of the embodiment of the present disclosure.
  • the menu display 806 in the display 800 (in the case of VR space sharing, the menu display in the display arranged in the virtual space) is selected by a gesture, and further selected by performing a different gesture.
  • the menu display 806 can be confirmed.
  • the detection unit 100a can recognize the gesture by the hand 602 from the captured image of the hand 602, the motion data acquired by the IMU, or the like.
  • Example 10 is an example common to VR space sharing and AR space sharing
  • FIG. 30 is an explanatory diagram for explaining Example 10 of the embodiment of the present disclosure.
  • sub-contents in the display 800 for example, moving images, slide shows, etc.
  • the selected sub-content 808 can be stopped by performing a different gesture (shown on the left side of FIG. 30) (shown on the right side of FIG. 30).
  • the detection unit 100a can recognize the gesture by the hand 602 from the captured image of the hand 602, the motion data acquired by the IMU, or the like.
  • the viewer selected by the artist 600 by the pointing gesture can be detected.
  • the reference point A indicating the position of the wrist of the hand 602 of the artist 600 and the reference point C indicating the position of the base of the arm 604 of the artist 600 corresponding to the pointing gesture can be easily detected by the sensor. Since it can be detected, the artist 600, which is a distributor, can easily select the viewer to be pointed to. Further, in the present embodiment, as the selection operation of the artist 600, a pointing gesture that points to the selected viewer is adopted, and the pointing gesture is used in daily life as described above.
  • the artist 600 Since it is an operation performed to point to an object or the like selected in the medium, and is a natural and intuitive operation, the artist 600 (particularly the arm 604) is not burdened. That is, according to the present embodiment, the viewer selected by the artist 600 by the natural and intuitive selection operation can be selected with high accuracy. As a result, the interactive viewing system 1 provides the viewer with an interactive communication experience in which the artist 600 and a specific viewer are present in the same space and it feels as if they are directly interacting with each other. The viewer can feel a sense of unity with the artist 600.
  • embodiments of the present disclosure are not limited to being applied only to events such as music and sports, entertainment, etc., but can also be applied to remote control of robots, education, medical support, and the like.
  • FIG. 31 is a hardware configuration diagram showing an example of a computer that realizes the function of the distributor side server 100.
  • the computer 1000 includes a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, an HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. Each part of the computer 1000 is connected by a bus 1050.
  • the CPU 1100 operates based on the program stored in the ROM 1300 or the HDD 1400, and controls each part. For example, the CPU 1100 expands the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to various programs.
  • the ROM 1300 stores a boot program such as a BIOS (Basic Input Output System) executed by the CPU 1100 when the computer 1000 is started, a program depending on the hardware of the computer 1000, and the like.
  • BIOS Basic Input Output System
  • the HDD 1400 is a computer-readable recording medium that non-temporarily records a program executed by the CPU 1100 and data used by the program.
  • the HDD 1400 is a recording medium for recording an information processing program according to the present disclosure, which is an example of program data 1450.
  • the communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (for example, the Internet).
  • the CPU 1100 receives data from another device or transmits data generated by the CPU 1100 to another device via the communication interface 1500.
  • the input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000.
  • the CPU 1100 receives data from an input device such as a keyboard or mouse via the input / output interface 1600. Further, the CPU 1100 transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600.
  • the input / output interface 1600 may function as a media interface for reading a program or the like recorded on a predetermined recording medium (media).
  • the media includes optical recording media such as DVD (Digital entirely Disc) and PD (Phase change rewritable Disc), magneto-optical recording media such as MO (Magnet-Optical disc), tape media, magnetic recording media, semiconductor memory, and the like. Is.
  • the CPU 1100 of the computer 1000 executes a function of the detection unit 100a or the like by executing an information processing program loaded on the RAM 1200.
  • the HDD 1400 stores an information processing program or the like according to the embodiment of the present disclosure.
  • the CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program, but as another example, these programs may be acquired from another device via the external network 1550.
  • the information processing device may be applied to a system including a plurality of devices, which is premised on connection to a network (or communication between each device), such as cloud computing. .. That is, the information processing device according to the present embodiment described above can be realized as, for example, an information processing system that performs processing related to the information processing method according to the present embodiment by a plurality of devices.
  • the embodiment described above may include, for example, a program for making the computer function as an information processing device according to the present embodiment, and a non-temporary tangible medium in which the program is recorded. Further, the above program may be distributed via a communication line (including wireless communication) such as the Internet.
  • each step in the processing of each of the above-described embodiments does not necessarily have to be processed in the order described.
  • each step may be processed in an appropriately reordered manner.
  • each step may be partially processed in parallel or individually instead of being processed in chronological order.
  • the processing method of each step does not necessarily have to be processed according to the described method, and may be processed by another method, for example, by another functional block.
  • the present technology can also have the following configurations.
  • (1) It is equipped with a detector that detects an object selected by the user by pointing gesture as a selected object.
  • the detection unit lowered the first reference point indicating the position of the wrist of the user's hand corresponding to the pointing gesture and the base of the arm of the user corresponding to the pointing gesture perpendicular to the floor surface.
  • the selected object is detected based on the first extension line extending from the first line segment connecting the second reference point corresponding to the height of the wrist on the first vertical line toward the wrist side.
  • Information processing device Information processing device.
  • (2) The information processing device according to (1) above, wherein the detection unit detects the object corresponding to the point on the first extension line as the selected object.
  • the information processing apparatus wherein the detection mode of the detection unit is switched according to the angle of an angle formed by the second line segment and the first perpendicular line. (7) When the angle of the angle is equal to or greater than the first angle and equal to or less than the second angle, The detection unit executes a first detection mode for detecting a selected object based on the first extension line. The information processing device according to (6) above. (8) When the angle of the angle is larger than the second angle, The detection unit executes a second detection mode of detecting the selected object based on the second extension line extending from the second line segment toward the wrist side. The information processing device according to (7) above.
  • the detection unit executes a third detection mode in which the selected object is not detected.
  • the information processing device according to (7) or (8) above.
  • a determination unit for determining whether the pointing action of the user selecting the selected object by the pointing gesture satisfies a predetermined stability condition is further provided.
  • the determination unit performs the pointing action when the angle of the angle or the distance from the spatial origin to the wrist of the user's hand corresponding to the pointing gesture is within a predetermined range within a predetermined time. Judging that the above stable conditions are satisfied, The detection unit detects the selected object based on the determination result.
  • the information processing device according to any one of (6) to (9) above.
  • the determination unit acquires a plurality of values of the angle of the angle or the distance, calculates a statistical value based on the acquired value, and the pointing action satisfies the stable condition based on the calculated statistical value.
  • Judge whether or not The information processing device according to (10) above.
  • (12) The information processing device according to any one of (1) to (11) above, wherein the object is a real object in real space.
  • the real object includes at least one of a human, an animal, a still life, a robot, a lighting, a communication device, and a display.
  • the information processing apparatus according to (13) above, further comprising a control unit that controls at least one of the robot, the lighting, the communication device, and the display to perform a predetermined operation.
  • the information processing apparatus according to (12) above, further comprising a display unit that superimposes and displays a predetermined image on the detected selected object.
  • the information processing device according to any one of (1) to (11) above, wherein the object is a virtual object in a virtual space.
  • the user and the object are located in the same real space.
  • the information processing device according to any one of (1) to (16) above, wherein the user and the object are located in different spaces.
  • the information processing device lowered the first reference point indicating the position of the wrist of the user's hand corresponding to the pointing gesture vertically from the base of the arm of the user corresponding to the pointing gesture to the floor surface.
  • the selected object is detected based on the first extension line extending from the first line segment connecting the second reference point corresponding to the height of the wrist on the first vertical line toward the wrist side. Including that Information processing method.
  • Bi-directional viewing system 100 Distributor side server 100a Detection unit 102 Distributor side data acquisition unit 104 Gesture recognition unit 106 Laser point output control unit 108 Viewer side data acquisition unit 110 Detection unit 112 Judgment unit 114 vs.
  • Image data generation unit 118 Video streaming processing unit 120 Body position information generation unit 122 3D model position information generation unit 124 Content data generation unit 126 Distribution unit 130 Distributor side data acquisition device 130a Multi-camera system 132 Camera 134a IR camera 134b Marker 136 Depth sensor 160 Distributor side output device 160a Distributor side display unit 200 Viewer side client 202 Distribution data acquisition unit 204 Content processing unit 206, 212 Timing synchronization unit 208 Output control unit 210 Viewer side data acquisition unit 214 Output data processing unit 216 Distribution Part 230 Viewer side data acquisition device 260 Viewer side output device 260a HMD 260b Viewer side display 260c Haptics device 260d Projector 260e Speaker 260f Smartphone 260g, 260h, 800 Display 260i Psyllium 260j Lighting 300 Internet 500, 502 Perpendicular line 510, 512 Line segment 520, 522 Extension line 530 Space 600 Artist 600a 602 Hand 604 Arm 610a, 610

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • User Interface Of Digital Computer (AREA)
  • Processing Or Creating Images (AREA)
  • Position Input By Displaying (AREA)
  • Studio Devices (AREA)
PCT/JP2021/002042 2020-01-29 2021-01-21 情報処理装置、情報処理システム及び情報処理方法 Ceased WO2021153413A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
EP21747621.7A EP4099275A4 (en) 2020-01-29 2021-01-21 Information processing device, information processing system, and information processing method
US17/790,413 US11907434B2 (en) 2020-01-29 2021-01-21 Information processing apparatus, information processing system, and information processing method
CN202180010474.2A CN115004132A (zh) 2020-01-29 2021-01-21 信息处理装置、信息处理系统和信息处理方法
JP2021574692A JP7544071B2 (ja) 2020-01-29 2021-01-21 情報処理装置、情報処理システム及び情報処理方法

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020-012995 2020-01-29
JP2020012995 2020-01-29

Publications (1)

Publication Number Publication Date
WO2021153413A1 true WO2021153413A1 (ja) 2021-08-05

Family

ID=77079883

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/002042 Ceased WO2021153413A1 (ja) 2020-01-29 2021-01-21 情報処理装置、情報処理システム及び情報処理方法

Country Status (5)

Country Link
US (1) US11907434B2 (cg-RX-API-DMAC7.html)
EP (1) EP4099275A4 (cg-RX-API-DMAC7.html)
JP (1) JP7544071B2 (cg-RX-API-DMAC7.html)
CN (1) CN115004132A (cg-RX-API-DMAC7.html)
WO (1) WO2021153413A1 (cg-RX-API-DMAC7.html)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPWO2023204159A1 (cg-RX-API-DMAC7.html) * 2022-04-21 2023-10-26
JP7697739B1 (ja) 2024-12-25 2025-06-24 コーセーエンジニアリング株式会社 部材レイアウト用設計システム及び部材レイアウト用設計方法
WO2026023426A1 (ja) * 2024-07-23 2026-01-29 ソニーグループ株式会社 情報処理装置、情報処理方法、プログラム

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115428032A (zh) * 2020-06-03 2022-12-02 索尼集团公司 信息处理装置、信息处理方法和程序
US11725959B2 (en) * 2020-11-13 2023-08-15 Nvidia Corporation Automatic graphical content recognition for vehicle applications
US12423915B2 (en) * 2021-04-30 2025-09-23 Nippon Telegraph And Telephone Corporation Distribution control system, distribution control apparatus, distribution control method, and program
EP4749413A1 (de) * 2024-11-20 2026-05-27 Deutsche Telekom AG Tragbares gerät und verfahren zur gestenbasierten auswahl von objekten im 3d-raum

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009037434A (ja) * 2007-08-02 2009-02-19 Tokyo Metropolitan Univ 制御機器操作ジェスチャ認識装置、制御機器操作ジェスチャ認識システムおよび制御機器操作ジェスチャ認識プログラム
JP2016184203A (ja) * 2015-03-25 2016-10-20 トヨタ自動車株式会社 ポインティング入力装置
JP2017090615A (ja) * 2015-11-09 2017-05-25 三菱自動車工業株式会社 音声認識制御システム

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4035610B2 (ja) * 2002-12-18 2008-01-23 独立行政法人産業技術総合研究所 インタフェース装置
US8971572B1 (en) * 2011-08-12 2015-03-03 The Research Foundation For The State University Of New York Hand pointing estimation for human computer interaction
US9684372B2 (en) * 2012-11-07 2017-06-20 Samsung Electronics Co., Ltd. System and method for human computer interaction
TW201510771A (zh) * 2013-09-05 2015-03-16 Utechzone Co Ltd 指向位置偵測裝置及其方法、程式及電腦可讀取紀錄媒體
JP6851133B2 (ja) * 2014-01-03 2021-03-31 ハーマン インターナショナル インダストリーズ インコーポレイテッド ユーザに方向付けられた個人情報アシスタント
US20150309681A1 (en) * 2014-04-23 2015-10-29 Google Inc. Depth-based mode switching for touchless gestural interfaces
US20180059811A1 (en) * 2015-03-31 2018-03-01 Sony Corporation Display control device, display control method, and recording medium
WO2016200197A1 (ko) * 2015-06-10 2016-12-15 (주)브이터치 사용자 기준 공간좌표계 상에서의 제스처 검출 방법 및 장치
JP6488946B2 (ja) 2015-08-21 2019-03-27 富士通株式会社 制御方法、プログラム及び制御装置
US11176745B2 (en) * 2019-09-20 2021-11-16 Facebook Technologies, Llc Projection casting in virtual environments

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009037434A (ja) * 2007-08-02 2009-02-19 Tokyo Metropolitan Univ 制御機器操作ジェスチャ認識装置、制御機器操作ジェスチャ認識システムおよび制御機器操作ジェスチャ認識プログラム
JP2016184203A (ja) * 2015-03-25 2016-10-20 トヨタ自動車株式会社 ポインティング入力装置
JP2017090615A (ja) * 2015-11-09 2017-05-25 三菱自動車工業株式会社 音声認識制御システム

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
PISHED BUNNUNWALTERIO W. MAYOL-CUEVAS: "7th IEEE/ACM International Symposium on Mixed and Augmented Reality", 2008, IEEE, article "OutlinAR: an assisted interactive model building system with reduced computational effort"
See also references of EP4099275A4
YATA, YUKIHIRO; WATANABE, KEN; NAGAO, KATASHI: "Natural Interface of Personal Intelligent Vehicles by Hand Gesture Recognition", PROCEEDINGS OF THE 75TH NATIONAL CONVENTION OF THE INFORMATION PROCESSING SOCIETY OF JAPAN, 6 March 2013 (2013-03-06), pages 4-101 - 4-102, XP009530202 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPWO2023204159A1 (cg-RX-API-DMAC7.html) * 2022-04-21 2023-10-26
WO2026023426A1 (ja) * 2024-07-23 2026-01-29 ソニーグループ株式会社 情報処理装置、情報処理方法、プログラム
JP7697739B1 (ja) 2024-12-25 2025-06-24 コーセーエンジニアリング株式会社 部材レイアウト用設計システム及び部材レイアウト用設計方法
WO2026140734A1 (ja) * 2024-12-25 2026-07-02 コーセーエンジニアリング株式会社 部材レイアウト用設計システム及び部材レイアウト用設計方法
JP2026112903A (ja) * 2024-12-25 2026-07-07 コーセーエンジニアリング株式会社 部材レイアウト用設計システム及び部材レイアウト用設計方法

Also Published As

Publication number Publication date
JP7544071B2 (ja) 2024-09-03
EP4099275A4 (en) 2023-07-05
EP4099275A1 (en) 2022-12-07
US20230071828A1 (en) 2023-03-09
JPWO2021153413A1 (cg-RX-API-DMAC7.html) 2021-08-05
US11907434B2 (en) 2024-02-20
CN115004132A (zh) 2022-09-02

Similar Documents

Publication Publication Date Title
JP7544071B2 (ja) 情報処理装置、情報処理システム及び情報処理方法
US12354204B2 (en) Audiovisual presence transitions in a collaborative reality environment
JP7366196B2 (ja) 広範囲同時遠隔ディジタル提示世界
WO2025024476A1 (en) Systems, devices, and methods for audio presentation in a three-dimensional environment
US7817104B2 (en) Augmented reality apparatus and method
CN121957350A (zh) 用于在三维环境中显示用户界面对象的方法
CN121532733A (zh) 用于显示物理位置视图的设备、方法和图形用户界面
JP2022549853A (ja) 共有空間内の個々の視認
JP2023139098A (ja) 画像生成装置および画像生成方法
WO2014204330A1 (en) Methods and systems for determining 6dof location and orientation of head-mounted display and associated user movements
US12223679B2 (en) Data processing
JP6470859B1 (ja) ユーザの動きをアバタに反映するためのプログラム、当該プログラムを実行するための情報処理装置、およびアバタを含む映像を配信するための方法
US20160320833A1 (en) Location-based system for sharing augmented reality content
EP4705862A1 (en) Devices, methods, and graphical user interfaces for presenting content
US20180299948A1 (en) Method for communicating via virtual space and system for executing the method
JP7111848B2 (ja) プログラム、情報処理装置、および方法
CN121569265A (zh) 用于显示演示应用的演示环境的设备、方法和图形用户界面
JP2019160295A (ja) ユーザの動きをアバタに反映するためのプログラム、当該プログラムを実行するための情報処理装置、およびアバタを含む映像を配信するための方法
WO2022107294A1 (ja) Vr映像空間生成システム
US11983306B2 (en) Peripheral tracking system and method
WO2024253867A1 (en) Devices, methods, and graphical user interfaces for presenting content
JP2024047587A (ja) 情報処理システムおよびプログラム
JP2024047008A (ja) 情報処理システム
EP4689850A1 (en) Devices, methods, and graphical user interfaces for providing environment tracking content
WO2024233224A1 (en) Devices, methods, and graphical user interfaces for providing environment tracking content

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21747621

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021574692

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2021747621

Country of ref document: EP

Effective date: 20220829