US20180278888A1 - Information processing device and information processing method - Google Patents

Information processing device and information processing method Download PDF

Info

Publication number
US20180278888A1
US20180278888A1 US15/760,060 US201615760060A US2018278888A1 US 20180278888 A1 US20180278888 A1 US 20180278888A1 US 201615760060 A US201615760060 A US 201615760060A US 2018278888 A1 US2018278888 A1 US 2018278888A1
Authority
US
United States
Prior art keywords
ghost
image
information processing
jackin
information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US15/760,060
Other languages
English (en)
Inventor
Shunichi Kasahara
Junichi Rekimoto
Jun Kimura
Taizo Shirai
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Corp
Original Assignee
Sony Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Corp filed Critical Sony Corp
Assigned to SONY CORPORATION reassignment SONY CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: KIMURA, JUN, SHIRAI, TAIZO, REKIMOTO, JUNICHI, KASAHARA, SHUNICHI
Publication of US20180278888A1 publication Critical patent/US20180278888A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/141—Systems for two-way working between two video terminals, e.g. videophone
    • H04N7/147—Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/21805—Source of audio or video content, e.g. local disk arrays enabling multiple viewpoints, e.g. using a plurality of cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/04817—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance using icons
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04842—Selection of displayed objects or displayed text elements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/1066—Session management
    • H04L65/1069—Session establishment or de-establishment
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/1066—Session management
    • H04L65/1083—In-session procedures
    • H04L65/1089—In-session procedures by adding media; by removing media
    • H04L67/18—
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/2866—Architectures; Arrangements
    • H04L67/30—Profiles
    • H04L67/306—User profiles
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/52—Network services specially adapted for the location of the user terminal
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M1/00—Substation equipment, e.g. for use by subscribers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/27—Server based end-user applications
    • H04N21/274—Storing end-user multimedia data in response to end-user request, e.g. network recorder
    • H04N21/2743—Video hosting of uploaded data from client
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/482—End-user interface for program selection
    • H04N21/4828—End-user interface for program selection for searching program descriptors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range

Definitions

  • a technology disclosed in the present specification relates to an information processing device and information processing method for performing matching between users and relates to, for example, an information processing device and information processing method for performing matching between a user who provides a first person view and a user who views the first person view.
  • a mobile camera system that remotely acquires an image captured by a mobile camera mounted on a mobile body such as a vehicle
  • an image processing system that provides, to a person who wears a head mounted display, information similar to visual information acquired by a person who wears eyeglasses including an imaging sensing wireless device (e.g., see Patent Literature 2).
  • Patent Literature 1 JP 2006-186645A
  • Patent Literature 2 JP 2004-222254A
  • Patent Literature 3 JP 2008-154192A
  • Patent Literature 4 JP 2014-522053T
  • Patent Literature 5 JP 2014-104185A
  • An object of the technology disclosed in the present specification is to provide an excellent information processing device and information processing method capable of performing matching between users.
  • a first aspect thereof is an information processing device including: a control unit configured to control connection between a first device that transmits an image and a second device that receives the image in accordance with which of the first device and the second device takes initiative.
  • the control unit of the information processing device is configured to receive a connection request from the second device, notify the first device in a standby state, and cause image transmission from the first device to the second device to start.
  • the control unit of the information processing device is configured to notify the first device of a connection request from the second device and cause image transmission from the first device to the second device to start.
  • the control unit of the information processing device is configured to notify the first device of connection requests from the plurality of second devices only in a case where the connection requests satisfy a predetermined start condition and cause image transmission from the first device to the plurality of second devices to start.
  • control unit of the information processing device is configured to control a start of image transmission from the first device to the second device and intervention in the first device by the second device.
  • a sixth aspect of the technology disclosed in the present specification is an information processing method including: a control step of controlling connection between a first device that transmits an image and a second device that receives the image in accordance with which of the first device and the second device takes initiative.
  • a seventh aspect of the technology disclosed in the present specification is an information processing device including: a selection unit configured to select a first device on a basis of position information of the first device, the first device transmitting an image to a second device.
  • the selection unit of the information processing device is configured to present a UI that shows a position of the first device on a map.
  • the selection unit of the information processing device is configured to select the first device further in consideration of behavior of a user.
  • the selection unit of the information processing device is configured to present only the first device on the UI, the first device being extracted on a basis of behavior of a user.
  • the selection unit of the information processing device is configured to present only the first device on the UI, the first device being extracted on a basis of behavior of a user.
  • the selection unit of the information processing device is configured to present information regarding intervention in the first device on the UI.
  • a thirteenth aspect of the technology disclosed in the present specification is an information processing method including: a selection step of selecting a first device on a basis of position information of the first device, the first device transmitting an image to a second device.
  • a fourteenth aspect of the technology disclosed in the present specification is an information processing device including: a selection unit configured to select a first device on a basis of behavior of a user of the first device, the first device transmitting an image to a second device.
  • the selection unit of the information processing device is configured to present a UI that shows information regarding the image transmitted from the first device.
  • the selection unit of the information processing device is configured to present information regarding the first device or the user of the first device on the UI.
  • the selection unit of the information processing device is configured to present only the image on the UI, the image being transmitted from the first device extracted on the basis of the behavior of the user.
  • an eighteenth aspect of the technology disclosed in the present specification is an information processing method including: a selection step of selecting a first device on a basis of behavior of a user of the first device, the first device transmitting an image to a second device.
  • a nineteenth aspect of the technology disclosed in the present specification is an information processing device including: a selection unit configured to select a second device to which a first device transmits an image on a basis of information regarding the second device or a user of the second device.
  • the selection unit of the information processing device is configured to present a UI that shows information regarding the second device or the user of the second device.
  • FIG. 1 illustrates an overview of a visual information sharing system 100 to which a technology disclosed in the present specification is applied.
  • FIG. 2 schematically illustrates a network topology of 1 to N.
  • FIG. 3 schematically illustrates a network topology of N to 1.
  • FIG. 4 schematically illustrates a network topology of N to N.
  • FIG. 5 illustrates a functional configuration example of an image provision device 101 and an image display device 102 .
  • FIG. 6 schematically illustrates a start flow of Body initiative start.
  • FIG. 7 schematically illustrates a start flow of ghost initiative start.
  • FIG. 8 illustrates a UI display example for selecting a Body.
  • FIG. 9 illustrates a UI display example for selecting a Body.
  • FIG. 10 illustrates a UI display example for selecting a Body.
  • FIG. 11 illustrates a UI display example for selecting a Body.
  • FIG. 12 illustrates a UI display example for selecting a Body.
  • FIG. 13 exemplifies a tag displayed on a Body selection UI.
  • FIG. 14A illustrates a UI display example for selecting a Body.
  • FIG. 14B illustrates a UI display example for selecting a Body.
  • FIG. 15 illustrates an example of a UI that allows a Body to select a ghost.
  • FIG. 1 illustrates an overview of a visual information sharing system 100 to which the technology disclosed in the present specification is applied.
  • the visual information sharing system 100 illustrated in FIG. 1 is configured by combining an image provision device 101 for providing an image obtained by capturing an image of a site and an image display device 102 for displaying the image provided from the image provision device 101 .
  • the image provision device 101 specifically includes a camera-equipped see-through head mounted display mounted on a head part of an observer 111 who actually acts on a site.
  • the “see-through” head mounted display herein is basically optical transmissive but may be a video see-through head mounted display.
  • the camera provided in the head mounted display captures an image of a substantially line-of-sight direction of the observer 111 and provides a first person view (FPV) thereof.
  • FMV first person view
  • the image display device 102 is assumed to be arranged separately from the site, i.e., from the image provision device 101 , and the image provision device 101 and the image display device 102 are assumed to communicate with each other via a network.
  • the term “separately” herein includes not only a remote location but also a situation in which the image provision device 101 and the image display device 102 are slightly (e.g., approximately several meters) separate in the same room. Further, the image provision device 101 and the image display device 102 are also assumed to exchange data via a server (not illustrated).
  • the image display device 102 is, for example, a head mounted display worn by a person who is not on the site (viewer of captured image) 112 .
  • a head mounted display worn by a person who is not on the site (viewer of captured image) 112 .
  • an immersive head mounted display as the image display device 102
  • the viewer 112 can experience the same sight as that of the observer 111 with more reality.
  • a see-through head mounted display may be used as the image display device 102 .
  • the image display device 102 is not limited to a head mounted display and may be, for example, a wrist-watch display.
  • the image display device 102 does not need to be a wearable terminal and may be a multifunctional information terminal such as a smartphone or a tablet, a general monitor display such as a computer screen or a television receiver, a game console, a projector for projecting an image on a screen, or the like.
  • the observer 111 Because the observer 111 is actually on the site and acts with his/her body, the observer 111 who is a user of the image provision device 101 (or the image provision device 101 ) will also be referred to as “Body” hereinafter. Meanwhile, the viewer 112 does not act with his/her body on the site but is conscious of being on the site by viewing a first person view of the observer 111 , and therefore the viewer 112 who is a user of the image display device 102 (or the image display device 102 ) will also be referred to as “Ghost” hereinafter.
  • a Body transmits the own peripheral situation to a ghost and further shares the situation with the ghost.
  • One of ghosts communicates with the Body and thus can achieve interactions such as operation support from a separate location.
  • Immersing the ghost in a first person experience of the Body to allow the ghost to perform interactions in the visual information sharing system 100 will also be referred to as “JackIn” hereinafter.
  • a start flow of JackIn is roughly classified into a case where the Body takes the initiative in performing JackIn (Body initiative start) and a case where the ghost takes the initiative in performing JackIn (Ghost initiative start). Details of the JackIn start flow will be described below.
  • the visual information sharing system 100 basically has a function of transmitting a first person view from the Body to the ghost to allow the ghost to view and experience the first person view and a function of allowing the Body and the ghost to communicate with each other.
  • the ghost can interact with the Body by intervention from a remote location, such as “visual intervention” that allows the ghost to intervene in vision of the Body, “auditory intervention” that allows the ghost to intervene in an auditory sensation of the Body, “body intervention” that allows the ghost to move or stimulate a body of the Body or a part of the body, and “alternative conversation” that allows the ghost to speak on a site, instead of the Body.
  • JackIn has a plurality of communication channels such as “visual intervention”, “auditory intervention”, “body intervention”, and “alternative conversation”. Details of “visual intervention”, “auditory intervention”, “body intervention”, and “alternative conversation” will be described below.
  • the Ghost can instruct the Body on behavior on a site through “visual intervention”, “auditory intervention”, “body intervention”, or “alternative conversation”.
  • the visual information sharing system 100 can be utilized for operation support in various industrial fields such as a medical site of a surgical operation and the like and a construction site of a construction work and the like, instructions on control of airplanes and helicopters and guidance thereof, navigation of drivers of automobiles, coaching or instructions in sports, and other uses.
  • the Body takes the initiative in implementing JackIn with an appropriate ghost (Body initiative start).
  • the ghost takes the initiative in implementing JackIn with a corresponding Body (Ghost initiative start).
  • the own behavior may be interrupted by the ghost, or the own behavior may be hindered and is therefore dangerous, and, in some cases, the Body's privacy is invaded.
  • the ghost may also have some videos that the ghost does not desire to view, or, in some cases, cannot provide services such as appropriate assistance, instruction, guidance, and navigation to the Body even in a case where the ghost is asked to. Therefore, JackIn to the Body by the ghost and intervention in the Body by the ghost in a JackIn state may be limited at a certain level.
  • FIG. 1 illustrates a network topology of a single Body to a single ghost, i.e., in which only a single image provision device 101 and a single image display device 102 exist.
  • the following are also assumed: a network topology of 1 to N in which a single Body and a plurality (N) of ghosts simultaneously perform JackIn as illustrated in FIG. 2 ; a network topology of N to 1 in which a plurality (N) of Bodies and a single ghost simultaneously perform JackIn as illustrated in FIG. 3 ; and a network topology of N to N in which a plurality (N) of Bodies and a plurality (N) of ghosts simultaneously perform JackIn as illustrated in FIG. 4 .
  • switching a single device from a Body to a ghost switching a single device from a ghost to a Body, and simultaneously having a role of a Body and a role of a ghost are also assumed.
  • a network topology (not illustrated) in which a single device performs JackIn to a Body as a ghost and, at the same time, functions as a Body for another ghost, i.e., three or more devices are daisy-chain connected.
  • a server may be interposed between a Body and a ghost.
  • FIG. 5 illustrates a functional configuration example of the image provision device 101 and the image display device 102 .
  • the image provision device 101 is a device to be used by a user (observer 112 ) who takes a role as a Body.
  • the image provision device 101 includes an imaging unit 501 , an image processing unit 502 , a display unit 503 , a first audio output unit 504 , a drive unit 505 , and a second audio output unit 506 serving as an output unit, a position detection unit 507 , a communication unit 508 , a control unit 509 , and an authentication unit 510 .
  • the imaging unit 501 includes a camera for capturing an image of a first person view of the Body.
  • the imaging unit 501 is attached to the head part of the observer 111 so as to capture an image of, for example, a line-of-sight direction of the Body, i.e., the observer 111 .
  • a whole-sky camera may be used as the imaging unit 501 to provide a 360-degree whole-sky image of an environment around the Body.
  • the whole-sky image does not necessarily need to be a 360-degree image, and a field of view may be narrower.
  • the whole-sky image may be a half celestial sphere image that does not include a floor surface containing little information (The same applies hereinafter.).
  • the image processing unit 502 processes image signals output from the imaging unit 501 .
  • the ghost views a video that shakes strongly because the Body looks out over a surrounding environment on his/her own and changes a line-of-sight direction.
  • health hazards such as virtual reality (VR) sickness and motion sickness are a matter of concern.
  • the ghost may desire to view a part on which the Body does not focus.
  • the image processing unit 502 simulatively forms a surrounding space on the basis of continuous images of the first person view of the Body captured by the imaging unit 501 .
  • the image processing unit 502 performs space recognition based on simultaneous localization and mapping (SLAM) recognition technology or the like in real time with respect to a video (whole-sky image) captured by the imaging unit 501 and spatially joins a current video frame and a past video frame together, thereby rendering a video seen from a viewpoint of a virtual camera controlled by the ghost.
  • the video rendered at the viewpoint of the virtual camera is a video seen from a viewpoint that is simulatively out of a body of the Body rather than the first person view of the Body. Therefore, the ghost can observe an environment around the Body independently from motion of the Body. This makes it possible to stabilize shaking of the video to prevent VR sickness and view a part on which the Body does not focus.
  • SLAM simultaneous localization and mapping
  • the display unit 503 displays and outputs information transmitted from the image display device 102 , thereby allowing the ghost to intervene in vision of the Body.
  • the display unit 503 superimposes and displays an augmented reality (AR) image that expresses consciousness of the ghost who shares a first person experience with the Body on vision of the observer 111 (i.e., scene of a real world).
  • the AR image includes images such as a pointer, an annotation, or the like showing a location indicated by the ghost. Therefore, the ghost can communicate with the Body to intervene in the vision thereof, thereby interacting with the Body on a site.
  • the first audio output unit 504 includes, for example, earphones, headphones, or the like and causes the Body to listen to information transmitted from the image display device 102 , thereby allowing the ghost to intervene in an auditory sensation of the Body.
  • the image display device 102 transmits information regarding consciousness of the ghost who shares a first person experience with the Body.
  • the image provision device 101 converts received information into audio signals and outputs audio from the first audio output unit 504 , thereby causing the Body, i.e., the observer 111 to listen to the audio.
  • audio signals uttered by the ghost who currently has a first person experience are transmitted from the image display device 102 as they are.
  • the image provision device 101 outputs the received audio signals in the form of audio from the first audio output unit 504 as they are, thereby causing the Body, i.e., the observer 111 to listen to the audio. Further, volume, quality, an output timing, and the like of audio output from the first audio output unit 504 may be appropriately adjusted. Alternatively, image information or character information transmitted from the image display device 102 may be converted into audio signals and be output in the form of audio from the first audio output unit 504 . Therefore, the ghost can communicate with the Body to intervene in the auditory sensation thereof, thereby interacting with the Body on a site.
  • the drive unit 505 moves or stimulates the body of the Body or a part of the body, thereby allowing the ghost to intervene in the body of the Body.
  • the drive unit 505 includes, for example, an actuator for applying tactile sensations or electrical stimulation (which is slight and thus does not harm health) to the body of the observer 111 .
  • the drive unit 505 includes a device (e.g., see Patent Literature 5) for supporting or restricting motion of the body by driving a powered suit or exoskeleton worn on arms, hands, legs, or the like of the observer 111 . Therefore, the ghost can communicate with the Body to intervene in the body thereof, thereby interacting with the Body on a site.
  • the second audio output unit 506 includes, for example, a wearable speaker or the like worn by the Body and outputs information or audio signals transmitted from the image display device 102 to the outside in the form of audio.
  • the audio output from the second audio output unit 506 is heard on a site as if the Body himself/herself spoke. Therefore, the ghost can have a conversation with people on a site where the Body exists or can give an instruction with audio, instead of the Body (alternative conversation).
  • the position detection unit 507 detects current position information of the image provision device 101 (i.e., Body) by using, for example, global positioning system (GPS) signals.
  • the detected position information is used in a case where, for example, the ghost searches for a Body who exists in a location desired by the ghost (described later).
  • the communication unit 508 which is mutually connected to the image display device 102 via a network, transmits an image of a first person view captured by the imaging unit 501 and space information and communicates with the image display device 102 .
  • Communication means of the communication unit 508 may be wireless or wired communication means and is not limited to a particular communication standard.
  • the authentication unit 510 performs authentication processing of the image display device 102 (or the ghost who is a user thereof) which is mutually connected via a network and determines an output unit for outputting information transmitted from the image display device 102 . Then, the control unit 509 controls output operation from the output unit on the basis of a result of authentication by the authentication unit 510 .
  • the control unit 509 has, for example, functions corresponding to a central processing unit (CPU) and a graphic processing unit (GPU).
  • the control unit 509 executes only display output from the display unit 503 . Further, in a case where the image display device 102 is permitted to perform not only visual intervention but also auditory intervention, the control unit 509 executes both display output from the display unit 503 and audio output from the first audio output unit 504 .
  • a range in which the Body permits intervention by the ghost is defined as a permission level. Meanwhile, a range in which the ghost intervenes in the Body is defined as a mission level (described below).
  • the visual information sharing system 100 it is also possible to form the visual information sharing system 100 so that the above processing performed by the authentication unit 510 and the control unit 509 is executed by the server (not illustrated) interposed between the image provision device 101 and the image display device 102 , instead of the image provision device 101 .
  • the image display device 102 is a device to be used by a user (viewer 112 ) who takes a role as a ghost.
  • the image display device 102 includes a communication unit 511 , an image decoding unit 512 , a display unit 513 , a user input unit 514 , and a position posture detection unit 515 .
  • the communication unit 511 which is mutually connected to the image provision device 101 via a network, receives a first person view from the image provision device 101 and communicates with the image provision device 101 .
  • Communication means of the communication unit 511 may be wireless or wired communication means and is not limited to a particular communication standard. However, the communication means is compatible with the communication unit 508 of the image provision device 101 .
  • the image decoding unit 512 performs decoding processing of image signals that the communication unit 511 receives from the image provision device 101 .
  • the display unit 513 displays and outputs the whole-sky image (first person view of the Body) which has been decoded in the image decoding unit 512 .
  • the processing for rendering a video seen from a viewpoint out of the body of the Body (described above) from the first person view of the Body may be performed by the image decoding unit 512 , instead of the image processing unit 502 of the image provision device 101 .
  • the position posture detection unit 515 detects a position and posture f a head part of the viewer 112 .
  • the detected position and posture correspond to a current viewpoint position and line-of-sight direction of the ghost.
  • a viewpoint position and line-of-sight direction of the virtual camera (described above) to create a video seen from a viewpoint simulatively out of the body of the Body on the basis of the first person view of the Body can be controlled on the basis of the position and posture of the head part of the viewer 112 detected by the position posture detection unit 515 .
  • the display unit 513 includes, for example, a head mounted display worn by the viewer 112 serving as a ghost.
  • a head mounted display worn by the viewer 112 serving as a ghost.
  • the viewer 112 can experience the same sight as that of the observer 111 with more reality.
  • a video viewed by the viewer 112 i.e., the ghost is not the first person view of the Body itself but is a surrounding space simulatively formed on the basis of continuous images of the first person view (video seen from a viewpoint simulatively out of the body of the Body) (described above).
  • the virtual camera performs head tracking of the ghost, i.e., follows the viewpoint position and line-of-sight direction of the viewer 112 detected by the position posture detection unit 515 , thereby moving an angle of view of display on the display unit 513 .
  • the display unit 513 may be a wearable terminal such as a see-through head mounted display or a wrist-watch display, instead of an immersive head mounted display.
  • the display unit 513 does not need to be a wearable terminal and may be a multifunctional information terminal such as a smartphone or a tablet, a general monitor display such as a computer screen or a television receiver, a game console, a projector for projecting an image on a screen, or the like.
  • the user input unit 514 is a device for allowing the viewer 112 serving as a ghost to input the ghost's own intention or consciousness in response to observation of the first person view of the Body displayed on the display unit 513 .
  • the user input unit 514 includes, for example, a coordinate input device such as a touchscreen, a mouse, or a joystick.
  • a coordinate input device such as a touchscreen, a mouse, or a joystick.
  • the ghost can directly indicate a location in which the ghost is particularly interested on a screen that displays the first person view of the Body.
  • the ghost gives an indication on a pixel coordinate of a video that the ghost currently views.
  • a captured video of the Body always changes, and therefore an indication on the pixel coordinate is meaningless.
  • the user input unit 514 specifies, by image analysis or the like, position information on a three-dimensional space corresponding to a pixel position that the ghost indicates by touching, click operation, or the like on the screen and transmits the position information in the three-dimensional space to the image provision device 101 . Therefore, the ghost can perform pointing that achieves fixation in a space, instead of on the pixel coordinate.
  • the user input unit 514 may capture eye movement by using an image of a face of the ghost captured by a camera or an eye potential, calculate a location at which the ghost gazes, and transmit information specifying the location to the image provision device 101 . Also at that time, the user input unit 514 specifies, by image analysis or the like, position information in the three-dimensional space corresponding to a pixel position at which the ghost gazes, and transmits the position information in the three-dimensional space to the image provision device 101 . Therefore, the ghost can perform pointing that achieves fixation in a space, instead of on the pixel coordinate.
  • the user input unit 514 includes a character input device such as a keyboard.
  • a character input device such as a keyboard.
  • the user input unit 514 may transmit the character information input by the ghost as it is to the image provision device 101 or may convert the character information into other forms of signals such as audio signals and then transmit the signals to the image provision device 101 .
  • the user input unit 514 includes an audio input device such as a microphone and inputs audio uttered by the ghost.
  • the user input unit 514 may transmit the input audio as they are in the form of audio signals from the communication unit 511 to the image provision device 101 .
  • the user input unit 514 may perform audio recognition of the input audio, convert the input audio into character information, and transmit the character information to the image provision device 101 .
  • the ghost is assumed to indicate an object by using a demonstrative pronoun such as “that” or “this” while viewing the first person view of the Body.
  • the user input unit 514 specifies, by language analysis, image analysis, or the like, position information of the object indicated by the demonstrative pronoun in the three-dimensional space and transmits the position information in the three-dimensional space to the image provision device 101 . Therefore, the ghost can perform pointing that achieves fixation in a space, instead of on the pixel coordinate.
  • the user input unit 514 may be a gesture input device for inputting body gestures and manual gestures of the ghost.
  • Means for capturing gestures is not particularly limited.
  • the user input unit 514 may include a camera for capturing an image of movement of arms and legs of the ghost and an image recognition device for processing the captured image. Further, in order to easily perform image recognition, a marker may be attached to the body of the ghost.
  • the user input unit 514 may transmit an input gesture from a communication unit 411 to the image provision device 101 as, for example, control signals to intervene in the body of the Body.
  • the user input unit 514 may convert the input gesture into image information to intervene in the vision of the Body (coordinate information, AR image to be superimposed and displayed, character information, or the like) or audio signals to intervene in the auditory sensation of the Body and transmit the image information or audio signals from the communication unit 511 to the image provision device 101 . Further, the user input unit 514 specifies, by image analysis or the like, position information in the three-dimensional space corresponding to a pixel position indicated by a gesture of the ghost and transmits the position information in the three-dimensional space to the image provision device 101 . Therefore, the ghost can perform pointing that achieves fixation in a space, instead of on the pixel coordinate.
  • JackIn developed in the visual information sharing system 100 resembles a general AR technology in view of superimposing and displaying an AR image. However, it is considered that JackIn is different from a normal AR technology performed by a computer in that a human being (Ghost) augments another human being (Body).
  • Ghost human being
  • Body human being
  • JackIn also resembles telepresence (described above).
  • normal telepresence and JackIn are different in that normal telepresence is an interface for viewing the world from a viewpoint of a machine such as a robot, whereas, in JackIn, a human being (Ghost) views the world from a viewpoint of another human being (Body).
  • telepresence presupposes that a human being is a master and a machine is a slave and the machine that is the slave truly reproduces motion of the human being.
  • the Body does not necessarily move in compliance with the ghost, i.e., is an independent interface.
  • a video provided from the image provision device 101 to the image display device 102 is not limited to a real-time video observed by the Body on a site (i.e., a live video captured by the imaging unit 501 ) and may be a past recorded video.
  • the image provision device 101 includes a mass storage device (not illustrated) for recording a past video, and the past video may be distributed from the image provision device 101 .
  • the past video recorded by the image provision device 101 may be accumulated in a JackIn server (provisional name) for controlling JackIn between the Body and the ghost or another recording server, and the past video may be streamed from the server to the ghost (image display device 102 ).
  • JackIn have a plurality of communication channels such as “visual intervention”, “auditory intervention”, “body intervention”, and “alternative conversation”. Therefore, by starting JackIn with the ghost, the Body can share the own vision with the ghost and can be assisted, instructed, guided, and navigated by the ghost regarding operation that is currently performed through visual intervention or the like. Further, by starting JackIn with the Body, the ghost can have a first person experience of the Body without visiting a site and can assist, instruct, guide, and navigate the Body regarding operation thereof through visual intervention or the like.
  • the Body's behavior may be interrupted by the ghost, or the Body's behavior may be hindered and is therefore dangerous, and, in some cases, the Body's privacy is invaded.
  • the ghost may also have some videos that the ghost does not desire to view or, in some cases, cannot provide services such as appropriate assistance, instruction, guidance, and navigation even in a case where the ghost is asked to by the Body. That is, a mismatch between the Body and the ghost is problematic.
  • “permission” and “mission” are defined.
  • a range in which the Body permits intervention by the ghost is defined as “permission”, and intervention by the ghost is limited to the range prescribed by the permission.
  • a range of operation in which the ghost intervenes in the Body is defined as “mission”, and a range in which the ghost intervenes in the Body is limited to the range prescribed by the mission.
  • Bodies can appropriately set permission having respective different levels at which intervention is permitted as exemplified below.
  • Level 1 Only exchange of vision (transmission of first person view) is permitted.
  • the image provision device 101 only transmits an image captured by the imaging unit 501 and operates no output unit.
  • Level 2 Only exchange of vision and visual intervention are permitted.
  • the image provision device 101 only transmits an image captured by the imaging unit 501 and performs display output on the display unit 503 .
  • the image provision device 101 transmits an image captured by the imaging unit 501 , performs display output on the display unit 503 , and performs audio output from the first audio output unit 504 .
  • the image provision device 101 can further drive the drive unit 505 and outputs audio to the outside from the second audio output unit 506 .
  • each Body may give individual permission to each ghost, instead of giving uniform permission to all the ghosts.
  • the Body may set permission based on a user attribute of the ghost.
  • the user attribute herein includes not only personal information such as age, sex, a human relationship with the Body (family or kinship relation, friend, boss and subordinate, or the like), a place of birth, an occupation, and a qualification, but also rating information of a skill of assistance target operation and information such as past performance (as an assistant, instructor, or the like) (how many hours the ghost has experienced the operation so far) and review of the ghost, and reputation by other Bodies (posting, voting result, or the like).
  • the Body may individually set permission (permission for Mr./Ms. A, permission for Mr./Ms. B, . . . , and the like), instead of setting permission based on an attribute.
  • permission may be set for each combination of a Body and a ghost.
  • the Body may set permission on the basis of a human relationship with the Body or may set permission on the basis of abilities of the ghost that the Body personally grasps.
  • there is also a method of giving temporary permission to a ghost by one-to-one negotiation, mediation, or the like between a Body and the ghost high-level permission is given to a certain ghost only in a predetermined period of time, and, when the period of time elapses, the permission is restored to original-level permission).
  • the Body may set a user who is prohibited from performing JackIn to the Body himself/herself.
  • the permission settings are cases where the Body charges for (i.e., monetizes) JackIn as a service. Any one of the above level-1 permission to level-4 permission is set for the ghost in accordance with a usage fee paid by the ghost, and therefore the ghost can perform JackIn with the Body.
  • a ghost who pays 10 dollars is permitted visual intervention and auditory intervention (level-2 or 3 permission).
  • a ghost who pays 100 dollars is permitted body intervention (level-4 permission).
  • alternative conversation is temporarily permitted.
  • a range of operation in which a ghost intervenes in a Body is defined as “mission”, and a range in which the ghost can intervene in the Body is limited to the range prescribed by the mission.
  • the mission of the ghost is set on the basis of, for example, a range of a mission to be carried out by the ghost or abilities thereof.
  • the mission is not arbitrarily determined by each ghost but is preferably permitted or authenticated by, for example, an authoritative organization or the like.
  • Missions having different levels exemplified below can be defined in accordance with a mission to be carried out by the ghost, a duty, an occupation, a qualification, rating of an intervention skill, past performance (experience time as a ghost or the like) (as an assistant, instructor, or the like) and review of the ghost, reputation by Bodies (posting, voting result, or the like), or the like.
  • Level 1 Only exchange of vision (transmission of first person view) is performed.
  • the image display device 102 only displays an image received from the image provision device 101 .
  • Level 2 Exchange of vision and visual intervention are performed.
  • the image display device 102 displays an image received from the image provision device 101 and transmits information regarding an image to be displayed in the image provision device 101 (image to be superimposed and displayed and to be used for visual intervention).
  • Level 3 Auditory intervention is further performed.
  • the image display device 102 further transmits information regarding audio to be output by the image provision device 101 (audio to be listened to by the Body).
  • the image display device 102 further transmits information for operating the drive unit 505 and information regarding audio to be output from the second audio output unit 506 to the outside.
  • Such filtering processing may be performed on the Body side (i.e., in the image provision device 101 ) or may be performed in a JackIn server (provisional name) for controlling JackIn between a large number of Bodies and a large number of ghosts.
  • the Body can automatically determine a level at which each ghost is permitted intervention, which is convenient.
  • whether or not JackIn can be performed or an intervention level may be determined on the spot on the basis of negotiation, mediation, or the like between the Body and the ghost, instead of being automatically determined on the basis of information such as permission and mission set in advance.
  • JackIn is a situation in which a ghost is immersed in a first person experience of a Body in the visual information sharing system 100 , and the ghost interacts with the Body.
  • JackIn is roughly classified into a case where the Body takes the initiative in starting JackIn (Body initiative start) and a case where the ghost takes the initiative in starting JackIn (Ghost initiative start).
  • JackIn can be classified into a case where a single or (specified) small number of ghosts perform JackIn (Single (or small number) ghost) and a case where a (unspecified) large number of ghosts perform JackIn (Large number ghosts).
  • the case where the Body takes the initiative in starting JackIn is assumed to be a situation in which the Body requests assistance, instruction, guidance, or navigation regarding operation that is currently performed.
  • the Body requests a person to teach car repair work
  • the Body requests assistance, instruction, guidance, or navigation regarding operation demanding a comparatively high-level technology or skill in a medical site of surgical operation and the like, a construction site of a construction work and the like, and other sites.
  • JackIn is basically started when the ghost enters (performs JackIn to) the Body. Therefore, in a case where the Body desires to take the initiative in starting JackIn, the Body requests a desired ghost (or a predetermined number of ghosts) to enter the Body himself/herself and then starts operation in a standby state.
  • FIG. 6 schematically illustrates a start flow of Body initiative start.
  • FIG. 6 illustrates only a single ghost for simplification. However, a plurality of ghosts are assumed to exist.
  • the Body starts operation in the above standby state while opening “acceptance” for accepting ghosts.
  • the Body may invite ghosts by posting comments such as “Need help!”, “Please teach me how to drive a vehicle.”, and “Please tell me the way to ⁇ .” with the use of a social networking service (SNS).
  • the ghost may charge for (monetize) a service to perform JackIn to assist, instruct, guide, or navigate the Body regarding the operation thereof.
  • the Body may also present a payable price at the time of inviting ghosts on an SNS or the like. A ghost who answers the invitation transmits a JackIn request.
  • an external device such as a wearable terminal worn by the user of the image provision device 101
  • the external device notifies the Body.
  • the Body When the Body receives the notification from the wearable terminal while opening acceptance, the Body establishes connection with the ghost.
  • the Body When the Body achieves JackIn with a desired ghost or the number of connected ghosts reaches a predetermined number, the Body closes acceptance and thus does not accept notifications from wearable terminals any more. Thereafter, the Body shares vision with the ghost who has performed JackIn to the Body and performs the operation while being subjected to visual intervention or another intervention by the ghost.
  • JackIn is basically started in accordance with a sequence similar to the sequence in FIG. 6 .
  • the case where the Body takes the initiative in performing JackIn with a (unspecified) large number of ghosts is assumed to be a situation in which the Body requests unspecified people to give pieces of advice or perform slight operation such as operation that an assistant can do.
  • the Body invites ghosts who perform JackIn to the Body himself/herself on an SNS or the like and starts operation in a standby state. Every time when the wearable terminal receives a JackIn request from a ghost, the wearable terminal notifies the Body. In a case where the Body is connected to a ghost, whether or not connection can be established is automatically determined on the basis of criteria for selection such as past performance and review of the ghost or is directly determined by the user. Further, in a case where a plurality of ghosts have performed JackIn to the Body himself/herself, it is also assumed that permission or mission to be set is different for each ghost.
  • a procedure in which a single ghost (or a specified small number of ghosts) takes the initiative in starting JackIn is basically achieved by the ghost entering (performing JackIn to) the Body. This action resembles operation in which the ghost makes a phone call to the Body.
  • FIG. 7 schematically illustrates a start flow of ghost initiative start.
  • a JackIn request is transmitted from the ghost to the Body, and therefore a JackIn state is achieved.
  • a first person view is transmitted from the Body to the ghost, and the ghost intervenes in the Body.
  • the Body may set permission to the ghost who has performed JackIn to the Body himself/herself, and the ghost may set the own mission.
  • the image provision device 101 and the image display device 102 may present a user interface (UI) for setting permission and a UI for setting a mission to the users, respectively.
  • UI user interface
  • the Body can set a start condition of JackIn in advance.
  • the wearable terminal is set to notify the Body only when the start condition is satisfied, instead of notifying the Body every time when a JackIn request is received from a ghost.
  • the number of ghosts who answer invitation can be set as the start condition.
  • the wearable terminal notifies the Body. Only when the number of ghosts is one hundred or larger, a first person view is distributed from the Body existing on a site.
  • a specific example is a use case where video distribution is started when a Body who participates in a festival writes a message such as “I'm attending the festival now” and the number of ghosts who desires to view the festival is larger than one hundred.
  • a Body opens acceptance of In a case where a Body requests initiative JackIn and starts operation. advice from unspecified ghosts, start When a ghost receives a the Body opens acceptance of JackIn REQ and starts JackIn, JackIn and starts operation. the Body is notified, and then When a ghost enters the Body, sharing of vision and operation the Body is notified. for intervention are started. Ghost A ghost transmits a JackIn A Body sets a start condition. initiative REQ to a specified Body, and, When the start condition is start when the Body responds satisfied, a JackIn REQ is thereto, JackIn is started. transmitted to the Body.
  • a ghost can select or filter a Body to whom the ghost desires to perform JackIn on the basis of a current position of the Body or behavior (operation) that the Body currently performs. Processing for selecting a Body may be implemented by each ghost, or the JackIn server (provisional name) for controlling JackIn between Bodies and ghosts may be interposed for the selection processing.
  • the Body side i.e., the image provision device 101 measures a current position on the basis of a GPS or the like or recognizes behavior that the user currently performs on the basis behavior (activity) recognition, thereby notifying the ghost or JackIn server of the position and behavior.
  • behavior recognition of the Body may not be automatized and may be a method based on character input (writing) or audio input by the Body himself/herself.
  • description will be provided without limiting a mechanism for specifying a position and behavior of each Body.
  • FIG. 8 illustrates an example of a UI that allows the ghost to select a Body on the basis of position information of Bodies.
  • an icon (or character) indicating a current position of each Body is displayed on a map of a range that is currently specified.
  • Such a UI is displayed on, for example, the display unit 514 of the image display device 102 , and the user, i.e., the ghost can select a Body to whom the ghost desires to perform JackIn by specifying an icon in a desired position by UI operation such as a touch or click.
  • An area displayed as a map can be caused to transition by operation such as dragging or moving a cursor.
  • the UI screen illustrated in FIG. 8 may be displayed on a screen of another terminal possessed by the ghost, instead of the display unit 514 of a main body of the image display device 102 . Then, when selection is settled, a JackIn request is transmitted from the ghost to the selected Body.
  • FIG. 9 illustrates an example of a UI that allows the ghost to select a Body on the basis of not only position information of Bodies but also behavior thereof.
  • FIG. 9 is a display example where “person who is watching fireworks” is input to a search field and a target to be subjected to JackIn is limited to a “person who is watching fireworks”.
  • the JackIn server provisional name
  • the JackIn server searches for a Body matching with a keyword (herein, behavior of Body) input to the search field from among a Body group displayed on the map.
  • a keyword herein, behavior of Body
  • Input to the search field can be performed via character input or audio input.
  • Bodies who are not watching fireworks disappear, and therefore the ghost can reduce Bodies to be selected.
  • FIG. 10 illustrates another example of a UI that allows the ghost to select a Body on the basis of position information of Bodies.
  • FIG. 10 is a modification example of the UI illustrated in FIG. 8 , and a tag indicating behavior or the like of each Body is added to the icon of the Body.
  • the ghost can recognize behavior that each Body currently performs or the like on the basis of content of display of the tag and select a Body to whom the ghost desires to perform JackIn without performing operation to conduct a search by a search word as in the case illustrated in FIG. 9 .
  • tags when tags are constantly displayed on all the icons in the UI display example illustrated in FIG. 10 , display becomes complicated, and therefore the map cannot be easily read.
  • the number of tags that are simultaneously displayed may be restricted by, for example, displaying only a tag of an icon that is provisionally selected by a touch, click, hovering, or the like.
  • the tag may indicate not only information regarding behavior of the Body but also information regarding whether or not acceptance is opened (described above) and permission (a range in which intervention is permitted), charge information (sharing of vision is free or charged; charge information in a case where sharing of vision is charged), and the like.
  • FIG. 13 illustrates a display example of a tag added to the icon of the Body.
  • the example illustrated in FIG. 13 shows whether or not the Body permits each intervention operation such as visual intervention, auditory intervention, body intervention, and alternative conversation.
  • the ghost can easily determine what the ghost can do at that location when the ghost performs JackIn to the Body.
  • the ghost can find a position of a Body on map display and perform JackIn thereto (i.e., perform JackIn to the Body associated with the location), i.e., can achieve operation that the ghost visually understands with ease. Further, by using the UI illustrated in FIG. 9 , the ghost can smoothly perform JackIn to a Body who performs specified behavior.
  • FIG. 11 illustrates further another example of a UI that allows the ghost to select a Body.
  • FIG. 11 displays thumbnails of first person views of respective Bodies in detail in the form of a list, instead of displaying an icon indicating a current position of each Body on a map.
  • the thumbnail of each first person view may be a real-time video or representative image (still image). Further, the thumbnail of each first person view may be displayed together with tag information such as behavior of the Body, a current position of the Body, an acceptance state, a permission setting, and charge information.
  • FIG. 12 illustrates still further another example of a UI that allows the ghost to select a Body.
  • FIG. 12 is a modification example of the UI illustrated in FIG. 11 and displays thumbnails of first person views of respective Bodies in the form of a catalog instead of the form of a list.
  • the thumbnail of each first person view may be displayed together with tag information such as behavior of the Body, a current position of the Body, an acceptance state, a permission setting, and charge information.
  • FIG. 12 is a display example where Bodies serving as a target to be subjected to JackIn are limited to “people who are watching fireworks”.
  • the JackIn server provisional name
  • the JackIn server searches for a Body matching with a keyword (herein, behavior of Body) input to a search field.
  • the JackIn server searches for Bodies without association with a location, which is different from the example illustrated in FIG. 9 . Therefore, Bodies existing in separate locations such as Hokkaido and Okinawa are simultaneously displayed as search results in some cases as long as the Bodies are “watching fireworks”.
  • a video provided from a Body is not limited to a real-time video observed by the Body on a site and is a recorded past video in some cases.
  • the ghost is not permitted any intervention in the Body including visual intervention and auditory intervention. Therefore, in the UI example or the like illustrated in FIG. 11 or FIG. 12 , in order to prevent intervention caused by misunderstanding by the ghost, it is preferable that which one of the real-time and recorded past videos is displayed be indicated together with the thumbnail of the first person view.
  • the ghost can perform JackIn while visually recognizing behavior performed by each Body on the basis of a displayed thumbnail of a first person view. Further, by using the UI illustrated in FIG. 12 , the ghost can smoothly perform JackIn to a Body who performs specified behavior.
  • the ghost can efficiently select a Body in association with a location.
  • the Body selection UIs that display thumbnails of first person views in the form of a list or catalog illustrated in FIGS. 11 and 12 , the ghost can efficiently select a Body while visually recognizing behavior (activity).
  • those two types of Body selection UIs may be superimposed and the UIs may be switched by using a tab.
  • FIG. 14A when a “MAP” tab is selected, the map-based Body selection UI is displayed in front. Therefore, the ghost can select a Body to whom the ghost desires to perform JackIn in association with a location.
  • FIG. 14B when an “Activity” tab is selected, the Body selection UI that displays thumbnails of first person views of respective Bodies in the form of a catalog is displayed in front. Therefore, the ghost can select a Body selection UI while visually recognizing behavior of each Body.
  • the Body invites ghosts who perform JackIn to the Body himself/herself to assist the Body.
  • the Body may invite ghosts by posting comments such as “Need help!”, “Please teach me how to drive a vehicle.”, and “Please tell me the way to ⁇ .” with the use of a social networking service (SNS).
  • SNS social networking service
  • a ghost may charge for (monetize) a service to perform JackIn to assist, instruct, guide, or navigate a Body regarding the operation thereof.
  • a Body may also present a payable price at the time of inviting ghosts.
  • Ghosts who attempt to answer invitation can refer to the Body who has issued the invitation via, for example, the UI screens illustrated in FIGS. 8 to 12 .
  • description of the UI on the ghost side is omitted.
  • FIG. 15 illustrates an example of the UI that allows a Body to select a ghost.
  • the UI illustrated in FIG. 15 includes a list of ghosts to be selected and displays information of each ghost.
  • the listed ghosts are users who answer invitation of the Body.
  • the listed ghosts may be people who are selected by the JackIn server (provisional name) for controlling JackIn between Bodies and ghosts in accordance with content of invitation of the Body.
  • Each ghost listed on the UI illustrated in FIG. 15 is, for example, a user who has specified behavior such as a “person who is watching fireworks” and applies to JackIn to a Body.
  • Information of the ghosts displayed on the ghost selection UI illustrated in FIG. 15 includes not only personal information such as age, sex, a human relationship with the Body (family or kinship relation, friend, boss and subordinate, or the like), a place of birth, an occupation, and a qualification but also rating information of a skill of assistance target operation and information such as past performance (as an assistant, instructor, or the like) (how many hours the ghost has experienced the operation so far) and review of the ghost, and reputation by other Bodies (posting, voting result, or the like). Further, in a case where a list of ghosts is displayed on the ghost selection UI, display order of the ghosts may be sorted on the basis of correspondence between permission and mission, past performance, review, reputation, or the like.
  • the Body can select, via the ghost selection UI illustrated in FIG. 15 , a ghost by whom the Body desires to be, for example, assisted, instructed (coaching and the like in sport competition), guided, or navigated.
  • the technology disclosed in the present specification can be utilized for, for example, operation support and the like in various industrial fields such as a medical site of a surgical operation and the like, a construction site of a construction work and the like, control of airplanes and helicopters, navigation of drivers of automobiles, instructions in sports, and other uses.
  • An information processing device including:
  • control unit configured to control connection between a first device that transmits an image and a second device that receives the image in accordance with which of the first device and the second device takes initiative.
  • the control unit receives a connection request from the second device, notifies the first device in a standby state, and causes image transmission from the first device to the second device to start.
  • the control unit notifies the first device of a connection request from the second device and causes image transmission from the first device to the second device to start.
  • the control unit notifies the first device of connection requests from the plurality of second devices only in a case where the connection requests satisfy a predetermined start condition and causes image transmission from the first device to the plurality of second devices to start.
  • control unit controls a start of image transmission from the first device to the second device and intervention in the first device by the second device.
  • An information processing method including:
  • An information processing device including:
  • a selection unit configured to select a first device on a basis of position information of the first device, the first device transmitting an image to a second device.
  • the selection unit presents a UI that shows a position of the first device on a map.
  • the selection unit selects the first device further in consideration of behavior of a user.
  • the selection unit presents only the first device on the UI, the first device being extracted on a basis of behavior of a user.
  • the selection unit presents behavior of a user of the first device on the UI.
  • the selection unit presents information regarding intervention in the first device on the UI.
  • An information processing method including:
  • An information processing device including:
  • a selection unit configured to select a first device on a basis of behavior of a user of the first device, the first device transmitting an image to a second device.
  • the selection unit presents a UI that shows information regarding the image transmitted from the first device.
  • the selection unit presents information regarding the first device or the user of the first device on the UI.
  • the selection unit presents only the image on the UI, the image being transmitted from the first device extracted on the basis of the behavior of the user.
  • An information processing method including:
  • An information processing device including:
  • a selection unit configured to select a second device to which a first device transmits an image on a basis of information regarding the second device or a user of the second device.
  • the selection unit presents a UI that shows information regarding the second device or the user of the second device.
  • An information processing method including:
  • a selecting step of selecting a second device to which a first device transmits an image on a basis of behavior of a user of the first device a selecting step of selecting a second device to which a first device transmits an image on a basis of behavior of a user of the first device.

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • General Business, Economics & Management (AREA)
  • Business, Economics & Management (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • User Interface Of Digital Computer (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
US15/760,060 2015-09-30 2016-07-11 Information processing device and information processing method Abandoned US20180278888A1 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
JP2015-195193 2015-09-30
JP2015195193 2015-09-30
PCT/JP2016/070483 WO2017056632A1 (fr) 2015-09-30 2016-07-11 Dispositif de traitement d'informations et procédé de traitement d'informations

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2016/070483 A-371-Of-International WO2017056632A1 (fr) 2015-09-30 2016-07-11 Dispositif de traitement d'informations et procédé de traitement d'informations

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/381,593 Division US10771739B2 (en) 2015-09-30 2019-04-11 Information processing device and information processing method

Publications (1)

Publication Number Publication Date
US20180278888A1 true US20180278888A1 (en) 2018-09-27

Family

ID=58427403

Family Applications (2)

Application Number Title Priority Date Filing Date
US15/760,060 Abandoned US20180278888A1 (en) 2015-09-30 2016-07-11 Information processing device and information processing method
US16/381,593 Active US10771739B2 (en) 2015-09-30 2019-04-11 Information processing device and information processing method

Family Applications After (1)

Application Number Title Priority Date Filing Date
US16/381,593 Active US10771739B2 (en) 2015-09-30 2019-04-11 Information processing device and information processing method

Country Status (6)

Country Link
US (2) US20180278888A1 (fr)
EP (1) EP3358837A4 (fr)
JP (1) JPWO2017056632A1 (fr)
KR (1) KR102512855B1 (fr)
CN (1) CN108141565A (fr)
WO (1) WO2017056632A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11768578B2 (en) 2019-04-17 2023-09-26 Apple Inc. User interfaces for tracking and finding items
US11778421B2 (en) 2020-09-25 2023-10-03 Apple Inc. User interfaces for tracking and finding items
US11823558B2 (en) 2019-04-28 2023-11-21 Apple Inc. Generating tactile output sequences associated with an object

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110455304A (zh) * 2019-08-05 2019-11-15 深圳市大拿科技有限公司 车辆导航方法、装置及系统
CN112423084B (zh) * 2020-11-11 2022-11-01 北京字跳网络技术有限公司 热点榜单的显示方法、装置、电子设备和存储介质

Family Cites Families (34)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3393143B2 (ja) 1997-02-26 2003-04-07 三菱電機株式会社 ビデオデータ配信方法、ビデオデータ配信システム、並びに、そのビデオデータ配信方法
JP2003345909A (ja) * 2002-05-28 2003-12-05 Tokio Deguchi 学業指導方法および学業指導システム
JP2004222254A (ja) 2002-12-27 2004-08-05 Canon Inc 画像処理システム、方法及びプログラム
US20040254982A1 (en) * 2003-06-12 2004-12-16 Hoffman Robert G. Receiving system for video conferencing system
JP2005222254A (ja) 2004-02-04 2005-08-18 Haisaabu Ueno:Kk キャッシュレジスタ装置
JP4926400B2 (ja) 2004-12-27 2012-05-09 京セラ株式会社 移動カメラシステム
JP5245257B2 (ja) 2006-11-22 2013-07-24 ソニー株式会社 画像表示システム、表示装置、表示方法
US20100097463A1 (en) * 2007-04-17 2010-04-22 Panasonic Corporation Monitoring unit control system
CN101163160B (zh) * 2007-11-05 2011-04-06 中兴通讯股份有限公司 网络电视系统中融合多方网络游戏业务的方法及系统
US10875182B2 (en) * 2008-03-20 2020-12-29 Teladoc Health, Inc. Remote presence system mounted to operating room hardware
US10808882B2 (en) * 2010-05-26 2020-10-20 Intouch Technologies, Inc. Tele-robotic system with a robot face placed on a chair
JP2012133534A (ja) * 2010-12-21 2012-07-12 Mitsubishi Electric Corp 遠隔作業支援システム、遠隔作業支援端末及び遠隔作業支援方法
JP5750935B2 (ja) 2011-02-24 2015-07-22 富士ゼロックス株式会社 情報処理システム、情報処理装置、サーバ装置およびプログラム
US8874474B2 (en) * 2011-03-23 2014-10-28 Panasonic Intellectual Property Corporation Of America Communication server, communication method, memory medium and integrated circuit for mediating requests for content delivery according to expectation values of a probability of acceptance of the request, desired location, and history information
US20130027561A1 (en) * 2011-07-29 2013-01-31 Panasonic Corporation System and method for improving site operations by detecting abnormalities
US8761933B2 (en) 2011-08-02 2014-06-24 Microsoft Corporation Finding a called party
US20130249947A1 (en) * 2011-08-26 2013-09-26 Reincloud Corporation Communication using augmented reality
JP5741358B2 (ja) 2011-10-04 2015-07-01 トヨタ自動車株式会社 樹脂成形部品及び製造方法
JP2013078893A (ja) 2011-10-04 2013-05-02 Canon Inc 記録装置および記録方法
JP5114807B1 (ja) 2011-10-04 2013-01-09 株式会社新盛インダストリーズ プリンター
JP2013191464A (ja) 2012-03-14 2013-09-26 Sharp Corp 有機エレクトロルミネッセンス素子及びその製造方法、液晶表示装置。
JP5334145B1 (ja) * 2012-06-29 2013-11-06 トーヨーカネツソリューションズ株式会社 物品のピッキング作業の支援システム
JP2014104185A (ja) 2012-11-28 2014-06-09 Sony Corp 運動補助装置及び運動補助方法
US20160132046A1 (en) * 2013-03-15 2016-05-12 Fisher-Rosemount Systems, Inc. Method and apparatus for controlling a process plant with wearable mobile control devices
US9699500B2 (en) * 2013-12-13 2017-07-04 Qualcomm Incorporated Session management and control procedures for supporting multiple groups of sink devices in a peer-to-peer wireless display system
KR102159353B1 (ko) * 2014-04-24 2020-09-23 현대모비스 주식회사 어라운드 뷰 시스템의 동작방법
US9818225B2 (en) * 2014-09-30 2017-11-14 Sony Interactive Entertainment Inc. Synchronizing multiple head-mounted displays to a unified space and correlating movement of objects in the unified space
US10187692B2 (en) * 2014-12-15 2019-01-22 Rovi Guides, Inc. Methods and systems for distributing media guidance among multiple devices
CN104657099B (zh) 2015-01-15 2019-04-12 小米科技有限责任公司 屏幕投射方法、装置及系统
US9690103B2 (en) * 2015-02-16 2017-06-27 Philip Lyren Display an image during a communication
US9298283B1 (en) * 2015-09-10 2016-03-29 Connectivity Labs Inc. Sedentary virtual reality method and systems
KR101844885B1 (ko) * 2016-07-11 2018-05-18 엘지전자 주식회사 차량 운전 보조장치 및 이를 포함하는 차량
US11062243B2 (en) * 2017-07-25 2021-07-13 Bank Of America Corporation Activity integration associated with resource sharing management application
KR102188721B1 (ko) 2020-04-27 2020-12-08 현대모비스 주식회사 탑-뷰 영상 생성 장치 및 그 방법

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11768578B2 (en) 2019-04-17 2023-09-26 Apple Inc. User interfaces for tracking and finding items
US11960699B2 (en) * 2019-04-17 2024-04-16 Apple Inc. User interfaces for tracking and finding items
US11966556B2 (en) * 2019-04-17 2024-04-23 Apple Inc. User interfaces for tracking and finding items
US11823558B2 (en) 2019-04-28 2023-11-21 Apple Inc. Generating tactile output sequences associated with an object
US11778421B2 (en) 2020-09-25 2023-10-03 Apple Inc. User interfaces for tracking and finding items
US11968594B2 (en) 2020-09-25 2024-04-23 Apple Inc. User interfaces for tracking and finding items
US12041514B2 (en) 2020-09-25 2024-07-16 Apple Inc. User interfaces for tracking and finding items

Also Published As

Publication number Publication date
EP3358837A4 (fr) 2019-07-31
KR102512855B1 (ko) 2023-03-23
KR20180063040A (ko) 2018-06-11
US20190238793A1 (en) 2019-08-01
US10771739B2 (en) 2020-09-08
EP3358837A1 (fr) 2018-08-08
JPWO2017056632A1 (ja) 2018-07-19
WO2017056632A1 (fr) 2017-04-06
CN108141565A (zh) 2018-06-08

Similar Documents

Publication Publication Date Title
US10771739B2 (en) Information processing device and information processing method
Kurata et al. Remote collaboration using a shoulder-worn active camera/laser
US10628114B2 (en) Displaying images with integrated information
JP6822413B2 (ja) サーバ装置及び情報処理方法、並びにコンピュータ・プログラム
US20160188585A1 (en) Technologies for shared augmented reality presentations
EP2731348A2 (fr) Appareil et procédé de fourniture de service de réseau social utilisant une réalité augmentée
KR20190004088A (ko) 생체신호연동 가상현실 교육 시스템 및 방법
US20190121515A1 (en) Information processing device and information processing method
US20180278995A1 (en) Information processing apparatus, information processing method, and program
JPWO2018216355A1 (ja) 情報処理装置、情報処理方法、及びプログラム
US10986206B2 (en) Information processing apparatus, control method thereof, and computer readable medium for visual information sharing
WO2018075523A9 (fr) Système informatique vestimentaire audio/vidéo à projecteur intégré
WO2017068928A1 (fr) Dispositif de traitement d'informations, procédé de commande associé, et programme informatique
JP2024140478A (ja) プログラム、情報処理装置、及び、情報処理システム

Legal Events

Date Code Title Description
AS Assignment

Owner name: SONY CORPORATION, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:KASAHARA, SHUNICHI;REKIMOTO, JUNICHI;KIMURA, JUN;AND OTHERS;SIGNING DATES FROM 20180212 TO 20180221;REEL/FRAME:045228/0349

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION