EP1435737A1 - An augmented reality system and method - Google Patents

An augmented reality system and method Download PDF

Info

Publication number
EP1435737A1
EP1435737A1 EP03079135A EP03079135A EP1435737A1 EP 1435737 A1 EP1435737 A1 EP 1435737A1 EP 03079135 A EP03079135 A EP 03079135A EP 03079135 A EP03079135 A EP 03079135A EP 1435737 A1 EP1435737 A1 EP 1435737A1
Authority
EP
European Patent Office
Prior art keywords
local
orientation
remote
graphics
camera
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
EP03079135A
Other languages
German (de)
English (en)
French (fr)
Inventor
John Pretlove
Thomas Pettersen
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
ABB Research Ltd Switzerland
ABB Research Ltd Sweden
Original Assignee
ABB Research Ltd Switzerland
ABB Research Ltd Sweden
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ABB Research Ltd Switzerland, ABB Research Ltd Sweden filed Critical ABB Research Ltd Switzerland
Publication of EP1435737A1 publication Critical patent/EP1435737A1/en
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B19/00—Programme-control systems
    • G05B19/02—Programme-control systems electric
    • G05B19/42—Recording and playback systems, i.e. in which the programme is recorded from a cycle of operations, e.g. the cycle of operations being manually controlled, after which this record is played back on the same machine
    • G05B19/427—Teaching successive positions by tracking the position of a joystick or handle to control the positioning servo of the tool head, master-slave control
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1454—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/183—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a single remote source
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/35—Nc in input of data, input till input file format
    • G05B2219/35482—Eyephone, head-mounted 2-D or 3-D display, also voice and other control
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/36—Nc in input of data, input key till input tape
    • G05B2219/36452—Touch points with handheld probe, camera detects position and orientation probe
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/39—Robotics, robotics to robotics hand
    • G05B2219/39449—Pendant, pda displaying camera images overlayed with graphics, augmented reality

Definitions

  • the present invention relates to an augmented reality system comprising: a camera for capturing an image, the camera being movably located at a local site, a registering unit generating graphics and registering the generated graphics to the image from the camera, to provide a composite augmented reality image, a display device located at a remote site, physically separated from the local site, for displaying a view comprising the composite augmented reality image, and a communication link, for communication of information between the local and the remote site.
  • the present invention also relates to a method for remotely displaying an augmented reality view comprising graphical information overlaid an image captured at a local location.
  • the system and the method according to the invention is suitable for performing remote tasks and processes, where the operator is physically separated from the location where the tasks or processes are preformed.
  • the invention is, for example, useful in connection with remote robot programming, remote maintenance, inspection as well as remote collaboration.
  • the present invention also relates to a use of a system according to the invention for remote programming of an industrial robot.
  • Augmented reality is a well-known method of overlaying real world representations with computer-generated information. Ideally, for vision based augmented reality, the user will not be able to recognize the difference between the real and the computer generated graphics, and thereby getting an improved perception of the real world environment.
  • augmented reality techniques are used in a number of applications. New areas, where augmented reality is used is for instance within media (weather reporting), medicine (visualization of internal organs), for collaborative environments (virtual meeting room), and in the process industry for maintenance and service tasks.
  • remote augmented reality With traditional vision based augmented reality systems, computer generated information is added to a real, local scene. Visualizing a remote environment with overlaid computer generated graphics is hereinafter referred to as remote augmented reality. Remote augmented reality is useful for tasks, where an expert is located at a distance from where a physical task is to be executed, such as inspection in hazardous environments, and service and support tasks.
  • the document describes an internet worked reality system comprising one or more local stations, which may be AR or non-AR, but at least one of the local stations must be AR, and one or more remote stations, which may be AR or non-AR.
  • the local and the remote stations are networked together.
  • the remote stations can provide resources not available at the local AR station, such as data bases, high performance computing, and methods by which, a person at the remote station can interact with a person at the local AR station.
  • a trainee is located at the local AR station, while an instructor, located at the remote station, monitors and controls the training.
  • the operator performs tasks at the local AR station, while information and assistance are located at the remote AR station.
  • the local AR station comprises a head mounted display for use by a person at the local station, a camera arranged in a fixed relation to the head mounted display, a computer comprising software for generating graphical representations, a video mixer mixing the generated graphical representation with the image to achieve a composed augmented reality image.
  • the head mounted display is adapted for showing a view comprising the composed augmented reality image.
  • the view shown on the head mounted display depends on the orientation and position of the head of the user in the local AR station.
  • Information about the composed augmented reality is transferred to the remote station through a network connecting the local and the remote station.
  • the remote station comprises a display device adapted for displaying the composed augmented reality transferred thereto. A person located in the remote station will be able to view exactly the same view as a person wearing the head mounted display in the local station.
  • U.S. Patent No. 5,745,387 shows an augmented reality maintenance system for operating in a hazardous environment, comprising an environment modeler, which creates a computer model of the environment.
  • An environment renderer creates a plurality of images, each corresponding to a viewing location and orientation.
  • a remotely operated manipulator arm is attached at a fixed end to a stationary structure and is provided with a utility package, such as a video camera attached to a distal end of the manipulator arm. Actuators move the manipulator arm to desired locations within the environment. The position and orientation of the manipulator arm are determined by a unit adapted for sensing position and attitude.
  • This information is passed to a manipulator arm renderer, which creates an image from a pre-stored model of the manipulator arm, viewed from several different viewpoints.
  • a viewpoint for rendering a computed view of the environment is determined by the operator, either by inputting coordinates or by selecting from a pre-established list of viewpoint coordinates and provide it to the environment renderer, which generates an image corresponding to the shapes defined by the environment geometry when viewed from the selected viewpoint.
  • the resulting viewpoint formed by combining the position and orientation of the distal end of the manipulator arm and the offset displacements of the utility package, will allow an environment rendered to produce images corresponding to views of the environment as seen from the viewpoint of the utility package.
  • position and orientation of the distal end of the manipulator arm is provided to a manipulator arm renderer.
  • the manipulator arm renderer generates an image of the manipulator arm as seen from the same viewpoint used by the environment renderer.
  • a video mixer superimposes an image of the manipulator arm and the image of the environment and displays the superimposed images on a monitor, thereby allowing an operator to visualize a position of the manipulator arm relative to its environment.
  • the invention is particularly suitable in connection with remote control of an industrial robot.
  • Generation of high quality robot programs e.g. for painting, grinding, and polishing, requires skilled programmers with extended process experience. This is due to the nature of todays programming systems, where process specific knowledge is important in order to meet quality requirements. Building up and keeping this competence within the end customer's organization is both time consuming and expensive. Cost savings can be accomplished if experts are located remotely and thereby can support or perform robot programming to several clients.
  • the object of the present invention is to provide an improved augmented reality system, which allows a remote operator to perform remote tasks or processes in an augmented reality that is displayed to the operator in such a way that he experiences that he is actually on site.
  • the object is achieved by the initially defined system, characterized in that the system further comprises a specifying unit, for specification of a position and an orientation in the remote site, the camera is arranged such that its position and orientation is dependent on the specified position and orientation, and the registering unit is adapted for registering the generated graphics to the image in dependence of the specified position and orientation.
  • a local site refers to where the process or task is to be executed and a remote site refers to where the operator performs the process or task.
  • the system according to the invention allows a remote operator to determine which viewpoint to be displayed.
  • the operator specifies a position and an orientation corresponding to the desired viewpoint and the position and orientation of the camera is adjusted based on the specified position and orientation and the graphics are registered to the image based on the specified position and orientation.
  • the system according to the invention allows a remote operator to visualize the environment of a local site and overlaid computer generated graphics from different viewpoints and thus enables the remote operator to be virtually present at the local site.
  • An advantage gained by the invention is that the remote operators may see the local scene from different angels and get an improved perception of the environment by means of overlaid virtual graphics. There is no need for an expert to travel to the local site for service and support.
  • said specifying unit comprises a tracking unit adapted for determining the position and orientation of a movable device located in the remote site, in relation to a fixed pre-specified remote coordinate system.
  • the movable device is attached to or carried by the operator and thus the position and orientation of the operator determine the view displayed.
  • the tracking unit is either a separate unit or a part of the movable device.
  • the tracking unit could be any known type of sensor or technique used for determining position and orientation.
  • said movable device is the display device.
  • the display device is preferably a hand-held display device or a head-mounted display device.
  • the viewpoint displayed depends on the position and orientation of the display device. If the operator carries the display device, the actual movements of the operator control the position and orientation of the camera and accordingly the viewpoint displayed.
  • the system further comprises a robot located at the local site, the camera is mounted on the robot and the robot is arranged in such a manner that a movement of the robot, in relation to a fixed pre-specified local coordinate system depends on the specified position and orientation.
  • the movements of the robot depend of the movements of the movable device.
  • Whit a robot is meant an industrial robot.
  • a robot comprises a manipulator and a control system controlling the movements of the manipulator.
  • the system comprises a graphical generator, for generation of a graphical representation, and the registering unit is adapted for generating graphics based on the graphical representation. It is advantageous to firstly generate a graphical representation and then generate graphics based on the graphical representation, since the graphical representation can easily be transferred between the remote and the local site without any high demand on the bandwidth.
  • the system comprises operator input means located at the remote site, provided for feeding data related to graphics to be displayed, to the system.
  • operator input means located at the remote site, provided for feeding data related to graphics to be displayed, to the system.
  • the operator is able to add virtual information to the virtual environment, e.g. annotations or graphical illustrations.
  • virtual information e.g. annotations or graphical illustrations.
  • the remote augmented system will be able to provide the remote operator with the required images of the object to be inspected.
  • This embodiment of the invention provides the remote operator with possibility to add virtual information to the real environment, providing the operator with enhanced perception of the local scene.
  • the virtual information may comprise the programmed robot path and process related information describing the robot processing result.
  • the virtual information may comprise a temperature or radiation map of the real environment, or status information of an object or structure.
  • the system further comprises a pointing device and a tracking unit for determining the position of the pointing device and the system is adapted for generating a graphical representation of a point being presently pointed out by the pointing member based on the position of the pointing device.
  • the pointing device is handheld. Such a pointing device can for example be use by an operator at the remote site for teaching point a robot path.
  • the system comprises two pointing devices and two tracking units, the second pointing device and second tracking unit is used by an operator located at the local site.
  • the pointing device is particularly useful in connection with teaching a robot program when there is no personnel with robot programming skills at the local site, where the processing is to be performed, or if there is no model of a target object to be processed.
  • the pointing device allows a process expert, located remotely from where the robot is, to perform a certain task on an image of the actual target object.
  • a remote operator can perform the robot programming task by teaching points in the virtual world and the points taught are shown by a graphical representation.
  • the remote operator may wear a head-mounted display, showing a graphical representation of the real world as seen from the camera with overlaid computer generated graphics.
  • the computer-generated graphics may, for example, represent the pointing device, process related information, and the programmed robot path.
  • the pointing device is used to specify waypoints of the robot path and the operator input means is used to specify process related information to the system.
  • the system comprises a second specifying unit, for specifying a position and an orientation in the local site, a second registering unit for generation of graphics and for registering of the generated graphics to the real environment or to an image of the environment of the local site in dependence of the position and orientation specified by the second specifying unit, and a local display device adapted for displaying a view comprising the environment of the local site and the graphics projected on the environment.
  • a second display device at the local site makes it possible for a local operator to supervise the remote operator's actions.
  • This embodiment of the invention allows the remote operator and the local operator to simultaneously view the same environment with the same overlaid computer generated graphics from their own viewpoints. By sharing the same augmented environment, the local operator may support the remote operator.
  • the viewpoint displayed to the remote operator depends on the position and orientation specified by him, e.g. the position of his hand or head, and the viewpoint of the local operator depends on the position and orientation specified by him.
  • the system comprises a second movable device located at the local site, and the second specifying unit comprises a second tracking unit, determining the position and orientation of the second movable device.
  • said second movable device is the local display device.
  • the local display device is, for example, a handheld display device or a head-mounted display.
  • the system also comprises a second graphical generator, located at the local site, for generating a second graphical representation, and the local display device is adapted for displaying a view comprising the second graphical representation.
  • a second graphical generator located at the local site, for generating a second graphical representation
  • the local display device is adapted for displaying a view comprising the second graphical representation.
  • the communication link is adapted for communication of the generated graphical representations between the local and the remote site.
  • both the local and the remote operator to modify the graphic, but still both operators view the same graphics but from different viewpoints. Since only the graphical representation and the video signals from the camera are transferred between the local and the remote site, instead of the graphics or the entire composite augmented reality image, the communication bandwidth is reduced.
  • the system comprises a second camera for capturing an image, the camera being arranged in a fixed relation to the second movable device, and a second registering unit, registering the generated graphical representation to the image from the second camera, to provide a composed augmented reality image and that the local display device is adapted for displaying a view comprising the composed augmented reality image.
  • the computer-generated graphics are merged with an image of the real world.
  • the computer-generated graphics are displayed on see-through glasses worn by the operator. The generated graphics are projected on the glasses, so that the operator can see the graphics in relation to the real world and the object to be processed.
  • the remote display device is adapted for displaying a view seen from a first visual angle that depends on a position and an orientation specified at the remote site and the local display device is adapted for displaying the same view as the remote display device, seen from a second visual angle that depends on a position and orientation specified at the local site.
  • the system comprises means for transferring voices between the local and the remote site via the communication link.
  • the remote operator and the local operator may communicate by talk to each other.
  • said communication means comprises microphones or headsets located at the remote and the local site.
  • the communication link is a network such as the Internet.
  • the object is achieved by the initially defined method, comprising specifying a position and an orientation in a remote site that is physically separated from the local site, positioning and orientating a camera, located at the local site, according to the specified position and orientation, obtaining said image from the camera, generating graphics, generating a composite augmented reality image based on the image, the graphics, and the specified position and orientation, and displaying a view comprising the composite augmented reality image.
  • the object is achieved by a computer program directly loadable into the internal memory of a computer, comprising software code portions for performing the steps of the method according to the invention, when said product is run on a computer.
  • the computer program product is provided either on a computer readable medium or through a network, such as the Internet.
  • the object is achieved by a computer-readable medium having a program recorded thereon, where the program is to make a computer perform the steps of the method according to the invention, when said program is run on the computer.
  • the system comprises a handheld display device comprising the display member and a camera.
  • the handheld display device is arranged so that the user seems to look directly through the display.
  • a handheld display device is useful when a user needs to see the computer-generated information from time to time.
  • the invention allows tasks or processes, e.g. generation of robot programs, to be performed from remote locations.
  • Advantages gained by the invention when used for remote robot programming is: no need for an end-customer to hire and train personnel with robot programming competence and the robot programming will become more efficient with a pool of programming experts in a collaborative environment.
  • Fig. 1 illustrates an augmented reality system according to an embodiment of the invention.
  • the system comprises one section located at a remote site and one section located at a local site.
  • the local and the remote sections are connected to each other via a communication link 1.
  • One or more remote operators 3 are located at the remote site.
  • the remote section comprises a graphical display device 5, e.g. a head-mounted display device, a tracking unit 7 for determining the position and orientation of the display device 5 in relation to a fixed remote coordinate system (xR, yR, zR), a computer 9 containing the necessary software in order to generate a graphical representation and an augmented reality image, and an communication unit 11 for communication with the local site and the computer 9.
  • the tracking unit 7 has a fixed position and orientation in relation to the display device 5.
  • the computer 9 is for instance a wearable computer. Different types of handhold displays, such as a Personal Digital Assistant (PDA) screen could also be used.
  • PDA Personal Digital Assistant
  • the remote operator 3 also has a pointing device 13, which makes it possible for the operator to point out positions and orientations at the remote site.
  • the pointing device 13 comprises a tracking unit for determining the position and orientation of the pointing device 13. Positions and orientations from the pointing device tracking unit and the display tracking unit will be related to a remote coordinate system (xR, yR, zR), corresponding to a local coordinate system (xL, yL, zL) at the local site.
  • the pointing device comprises a number of interaction means, such as pushbuttons, which are adapted for interaction between the operator and the system.
  • the interaction means are, for example, adapted for input of information related to the process to the system.
  • the operator feds information to the system via a keyboard or any other known input means.
  • the display tracking unit 7 and the pointing device tracking unit is based on any known technology for determining position and orientation, such as image recognition, accelerometers, inertial sensors, gyros, magnetic trackers, ultrasound, laser technology, and a global positioning system (GPS).
  • GPS global positioning system
  • the local section of the system comprises one or a plurality of robot manipulators 17, and a camera unit 19 mounted at one end of a robot arm for capturing stereoscopic or monoscopic video streams of the object 15.
  • a tracking unit may be provided for determining the position and orientation of the robot-mounted camera.
  • a system for calculating the position and orientation of the camera based upon information from the robot control system is provided.
  • a robot controller 20 is connected to the robot manipulator or each of them, making it possible to move the robot arm according to a specified position and orientation in relation to the fixed local coordinate frame (xL, yL, zL) in real time. In the following a position and an orientation is denoted a pose.
  • the local section further comprises a communication unit 22 for communicating with the remote communication unit 11.
  • An industrial robot comprises a manipulator having a number of arms movable relative to each other and a control system controlling the movements of the arms.
  • the system is operated without a local operator present.
  • a local operator is present in order to support the remote operator.
  • the local section further comprises a graphical display device 26, e.g. a head-mounted display, a camera 28 mounted on the display device, for capturing a stream of images of the environment and the object 15, a tracking unit 27 for determining the position and orientation of the display device 26 and the camera 28 in relation to the local coordinate system, and a computer 32 containing the necessary software in order to generate a graphical representation and an augmented reality image based upon the video stream from the camera 28 and the generated graphical representation.
  • a graphical display device 26 e.g. a head-mounted display
  • a camera 28 mounted on the display device for capturing a stream of images of the environment and the object 15
  • a tracking unit 27 for determining the position and orientation of the display device 26 and the camera 28 in relation to the local coordinate system
  • a computer 32 containing the necessary software in order to generate a graphical representation and an augmented reality image based upon
  • the camera 28 is mounted in a fixed position in relation to the display device 26.
  • the display device will be located along the camera view axis and at the image plan of the camera.
  • the display device 26 is an optical see-through display and then, the camera is not needed, since the local operator 24 sees the real world environment and the object directly through the display device without the camera stream.
  • the local section also comprises a hand-held pointing device 34 with a tracking unit for determining its position and orientation in relation to the local coordinate system and means for interacting with the system.
  • the computer 32 is a wearable computer.
  • the remote operator 3 wears the display device 5 and the movements of the operator are sensed by means of the tracking unit 7.
  • the communication between the display unit 5 and the wearable computer 9 is done through a wired or wireless link.
  • the computer 9 contains the necessary software in order to generate a graphical representation and an augmented reality image based upon the video stream from the camera 19 and the generated graphical representation.
  • the computer 9 also contains the software needed to perform a desired task or process, e.g. generating a robot program.
  • the computer 9 will contain a storage medium in order to save and restore previously saved information.
  • the communication of information from the pointing device 13 to the computer 9 is done through a wired or a wireless link.
  • the pose of the pointing device 13 can come from the pointing device itself or from an external tracking unit.
  • the communication unit 11 contains the necessary software and hardware to allow communication from the computer 9 to the local communication unit 22. Communication from the computer 9 to the communication unit 11 can be provided by means of a wired or a wireless communication link. Communication between the remote 11 and the local 22 communication units may be done through the Internet or some other appropriate communication link.
  • the camera 19 is mounted at a distal end of the robot 17 and transmits images of the environment and the object 15 to the remote operator 3.
  • the robot controller 20 receives continuously updated poses from the remote display tracking unit 7, so that the camera is positioned according to the pose of the remote operator 3.
  • the position of the pointing device 34 and the display device 26 is related to the local coordinate system.
  • the local operator 24 wears the display device 26, which will provide the operator with an augmented reality view of the local environment and the object 15. If the display device 26 is of a video see-through type, the camera 28 is used for capturing images of the local environment. If the display device 26 is of an optical see-through type, the camera is not needed.
  • the local operator 24 uses the pointing device 34 to specify poses and interact in the shared virtual environment.
  • the position of the pointing device 34 is tracked in relation to the local coordinate system.
  • the computer 32 contains the necessary software to produce the local augmented reality view.
  • the computer 32 generates the process or task related graphics as seen by the remote operator.
  • the computer 32 also contains a graphics unit containing a graphical representation of the virtual world transferred from the remote section or other local sections via the communication link 1. Thus, it is not necessary to transfer the graphics in video format between the local and the remote site, thereby reducing the communication bandwidth requirements.
  • the computer 32 further comprises a storage device used to save process related information, e.g. virtual graphical representations, or process or task related information provided by the remote or local operator.
  • the local site may also contain more local operator interacting with the system in the same manner.
  • Fig. 2 is a block diagram of an augmented reality system according to an embodiment of the invention.
  • the remote section of the system located at the remote site, receives a video stream of images from the robot-mounted local camera 19.
  • the remote operator 3 is able to interact with the system through the pointing and interaction device 13.
  • the tracking unit makes it possible to trace the pose of the pointing device in 3D, and can also be used for specifying the remote coordinate system.
  • the pointing device 13 can be used for specifying waypoints in a robot path and for specifying process related information, while for remote inspection, the pointing device may be used for adding virtual information to the object or structure to be inspected.
  • the output from the pointing device 13 and the information fed to the system by the operator are transferred to an application unit 36.
  • the application unit 36 contains the necessary software to perform the desired process or task, e.g. generate a robot program or support remote inspections.
  • a graphics unit 37 comprises a graphical generator for generating a 3D graphical representation of the visual information that is to be displayed from the application unit 36.
  • the graphics unit 37 comprises specification of a number of 3D graphical elements to be visualized.
  • the graphics unit 37 also contains the entire virtual graphical representation from other remote or local sections. The positions of all the graphical elements are specified in relation to the remote coordinate system.
  • relevant visual information are operator specified waypoints, the actual robot path, and task specific information. Additionally, the pose of the pointing device will be rendered so that an operator may get visual indications of what other operators are pointing at.
  • the generated graphical representation is transmitted to a registering unit 38.
  • a storage unit 39 enables saving and loading of application related information.
  • the display-tracking unit 7 continuously delivers updated data regarding the pose of the display device 5.
  • the pose data are transmitted to the communication unit 11 for further transmission to the robot controller unit 20.
  • the pose data from the display-tracking unit 7 are also transmitted to the registering unit 38.
  • the registering unit 38 generates the graphics and based on the graphical representation in the graphics unit 37 it registers the graphics to the image from the camera unit 19 to provide a composed augmented reality image.
  • the registering unit 38 uses the pose data from the display-tracking unit 7 to overlay the computer generated graphics with the real world images. With the registration correctly done, the computer generated graphics will be virtually attached to the real world scene.
  • the combined images are displayed on the display device 5.
  • the local communication unit 22 receives continuous information regarding the pose of the remote display device 5 in relation to the remote coordinate system and feeds this data to the robot controller unit 20 steering the robot-mounted camera 19 to the same pose in relation to the local coordinate system.
  • the camera 19 continuously captures images of the scene, which are transmitted back to the communication unit 22 and further to the registering unit 38.
  • Fig. 3 shows another embodiment of the local section at the local site. This embodiment is provided for collaboration between a remote and a local operator.
  • the communication unit 22 receives the current pose of the remote display device 5, the current pose of the remote pointing device 13, and process specific information specified by the remote operator, and the graphical representations generated in the remote graphics unit 37.
  • the received graphical representations are transmitted to a local server unit 40.
  • a camera unit 28, which is attached to the display device 26, produces local real world images.
  • the pose of the camera 28 and the display device 26 are determined according to the local coordinate system and is transmitted to a registering unit 42 to overlay the real world scene with the computer generated graphics generated from a graphics module 44.
  • the graphics module 44 receives information from the remote site regarding the virtual graphical information to be displayed, which is a representation of the graphical elements to be rendered in the shared virtual environment.
  • the local operator uses the pointing and interaction device 34 to interact with the system.
  • the pose of the pointing device will be rendered with graphical information in such a way that it is visible through the shared virtual environment.
  • Graphical representations related to the local operator are transmitted by means of the communication unit 22 to the other operators.
  • Information from the pointing and interaction device 34 is transferred to the local graphics module 44 that generates a graphical representation of the pointing device.
  • This graphical representation is transferred to the local registering unit 42, to the communication unit 22, via the server unit 40, and further to the remote graphics unit 37, via the remote communication unit 11.
  • the remote graphics module adds the received graphical representation to the graphical representation generated based on the remote information.
  • the remote and the local operator are able to see the same view seen from different viewpoints.
  • the server unit 40 receives data from the remote site through the communication unit 22. Further, the server unit 40 holds information regarding the local coordinate system.
  • a storage unit 46 in communication with the server unit 40, is used to store the system information, e.g. application related information, graphical representations, and system configuration parameters, such as the local coordinate system.
  • One or several local operators may be attached to the server unit 40.
  • the system is adapted for transmitting voices to/from the remote operator via the communication link to/from the local operator.
  • voice By equipping both local and remote operators with microphones and headsets, it will be possible for the operators to communicate with each other by means of voice.
  • the following comprises a description of how the augmented reality system according to the invention could be used.
  • Either the local or the remote operator can initiate a session. Alternatively, a session is scheduled in advance.
  • the local coordinate system needs to be specified. One way of doing this by a six point calibration method, where the operator specifies six points describing the position and orientation of the coordinate system.
  • All pose information provided by the local tracking units will be in relation to this local coordinate system.
  • the position and orientation of the robot or robots in relation to this local coordinate system also needs to be known. This way, a remotely specified point can be mapped to the robot coordinate system.
  • a remote operator starts operating the system by wearing the display device.
  • the remote operator then needs to specify the remote coordinate system, which will be corresponding with the local coordinate system.
  • the remote coordinate system may be specified in the same manner as the local word coordinate system. All pose information provided by the remote tracking units will be in relation to this remote coordinate system.
  • a target object is positioned at a desired position in relation to the local coordinate system.
  • the movements of the remote display device will be captures by the display tracking unit and transmitted to the local application server by means of the communication link.
  • the local communication server then sends a command to the robot controller to move the camera to the pose specified by the remote display-tracking unit.
  • the pose of the robot-mounted camera will move to the same pose related to the local coordinate system, as the tracking unit will be in relation to the remote coordinate system.
  • the pose of the remote pointing device is tracked in relation to the remote coordinate system.
  • the pose information regarding the remote pointing device is transferred to the local communication unit. If a local operator is present, the position of the remote operator's pointing device will be used to render a computer generator graphics on the local display device of the local operator. Thus, the local operator will be able to visualize the movements of the remote operator's pointing device in real time by means of virtual 3D graphics overlaid the real scene.
  • the remote user may now perform the desired process, e.g. generate a robot program.
  • process related information is visualized with computer generated 3D graphics on the remote display device.
  • the graphics is overlaid the world representation from the local camera and is fixed with respect to the remote coordinate system. This makes it possible for the remote user to virtually move around the target object with a computer-generated graphics attached to the image of the target object.
  • the movement of the remote operator will be captured and transmitted to the local robot-mounted camera in such a way that his camera moves according to the remote operator.
  • the local operator will also be able to visualize the process result in the same manner as the remote operator by means of 3D computer-generated graphics displayed on the local display device.
  • the process graphics will be registered and overlaid the real world representation as seen from the local operator, hence when the local operator moves to a new position, the computer-generated process graphics will be visualized as fixed with respect to the local coordinate system.
  • the position of the local pointing device is visualized to the local and the remote operator.
  • the position of the local pointing device is tracked in relation to the local coordinate system.
  • Pose information regarding the local pointing device is transferred to the remote site via the local and remote communication units.
  • the pose information of the local operator's pointing device is used to render computer-generated graphics on the remote display device and the local display device.
  • the remote operator will be able to visualize the movements of the pointing device of the local operator. If there are more remote operators, each remote operator will be able to visualize the movements of the pointing devices of the other remote operators
  • the software used for implementing the method according to the invention is partly based on software known to those skilled in the art.
  • the positions and orientations of the pointing member may be generated in ARToolKit.
  • the ARToolKit is developed by Washington University and the University of Hiroshima and is an open-source software library that enables augmented reality applications to be built using accurate computer vision-based tracking techniques.
  • the software Open GL may be used. Open GL provides a library of 2D and 3D functions including modeling alterations, color, light and shade functions.
  • Microsoft Vision SDK is a library for writing programs to perform image manipulation and analyses on computers.
  • the augmented reality software includes algorithms for drawing graphics, such as points and lines, transferring positions and orientations between different coordinate systems, extracting and generating a sequence list of positions and orientations, capturing process related information, and drawing advanced graphics, such as color-coded points and lines representing paint strokes with different widths and orientations.
  • the remote site is provided with more than one camera, more than one display device, more than one specifying unit, more than one graphical generator, and more than one registering unit to enable two or more remote operators to simultaneously view the environment of the local site and the overlaid graphics from their own view.
  • the local site is also provided with more than one camera, more than one display device, more than one specifying unit, more than one graphical generator, and more than one registering unit to enable two or more local operators to simultaneously view the environment of the local site and the overlaid graphics, from different view points.
  • the operator feeds the position and the orientation to the system.
  • the operator specifies the position and orientation by means of one or more joy-sticks.
  • the display device is a stationary computer screen.
  • the system comprises a handheld augmented reality display device.
  • the operator holds the handheld display device, showing the real world combined with overlaid computer-generated graphics.
  • the computer-generated graphics may represent process related information, e.g. the programmed robot path, process specific information, target points and events. Either the operator's movements of his hand with a gesture-based recognition system or a pointing and interaction device is used to specify the process related information to the system.
  • the computer-generated information represents operator input for robot programming.
  • the operator utilizes the handheld AR display to see the result of what he is doing while he is making a new robot program.
  • the handheld display has a camera integrated with the same field of view as the display. The camera captures live video of the real world.
  • the system combines and synchronizes the live video with computer-generated graphics representing the operator's input and presents it on the hand-held AR display.
  • the operator can freely move the handheld AR display in the environment, allowing the operator to see the local environment including the object from different viewpoints with overlaid computer generated graphics.
  • the operator "looks" through the handheld AR display to see the generated robot program in relation to the real object.
  • the system comprises a handheld interaction/pointing device 1 with a tracking system for determining its position and orientation in relation to a world coordinate system 60.
  • the system comprises a gesture-based recognition system comprising a recognition system for recognizing and determining the position and orientation of a hand or fingers in relation to the world coordinate system.
  • the system further comprises a handheld display device 62, e.g. a tablet PC or a PDA (Personal Digital Assistant).
  • the display device comprises a display member 64, a camera 8 for capturing a stream of images of the environment mounted on or integrated with the display device 64.
  • the camera 8 will be mounted in a fixed position in relation to the display device 64.
  • the display device will be located along the camera view axis and at the camera's image plane.
  • the system further comprises a tracking system 66 for determining the position and orientation of the display devices.
  • the tracking system is preferable mounted on the display device.
  • the system also comprises a system for generating an augmented reality representation of computer-generated graphical information overlaid the real world representation. The operator 2 movements are sensed through the display tracking system 66.
  • a wearable computer 68 contains the necessary software in order to generate the augmented reality environment based upon the video stream from the camera 8 and the computer-generated graphics.
  • the wearable computer 68 also contains the necessary software needed to perform the desired task or process, e.g. generating a robot program and performing reachability check. Further, the wearable computer will generate the graphics, which will provide the augmented reality view. Finally, the wearable computer will contain a storage media in order to save, and restore previously saved, information. Communication of the information from the interaction/pointing device 1 to the wearable computer 68 is done through a wired or wireless link.
  • the operator carries the display device 62 which will provide him/her with an augmented reality view of the environment.
  • the display device is of "video see through" type.
  • Video see-through is used to generate and present an augmented reality world at the handheld AR display device.
  • the camera integrated with the display device is used to capture a live video stream of the real world.
  • the camera is located in relation with the display in such a way that it provides the same view, as the user would get by looking "through" the display device.
  • the live video stream combined with computer-generated graphics is presented in real-time at the display device. Additional functionality includes camera zooming with output of the actual camera focal length. This will enable the system to display the computer-generated graphics correctly while zooming.
  • the camera may also be used for vision-based tracking if vision-based tracking is used as the tracking system.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Human Computer Interaction (AREA)
  • General Engineering & Computer Science (AREA)
  • Automation & Control Theory (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Processing Or Creating Images (AREA)
  • Manipulator (AREA)
  • User Interface Of Digital Computer (AREA)
EP03079135A 2002-12-30 2003-12-19 An augmented reality system and method Ceased EP1435737A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
SE0203908A SE0203908D0 (sv) 2002-12-30 2002-12-30 An augmented reality system and method
SE0203908 2002-12-30

Publications (1)

Publication Number Publication Date
EP1435737A1 true EP1435737A1 (en) 2004-07-07

Family

ID=20290053

Family Applications (1)

Application Number Title Priority Date Filing Date
EP03079135A Ceased EP1435737A1 (en) 2002-12-30 2003-12-19 An augmented reality system and method

Country Status (4)

Country Link
US (1) US7714895B2 (ja)
EP (1) EP1435737A1 (ja)
JP (1) JP2004213673A (ja)
SE (1) SE0203908D0 (ja)

Cited By (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1537959A3 (en) * 2003-11-24 2008-01-23 Abb Research Ltd. A method and a system for programming an industrial robot
WO2008059086A1 (es) * 2006-11-16 2008-05-22 The Movie Virtual, S.L. Sistema y metodo para la visualizacion de una imagen aumentada aplicando tecnicas de realidad aumentada
GB2456802A (en) * 2008-01-24 2009-07-29 Areograph Ltd Image capture and motion picture generation using both motion camera and scene scanning imaging systems
EP2071841A3 (en) * 2007-12-12 2009-12-16 Nokia Corp. Method, apparatus and computer program product for displaying virtual media items in a visual media
WO2010008518A1 (en) * 2008-07-15 2010-01-21 Eastman Kodak Company Image capture and display configuration
WO2011025450A1 (en) * 2009-08-25 2011-03-03 Xmreality Research Ab Methods and systems for visual interaction
AU2011205223B1 (en) * 2011-08-09 2012-09-13 Microsoft Technology Licensing, Llc Physical interaction with virtual objects for DRM
WO2012131148A1 (en) * 2011-03-31 2012-10-04 Nokia Corporation Method and apparatus for providing collaboration between remote and on-site users of indirect augmented reality
WO2013113984A1 (en) * 2012-02-02 2013-08-08 Nokia Corporation Methods, apparatuses, and computer-readable storage media for providing interactive navigational assistance using movable guidance markers
EP1763845B1 (de) * 2005-03-02 2013-10-09 KUKA Laboratories GmbH Verfahren und vorrichtung zur bestimmung von optischen überdeckungen mit ar-objekten
US8676615B2 (en) 2010-06-15 2014-03-18 Ticketmaster Llc Methods and systems for computer aided event and venue setup and modeling and interactive maps
WO2014049200A1 (en) * 2012-09-28 2014-04-03 Nokia Corporation Method and apparatus for providing an indication regarding content presented to another user
US9268406B2 (en) 2011-09-30 2016-02-23 Microsoft Technology Licensing, Llc Virtual spectator experience with a personal audio/visual apparatus
US9286711B2 (en) 2011-09-30 2016-03-15 Microsoft Technology Licensing, Llc Representing a location at a previous time period using an augmented reality display
CN106128212A (zh) * 2016-08-27 2016-11-16 大连新锐天地传媒有限公司 基于增强现实技术的书法学习系统及方法
US9606992B2 (en) 2011-09-30 2017-03-28 Microsoft Technology Licensing, Llc Personal audio/visual apparatus providing resource management
WO2017060539A1 (de) * 2015-10-08 2017-04-13 Sami Haddadin Robotersystem
CN106569671A (zh) * 2016-11-09 2017-04-19 唐雪松 基于ar技术的消费品使用说明信息显示方法及装置
WO2017093289A3 (en) * 2015-12-01 2017-07-27 Bragi GmbH Robotic safety using wearables
US20170278262A1 (en) 2014-07-31 2017-09-28 Sony Corporation Information processing device, method of information processing, and image display system
US9781170B2 (en) 2010-06-15 2017-10-03 Live Nation Entertainment, Inc. Establishing communication links using routing protocols
KR20180064494A (ko) * 2015-10-08 2018-06-14 카스타니엔바움 게엠바하 로보틱 시스템
NO342793B1 (en) * 2017-06-20 2018-08-06 Augmenti As Augmented reality system and method of displaying an augmented reality image
EP3553730A1 (en) * 2018-04-10 2019-10-16 Prisma Systems Corporation System and method for the creation and management of digital product visuals
US10573084B2 (en) 2010-06-15 2020-02-25 Live Nation Entertainment, Inc. Generating augmented reality images using sensor and location data
US10625414B2 (en) 2015-08-14 2020-04-21 Franka Emika Gmbh Robotic system and housing part for such robotic system
EP3610996A4 (en) * 2017-04-11 2021-01-13 Kawasaki Jukogyo Kabushiki Kaisha ROBOTIC SYSTEM AND METHOD OF OPERATING IT
US11040455B2 (en) 2015-10-08 2021-06-22 Haddadin Beteiligungs Ug Robot system and method for controlling a robot system
US11358275B2 (en) 2016-04-20 2022-06-14 Franka Emika Gmbh Drive unit for a robot and method for manufacturing the same
US11623355B2 (en) 2016-04-20 2023-04-11 Kastanienbaum GmbH Method for producing a robot and device for carrying out said method

Families Citing this family (233)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8944070B2 (en) 1999-04-07 2015-02-03 Intuitive Surgical Operations, Inc. Non-force reflecting method for providing tool force information to a user of a telesurgical system
US7046214B2 (en) * 2003-12-17 2006-05-16 Information Decision Technologies, Llc Method and system for accomplishing a scalable, multi-user, extended range, distributed, augmented reality environment
NO317898B1 (no) * 2002-05-24 2004-12-27 Abb Research Ltd Fremgangsmate og system for a programmere en industrirobot
SE525108C2 (sv) * 2002-12-30 2004-11-30 Abb Research Ltd Metod och system för programmering av en industrirobot, datorprogramprodukt, datorläsbart medium samt användning
US11033821B2 (en) 2003-09-02 2021-06-15 Jeffrey D. Mullen Systems and methods for location based games and employment of the same on location enabled devices
WO2005066744A1 (en) * 2003-12-31 2005-07-21 Abb Research Ltd A virtual control panel
US7236854B2 (en) * 2004-01-05 2007-06-26 Abb Research Ltd. Method and a system for programming an industrial robot
US7961909B2 (en) 2006-03-08 2011-06-14 Electronic Scripting Products, Inc. Computer interface employing a manipulated object with absolute pose detection component and a display
KR100590549B1 (ko) * 2004-03-12 2006-06-19 삼성전자주식회사 3차원 포인팅 방법을 이용한 로봇의 원격 제어 방법 및이를 구현한 로봇 제어 시스템
JP4553362B2 (ja) * 2005-01-31 2010-09-29 キヤノン株式会社 システム、画像処理装置、情報処理方法
EP1686554A3 (en) * 2005-01-31 2008-06-18 Canon Kabushiki Kaisha Virtual space generating system, image processing apparatus and information processing method
WO2006086223A2 (en) * 2005-02-08 2006-08-17 Blue Belt Technologies, Inc. Augmented reality device and method
JP4738870B2 (ja) * 2005-04-08 2011-08-03 キヤノン株式会社 情報処理方法、情報処理装置および遠隔複合現実感共有装置
US9789608B2 (en) 2006-06-29 2017-10-17 Intuitive Surgical Operations, Inc. Synthetic representation of a surgical robot
JP4933164B2 (ja) * 2005-07-01 2012-05-16 キヤノン株式会社 情報処理装置、情報処理方法、プログラムおよび記憶媒体
FR2889761A1 (fr) * 2005-08-09 2007-02-16 Total Immersion Sa Systeme permettant a un utilisateur de localiser une camera afin de pouvoir inserer, rapidement de maniere ajustee, des images d'elements virtuels dans des images video d'elements reels captees par la camera
US9250703B2 (en) 2006-03-06 2016-02-02 Sony Computer Entertainment Inc. Interface with gaze detection and voice input
US8730156B2 (en) 2010-03-05 2014-05-20 Sony Computer Entertainment America Llc Maintaining multiple views on a shared stable virtual space
DE102006022483A1 (de) * 2006-05-13 2007-11-29 Kuka Roboter Gmbh Verfahren und Vorrichtung zum Anzeigen einer Roboterbahn zur Unterstützung einer Ortsänderung eines Stützpunktes
EP2024762A4 (en) * 2006-06-09 2012-11-07 Inova Ltd ACQUISITION OF SEISMIC DATA
CN104688349B (zh) 2006-06-13 2017-05-10 直观外科手术操作公司 微创手术系统
US10258425B2 (en) 2008-06-27 2019-04-16 Intuitive Surgical Operations, Inc. Medical robotic system providing an auxiliary view of articulatable instruments extending out of a distal end of an entry guide
US9718190B2 (en) 2006-06-29 2017-08-01 Intuitive Surgical Operations, Inc. Tool position and identification indicator displayed in a boundary area of a computer display screen
US20090192523A1 (en) 2006-06-29 2009-07-30 Intuitive Surgical, Inc. Synthetic representation of a surgical instrument
US10008017B2 (en) 2006-06-29 2018-06-26 Intuitive Surgical Operations, Inc. Rendering tool information as graphic overlays on displayed images of tools
JP5228307B2 (ja) 2006-10-16 2013-07-03 ソニー株式会社 表示装置、表示方法
US9138129B2 (en) 2007-06-13 2015-09-22 Intuitive Surgical Operations, Inc. Method and system for moving a plurality of articulated instruments in tandem back towards an entry guide
US9469034B2 (en) 2007-06-13 2016-10-18 Intuitive Surgical Operations, Inc. Method and system for switching modes of a robotic system
US9084623B2 (en) 2009-08-15 2015-07-21 Intuitive Surgical Operations, Inc. Controller assisted reconfiguration of an articulated instrument during movement into and out of an entry guide
US8620473B2 (en) 2007-06-13 2013-12-31 Intuitive Surgical Operations, Inc. Medical robotic system with coupled control modes
US9089256B2 (en) 2008-06-27 2015-07-28 Intuitive Surgical Operations, Inc. Medical robotic system providing an auxiliary view including range of motion limitations for articulatable instruments extending out of a distal end of an entry guide
NO327577B1 (no) * 2007-07-31 2009-08-24 Kongsberg Defence & Aerospace Naerobservasjonssensor med folgestyring og malbestemmelse for vapenstasjon
US8094090B2 (en) * 2007-10-19 2012-01-10 Southwest Research Institute Real-time self-visualization system
US8485038B2 (en) * 2007-12-18 2013-07-16 General Electric Company System and method for augmented reality inspection and data visualization
US8838489B2 (en) 2007-12-27 2014-09-16 Amazon Technologies, Inc. On-demand generating E-book content with advertising
JP2009237878A (ja) * 2008-03-27 2009-10-15 Dainippon Printing Co Ltd 複合映像生成システム、重畳態様決定方法、映像処理装置及び映像処理プログラム
DE102008020772A1 (de) * 2008-04-21 2009-10-22 Carl Zeiss 3D Metrology Services Gmbh Darstellung von Ergebnissen einer Vermessung von Werkstücken
DE102008021160A1 (de) * 2008-04-28 2009-10-29 Beckhoff Automation Gmbh Fernbedienung
US20110112934A1 (en) * 2008-06-10 2011-05-12 Junichi Ishihara Sensory three-dimensional virtual real space system
US8864652B2 (en) 2008-06-27 2014-10-21 Intuitive Surgical Operations, Inc. Medical robotic system providing computer generated auxiliary views of a camera instrument for controlling the positioning and orienting of its tip
US9600067B2 (en) * 2008-10-27 2017-03-21 Sri International System and method for generating a mixed reality environment
US8817092B2 (en) * 2008-11-25 2014-08-26 Stuart Leslie Wilkinson Method and apparatus for generating and viewing combined images
KR101590331B1 (ko) * 2009-01-20 2016-02-01 삼성전자 주식회사 이동 가능한 디스플레이 장치와 이를 구비한 로봇 및 그 디스플레이 방법
US8970690B2 (en) * 2009-02-13 2015-03-03 Metaio Gmbh Methods and systems for determining the pose of a camera with respect to at least one object of a real environment
US9052710B1 (en) * 2009-03-20 2015-06-09 Exelis Inc. Manipulation control based upon mimic of human gestures
US20100259619A1 (en) * 2009-04-10 2010-10-14 Nicholson Timothy J Hmd with elevated camera
US8839121B2 (en) * 2009-05-06 2014-09-16 Joseph Bertolami Systems and methods for unifying coordinate systems in augmented reality applications
US8918211B2 (en) 2010-02-12 2014-12-23 Intuitive Surgical Operations, Inc. Medical robotic system providing sensory feedback indicating a difference between a commanded state and a preferred pose of an articulated instrument
US9492927B2 (en) 2009-08-15 2016-11-15 Intuitive Surgical Operations, Inc. Application of force feedback on an input device to urge its operator to command an articulated instrument to a preferred pose
US8473101B2 (en) * 2009-08-21 2013-06-25 Harris Corporation Coordinated action robotic system and related methods
US8542267B1 (en) * 2009-10-01 2013-09-24 Hewlett-Packard Development Company, L.P. Calibrating a visual-collaborative system
WO2011080882A1 (ja) * 2009-12-28 2011-07-07 パナソニック株式会社 動作空間提示装置、動作空間提示方法およびプログラム
US8400548B2 (en) 2010-01-05 2013-03-19 Apple Inc. Synchronized, interactive augmented reality displays for multifunction devices
KR101487944B1 (ko) 2010-02-24 2015-01-30 아이피플렉 홀딩스 코포레이션 시각 장애인들을 지원하는 증강 현실 파노라마
KR20110116525A (ko) * 2010-04-19 2011-10-26 엘지전자 주식회사 3ｄ 오브젝트를 제공하는 영상표시장치, 그 시스템 및 그 동작 제어방법
US9858475B2 (en) * 2010-05-14 2018-01-02 Intuitive Surgical Operations, Inc. Method and system of hand segmentation and overlay using depth data
US9149929B2 (en) * 2010-05-26 2015-10-06 The Boeing Company Methods and systems for inspection sensor placement
US9122707B2 (en) * 2010-05-28 2015-09-01 Nokia Technologies Oy Method and apparatus for providing a localized virtual reality environment
US20110316845A1 (en) * 2010-06-25 2011-12-29 Palo Alto Research Center Incorporated Spatial association between virtual and augmented reality
US8559030B2 (en) 2010-07-27 2013-10-15 Xerox Corporation Augmented reality system and method for device management and service
KR101299910B1 (ko) * 2010-08-18 2013-08-23 주식회사 팬택 증강 현실 서비스의 공유 방법 및 그를 위한 사용자 단말기와 원격자 단말기
US8902254B1 (en) 2010-09-02 2014-12-02 The Boeing Company Portable augmented reality
US10026227B2 (en) 2010-09-02 2018-07-17 The Boeing Company Portable augmented reality
CN102802884B (zh) * 2010-11-12 2015-04-08 松下电器产业株式会社 移动路径搜索装置及移动路径搜索方法
US9264515B2 (en) 2010-12-22 2016-02-16 Intel Corporation Techniques for mobile augmented reality applications
US8922645B1 (en) * 2010-12-22 2014-12-30 Google Inc. Environmental reproduction system for representing an environment using one or more environmental sensors
US9090214B2 (en) 2011-01-05 2015-07-28 Orbotix, Inc. Magnetically coupled accessory for a self-propelled device
US9150263B2 (en) 2011-01-05 2015-10-06 Sphero, Inc. Self-propelled device implementing three-dimensional control
US10281915B2 (en) 2011-01-05 2019-05-07 Sphero, Inc. Multi-purposed self-propelled device
US9218316B2 (en) 2011-01-05 2015-12-22 Sphero, Inc. Remotely controlling a self-propelled device in a virtualized environment
US9429940B2 (en) 2011-01-05 2016-08-30 Sphero, Inc. Self propelled device with magnetic coupling
US9113050B2 (en) * 2011-01-13 2015-08-18 The Boeing Company Augmented collaboration system
US8918214B2 (en) 2011-01-19 2014-12-23 Harris Corporation Telematic interface with directional translation
US8918215B2 (en) 2011-01-19 2014-12-23 Harris Corporation Telematic interface with control signal scaling based on force sensor feedback
KR101292463B1 (ko) * 2011-01-27 2013-07-31 주식회사 팬택 원격으로 증강현실 서비스를 공유하는 증강현실 시스템 및 그 방법
KR101329935B1 (ko) * 2011-01-27 2013-11-14 주식회사 팬택 이종 마커를 이용해서 원격으로 증강현실 서비스를 공유하는 증강현실 시스템 및 그 방법
CN102147658B (zh) * 2011-02-12 2013-01-09 华为终端有限公司 实现扩增实境互动的方法、扩增实境互动装置及移动终端
US9205555B2 (en) 2011-03-22 2015-12-08 Harris Corporation Manipulator joint-limit handling algorithm
US8810598B2 (en) 2011-04-08 2014-08-19 Nant Holdings Ip, Llc Interference based augmented reality hosting platforms
CA2833544A1 (en) * 2011-04-18 2012-10-26 Eyesee360, Inc. Apparatus and method for panoramic video imaging with mobile computing devices
US20170028557A1 (en) 2015-07-28 2017-02-02 Comprehensive Engineering Solutions, Inc. Robotic navigation system and method
US8694134B2 (en) 2011-05-05 2014-04-08 Harris Corporation Remote control interface
WO2012154938A1 (en) 2011-05-10 2012-11-15 Kopin Corporation Headset computer that uses motion and voice commands to control information display and remote devices
US8639386B2 (en) 2011-05-20 2014-01-28 Harris Corporation Haptic device for manipulator and vehicle control
US10120438B2 (en) 2011-05-25 2018-11-06 Sony Interactive Entertainment Inc. Eye gaze to alter device behavior
WO2012166814A1 (en) * 2011-05-31 2012-12-06 Honda Motor Co., Ltd. Online environment mapping
US9026250B2 (en) 2011-08-17 2015-05-05 Harris Corporation Haptic manipulation system for wheelchairs
US9128520B2 (en) 2011-09-30 2015-09-08 Microsoft Technology Licensing, Llc Service provision using personal audio/visual system
US8996244B2 (en) 2011-10-06 2015-03-31 Harris Corporation Improvised explosive device defeat system
WO2013078345A1 (en) 2011-11-21 2013-05-30 Nant Holdings Ip, Llc Subscription bill service, systems and methods
KR101941844B1 (ko) 2012-01-10 2019-04-11 삼성전자주식회사 로봇 및 그 제어방법
JP5516610B2 (ja) * 2012-01-19 2014-06-11 株式会社安川電機 ロボット、ロボットハンドおよびロボットハンドの保持位置調整方法
KR101978740B1 (ko) 2012-02-15 2019-05-15 삼성전자주식회사 원격조종시스템 및 그 제어방법
WO2013131036A1 (en) * 2012-03-01 2013-09-06 H4 Engineering, Inc. Apparatus and method for automatic video recording
US8947456B2 (en) 2012-03-22 2015-02-03 Empire Technology Development Llc Augmented reality process for sorting materials
JPWO2013145614A1 (ja) * 2012-03-27 2015-12-10 パナソニックＩｐマネジメント株式会社 情報処理装置、サーバ装置、情報処理方法、およびプログラム
US9827487B2 (en) * 2012-05-14 2017-11-28 Sphero, Inc. Interactive augmented reality using a self-propelled device
CN104428791A (zh) 2012-05-14 2015-03-18 澳宝提克斯公司 通过检测图像中的圆形物体操作计算装置
US20130321245A1 (en) * 2012-06-04 2013-12-05 Fluor Technologies Corporation Mobile device for monitoring and controlling facility systems
US10056791B2 (en) 2012-07-13 2018-08-21 Sphero, Inc. Self-optimizing power transfer
ITTV20120139A1 (it) * 2012-07-25 2014-01-26 Isis S R L Metodo per il controllo e l'attivazione di una interfaccia utente contestuale e dispositivo e impianto con tale metodo e interfaccia
US9245428B2 (en) * 2012-08-02 2016-01-26 Immersion Corporation Systems and methods for haptic remote control gaming
US20150193977A1 (en) * 2012-08-31 2015-07-09 Google Inc. Self-Describing Three-Dimensional (3D) Object Recognition and Control Descriptors for Augmented Reality Interfaces
US9025856B2 (en) 2012-09-05 2015-05-05 Qualcomm Incorporated Robot control information
US8954195B2 (en) 2012-11-09 2015-02-10 Harris Corporation Hybrid gesture control haptic system
WO2014084858A1 (en) * 2012-11-30 2014-06-05 Empire Technology Development Llc Energy savings using augmented reality
US20140181678A1 (en) * 2012-12-20 2014-06-26 Sigal Louchheim Interactive augmented reality system, devices and methods using the same
US20140198130A1 (en) * 2013-01-15 2014-07-17 Immersion Corporation Augmented reality user interface with haptic feedback
JP6428268B2 (ja) * 2013-01-24 2018-11-28 ソニー株式会社 画像表示装置及び画像表示方法、並びに画像表示システム
US8965620B2 (en) 2013-02-07 2015-02-24 Harris Corporation Systems and methods for controlling movement of unmanned vehicles
US10507066B2 (en) 2013-02-15 2019-12-17 Intuitive Surgical Operations, Inc. Providing information of tools by filtering image areas adjacent to or on displayed images of the tools
US20140253574A1 (en) * 2013-03-05 2014-09-11 Research In Motion Limited Development Environment For Capture Of Image Data From A Mobile Device
US9158518B2 (en) 2013-03-11 2015-10-13 Blackberry Limited Collaborative application development environment using a connected device
US9779517B2 (en) * 2013-03-15 2017-10-03 Upskill, Inc. Method and system for representing and interacting with augmented reality content
US9773264B2 (en) 2013-03-26 2017-09-26 Blackberry Limited Method for providing composite user interface controls and an online storefront for same
JP5796726B2 (ja) * 2013-03-29 2015-10-21 コニカミノルタ株式会社 ジョブ情報表示装置
US20140320529A1 (en) * 2013-04-26 2014-10-30 Palo Alto Research Center Incorporated View steering in a combined virtual augmented reality system
US9258521B2 (en) 2013-05-06 2016-02-09 Globalfoundries Inc. Real-time advisor system with projected augmentable annotations
US9904356B2 (en) * 2013-05-28 2018-02-27 The Boeing Company Tracking a user to support tasks performed on complex-system components
US9395810B2 (en) * 2013-05-28 2016-07-19 The Boeing Company Ubiquitous natural user system
US9645652B2 (en) * 2013-05-28 2017-05-09 The Boeing Company Ubiquitous natural user system for human-machine interaction
US10262462B2 (en) 2014-04-18 2019-04-16 Magic Leap, Inc. Systems and methods for augmented and virtual reality
US9256072B2 (en) * 2013-10-02 2016-02-09 Philip Scott Lyren Wearable electronic glasses that detect movement of a real object copies movement of a virtual object
AU2014334669A1 (en) * 2013-10-15 2016-05-05 Sphero, Inc. Interactive augmented reality using a self-propelled device
US9582516B2 (en) 2013-10-17 2017-02-28 Nant Holdings Ip, Llc Wide area augmented reality location-based services
WO2015066037A1 (en) * 2013-10-28 2015-05-07 Brown University Virtual reality methods and systems
CN105792996B (zh) * 2013-11-28 2017-07-25 三菱电机株式会社 机器人系统以及机器人系统的控制方法
US9829882B2 (en) 2013-12-20 2017-11-28 Sphero, Inc. Self-propelled device with center of mass drive system
US9128507B2 (en) 2013-12-30 2015-09-08 Harris Corporation Compact haptic interface
JP5850958B2 (ja) * 2014-01-24 2016-02-03 ファナック株式会社 ワークを撮像するためのロボットプログラムを作成するロボットプログラミング装置
US9524588B2 (en) * 2014-01-24 2016-12-20 Avaya Inc. Enhanced communication between remote participants using augmented and virtual reality
US9613448B1 (en) 2014-03-14 2017-04-04 Google Inc. Augmented display of information in a device view of a display screen
US9690370B2 (en) 2014-05-05 2017-06-27 Immersion Corporation Systems and methods for viewport-based augmented reality haptic effects
US10068173B2 (en) * 2014-05-22 2018-09-04 Invuity, Inc. Medical device featuring cladded waveguide
JP6004051B2 (ja) * 2014-07-31 2016-10-05 キヤノンマーケティングジャパン株式会社 情報処理システム、その制御方法、及びプログラム、並びに、情報処理装置、その制御方法、及びプログラム
US20190347865A1 (en) 2014-09-18 2019-11-14 Google Inc. Three-dimensional drawing inside virtual reality environment
WO2016077506A1 (en) 2014-11-11 2016-05-19 Bent Image Lab, Llc Accurate positioning of augmented reality content
US20170243403A1 (en) * 2014-11-11 2017-08-24 Bent Image Lab, Llc Real-time shared augmented reality experience
US20160133230A1 (en) * 2014-11-11 2016-05-12 Bent Image Lab, Llc Real-time shared augmented reality experience
US10773329B2 (en) 2015-01-20 2020-09-15 Illinois Tool Works Inc. Multiple input welding vision system
BR112017016108A2 (pt) * 2015-02-13 2018-03-27 Halliburton Energy Services Inc sistema e método para facilitar a comunicação
US9643314B2 (en) * 2015-03-04 2017-05-09 The Johns Hopkins University Robot control, training and collaboration in an immersive virtual reality environment
US10448692B2 (en) 2015-03-06 2019-10-22 Illinois Tool Works Inc. Sensor assisted head mounted displays for welding
US10380911B2 (en) 2015-03-09 2019-08-13 Illinois Tool Works Inc. Methods and apparatus to provide visual information associated with welding operations
US9977242B2 (en) 2015-03-26 2018-05-22 Illinois Tool Works Inc. Control of mediated reality welding system based on lighting conditions
US9666160B2 (en) * 2015-03-26 2017-05-30 Illinois Tool Works Inc. Control of mediated reality welding system based on lighting conditions
JP6653526B2 (ja) * 2015-04-21 2020-02-26 株式会社ミツトヨ 測定システムおよびユーザインタフェース装置
US10007413B2 (en) 2015-04-27 2018-06-26 Microsoft Technology Licensing, Llc Mixed environment display of attached control elements
US10099382B2 (en) 2015-04-27 2018-10-16 Microsoft Technology Licensing, Llc Mixed environment display of robotic actions
DE102015211515A1 (de) * 2015-06-23 2016-12-29 Siemens Aktiengesellschaft Interaktionssystem
US10363632B2 (en) 2015-06-24 2019-07-30 Illinois Tool Works Inc. Time of flight camera for welding machine vision
EP3112965A1 (en) * 2015-07-02 2017-01-04 Accenture Global Services Limited Robotic process automation
US9916506B1 (en) 2015-07-25 2018-03-13 X Development Llc Invisible fiducial markers on a robot to visualize the robot in augmented reality
US9919427B1 (en) 2015-07-25 2018-03-20 X Development Llc Visualizing robot trajectory points in augmented reality
US20170028549A1 (en) * 2015-07-28 2017-02-02 Comprehensive Engineering Solutions, Inc. Robotic navigation system and method
US10454943B2 (en) 2015-08-17 2019-10-22 The Toronto-Dominion Bank Augmented and virtual reality based process oversight
JP6730577B2 (ja) * 2015-09-25 2020-07-29 キヤノンマーケティングジャパン株式会社 情報処理装置、情報処理システム、その制御方法及びプログラム
US10146194B2 (en) * 2015-10-14 2018-12-04 Hand Held Products, Inc. Building lighting and temperature control with an augmented reality system
US10600249B2 (en) 2015-10-16 2020-03-24 Youar Inc. Augmented reality platform
CN105491416B (zh) * 2015-11-25 2020-03-03 腾讯科技（深圳）有限公司 增强现实的信息传输方法和装置
JP6420229B2 (ja) 2015-12-10 2018-11-07 ファナック株式会社 仮想物体の画像をロボットの映像に重畳表示する映像表示装置を備えるロボットシステム
US10456910B2 (en) * 2016-01-14 2019-10-29 Purdue Research Foundation Educational systems comprising programmable controllers and methods of teaching therewith
US10244211B2 (en) 2016-02-29 2019-03-26 Microsoft Technology Licensing, Llc Immersive interactive telepresence
WO2017165705A1 (en) 2016-03-23 2017-09-28 Bent Image Lab, Llc Augmented reality for the internet of things
GB2549264B (en) * 2016-04-06 2020-09-23 Rolls Royce Power Eng Plc Apparatus, methods, computer programs, and non-transitory computer readable storage mediums for enabling remote control of one or more devices
US10168688B2 (en) * 2016-04-29 2019-01-01 Taylor BRUSKY Systems and methods for implementing a pointer-guided tracking system and a pointer-guided mechanical movable device control system
US11577159B2 (en) 2016-05-26 2023-02-14 Electronic Scripting Products Inc. Realistic virtual/augmented/mixed reality viewing and interactions
PT3260255T (pt) * 2016-06-24 2019-11-29 Zuend Systemtechnik Ag Sistema de corte de produto para corte
CN106383578B (zh) * 2016-09-13 2020-02-04 网易（杭州）网络有限公司 虚拟现实系统、虚拟现实交互装置及方法
US10950049B1 (en) * 2016-09-30 2021-03-16 Amazon Technologies, Inc. Augmenting transmitted video data
US10943396B1 (en) 2016-09-30 2021-03-09 Amazon Technologies, Inc. Synchronizing transmitted video data and enhancements
US11295525B1 (en) 2016-09-30 2022-04-05 Amazon Technologies, Inc. Augmenting transmitted video data
US10571902B2 (en) * 2016-10-12 2020-02-25 Sisu Devices Llc Robotic programming and motion control
US9805306B1 (en) 2016-11-23 2017-10-31 Accenture Global Solutions Limited Cognitive robotics analyzer
US10999602B2 (en) 2016-12-23 2021-05-04 Apple Inc. Sphere projected motion estimation/compensation and mode decision
US11259046B2 (en) 2017-02-15 2022-02-22 Apple Inc. Processing of equirectangular object data to compensate for distortion by spherical projections
US10924747B2 (en) 2017-02-27 2021-02-16 Apple Inc. Video coding techniques for multi-view video
US10979676B1 (en) 2017-02-27 2021-04-13 Amazon Technologies, Inc. Adjusting the presented field of view in transmitted data
US10646994B2 (en) * 2017-04-25 2020-05-12 At&T Intellectual Property I, L.P. Robot virtualization leveraging Geo analytics and augmented reality
WO2018200637A1 (en) * 2017-04-28 2018-11-01 Southie Autonomy Works, Llc Automated personalized feedback for interactive learning applications
US11093752B2 (en) * 2017-06-02 2021-08-17 Apple Inc. Object tracking in multi-view video
US20180357922A1 (en) 2017-06-08 2018-12-13 Honeywell International Inc. Apparatus and method for assessing and tracking user competency in augmented/virtual reality-based training in industrial automation systems and other systems
JP6506348B2 (ja) * 2017-06-14 2019-04-24 ファナック株式会社 ロボットの軌道を修正するロボットの教示装置
US10304251B2 (en) 2017-06-15 2019-05-28 Microsoft Technology Licensing, Llc Virtually representing spaces and objects while maintaining physical properties
US10235192B2 (en) 2017-06-23 2019-03-19 Accenture Global Solutions Limited Self-learning robotic process automation
TWI639960B (zh) * 2017-06-27 2018-11-01 行政院原子能委員會核能硏究所 擴增實境系統及其方法
US10754242B2 (en) 2017-06-30 2020-08-25 Apple Inc. Adaptive resolution and projection format in multi-direction video
US10304239B2 (en) 2017-07-20 2019-05-28 Qualcomm Incorporated Extended reality virtual assistant
US10623453B2 (en) * 2017-07-25 2020-04-14 Unity IPR ApS System and method for device synchronization in augmented reality
US10970930B1 (en) 2017-08-07 2021-04-06 Amazon Technologies, Inc. Alignment and concurrent presentation of guide device video and enhancements
JP6795471B2 (ja) 2017-08-25 2020-12-02 ファナック株式会社 ロボットシステム
WO2019046559A1 (en) * 2017-08-30 2019-03-07 Linkedwyz UTILIZING INCREASED REALITY TO CONTROL INTELLIGENT DEVICES
US10970545B1 (en) 2017-08-31 2021-04-06 Amazon Technologies, Inc. Generating and surfacing augmented reality signals for associated physical items
US10460748B2 (en) 2017-10-04 2019-10-29 The Toronto-Dominion Bank Conversational interface determining lexical personality score for response generation with synonym replacement
US10339931B2 (en) 2017-10-04 2019-07-02 The Toronto-Dominion Bank Persona-based conversational interface personalization using social network preferences
KR102082433B1 (ko) * 2017-10-19 2020-02-27 한국과학기술연구원 프로젝터-카메라 기반의 로봇형 디바이스와 헤드 마운트 디스플레이를 사용하는 원격 협업 시스템 및 이를 이용한 원격 인터랙션 방법
DE102017219067A1 (de) * 2017-10-25 2019-04-25 Bayerische Motoren Werke Aktiengesellschaft Vorrichtung und verfahren zur visuellen unterstützung eines benutzers in einem arbeitsumfeld
US10878838B1 (en) 2017-11-16 2020-12-29 Amazon Technologies, Inc. Systems and methods to trigger actions based on encoded sounds associated with containers
US11472598B1 (en) 2017-11-16 2022-10-18 Amazon Technologies, Inc. Systems and methods to encode sounds in association with containers
US10713840B2 (en) 2017-12-22 2020-07-14 Sony Interactive Entertainment Inc. Space capture, modeling, and texture reconstruction through dynamic camera positioning and lighting using a mobile robot
US10751877B2 (en) 2017-12-31 2020-08-25 Abb Schweiz Ag Industrial robot training using mixed reality
US11010975B1 (en) 2018-03-06 2021-05-18 Velan Studios, Inc. Remote camera augmented reality system
DE102018204508B4 (de) * 2018-03-23 2024-10-31 Kuka Deutschland Gmbh Verfahren und System zum Betreiben eines Roboters
JP6826069B2 (ja) * 2018-04-18 2021-02-03 ファナック株式会社 ロボットの動作教示装置、ロボットシステムおよびロボット制御装置
US20190346842A1 (en) * 2018-05-11 2019-11-14 Honeywell International Inc. Transferring annotations to images captured by remote vehicles between displays
US11429086B1 (en) 2018-05-31 2022-08-30 Amazon Technologies, Inc. Modifying functions of computing devices based on environment
DE102018113336A1 (de) * 2018-06-05 2019-12-05 GESTALT Robotics GmbH Verfahren zum Verwenden mit einer Maschine zum Einstellen einer Erweiterte-Realität-Anzeigeumgebung
JP6773084B2 (ja) * 2018-07-02 2020-10-21 株式会社安川電機 動作教示装置、ロボットシステム及び動作教示方法
CN112739244B (zh) * 2018-07-13 2024-02-09 美国iRobot公司 移动机器人清洁系统
EP3857291A4 (en) 2018-09-25 2021-11-24 Magic Leap, Inc. SYSTEMS AND PROCEDURES FOR EXTENDED REALITY
US11191609B2 (en) 2018-10-08 2021-12-07 The University Of Wyoming Augmented reality based real-time ultrasonography image rendering for surgical assistance
CN113227940A (zh) 2018-11-09 2021-08-06 贝克曼库尔特有限公司 具有选择性数据提供的服务眼镜
CN113412479A (zh) * 2019-02-06 2021-09-17 麦克赛尔株式会社 混合现实显示装置和混合现实显示方法
US11450233B2 (en) 2019-02-19 2022-09-20 Illinois Tool Works Inc. Systems for simulating joining operations using mobile devices
US11521512B2 (en) 2019-02-19 2022-12-06 Illinois Tool Works Inc. Systems for simulating joining operations using mobile devices
US11457197B2 (en) * 2019-02-22 2022-09-27 Avalon Holographics Inc. Layered scene decomposition CODEC with view independent rasterization
JP2020162916A (ja) 2019-03-29 2020-10-08 ソニー株式会社 制御装置及びマスタスレーブシステム
US11983959B2 (en) 2019-04-18 2024-05-14 Beckman Coulter, Inc. Securing data of objects in a laboratory environment
JP7359577B2 (ja) * 2019-06-21 2023-10-11 ファナック株式会社 ロボット教示装置及びロボットシステム
JP6837109B2 (ja) * 2019-09-04 2021-03-03 株式会社ミツトヨ 制御システム
US11159766B2 (en) 2019-09-16 2021-10-26 Qualcomm Incorporated Placement of virtual content in environments with a plurality of physical participants
US11958183B2 (en) * 2019-09-19 2024-04-16 The Research Foundation For The State University Of New York Negotiation-based human-robot collaboration via augmented reality
US11322037B2 (en) 2019-11-25 2022-05-03 Illinois Tool Works Inc. Weld training simulations using mobile devices, modular workpieces, and simulated welding equipment
US11721231B2 (en) 2019-11-25 2023-08-08 Illinois Tool Works Inc. Weld training simulations using mobile devices, modular workpieces, and simulated welding equipment
KR20210072463A (ko) 2019-12-09 2021-06-17 한국전자통신연구원 인간-머신 상호작용 방법 및 이를 위한 장치
US11529737B2 (en) 2020-01-30 2022-12-20 Raytheon Company System and method for using virtual/augmented reality for interaction with collaborative robots in manufacturing or industrial environment
WO2021220915A1 (ja) * 2020-04-27 2021-11-04 ファナック株式会社 産業機械の表示装置
US20230249341A1 (en) * 2020-06-23 2023-08-10 Kawasaki Jukogyo Kabushiki Kaisha Robot teaching method and robot working method
US11774954B2 (en) * 2020-12-02 2023-10-03 Westinghouse Electric Company Llc Systems and methods for wireless remote control of automated equipment
US11926064B2 (en) * 2020-12-10 2024-03-12 Mitsubishi Electric Corporation Remote control manipulator system and remote control assistance system
JP2022156506A (ja) * 2021-03-31 2022-10-14 Ｊｏｈｎａｎ株式会社 ロボット制御システム、および制御装置
JP7573505B2 (ja) 2021-10-08 2024-10-25 三菱電機株式会社 遠隔検査装置及び遠隔検査方法
US12073585B2 (en) * 2023-01-09 2024-08-27 Chengdu University Of Technology Pose estimation apparatus and method for robotic arm to grasp target based on monocular infrared thermal imaging vision

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4992866A (en) * 1989-06-29 1991-02-12 Morgan Jack B Camera selection and positioning system and method
EP0729275A2 (en) * 1995-02-24 1996-08-28 Canon Kabushiki Kaisha Image input system
US5825982A (en) * 1995-09-15 1998-10-20 Wright; James Head cursor control interface for an automated endoscope system for optimal positioning
US20020010734A1 (en) * 2000-02-03 2002-01-24 Ebersole John Franklin Internetworked augmented reality system and method
US20020057279A1 (en) * 1999-05-20 2002-05-16 Compaq Computer Corporation System and method for displaying images using foveal video
US20020097322A1 (en) * 2000-11-29 2002-07-25 Monroe David A. Multiple video display configurations and remote control of multiple video signals transmitted to a monitoring station over a network

Family Cites Families (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5579444A (en) * 1987-08-28 1996-11-26 Axiom Bildverarbeitungssysteme Gmbh Adaptive vision-based controller
US5155683A (en) * 1991-04-11 1992-10-13 Wadiatur Rahim Vehicle remote guidance with path control
JPH06134687A (ja) 1992-10-23 1994-05-17 Mitsubishi Heavy Ind Ltd 移動ロボットの遠隔操作装置
US5815411A (en) 1993-09-10 1998-09-29 Criticom Corporation Electro-optic vision system which exploits position and attitude
US6690338B1 (en) * 1993-08-23 2004-02-10 Francis J. Maguire, Jr. Apparatus and method for providing images of real and virtual objects in a head mounted display
JPH08257948A (ja) 1995-03-20 1996-10-08 Yaskawa Electric Corp ロボットの遠隔操作装置
US6535210B1 (en) * 1995-06-07 2003-03-18 Geovector Corp. Vision system computer modeling apparatus including interaction with real scenes with respect to perspective and spatial relationship as measured in real-time
US5706195A (en) * 1995-09-05 1998-01-06 General Electric Company Augmented reality maintenance system for multiple rovs
US5745387A (en) * 1995-09-28 1998-04-28 General Electric Company Augmented reality maintenance system employing manipulator arm with archive and comparison device
JPH11309269A (ja) 1998-04-27 1999-11-09 Sony Corp ゲーム装置、シミュレーション装置及びゲーム画像表示方法
US7324081B2 (en) * 1999-03-02 2008-01-29 Siemens Aktiengesellschaft Augmented-reality system for situation-related support of the interaction between a user and an engineering apparatus
WO2000052541A1 (de) * 1999-03-02 2000-09-08 Siemens Aktiengesellschaft System und verfahren zur situationsgerechten unterstützung der interaktion mit hilfe von augmented-reality-technologien
DE19953739C2 (de) 1999-11-09 2001-10-11 Siemens Ag Einrichtung und Verfahren zur objektorientierten Markierung und Zuordnung von Information zu selektierten technologischen Komponenten
EP2363774B1 (en) * 2000-05-01 2017-06-21 iRobot Corporation Method and system for remote control of mobile robot
EP1356413A2 (en) * 2000-10-05 2003-10-29 Siemens Corporate Research, Inc. Intra-operative image-guided neurosurgery with augmented reality visualization
JP2002247602A (ja) 2001-02-15 2002-08-30 Mixed Reality Systems Laboratory Inc 画像生成装置及びその制御方法並びにそのコンピュータプログラム
JP4649050B2 (ja) * 2001-03-13 2011-03-09 キヤノン株式会社 画像処理装置、画像処理方法、及び制御プログラム
US20030179308A1 (en) * 2002-03-19 2003-09-25 Lucia Zamorano Augmented tracking using video, computed data and/or sensing technologies
NO317898B1 (no) * 2002-05-24 2004-12-27 Abb Research Ltd Fremgangsmate og system for a programmere en industrirobot
SE525108C2 (sv) * 2002-12-30 2004-11-30 Abb Research Ltd Metod och system för programmering av en industrirobot, datorprogramprodukt, datorläsbart medium samt användning
US7391424B2 (en) * 2003-08-15 2008-06-24 Werner Gerhard Lonsing Method and apparatus for producing composite images which contain virtual objects
SE526119C2 (sv) * 2003-11-24 2005-07-05 Abb Research Ltd Metod och system för programmering av en industrirobot
US7236854B2 (en) * 2004-01-05 2007-06-26 Abb Research Ltd. Method and a system for programming an industrial robot

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4992866A (en) * 1989-06-29 1991-02-12 Morgan Jack B Camera selection and positioning system and method
EP0729275A2 (en) * 1995-02-24 1996-08-28 Canon Kabushiki Kaisha Image input system
US5825982A (en) * 1995-09-15 1998-10-20 Wright; James Head cursor control interface for an automated endoscope system for optimal positioning
US20020057279A1 (en) * 1999-05-20 2002-05-16 Compaq Computer Corporation System and method for displaying images using foveal video
US20020010734A1 (en) * 2000-02-03 2002-01-24 Ebersole John Franklin Internetworked augmented reality system and method
US20020097322A1 (en) * 2000-11-29 2002-07-25 Monroe David A. Multiple video display configurations and remote control of multiple video signals transmitted to a monitoring station over a network

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
STOKER C R ET AL: "ANTARCTIC UNDERSEA EXPLORATION USING A ROBOTIC SUBMARINE WITH A TELEPRESENCE USER INTERFACE", IEEE EXPERT, IEEE INC. NEW YORK, US, VOL. 10, NR. 6, PAGE(S) 14-23, ISSN: 0885-9000, XP000539881 *

Cited By (50)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1537959A3 (en) * 2003-11-24 2008-01-23 Abb Research Ltd. A method and a system for programming an industrial robot
EP1763845B1 (de) * 2005-03-02 2013-10-09 KUKA Laboratories GmbH Verfahren und vorrichtung zur bestimmung von optischen überdeckungen mit ar-objekten
WO2008059086A1 (es) * 2006-11-16 2008-05-22 The Movie Virtual, S.L. Sistema y metodo para la visualizacion de una imagen aumentada aplicando tecnicas de realidad aumentada
EP2071841A3 (en) * 2007-12-12 2009-12-16 Nokia Corp. Method, apparatus and computer program product for displaying virtual media items in a visual media
US8769437B2 (en) 2007-12-12 2014-07-01 Nokia Corporation Method, apparatus and computer program product for displaying virtual media items in a visual media
GB2456802A (en) * 2008-01-24 2009-07-29 Areograph Ltd Image capture and motion picture generation using both motion camera and scene scanning imaging systems
WO2010008518A1 (en) * 2008-07-15 2010-01-21 Eastman Kodak Company Image capture and display configuration
WO2011025450A1 (en) * 2009-08-25 2011-03-03 Xmreality Research Ab Methods and systems for visual interaction
US10051018B2 (en) 2010-06-15 2018-08-14 Live Nation Entertainment, Inc. Establishing communication links using routing protocols
US9954907B2 (en) 2010-06-15 2018-04-24 Live Nation Entertainment, Inc. Establishing communication links using routing protocols
US9781170B2 (en) 2010-06-15 2017-10-03 Live Nation Entertainment, Inc. Establishing communication links using routing protocols
US10573084B2 (en) 2010-06-15 2020-02-25 Live Nation Entertainment, Inc. Generating augmented reality images using sensor and location data
US8676615B2 (en) 2010-06-15 2014-03-18 Ticketmaster Llc Methods and systems for computer aided event and venue setup and modeling and interactive maps
US11532131B2 (en) 2010-06-15 2022-12-20 Live Nation Entertainment, Inc. Generating augmented reality images using sensor and location data
US10778730B2 (en) 2010-06-15 2020-09-15 Live Nation Entertainment, Inc. Establishing communication links using routing protocols
US11223660B2 (en) 2010-06-15 2022-01-11 Live Nation Entertainment, Inc. Establishing communication links using routing protocols
WO2012131148A1 (en) * 2011-03-31 2012-10-04 Nokia Corporation Method and apparatus for providing collaboration between remote and on-site users of indirect augmented reality
US9071709B2 (en) 2011-03-31 2015-06-30 Nokia Technologies Oy Method and apparatus for providing collaboration between remote and on-site users of indirect augmented reality
AU2011205223B1 (en) * 2011-08-09 2012-09-13 Microsoft Technology Licensing, Llc Physical interaction with virtual objects for DRM
US9038127B2 (en) 2011-08-09 2015-05-19 Microsoft Technology Licensing, Llc Physical interaction with virtual objects for DRM
AU2011205223C1 (en) * 2011-08-09 2013-03-28 Microsoft Technology Licensing, Llc Physical interaction with virtual objects for DRM
US9767524B2 (en) 2011-08-09 2017-09-19 Microsoft Technology Licensing, Llc Interaction with virtual objects causing change of legal status
US9286711B2 (en) 2011-09-30 2016-03-15 Microsoft Technology Licensing, Llc Representing a location at a previous time period using an augmented reality display
US9268406B2 (en) 2011-09-30 2016-02-23 Microsoft Technology Licensing, Llc Virtual spectator experience with a personal audio/visual apparatus
US9606992B2 (en) 2011-09-30 2017-03-28 Microsoft Technology Licensing, Llc Personal audio/visual apparatus providing resource management
WO2013113984A1 (en) * 2012-02-02 2013-08-08 Nokia Corporation Methods, apparatuses, and computer-readable storage media for providing interactive navigational assistance using movable guidance markers
US9525964B2 (en) 2012-02-02 2016-12-20 Nokia Technologies Oy Methods, apparatuses, and computer-readable storage media for providing interactive navigational assistance using movable guidance markers
CN109584374A (zh) * 2012-02-02 2019-04-05 诺基亚技术有限公司 用于使用可移动引导标记来提供交互导航辅助的方法、装置和计算机可读存储介质
WO2014049200A1 (en) * 2012-09-28 2014-04-03 Nokia Corporation Method and apparatus for providing an indication regarding content presented to another user
US10620902B2 (en) 2012-09-28 2020-04-14 Nokia Technologies Oy Method and apparatus for providing an indication regarding content presented to another user
US20170278262A1 (en) 2014-07-31 2017-09-28 Sony Corporation Information processing device, method of information processing, and image display system
US10269132B2 (en) 2014-07-31 2019-04-23 Sony Corporation Displaying images according to head posture and camera posture
EP3177010A4 (en) * 2014-07-31 2018-04-25 Sony Corporation Information processing device, information processing method, and image display system
US10625414B2 (en) 2015-08-14 2020-04-21 Franka Emika Gmbh Robotic system and housing part for such robotic system
US10843344B2 (en) 2015-10-08 2020-11-24 Sami Haddadin Robot system
KR20180064494A (ko) * 2015-10-08 2018-06-14 카스타니엔바움 게엠바하 로보틱 시스템
WO2017060539A1 (de) * 2015-10-08 2017-04-13 Sami Haddadin Robotersystem
US11040455B2 (en) 2015-10-08 2021-06-22 Haddadin Beteiligungs Ug Robot system and method for controlling a robot system
US10981278B2 (en) 2015-10-08 2021-04-20 Kastanienbaum GmbH Robot system
US10099374B2 (en) 2015-12-01 2018-10-16 Bragi GmbH Robotic safety using wearables
WO2017093289A3 (en) * 2015-12-01 2017-07-27 Bragi GmbH Robotic safety using wearables
US11358275B2 (en) 2016-04-20 2022-06-14 Franka Emika Gmbh Drive unit for a robot and method for manufacturing the same
US11623355B2 (en) 2016-04-20 2023-04-11 Kastanienbaum GmbH Method for producing a robot and device for carrying out said method
CN106128212A (zh) * 2016-08-27 2016-11-16 大连新锐天地传媒有限公司 基于增强现实技术的书法学习系统及方法
CN106569671A (zh) * 2016-11-09 2017-04-19 唐雪松 基于ar技术的消费品使用说明信息显示方法及装置
EP3610996A4 (en) * 2017-04-11 2021-01-13 Kawasaki Jukogyo Kabushiki Kaisha ROBOTIC SYSTEM AND METHOD OF OPERATING IT
US11358286B2 (en) 2017-04-11 2022-06-14 Kawasaki Jukogyo Kabushiki Kaisha Robot system and method of operating the same
NO20171008A1 (en) * 2017-06-20 2018-08-06 Augmenti As Augmented reality system and method of displaying an augmented reality image
NO342793B1 (en) * 2017-06-20 2018-08-06 Augmenti As Augmented reality system and method of displaying an augmented reality image
EP3553730A1 (en) * 2018-04-10 2019-10-16 Prisma Systems Corporation System and method for the creation and management of digital product visuals

Also Published As

Publication number Publication date
JP2004213673A (ja) 2004-07-29
US20040189675A1 (en) 2004-09-30
SE0203908D0 (sv) 2002-12-30
US7714895B2 (en) 2010-05-11

Similar Documents

Publication Publication Date Title
US7714895B2 (en) Interactive and shared augmented reality system and method having local and remote access
CN108422435B (zh) 一种基于增强现实的远程监测及控制系统
CA3016539C (en) Image processing method, display device, and inspection system
Wang et al. An augmented reality based system for remote collaborative maintenance instruction of complex products
EP1435280B1 (en) A method and a system for programming an industrial robot
US20050149231A1 (en) Method and a system for programming an industrial robot
EP1537959A2 (en) A method and a system for programming an industrial robot
Buss et al. Development of a multi-modal multi-user telepresence and teleaction system
Huy et al. See-through and spatial augmented reality-a novel framework for human-robot interaction
CN107656505A (zh) 使用增强现实设备控制人机协作的方法、装置和系统
CN102221884A (zh) 一种基于摄像机实时定标的可视遥在装置及其工作方法
JP2017100234A (ja) ティーチング結果表示システム
CN111947650A (zh) 基于光学追踪与惯性追踪的融合定位系统及方法
Stone Virtual reality and telepresence
CN116160440A (zh) 基于mr远程控制的双臂智能机器人遥操作系统
JPH0421105A (ja) マニピユレータの立体教示装置
Makita et al. Offline direct teaching for a robotic manipulator in the computational space
Gallala et al. Human-robot interaction using mixed reality
JPH11338532A (ja) 教示装置
JPH1177568A (ja) 教示支援方法及び装置
Wen et al. Teleyes: A telepresence system based on stereoscopic vision and head motion tracking
EP3502836B1 (en) Method for operating an augmented interactive reality system
Moon et al. Development of immersive augmented reality interface for construction robotic system
Stone Virtual reality: A tool for telepresence and human factors research
Kobayashi et al. Viewing and reviewing how humanoids sensed, planned and behaved with mixed reality technology

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LI LU MC NL PT RO SE SI SK TR

AX Request for extension of the european patent

Extension state: AL LT LV MK

17P Request for examination filed

Effective date: 20041217

AKX Designation fees paid

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LI LU MC NL PT RO SE SI SK TR

17Q First examination report despatched

Effective date: 20050303

17Q First examination report despatched

Effective date: 20050303

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN REFUSED

18R Application refused

Effective date: 20080209